True Score Variance Research Articles

Consistency of Limited Prosocial Emotions Across Occasions, Sources, and Settings: Trait- or State-Like Construct in a Young Community Sample?

Limited prosocial emotions (LPE, also referred to as callous-unemotional [CU] traits) are considered to reflect a more trait- than state-like construct. Our first objective was to determine the amount true score variance in CU/LPE that was consistent (trait consistency) over two occasions (12-month interval) of measurement versus specific (occasion-specificity) to each occasion. Our second objective was to determine the convergent validity of the consistent (trait) and occasion-specific (state) variance in CU/LPE symptom ratings within and across settings. Mothers, fathers, primary teachers, and ancillary teachers rated the CU/LPE symptoms in sample of 811 Spanish children (55% boys) on two occasions (i.e., end of first and second grades). CU/LPE symptom ratings showed more trait consistency than occasion-specificity for mothers and fathers, slightly more occasion-specificity than trait consistency for primary teachers, and much more occasion-specificity than trait consistency for ancillary teachers. Convergent validity for trait consistency was strong for fathers with mothers but weaker for primary with ancillary teachers. There was essentially no convergent validity for either trait consistency or occasion-specificity across home and school settings. CU/LPE symptom ratings within this age range represented a more trait-like construct for mothers and fathers and more state-like construct for primary teachers and ancillary teachers. In contrast, earlier studies showed ADHD and ODD ratings to be trait-like within and across home and school. The study of CU/LPE in young children should therefore include multiple sources in multiple settings across occasions to better understand the consistent and occasion-specific nature of the CU/LPE construct.

Relating Measurement Invariance, Cross-Level Invariance, and Multilevel Reliability.

Data often have a nested, multilevel structure, for example when data are collected from children in classrooms. This kind of data complicate the evaluation of reliability and measurement invariance, because several properties can be evaluated at both the individual level and the cluster level, as well as across levels. For example, cross-level invariance implies equal factor loadings across levels, which is needed to give latent variables at the two levels a similar interpretation. Reliability at a specific level refers to the ratio of true score variance over total variance at that level. This paper aims to shine light on the relation between reliability, cross-level invariance, and strong factorial invariance across clusters in multilevel data. Specifically, we will illustrate how strong factorial invariance across clusters implies cross-level invariance and perfect reliability at the between level in multilevel factor models.

Validity Generalization as a Continuum

Tett, Hundley, and Christiansen (2017) make a compelling case against meta-analyses that focus on mean effect sizes (e.g., rxy and ρ) while largely disregarding the precision of the estimate and true score variance. This is a reasonable point, but meta-analyses that myopically focus on mean effects at the expense of variance are not examples of validity generalization (VG)—they are examples of bad meta-analyses. VG and situational specificity (SS) fall along a continuum, and claims about generalization are confined to the research question and the type of generalization one is seeking (e.g., directional generalization, magnitude generalization). What Tett et al. (2017) successfully debunk is an extreme position along the generalization continuum significantly beyond the tenets of VG that few, if any, in the research community hold. The position they argue against is essentially a fixed-effects assumption, which runs counter to VG. Describing VG in this way is akin to describing SS as a position that completely ignores sampling error and treats every between-sample difference in effect size as true score variance. Both are strawmen that were knocked down decades ago (Schmidt et al., 1985). There is great value in debating whether a researcher should or can argue for generalization, but this debate must start with (a) an accurate portrayal of VG, (b) a discussion of different forms of generalization, and (c) the costs of trying to establish universal thresholds for VG.

Evidence for the Trait-Impulsivity Etiological Model in a Clinical Sample: Bifactor Structure and Its Relation to Impairment and Environmental Risk.

The trait-impulsivity etiological model assumes that a general factor (trait-impulsivity) underlies attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), and other externalizing disorders. We investigated the plausibility of this assumption by testing the factor structure of ADHD and ODD in a bifactor framework for a clinical sample of 1420 children between 6 and 18years of age (M=9.99, SD=3.34; 85% male). Further, the trait-impulsivity etiological model assumes that ODD emerges only if environmental risk factors are present. Our results support the validity of the trait-impulsivity etiological model, as they confirm that ADHD and ODD share a strong general factor of disruptive behavior (DB) in this clinical sample. Furthermore, unlike the subdimensions of ADHD, we found that the specific ODD factor explained as much true score variance as the general DB factor. This suggests that a common scale of ADHD and ODD may prove to be as important as a separate ODD subscale to assess externalizing problems in school-age children. However, all other subscales of ADHD may not explain sufficient true score variance once the impact of the general DB factor has been taken into consideration. In accordance with the trait-impulsivity model, we also showed that all factors, but predominantly the general factor and specific inattention factor, predicted parent-rated impairment, and that predominantly ODD and impulsivity are predicted by environmental risk factors.

Investigating a self-scoring interview simulation for learning and assessment in the medical consultation.

Experience with simulated patients supports undergraduate learning of medical consultation skills. Adaptive simulations are being introduced into this environment. The authors investigate whether it can underpin valid and reliable assessment by conducting a generalizability analysis using IT data analytics from the interaction of medical students (in psychiatry) with adaptive simulations to explore the feasibility of adaptive simulations for supporting automated learning and assessment. The generalizability (G) study was focused on two clinically relevant variables: clinical decision points and communication skills. While the G study on the communication skills score yielded low levels of true score variance, the results produced by the decision points, indicating clinical decision-making and confirming user knowledge of the process of the Calgary–Cambridge model of consultation, produced reliability levels similar to what might be expected with rater-based scoring. The findings indicate that adaptive simulations have potential as a teaching and assessment tool for medical consultations.

The Development and Evaluation of a Novel Instrument Assessing Residents' Discharge Summaries.

To develop and determine the reliability of a novel measurement instrument assessing the quality of residents' discharge summaries. In 2014, the authors created a discharge summary evaluation instrument based on consensus recommendations from national regulatory bodies and input from primary care providers at their institution. After a brief pilot, they used the instrument to evaluate discharge summaries written by first-year internal medicine residents (n = 24) at a single U.S. teaching hospital during the 2013-2014 academic year. They conducted a generalizability study to determine the reliability of the instrument and a series of decision studies to determine the number of discharge summaries and raters needed to achieve a reliable evaluation score. The generalizability study demonstrated that 37% of the variance reflected residents' ability to generate an adequate discharge summary (true score variance). The decision studies estimated that the mean score from six discharge summary reviews completed by a unique rater for each review would yield a reliability coefficient of 0.75. Because of high interrater reliability, multiple raters per discharge summary would not significantly enhance the reliability of the mean rating. This evaluation instrument reliably measured residents' performance writing discharge summaries. A single rating of six discharge summaries can achieve a reliable mean evaluation score. Using this instrument is feasible even for programs with a limited number of inpatient encounters and a small pool of faculty preceptors.

Effects of Differential Item Functioning on Examinees' Test Performance and Reliability of Test

Simulations were conducted to examine the effect of differential item functioning (DIF) on measurement consequences such as total scores, item response theory (IRT) ability estimates, and test reliability in terms of the ratio of true-score variance to observed-score variance and the standard error of estimation for the IRT ability parameter. The objective was to provide bounds of the likely DIF effects on these measurement consequences. Five factors were manipulated: test length, percentage of DIF items per form, item type, sample size, and level of group ability difference. Results indicate that the greatest DIF effect was less than 2 points on the 0 to 60 total score scale and about 0.15 on the IRT ability scale. DIF had a limited effect on the ratio of true-score variance to observed-score variance, but its influence on the standard error of estimation for the IRT ability parameter was evident for certain ability values.

Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework.

Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework.

Statistical power as a function of Cronbach alpha of instrument questionnaire items.

BackgroundIn countless number of clinical trials, measurements of outcomes rely on instrument questionnaire items which however often suffer measurement error problems which in turn affect statistical power of study designs. The Cronbach alpha or coefficient alpha, here denoted by Cα, can be used as a measure of internal consistency of parallel instrument items that are developed to measure a target unidimensional outcome construct. Scale score for the target construct is often represented by the sum of the item scores. However, power functions based on Cα have been lacking for various study designs.MethodsWe formulate a statistical model for parallel items to derive power functions as a function of Cα under several study designs. To this end, we assume fixed true score variance assumption as opposed to usual fixed total variance assumption. That assumption is critical and practically relevant to show that smaller measurement errors are inversely associated with higher inter-item correlations, and thus that greater Cα is associated with greater statistical power. We compare the derived theoretical statistical power with empirical power obtained through Monte Carlo simulations for the following comparisons: one-sample comparison of pre- and post-treatment mean differences, two-sample comparison of pre-post mean differences between groups, and two-sample comparison of mean differences between groups.ResultsIt is shown that Cα is the same as a test-retest correlation of the scale scores of parallel items, which enables testing significance of Cα. Closed-form power functions and samples size determination formulas are derived in terms of Cα, for all of the aforementioned comparisons. Power functions are shown to be an increasing function of Cα, regardless of comparison of interest. The derived power functions are well validated by simulation studies that show that the magnitudes of theoretical power are virtually identical to those of the empirical power.ConclusionRegardless of research designs or settings, in order to increase statistical power, development and use of instruments with greater Cα, or equivalently with greater inter-item correlations, is crucial for trials that intend to use questionnaire items for measuring research outcomes.DiscussionFurther development of the power functions for binary or ordinal item scores and under more general item correlation strutures reflecting more real world situations would be a valuable future study.

Exploring sources of construct-relevant multidimensionality in psychiatric measurement: A tutorial and illustration using the Composite Scale of Morningness.

This paper illustrates a psychometric approach of broad relevance to psychiatric research instruments. Many instruments include indicators related to more than one source of true-score variance due to the: (1) assessment of conceptually adjacent constructs; (2) the presence of a global construct underlying answers to items designed to assess multiple dimensions. Exploratory structural equation modelling (ESEM) is naturally suited to the investigation of the first source, whereas bifactor models are particularly suited to the investigation of the second source. When both sources are present, bifactor-ESEM becomes the model of choice. To illustrate this framework, we use the responses of 1159 adults [655 female, 504 male, mean age (Mage ) = 41.84] who completed the French Version of the Composite Scale of Morningness (CSM). We investigate the factor structure of the CSM, test the relations between CSM factors and body mass index, and verify the measurement invariance of the model across gender and age groups. Copyright © 2015 John Wiley & Sons, Ltd.

Psychometric Properties of "Community Assessment of Psychic Experiences": Review and Meta-analyses.

The Community Assessment of Psychic Experiences (CAPE) has been used extensively as a measurement for psychosis proneness in clinical and research settings. However, no prior review and meta-analysis have comprehensively examined psychometric properties (reliability and validity) of CAPE scores across different studies. To study CAPE's internal reliability--ie, how well scale items correlate with one another--111 studies were reviewed. Of these, 18 reported unique internal reliability coefficients using data at hand, which were aggregated in a meta-analysis. Furthermore, to confirm the number and nature of factors tapped by CAPE, 17 factor analytic studies were reviewed and subjected to meta-analysis in cases of discrepancy. Results suggested that CAPE scores were psychometrically reliable--ie, scores obtained could be attributed to true score variance. Our review of factor analytic studies supported a 3-factor model for CAPE consisting of "Positive", "Negative", and "Depressive" subscales; and a tripartite structure for the Negative dimension consisting of "Social withdrawal", "Affective flattening", and "Avolition" subdimensions. Meta-analysis of factor analytic studies of the Positive dimension revealed a tridimensional structure consisting of "Bizarre experiences", "Delusional ideations", and "Perceptual anomalies". Information on reliability and validity of CAPE scores is important for ensuring accurate measurement of the psychosis proneness phenotype, which in turn facilitates early detection and intervention for psychotic disorders. Apart from enhancing the understanding of psychometric properties of CAPE scores, our review revealed questionable reporting practices possibly reflecting insufficient understanding regarding the significance of psychometric properties. We recommend increased focus on psychometrics in psychology programmes and clinical journals.

Fluid intelligence shares closer to 60% of its variance with working memory capacity and is a better indicator of general intelligence

A number of empirical studies have examined the association between working memory capacity (WMC) and fluid intelligence (gf), with conclusions varying from constructs that are substantially distinct to constructs that are isomorphic. A review of the empirical literature suggests that these disparate conclusions are likely due to a number of factors, including the use of tests of varying quality, tests which share method variance, small sample sizes, and samples of varying representativeness of the population. Consequently, in this investigation, the association between WMC and gf was estimated based on well validated adult Wechsler scale subtests and a large, normative sample (N=2200). Based on a correlated two-factor model, the correlation between WMC and gf was estimated at r=.77 (95% CI: .74/.80), suggesting a substantial level of shared variance, but a meaningful level of uniqueness, as well. Furthermore, based on a bifactor model and omega specific (ωs), both WMC and gf were found to be associated with approximately equal standardized levels of unique common variance: WMC ωs=.18 (95% CI: .12/.23) and gf ωs=.19 (95% CI: .13/.25). Finally, further evidence of divergent validity was obtained, as the gf subtests were observed to be more substantial indicators of general intelligence (g) than the WMC subtests. It is concluded that WMC and gf share approximately 60% of their true score variance, rather than the commonly cited 50%. Additionally, gf should be considered a better indicator of g than WMC on both empirical and theoretical grounds.

Estimating true standard deviations.

Estimating true standard deviations.

Modeling the Balanced Inventory of Desirable Responding: Evidence in Favor of a Revised Model of Socially Desirable Responding

Paulhus (1984) proposed a 2-factor model of socially desirable responding (SDR) and created the Balanced Inventory of Desirable Responding (BIDR) to capture the 2 dimensions: self-deceptive enhancement and impression management. However, the 2-factor model has yet to be supported via confirmatory factor analysis (CFA) of the BIDR. Paulhus and Reid (1991) proposed a revised model of SDR, which bifurcated the self-deceptive enhancement dimension into 2 factors: self-deceptive enhancement (SDE; positively keyed items) and self-deceptive denial (SDD; negatively keyed items). Thus, it was considered useful to test the revised SDR model on the BIDR, as this might have implications for the manner in which the BIDR should be scored. Additionally, as the BIDR subscales might be positively correlated, it was considered useful from a both a theoretical and practical perspective to test the possibility that the BIDR might measure a general SDR process. Based on a sample of 466 adults, Paulhus and Reid's revised model of SDR was largely supported when tested via CFA on a bifactor model, which included a first-order general SDR factor and 2 nested factors. Thus, applied researchers might consider using total BIDR composite scores, a self-deceptive enhancement composite score (positively keyed items), and impression management composite scores. However, ideally, researchers would use a bifactor model to test substantive hypotheses, as the bifactor model partitions true score variance into unique sources, which facilitates less ambiguous interpretations of effects.

Developing and testing a new outcome measure

Health outcome measures are used to assess treatment effectiveness. Historically, survival was the most important health outcome. Treatments were assessed based on whether the patient lived or died. Now in the 21st century, treatments for many injuries and illnesses are so effective that many, if not most, patients survive. If patients survive, the quality of their survival becomes an important health outcome. The severity of impairments in body structure (limb loss) and function (weakness) are one way to assess the quality of survival. However, from the perspective of the patient, discomfort and disability are probably more important health outcomes. Limitation in mobility activities is an important component of disability, particularly for individuals with lower-limb loss (LLL). The treatment effectiveness of the rehabilitative care provided to servicemembers with LLL should be assessed by examining its effect on mobility limitations. Unfortunately, it is much more difficult to accurately measure mobility limitations than it is to measure survival. Measurement requires developing a set of rules to assign numbers to represent a concept or health outcome. When outcome measures contain multiple items, rules must also be developed to combine item scores to generate total and subscale scores. Determining the set of rules that will best represent a particular health outcome is affected by both the reason for measuring the outcome and the types of individuals being measured. Outcome measures can be used to examine small changes resulting from a treatment or to place individuals into broad categories. The purpose for using the outcome measure will dictate the types of items selected and the measurement dimension attached to these items. Measures of mobility activity limitations can be either performance-based or self-report. The types of individuals being measured will often determine which approach is best. A performance-based outcome measure requires both a set of rules for performing the test and a set of rules for scoring the test. It is not enough to simply create an outcome measure. It is essential to determine whether the rules used to create the outcome measure work to consistently and accurately represent the concept being measured. RELIABILITY For an outcome measure to be useful, it must be reliable in that it produces consistent findings if no real change has occurred. Performance-based outcome measures use raters. Raters must be trained to follow a standard set of rules to administer and score the measure. If raters do not adhere to rules, measurement errors may occur that adversely affect the reliability of the measure. Two types of rater reliability can be examined. Intrarater reliability indicates how consistently a rater administers and scores an outcome measure. Interrater reliability indicates how well two raters agree in the way they administer and score an outcome measure. To evaluate a measure's ability to detect real change, we must also examine score consistency over short periods in which no real change should occur. This is called test-retest reliability. An outcome measure used to evaluate progress over time must also be responsive in its ability to detect real change. Test-retest reliability is a critical factor in determining how well an outcome measure will detect real change. Reliability can be examined experimentally by testing how well scores agree between raters and time periods. Agreement is expressed mathematically by calculating a reliability coefficient representing the ratio of true score variance divided by true score variance plus error variance. A reliability coefficient of 1.0 represents perfect reliability, indicating that all of the differences between scores represent real differences between individuals. A reliability coefficient of 0.43 indicates that 43 percent of the variance is due to true score and 57 percent of the variance is due to measurement error. …

Relationships of Measurement Error and Prediction Error in Observed‐Score Regression

The focus of this paper is assessing the impact of measurement errors on the prediction error of an observed‐score regression. Measures are presented and described for decomposing the linear regression's prediction error variance into parts attributable to the true score variance and the error variances of the dependent variable and the predictor variable(s). These measures are demonstrated for regression situations reflecting a range of true score correlations and reliabilities and using one and two predictors. Simulation results also are presented which show that the measures of prediction error variance and its parts are generally well estimated for the considered ranges of true score correlations and reliabilities and for homoscedastic and heteroscedastic data. The final discussion considers how the decomposition might be useful for addressing additional questions about regression functions’ prediction error variances.

The World of Assessment: How Simple Is It?

The World of Assessment: How Simple Is It?

Magnitude of Impact of Executive Functioning and IQ on Episodic Memory in Schizophrenia

Research has implicated IQ and executive function (EF) as contributors to episodic memory impairments in schizophrenia. However, it has been difficult to quantitatively apportion the respective contributions of these factors. We conducted a series of analyses to objectively parse the associated variance and to determine to what extent episodic memory impairment in schizophrenia is independent of IQ and EF. Participants included 323 schizophrenia patients and 327 healthy controls from the National Insitute of Mental Health Sibling Study. Neurocognitive tests assessing IQ, EF, and episodic memory were administered. We examined group differences while controlling for IQ or EF in analyses of covariance, we used linear regression to quantify the amount of variance not explained by IQ or EF, and we matched control and patient subgroups on IQ or EF to determine if memory measures remained different. Analyses of covariance revealed significant group differences between schizophrenia individuals and healthy control subjects across multiple episodic memory measures after controlling for IQ or EF. Furthermore, regressions with IQ and/or EF factors entered left more than 50% of variance in memory unaccounted. Follow-up true score variance analyses indicated that the majority of this variance was directly related to memory function. Matched subgroups also yielded subgroup differences on all memory measures. Findings across the multiple statistical strategies suggested that the mechanisms underlying the memory impairment in schizophrenia are fully attributable neither to IQ nor EF. Rather, they most likely reflect compromises in episodic memory processing itself and, by inference, the medial temporal system.

Scalar Equivalence of OPQ32

Scalar equivalence of Big Five scale scores, derived from OPQ32i data, for over one million people are reviewed in terms of differences between 31 countries involving over 20 different languages. Strong relationships are found between country average scale scores and country standard deviations ( SDs), on the one hand, and two of Hofstede’s cultural dimensions, on the other. Country SDs are also seen to vary with cultural “tightness” ratings. Country-level performance indicators are also examined (the World Economic Forum Global Competitiveness Index and UN Human Development indices). Strong correlations are found between these indicators and both country-level mean personality scores and SDs of personality scores. While Hofstede’s dimensions also predict variation in global competitiveness ( R = 0.66), adding OPQ32 Big Five personality scale data increases the level of prediction to R = 0.84. It is argued that the strength of these relationships with independent country-level metrics supports the view that between-country differences represent true score variance rather than systematic instrument-related biases.

Reliability Generalization

The assessment of positive affect (PA) and negative affect (NA) by means of the Positive Affect and Negative Affect Schedule has received a remarkable popularity in the social sciences. Using a meta-analytic tool-namely, reliability generalization (RG)-population reliability scores of both scales have been investigated on the basis of a random effects model in 147 studies. Correcting for measurement errors, the results demonstrate moderate to high internal consistency coefficients and variations of the PA and NA reliability scores with regard to time frame instructions, language of items, and sample characteristics. The percentage of PA and NA true score variance differs in subpopulations up to 11%. RG analysis of test-retest coefficients illustrates state-like fluctuations and trait-like stability of both scales. Calculations of the fail-safe number point at the robustness of the results. Applications of RG coefficients compared to single-study coefficients highlight the relevance of population coefficients for research and assessment situations.