Reliability Of Difference Scores Research Articles

The reward positivity shows increased amplitude and decreased latency with increasing age in early childhood.

The reward positivity (RewP) is a widely studied measure of neural response to rewards, yet little is known about normative developmental characteristics of the RewP during early childhood. The present study utilized a pooled community sample of 309 4- to 6-year-old children who participated in the Doors guessing game to examine the latency and amplitude of the RewP. Peak detection of the gain-loss difference waveform was conducted for electrodes Fz, Cz, Pz, Oz and the mean activity in a 100ms window centered around this peak was analyzed. There was a significant decrease in RewP latency (RewP was earlier) and increase in RewP amplitude (RewP magnitude was larger) with advancing age in this cross-sectional analysis. Further, these were independent effects, as both RewP latency and RewP amplitude were uniquely associated with children's age. Moreover, our results indicate that the RewP latency in 4- to 6-year-olds falls outside the 250-350ms window typically used to quantify the RewP (RewP latency in our sample=381ms; SD=60.15). The internal consistency for latency (.64) and amplitude (.27) of the RewP were characterized by moderate to low reliability, consistent with previous work on the reliability of difference scores. Overall, results demonstrate RewP differences in both timing and amplitude across age in early childhood, and suggest that both amplitude and latency of the RewP might function as individual difference measures of reward processing. These findings are discussed in the context of methodological considerations and the development of reward processing across early childhood.

Estimating Difference-Score Reliability in Pretest–Posttest Settings

Clinical, medical, and health psychologists use difference scores obtained from pretest–posttest designs employing the same test to assess intraindividual change possibly caused by an intervention addressing, for example, anxiety, depression, eating disorder, or addiction. Reliability of difference scores is important for interpreting observed change. This article compares the well-documented traditional method and the unfamiliar, rarely used item-level method for estimating difference-score reliability. We simulated data under various conditions that are typical of change assessment in pretest–posttest designs. The item-level method had smaller bias and greater precision than the traditional method and may be recommended for practical use.

Evaluating the internal consistency of subtraction-based and residualized difference scores: Considerations for psychometric reliability analyses of event-related potentials.

In studies of event-related brain potentials (ERPs), difference scores between conditions in a task are frequently used to isolate neural activity for use as a dependent or independent variable. Adequate score reliability is a prerequisite for studies examining relationships between ERPs and external correlates, but there is no extensive treatment on the suitability of the various available approaches to estimating difference score reliability that focus on ERP research. In the present study, we provide formulas from classical test theory and generalizability theory for estimating the internal consistency of subtraction-based and residualized difference scores. These formulas are then applied to error-related negativity (ERN) and reward positivity (RewP) difference scores from the same sample of 117 participants. Analyses demonstrate that ERN difference scores can be reliable, which supports their use in studies of individual differences. However, RewP difference scores yielded poor reliability due to the high correlation between the constituent reward and non-reward ERPs. Findings emphasize that difference score reliability largely depends on the internal consistency of constituent scores and the correlation between those scores. Furthermore, generalizability theory yields more suitable estimates of internal consistency for subtraction-based difference scores than classical test theory. We conclude that ERP difference scores can show adequate reliability and be useful for isolating neural activity in studies of individual differences.

Difference score reliabilities within the RIAS‐2 and WISC‐V

AbstractMany prominent intelligence tests (e.g., Wechsler Intelligence Scale for Children, Fifth Edition [WISC‐V] and Reynolds Intellectual Abilities Scale, Second Edition [RIAS‐2]) offer methods for computing subtest‐ and composite‐level difference scores. This study uses data provided in the technical manual of the WISC‐V and RIAS‐2 to calculate reliability coefficients for difference scores. Subtest‐level difference score reliabilities range from 0.59 to 0.99 for the RIAS‐2 and from 0.53 to 0.87 for the WISC‐V. Composite‐level difference score reliabilities generally range from 0.23 to 0.95 for the RIAS‐2 and from 0.36 to 0.87 for the WISC‐V. Emphasis is placed on comparisons recommended by test publishers and a discussion of minimum requirements for interpretation of differences scores is provided.

Over- and Under-Perceiving Social Support from One’s Partner and Relationship Quality Over Time

Using dyadic data from a national longitudinal survey in Germany (N = 3,674 couples), we tested associations between congruence/discrepancy in partners’ reports of providing and receiving social support (i.e., whether a recipient reports more or less than the provider) and relationship quality. All participants reported how much social support they provided to, and received from, their partner. Latent Congruence Modeling was used to enhance the reliability of difference scores between one partner’s reported provision and the other's reported receipt. In most instances, perceiving more support than one’s partner reported giving (over-perception) and extensive support reported jointly by providers and recipients were positively associated with recipients’ and providers’ relationship quality. Conceptual, methodological, and possible clinical issues are discussed.

A defense against the alleged unreliability of difference scores

AbstractBased on a classical true score theory (classical test theory, CTT) equation, indicating that as the observed correlation between two tests increases, the reliability of the difference scores decreases, researchers have concluded that difference scores are unreliable. But CTT shows that the reliabilities of the two tests and the true correlation between them influence the observed correlation and previous analyses have not taken the true correlation sufficiently into account. In turn, the reliability of difference scores depends on the interaction of the reliabilities of the individual tests and their true correlation when the variances of the tests are equal, and on a more complicated interaction between them and the deviation ratio when the variances of the tests are not equal. The upshot is that difference scores likely are more reliable, on more occasions, than researchers have realized. I show how researchers can predict what the reliability of the difference scores is likely to be, to aid in...

Individual differences make a difference: On the use and the psychometric properties of difference scores in social psychology

AbstractMuch social psychological research is concerned with the question whether and how behavior changes because of a “treatment” (e.g., a situation that triggers a psychological reaction). One easy way to investigate such changes would be to analyze intraindividual differences before (Time 1) and after the treatment (Time 2). Interestingly, many scholars refrain from using difference scores because they think they are inherently unreliable. However, the bad reputation of difference scores is, in many cases, unwarranted: difference scores can be sufficiently reliable when standard deviations differ between measurement occasions, and standard deviations are likely to differ between measurement occasions because of differential treatment effects (i.e., interindividual differences in responsiveness to a treatment) and/or “strong situation” treatments. In the present article, we will (1) summarize classic and current arguments regarding the reliability of difference scores, (2) discuss the use of residual change scores as an alternative to difference scores, and (3) argue that latent difference score models are a particularly useful tool that social psychologists should consider using more frequently. Copyright © 2014 John Wiley & Sons, Ltd.

A Proposal for Evaluating the Validity of Holistic-Based Admission Processes

Background. Admission decisions require that information about an applicant be combined using either holistic (human judges) or statistical (actuarial) methods. For optimizing a defined measureable outcome, there is a consistent body of research evidence demonstrating that statistical methods yield superior decisions compared to those generated by judges. It is possible, however, that the benefits of holistic decisions are reflected in unmeasured outcomes. If such benefits exist, they would necessarily appear as systematic variance in raters’ scores that deviate from statistically-based decisions. Purpose. To estimate this variance, we propose a design examining the interrater reliability of difference scores (i.e., the difference between observed committee rankings and rankings based on statistical approaches). Methods. Example calculations and G study models are presented to demonstrate how rater agreement on difference scores can be analyzed under various circumstances. High interrater reliability of difference scores would support but not prove the assertion that the holistic process adds useful information beyond that achieved by much less costly statistical approaches. Conversely, if the interrater reliability of difference scores is near zero, this would clearly demonstrate that committee judgments add random error to the decision process. Results. Evidence to conduct such studies already exists within most highly selective medical schools and graduate programs and the proposed validity research could be conducted on existing data. Conclusions. Such research evidence is critical for establishing the validity of widely used holistic admission approaches.

Difference Scores From the Point of View of Reliability and Repeated-Measures ANOVA

There is such doubt in research practice about the reliability of difference scores that granting agencies, journal editors, reviewers, and committees of graduate students’ theses have been known to deplore their use. This most maligned index can be used in studies of change, growth, or perhaps discrepancy between two measures taken on the same sampling unit. The most commonly stated problem with difference scores is the supposed associated increase in unreliability of difference scores. In this article, the authors examine difference scores from the point of view of reliability and repeated-measures ANOVA. The authors demonstrate that when the focus of difference scores is data analysis of aggregate models, their use should not be assessed in terms of reliability, and that the complete abolition of difference scores in research practice is unwarranted.

Neuropsychologists Must Keep Their Eyes on the Reliability of Difference Measures

Sanchez-Cubillo et al. (2009) conducted a study in which 41 healthy older subjects performed a battery of neuropsychological tests, including the Trail Making Test (TMT), the Digit Symbol subtest (WAIS-III), the Digits Forward and Backward subtests (WAIS-III), a Finger Tapping Test, a Stroop Test, and a task-switching paradigm akin to the Wisconsin Card Sorting Test (cf. Strauss, Sherman, and Spreen, 2006). The results of correlation and regression analyses suggested that TMTA requires mainly visuo-perceptual abilities, TMTB primarily reflects working memory and task-switching abilities, while the TMTB2A difference score provides a relatively pure indicator of task-switching abilities. The use of the TMTB2A difference score should help clinicians to interpret abnormal performance in terms of a failure of this specific cognitive mechanism. Unfortunately, there is the danger that the reliability of difference scores will be unacceptably low because the reliability of a difference score is simply a function of the average reliability of its two components and of the correlation between them (Crawford, Sutherland, and Garthwaite, 2008). Under the circumstance of a common standard deviation of the two components used to form the difference, the formula for the reliability of a difference score, r(B2A), is:

The Reliability of Difference Scores in Populations and Samples

This study was an investigation of the relation between the reliability of difference scores, considered as a parameter characterizing a population of examinees, and the reliability estimates obtained from random samples from the population. The parameters in familiar equations for the reliability of difference scores were redefined in such a way that determinants of reliability in both populations and samples become more transparent. Computer simulation was used to find sample values and to plot frequency distributions of various correlations and variance ratios relevant to the reliability of differences. The shape of frequency distributions resulting from the simulations and the means and standard deviations of these distributions reveal the extent to which reliability estimates based on sample data can be expected to meaningfully represent population reliability.

On the reliability and standard errors of measurement of contrast measures from the D-KEFS

A formula for the reliability of difference scores was used to estimate the reliability of Delis-Kaplan Executive Function System (D-KEFS; Delis et al., 2001) contrast measures from the reliabilities and correlations of their components. In turn these reliabilities were used to calculate standard errors of measurement. The majority of contrast measures had low reliabilities: of the 51 reliability coefficients calculated in the present study, none exceeded 0.7 and hence all failed to meet any of the criteria for acceptable reliability proposed by various experts in psychological measurement. The mean reliability of the contrast scores was 0.27, the median reliability was 0.30. The standard errors of measurement were large and, in many cases, equaled or were only marginally smaller than the contrast scores' standard deviations. The results suggest that, at present, D-KEFS contrast measures should not be used in neuropsychological decision making.

A Comparison of the Reliabilities of Four Types of Difference Scores for Five Cognitive Assessment Batteries

Summary: In recent years, studies have been conducted comparing the reliability of simple difference scores to differences computed via reliable component analysis (RCA). For the five cognitive assessment batteries examined here, RCA difference scores have been superior to simple difference scores in this regard. In the present study, the five cognitive assessment batteries are considered again, including a comparison of two alternative methods of difference score construction, the residualized and base-free difference methods, in order to determine if RCA outperforms the other methods as well. Results indicated that the RCA difference scores were more reliable in 68/70 comparisons between the four methods. When difference scores are a central part of test interpretation, clinicians should use the RCA scoring method because of the high reliability of the difference scores.

On the Relation between Power and Reliability of Difference Scores

The potential problems which may arise from the use of difference (a.k.a., gain) scores in the measurement of change are well documented, including (a) difference scores are often negatively correlated with initial ability, and (b) they often tend to be unreliable. Conversely, they make excellent dependent variables in a true experiment since they tend to reduce variability due to individual differences among persons. This brief didactic paper presents a conclusion similar to Nicewander and Price, using a perhaps more straightforward argument based on difference scores. We argue that the same reason difference scores provide powerful significance tests, namely, reduction of "true score" variance, is also the reason they tend to be unreliable. Further, we make the point that reducing true score variance will increase the power of a significance test (since it will reduce the denominator or "error term" of the observed statistic) but will decrease reliability (since it is the numerator and a component of the denominator of the reliability coefficient).

Some Problems, Pitfalls, and Paradoxes in Educational Measurement

This article provides brief discussions of an eclectic set of problems, pitfalls, and paradoxes that challenge measurement theory and practice, especially for K–12 achievement testing. The technical issues considered are: true scores and error scores, fixed versus random facets, lower limits to reliability, the reliability‐validity paradox, reliability of difference scores, equating and linking, and scaling. The practical issues considered are: validation, standard setting, instructional improvement and high‐stakes accountability, multiple measures, adequate yearly progress, the National Assessment of Educational Progress as a confirmation model, and capacity.

Difference score reliability for tactual performance test trials.

In further analysis of published data difference score reliabilities are presented for the difference between raw score and blocks-per-minute trial scores on the Tactual Performance Test.

DIFFERENCE SCORE RELIABILITY FOR TACTUAL PERFORMANCE TEST TRIALS

In further analysis of published data difference score reliabilities are presented for the difference between raw score and blocks-per-minute trial scores on the Tactual Performance Test.

Identifying submaximal muscular effort: reliability of difference scores calculated from isometric and isokinetic measurements.

The present investigation examined the reliability of a derived strength measurement and also how variability between test sessions affects the utility of this variable for judging an individual's muscular effort. 31 young healthy men (M age = 25 yr.) completed three isometric and isokinetic concentric contractions of the knee extensors, using maximal and self-selected submaximal efforts, on each of two test sessions. Difference scores between isometric and isokinetic measurements were calculated by subtracting the mean of the three isokinetic peak torques from the mean of the three isometric peak torques for each individual subject for maximal and submaximal efforts performed on both test sessions. For the group of subjects, difference scores were significantly greater during maximal (33 +/- 29 Nm) than submaximal (13 +/- 30 Nm) efforts, suggesting subjects could not maintain the same relationship between isometric and isokinetic muscular actions across the maximal and submaximal conditions. However, the test-retest reliability of the difference scores was only modest even when data were averaged over two test sessions (intraclass correlation coefficients were .82 for maximal, and .58 for submaximal). As a result, the range of scores within which an individual's true score might be expected to lie was large (+/- 25 Nm for maximal, and +/- 37 Nm for submaximal). Although derived strength parameters like difference scores may be effective in distinguishing submaximal from maximal efforts completed by groups of subjects, the test-retest reliability of the present scores suggests that their use in judging an individual's performance may be limited.

A geometric interpretation of the validity and reliability of difference scores

For several decades, psychometricians have been concerned about the unreliability and meagre validity of difference scores and gain scores. The present paper explores a geometric interpretation, in which observed scores are identified with vectors in a function space of random variables, and true and error components of scores are identified with orthogonal projections onto complementary subspaces. This point of view provides a simple and easily visualized geometric explanation for the widespread belief that differences are inherently unreliable. Furthermore, it discloses conditions under which difference scores are highly reliable and have substantial correlations with other measures.

Are Simple Gain Scores Obsolete?

It is widely believed that measures of gain, growth, or change, expressed as simple differences between pretest and posttest scores, are inherently unreliable. It is also believed that gain scores lack predictive validity with respect to other criteria. However, these conclu sions are based on misleading assumptions about the values of parameters in familiar equations in classical test theory. The present paper examines modified equa tions for the validity and reliability of difference scores that describe applied testing situations more realisti cally and reveal that simple gain scores can be more useful in research than commonly believed.