On the Relation between Power and Reliability of Difference Scores

Abstract

The potential problems which may arise from the use of difference (a.k.a., gain) scores in the measurement of change are well documented, including (a) difference scores are often negatively correlated with initial ability, and (b) they often tend to be unreliable. Conversely, they make excellent dependent variables in a true experiment since they tend to reduce variability due to individual differences among persons. This brief didactic paper presents a conclusion similar to Nicewander and Price, using a perhaps more straightforward argument based on difference scores. We argue that the same reason difference scores provide powerful significance tests, namely, reduction of "true score" variance, is also the reason they tend to be unreliable. Further, we make the point that reducing true score variance will increase the power of a significance test (since it will reduce the denominator or "error term" of the observed statistic) but will decrease reliability (since it is the numerator and a component of the denominator of the reliability coefficient).