DEMONSTRATING THE RELIABILITY THE DIFFERENCE SCORE IN THE MEASUREMENT OF CHANGE

Abstract

The reliability of difference score (a.k.a., gain score) has been one of most discussed, yet least understood, topics in measurement of change. Demonstrations of lack of reliability of difference score are prevalent. Most discussions of reliability of difference score in measurement of change literature have recited and served to document assertions by Lord that between scores tend to be much more unreliable than scores themselves (1956, p. 429) and that the difference between two fallible measures is frequently much more fallible than either (1963, p. 32). Rogosa, Brandt, and Zimowski (1982) explained that these assertions are accurate only in extreme, yet frequently examined, situations.' The reliability of a measure represents ability of that measure to distinguish among individuals on a particular trait or true score. Hence, when all individuals display nearly same true change, naturally difference score is unable to adequately distinguish true change among individuals. Consequently, reliability of difference score will be very low. As most previous investigations have emphasized situations for which true change varies little across individuals, it is not surprising that criticisms of difference score, because of its low reliability, dominate literature on change. On other hand, when individual differences in true change do exist, difference score does a good job in distinguishing among individuals (see Rogosa et al., 1982). The purpose of this paper is to demonstrate explicitly that difference score is reliable in many important situations. We consider measurements on each of n persons at time 1 and at time 2; observed score for person j (j = 1,... n) is Xij at time 1 and X2j at time 2. We assume a classical test theory model for observed scores: X,j = l,j + E,j and X2j = t2j + e2j where l,j and ~2j are true scores for person j at times 1 and 2, respectively, and e,j and E2j are measurement errors at times 1 and 2. True change for person j is written: Aj = ~2j lj. The difference (gain) score for person j, which estimates true change, is written: Dj = X2j Xj. The reliabilities of X, and X2 are denoted by p(X,) and p(X2). The reliability of difference score is denoted by p(D).