Rejoinder to Edwards’ s Comments

Abstract

Listen

Translate article

In his position paper, Edwards critiqued several of our comments concerning the reliability and validity of difference scores. We believe our differences of opinion occur not only because Edwards has endorsed historical arguments against difference scores, but also because he conceptualizes certain issues quite differently than we do. We address his major points of criticism and then reiterate (and perhaps clarify) our position. Edwards assumes that it is reasonable to assert a priori that difference scores will often exhibit poor reliabilities because the conditions under which poor reliabilities can occur (i.e., unreliable and highly positively intercorrelated component measures) are very common in empirical research. Although these circumstances may be common, they should not be sufficient to condemn the use of difference scores a priori because reliability may be empirically investigated and because, as we suggested, reliabilities can be improved. We take exception to Edwards’s statements, “. . . the reliability of a difference score should be evaluated not only in an absolute sense, but also in relation to viable alternatives, such as using both component measures jointly in multiple regression analysis . . . . If a difference score exhibits adequate reliability, then it is almost certain that its components will exhibit superior reliabilities, indicating that the latter should be used in place of the former.” To us, this presumes that the difference and component measures in question are conceptually interchangeable, a blanket assumption we are unwilling to make. For example, the concept of role conflict obtained from the differences between subordinate and supervisor job ratings is not the same as conceptualizations of the components of subordinate and supervisor job ratings. Also, we do not agree, given adequate difference score reliabilities, that difference scores should be discarded because their component measures show higher reliabilities. What about the theory being tested or research goals? Finally, notice that we and Edwards (1994) agree, that response surfaces do not eliminate reliability problems. We disagree with Edwards’s suggestion that the reliabilities of profile similarity measures can be “problematic” because dimensions are often formed by large numbers of heterogeneous items. Our position was never that