A defense against the alleged unreliability of difference scores

David Trafimow

doi:10.1080/23311835.2015.1064626

Abstract

AbstractBased on a classical true score theory (classical test theory, CTT) equation, indicating that as the observed correlation between two tests increases, the reliability of the difference scores decreases, researchers have concluded that difference scores are unreliable. But CTT shows that the reliabilities of the two tests and the true correlation between them influence the observed correlation and previous analyses have not taken the true correlation sufficiently into account. In turn, the reliability of difference scores depends on the interaction of the reliabilities of the individual tests and their true correlation when the variances of the tests are equal, and on a more complicated interaction between them and the deviation ratio when the variances of the tests are not equal. The upshot is that difference scores likely are more reliable, on more occasions, than researchers have realized. I show how researchers can predict what the reliability of the difference scores is likely to be, to aid in...

Highlights

Basic and applied researchers in education and psychology often have reason to be interested in the differences between people’s scores on two tests
The criticism that difference scores are unreliable comes directly from a theorem based on classical true score theory or classical test theory (CTT) that can be found in many places (e.g. Cronbach, 1990; Lord, 1963; see Gulliksen, 1987 for an excellent review of CTT theorems)
Given Equations 4 and 9, as well as Figures 1 and 2, what can we conclude about the reliability of difference scores? The analyses demonstrate that, if the reliabilities of the individual tests are good, the reliability of the difference scores will be at least reasonable so long as the true correlation is not too large

Summary

Introduction

Basic and applied researchers in education and psychology often have reason to be interested in the differences between people’s scores on two tests. For many decades, psychometric experts have warned that difference scores are not reliable. 3) stated the consequence of this warning with admirable clarity: it is commonplace for research to be stymied by some difficulty in experimental methodology, there are really not many instances in the behavioral sciences of promising questions going unresearched because of deficiencies in statistical methodology. Questions dealing with psychological change may well constitute the most important exceptions. If previous criticisms hold up, this would constitute an important reason to avoid doing research that depends on difference scores. If previous criticisms do not hold up, an enormous impediment to research can be removed. A close study of this three-way interaction suggests that the blanket conclusion that difference scores are inherently unreliable is too pessimistic

The basis of the criticism

The importance of CTT’s true correlations

Discussion