A Critical Examination of the "Reliability" and "Abnormality" Approaches to the Evaluation of Subtest Score Differences

Abstract

Two criteria for the evaluation of intra-individual subtest score differences have been suggested in the literature, namely "statistical significance" and "abnormality." The procedures for determining them involve a z-score transformation of the observed difference d—the division of d by the standard error of measurement of the difference (uMd) in the case of statistical significance, and by the standard deviation Cd of the observed differences in the case of abnormality. As oMd<od, statistical significance is usually a necessary but insufficient condition for abnormality and the two expressions are generally considered hierarchical criteria for the evaluation of the clinical significance of subtest score differences differing only in their stringency. This paper emphasizes that the criteria of statistical significance and of abnormality address two entirely different questions and shows that despite their hierarchial statistical relation, they can neither be considered alternatives nor be jointly used for the evaluation of score differences. Moreover, it is argued that statistical significance is conceptually inappropriate for this purpose and that abnormality has been applied to poor estimates of the true score difference (i.e., observed differences). Improved estimates of the true score difference are suggested, and applicability of the abnormality criterion to these estimates is discussed.