A Note on the Use of Confidence Bands to Evaluate the Reliability of a Difference between Two Scores

Abstract

In the past ten years testing authorities have increasingly recommended the use of confidence bands in plotting test score profiles (for example, see Anastasi, 1961, and Davis, 1964). The principal virtue of such bands is that they incorporate notions of error of measurement directly into the process of interpreting scores and profiles. When inferences must be drawn about the difference between scores in two areas, the use of confidence bands seems especially convenient. One may employ, as a rule-of-thumb, the simple criterion that a difference is trustworthy whenever the bands for the two scores do not overlap. When a profile of scores is presented in the form of percentile rank bands, attention is naturally drawn to the element of unreliability in the score information. This awareness, it is hoped, will discourage the tendency by either the examiner or the examinee to attach undue importance to untrustworthy score differences. But any procedure which achieves this goal will also have the effect of diverting attention away from true and possibly important differences when they exist. This possibility, which is not sufficiently well recognized, is the subject of this paper. The use of confidence bands for test scores is not new. In the early 1940's Lindquist developed a standard score scale for the Iowa Tests of Educational Development which was directly tied to the probable error of measurement. The standard score unit was, in fact, equal to one probable error. A fifty percent confidence interval could thus be obtained for an examinee's true score by adding and subtracting one standard score point to and from his observed score. The impetus for the current popularity of confidence bands undoubtedly stems, however, from their use in the SCAT (1955) and STEP (1957) series published by Educational Testing Service. In these series the technique of assessing the trustworthiness of a difference through a comparison of the percentile rank bands for the separate scores is heavily emphasized. The theory underlying the technique of comparing bands is relatively simple. If a test has a standard error of measurement equal to oe and the 100y percent confidence interval is to be obtained for an examinee's true score, the quantity c<e is added to and subtracted from his observed