A STUDY OF THE ACCURACY OF SUBKOVIAK'S SINGLE‐ADMINISTRATION ESTIMATE OF THE COEFFICIENT OF AGREEMENT USING TWO TRUE‐SCORE ESTIMATES

Abstract

Recently, Subkoviak (1976) developed a single-administration procedure for estimating the reliability of a criterion-referenced test composed of dichotomously scored items. The resulting reliability index is termed the coefficient of agreement. The procedure represents an important methodological development for criterion-referenced testing because, in line with suggestions by Hambleton and Novick (1973), the coefficient estimates the proportion of mastery classifications that are consistent on two test administrations, while avoiding the necessity of multiple test administrations. Application of the procedure requires an estimate of each examinee's relative true score (subsequently referred to as true score). The true score is defined as the expected value of the proportion-correct score. Subkoviak (1976) suggests using linear regression true-score estimates, but raises a question about the adequacy of the estimates. Although it is unlikely that the regression of true score on observed score is precisely linear, the regression function should be monotonically non-decreasing. Therefore, a linear regression function should provide a good approximation to the regression function (Dawes and Corrigan, 1974). In particular, when the true-score variance is small, a situation that is common in criterion-referenced testing (Hambleton and Novick, 1973; Popham and Husek, 1969), the approximation of the linear to the true regression function should be quite good. Thus, the use of linear regression estimates may be expected to produce reasonably accurate estimates of the coefficient of agreement.