A LATENT TRAIT METHOD FOR DETERMINING INTRAJUDGE INCONSISTENCY IN THE ANGOFF AND NEDELSKY TECHNIQUES OF STANDARD SETTING

Abstract

In objectives-based instructional programs, absolute rather than relative standards are used to evaluate student performances. These standards can be considered to be the translation of the learning objectives in the program into cutoff scores on the true score scale of the test. They constitute the predetermined levels that the student's true performance must exceed to be granted mastery status and, for instance, to be allowed to proceed with the next instructional unit. An important problem in mastery testing is how to set standards separating those students who meet learning objectives from those who do not. Several techniques of standard setting have been proposed, and an extensive literature on the problem is available which has recently been reviewed by several authors (Glass, 1978; Hambleton, 1980; Hambleton, Powell & Eignor, 1979; Jaeger, 1979; Shepard, 1980a, 1980b). In this paper, the emphasis will be on the Angoff (1971) and Nedelsky (1954) techniques; these are commonly classified as techniques based on judgement of test content. A review of the mastery testing literature is given in Hambleton, Swaminathan, Algina, and Coulson (1978) and van der Linden (1982). It has been argued that all standard setting is arbitrary (Glass, 1978; Shepard, 1979, 1980a, 1980b). This is correct since standards ought to reflect learning objectives and these ultimately rest on value judgments and norms. In addition, the various standard setting techniques available differ in varying degree according to the conception of mastery underlying the way standards are obtained. Therefore, different results can be expected both for different techniques and for different persons using the same technique. This has been confirmed in many experiments (Andrew & Hecht, 1976; Brennan & Lockwood, 1980; Koffler, 1980; Saunders, Ryan & Huynh, 1981; Skakun & Kling, 1980). That all standard setting is ultimately arbitrary led Glass to the pessimistic conclusion that we should abandon the use of these techniques. However, as Hambleton (1978) and Popham (1978) have put forward, arbitrariness does not necessarily have a negative meaning. There are many other instances in which arbitrary choices have to be made in which deliberate, defensible results are obtained. What should be avoided is capricious standard setting, that is, standard setting in which the learning objectives are inconsistently translated into the cutoff score and, in fact, erratic standards of mastery are obtained. In a sense, the present paper addresses this second, negatively loaded type of arbitrariness. Its concern is not with differences in standard setting between persons or techniques. There simply is no reason to expect the same result when persons with different interpretations of learning objectives or techniques based on different concep-