The Effect of Guessing on Item Reliability under Answer-Until-Correct Scoring

Abstract

The answer-until-correct (AUC) procedure re quires that examinees respond to a multiple-choice item until they answer it correctly. The examinee's score on the item is then based on the number of responses required for the item. It was expected that the additional responses obtained under the AUC procedure would improve reliability by pro viding additional information on those examinees who fail to choose the correct alternative on their first attempt. However, when compared to the zero- one (ZO) scoring procedure, the AUC procedure has failed to yield consistent improvements in relia bility. Using a modified version of Horst's model for examinee behavior, this paper compares the ef fect of guessing on item reliability for the AUC pro cedure and the ZO procedure. The analysis shows that the relative efficiency of the two procedures de pends strongly on the nature of the item alterna tives and implies that the appropriate criteria for item selection are different for each procedure. Conflicting results reported for empirical compari sons of the reliabilities of the two procedures may result from a failure to control for the characteris tics of the items.