The Maximum Reliability of a Multiple-Choice Test as a Function of Number of Items, Number of Choices, and Group Heterogeneity

Abstract

IN PREVIOUS papers (7, 8) it has been shown that chance success due to guessing introduces an unavoidable source of error into multiple-choice test scores. This particular class of error is neg atively correlated with true scores. The usual equa tions for test reliability and other intercorrelations among components of test scores depend upon the assumption that the correlations between true scores and error scores and between error scores and er ror scores on parallel forms of a test are zero. In previous papers (6, 8, 9, 10) more general equa tions for these intercorrelation terms, which do not depend upon the above assumptions, have been pre sented. Because of the presence of chance success due to guessing the reliability of a multiple-choice test has a maximum value. In other words, if all sources of error other than chance success due to guessing were eliminated, the reliability of a test would re main at some value less than unity because of the unavoidable error due to guessing. The computer simulation method described previously (8) gave re liabilities for several kinds of tests, under the as sumption that only error due to guessing is present. The purpose of this paper is to determine these val ues using analytic methods. An equation for the maximum reliability of a multiple-choice test, which involves only number of items, number of choices, and mean and variance of true scores (grouphetero geneity) is derived. Horst (2) derived equations indicating the maxi mum correlation between two different tests. Be ginning with these, Roberts (5) derived equations for maximum reliability of a test. These results in volve item difficulties and are based on assumptions concerning intercorrelations among items. The re lation of number of alternative choices to test reli ability has also been investigated by Carroll (1), Lord (3), and Plumlee (4). The present paper dif fers from these approaches to the problem in that it does not involve item difficulties, but considers only components of variance of test scores. It in volves no assumptions about intercorrelations among items and holds for the case in which there is a neg ative correlation between true scores and error scores introduced by guessing. The result is rela tively simple in form.