Reliability and Validity of Multiple-Choice Tests Developed by Four Distractor Selection Procedures 1

Abstract

Four experimental multiple-choice tests were developed by frequency, discrimination, individual judgmental, and group judgmental procedures of selecting item distractors. Two completion tests, similar in content, length, and difficulty, were based on interpreting graphic and tabular data. One of these tests served as the criterion measure. The other was administered to 1,391 examinees. Item foils for the two judgmental experimental tests were provided by classroom teachers, while distractors for the other two experimental tasks were developed from the responses of the 1,391 examinees. The criterion test was administered to 1,925 fifth and sixth grade pupils. Next, each student immediately took one of the experimental tests. Comparison of reliability and validity of resulting scores strongly suggests that the alleged advantage of the frequency procedure is illusory. It was recommended that classroom teachers use an individual judgmental procedure and that authors of standardized achievement tests use either of the judgmental procedures.