The use of free Response Data in Writing Choice-Type Items

Abstract

MOST TEXTBOOKS on educational measure ment present rules or suggestions for construct ing objective test items of various types. One suggestion often made is that the distracters (wrong responses) for choice-type items should be plausible. The application of this rule re quires that an item writer have some notion con cerning what alternatives are likely to serve as efficient distracters. For an experienced item writer this selection of plausible distracters may be relatively easy. A relatively inexper ienced item writer, however, may experience difficulty in devising plausible wrong choices. Some writers on educational test construction have suggested using students' responses to items set up in free response form as a source of possible distracters. In general, these writ ers suggest that using student responses to free response items as a source of distracters for a choice-type item often results in a better item than if such data is not used. Although the use of free response data has been recommended by many authors of textbooks in educational measurement, r e s e arch studies designed to evaluate the procedure have been rel atively few. Kelley (1) in 1937 used free re sponse data in constructing a vocabulary test and concluded that the procedure was questionable in view of the work involved and the results obtained. Frederiksen and Satter (2) in 1953 reported on the use of free response data in construe ting arithmetic problems and found the method was of some help. They did not, however, make any di rect comparison between items constructed with and without knowledge of such data. The results of these two research studies sug gest that the value of free response data is ques tionable. In addition, the procedure has appar ently been applied only to such factual school sub jects as vocabulary and arithmetic, where it is relatively easy to define incorrect answers. As far as it could be determined, no experimental evidence has been obtained in content areas such as the social studies. Furthermore, pre vious studies appeared to have only one person doing the item writing. As the method of using free response data is offered as a general sugges tion for all item writers, it is possible that the re sults of previous studies have been largely deter mined by the ability of a particular writer to use the free response data, rather than to the general value of the method itself. In view of the above considerations and the rather limited amount of