Flawed Items Research Articles

Multiple‐Choice Models: The Distractors Are Also Part of the Item

This paper describes an item response model for multiple‐choice items and illustrates its application in item analysis. The model provides parametric and graphical summaries of the performance of each alternative associated with a multiple‐choice item; the summaries describe each alternative's relationship to the proficiency being measured. The interpretation of the parameters of the multiple‐choice model and the use of the model in item analysis are illustrated using data obtained from a pilot test of mathematics achievement items. The use of such item analysis for the detection of flawed items, for item design and development, and for test construction is discussed.

An Alternative to Mandatory Test Disclosure

Challenges to tests and test items have led to demands and legislation for mandatory disclosure. Disclosure is tantamount to a loss of item security, a consequence that hampers subsequent test development and equating efforts. Examining boards therefore oppose disclosure. This article describes a new test critique procedure developed by the National Board of Examiners in Optometry that may be used by other credentialing boards. This procedure encourages candidates to identify potentially flawed items during rather than after the test. Candidates therefore retain the opportunity to challenge the test, while a board is able to retain item security. Other benefits of this procedure are discussed.

A Microcomputer-Based Evaluation Subsystem for the Textual Component of an Instructional System

The purpose of this study was to extend the notion of evaluation to determine and improve the reliability and validity of an evaluation instrument. This process was facilitated by a microcomputer testing program designed to collect performance data of both students and the evaluation instrument. The study was conducted in two cycles. In the first cycle, potentially flawed items were identified. These items were then pulled from the test pool and examined by the instructional staff. Thirty-four items out of the 120 test items were flagged as being potentially faulty items. These flagged items fell into five categories: 1) items containing more than one correct answer, 2) items which were incorrectly keyed, 3) items with no correct answers, 4) items with “give-away” answer foils, and 5) items in which no flaws were identified. The faulty items were revised by the instructional staff and reinserted in the test pool. In cycle two of the study, the entire test pool, including the revised items, was readministered. Of the thirty-four flagged items in cycle one, twenty items fell within the acceptable range during cycle two. An added benefit to these revisions was that the mean score for each of the six behavioral objectives reflected by the test pool increased, while the standard deviation for each objective decreased. In addition to the discussion of this process in terms of reliability and validity, a discussion of the use of microcomputers as a means of facilitating this evaluation and redesign process is also given.

THE IMPACT OF ITEM DELETION ON EQUATING CONVERSIONS AND REPORTED SCORE DISTRIBUTIONS

A formal analysis of the effects of item deletion on equating/scaling functions and reported score distributions is presented. There are two components of the present analysis: analytical and empirical. The analytical decomposition demonstrates how the effects of item characteristics, test properties, individual examinee responses, and rounding rules combine to produce the item deletion effect on the equating/scaling function and candidate scores, In addition to demonstrating how the deleted item's psychometric characteristics can affect the equating function, the analytical component of the report examines the effects of not scoring versus scoring all options correct, the effects of re‐equating versus not re‐equating, and the interaction between the decision to re‐equate or to not re‐equate and the scoring option chosen for the flawed item. The empirical portion of the report uses data from the May 1982 administration of the SA T, which contained the circles item, to illustrate the effects of item deletion on reported score distributions and equating functions. The empirical data verify what the analytical decomposition predicts.

Violation of Selected Item Construction Principles in Educational Measurement

The effects of violating four item construction principles were examined to assess the validity of the principles and the importance of students’ test wiseness. While flawed items were significantly less difficult than sound items, differences in item discrimination, test reliability, and concurrent validity were not observed. These results are interpreted to suggest that variance attributable to test wiseness may be relatively small, and item-writing guidelines may be robust to violation.

EFFECTS ON SCORE DISTRIBUTIONS OF DELETING AN UNKEYABLE ITEM FROM A TEST

ABSTRACTThe purpose of this report is to present a formal analysis of the effects of item deletion on equating/scaling functions and reported score distributions. The phrase “item deletion” shall be used to refer to the process of changing the original key of a flawed item to either all options correct, including omits, or to no options correct, i.e., not scoring the flawed item. There are two aspects to the present analysis. The first aspect is analytical, focusing on the development of a formal model for the item deletion effect by decomposing it into its constituent elements. The second component of the analysis is empirical, involving the use of actual data to illustrate and supplement the analytical results. The analytical decomposition demonstrates how the effects of item properties, test properties, individual examinee responses and rounding rules combine to produce the item deletion effect on the equating/scaling function and candidate scores. In addition to demonstrating how the deleted item's psychometric properties can affect the equating function, the analytical component of the report examines the effects of not scoring vs. scoring all options correct and the effects of re‐equating vs. not re‐equating, as well as the interaction between the decision to re‐equate or to not re‐equate and the scoring option chosen for the flawed item. The empirical portion of the report uses data from the May 1982 administration of the SAT, which contained the circles item, to illustrate the effects of item deletion on reported score distributions and equating functions. The empirical data verify what the analytical decomposition predicts.