Flawed Items Research Articles

Do item-writing flaws reduce examinations psychometric quality?

BackgroundThe psychometric characteristics of multiple-choice questions (MCQ) changed when taking into account their anatomical sites and the presence of item-writing flaws (IWF). The aim is to understand the impact of the anatomical sites and the presence of IWF in the psychometric qualities of the MCQ.Results800 Clinical Anatomy MCQ from eight examinations were classified as standard or flawed items and according to one of the eight anatomical sites. An item was classified as flawed if it violated at least one of the principles of item writing. The difficulty and discrimination indices of each item were obtained. 55.8 % of the MCQ were flawed items. The anatomical site of the items explained 6.2 and 3.2 % of the difficulty and discrimination parameters and the IWF explained 2.8 and 0.8 %, respectively.ConclusionsThe impact of the IWF was heterogeneous, the Writing the Stem and Writing the Choices categories had a negative impact (higher difficulty and lower discrimination) while the other categories did not have any impact. The anatomical site effect was higher than IWF effect in the psychometric characteristics of the examination. When constructing MCQ, the focus should be in the topic/area of the items and only after in the presence of IWF.Electronic supplementary materialThe online version of this article (doi:10.1186/s13104-016-2202-4) contains supplementary material, which is available to authorized users.

Flawed multiple-choice questions put on the scale: What is their impact on students′ achievement in a final undergraduate surgical examination?

Background : Violation of item-writing guidelines is still frequently encountered in assessments in medical colleges. Flawed multiple-choice (MC) items affect students' performance and distort examinations' results. Aims: The aim was to assess the frequency and impact of flawed MC items on students' achievement in our setting. Settings and Design: This is a quantitative descriptive study conducted at the Faculty of Medicine in the University of Tabuk, Saudi Arabia. Methods: We evaluated a 100 single-correct answer MC questions summative surgical examination administered to 44 6 th year final medical students in November 2014. MC items, which contain one or more violation of item-writing guidelines, were classified as flawed, those with no violation, were classified as standard. The passing rates and median scores of high- and low-achieving students were calculated on both standard and flawed test scales. Item performance parameters (difficulty index, discrimination power and internal consistency reliability (Kuder-Richardson formula 20) were calculated for standard and flawed items. Descriptive and comparative statistics with the relevant tests of significance were performed using the SPSS (IBM SPSS Inc. Chicago, Illinois) computer software version 16. Results: Thirty-nine flawed items were identified (39%) which contain 49 violations of the item-writing guidelines. The passing rate was 93.2% and 91.8% on the total and standard scales, respectively. Flawed items benefited low-achieving students and disadvantaged the high-achieving students. Overall, flawed items were less difficult, less discriminating and less reliable than standard items. Conclusions: The frequency of flawed items in our examination was high and reflects the need for more training and faculty development programmes.

LMS assessment: using IRT analysis to detect defective multiple-choice test items

Due to the computerisation of assessment tests, the use of Item Response Theory IRT has become commonplace for educational assessment development, evaluation and refinement. When used appropriately by a Learning Management System LMS, IRT can improve the assessment quality, increase the efficiency of the testing process and provide in-depth descriptions of item properties. This paper introduces a methodological and architectural framework which embeds an IRT analysis tool in an LMS so as to extend its functionality with assessment optimisation support. By applying a set of validity rules to the statistical indices produced by the IRT analysis, the enhanced LMS is able to detect several defective items from an item pool which are then reported for reviewing of their content. Assessment refinement is achieved by repeatedly employing this process until all flawed items are eliminated.

Identification of technical item flaws leads to improvement of the quality of single best Multiple Choice Questions

The purpose of the study was to identify technical item flaws in the multiple choice questions submitted for the final exams for the years 2009, 2010 and 2011. This descriptive analytical study was carried out in Islamic International Medical College (IIMC). The Data was collected from the MCQ's submitted by the faculty for the final exams for the year 2009, 2010 and 2011. The data was compiled and evaluated by a three member assessment committee. The data was analyzed for frequency and percentages the categorical data was analyzed by chi-square test. Overall percentage of flawed item was 67% for the year 2009 of which 21% were for testwiseness and 40% were for irrelevant difficulty. In year 2010 the total item flaws were 36% and 11% testwiseness and 22% were for irrelevant difficulty. The year 2011 data showed decreased overall flaws of 21%. The flaws of testwisness were 7%, irrelevant difficulty were 11%. Technical item flaws are frequently encountered during MCQ construction, and the identification of flaws leads to improved quality of the single best MCQ's.

A tool to reduce bias from flawed OSCE items at the writing and revieweing stage of OSCE development

A tool to reduce bias from flawed OSCE items at the writing and revieweing stage of OSCE development

Commentary on the National Mathematics Advisory Panel Recommendations on Assessment

Foundations for Success: The Final Report of the National Mathematics Advisory Panel (2008) recommends that National Assessment of Educational Progress and state tests focus on foundations of algebra and include fewer pattern-type algebra items. Greater mathematics expertise is needed in test development to prevent flawed items. The Panel reached the erroneous conclusion that multiple-choice and constructed response items measure the same thing because it relied on studies where the two item types were constrained to be identical. The Panel’s recommendations on standard setting are misleading because it relied on a single simulation study. By focusing only on special education studies, its discussion of formative assessment implies that formative assessment requires formal tools administered and scored frequently and fails to recognize more interactive forms of feedback found in studies from cognitive science.

Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments

Multiple-choice questions (MCQs) are frequently used to assess students in health science disciplines. However, few educators have formal instruction in writing MCQs and MCQ items often have item-writing flaws. The purpose of this study was to examine the impact of item-writing flaws on student achievement in high-stakes assessments in a nursing programme in an English-language university in Hong Kong. From a larger sample, we selected 10 summative test papers that were administered to undergraduate nursing students in 1 nursing department. All test items were reviewed for item-writing flaws by a 4-person consensus panel. Items were classified as 'flawed' if they contained > or = 1 flaw. Items not containing item-writing violations were classified as 'standard'. For each paper, 2 separate scales were computed: a total scale which reflected the characteristics of the assessment as administered and a standard scale which reflected the characteristics of a hypothetical assessment including only unflawed items. The proportion of flawed items on the 10 test papers ranged from 28-75%; 47.3% of all items were flawed. Fewer examinees passed the standard scale than the total scale (748 [90.6%] versus 779 [94.3%]). Conversely, the proportion of examinees obtaining a score > or = 80% was higher on the standard scale than the total scale (173 [20.9%] versus 120 [14.5%]). Flawed MCQ items were common in high-stakes nursing assessments but did not disadvantage borderline students, as has been previously demonstrated. Conversely, high-achieving students were more likely than borderline students to be penalised by flawed items.

Use of a Committee Review Process to Improve the Quality of Course Examinations

In conjunction with curricular changes, a process to develop integrated examinations was implemented. Pre-established guidelines were provided favoring vignettes, clinically relevant material, and application of knowledge rather than simple recall. Questions were read aloud in a committee including all course directors, and a reviewer with National Board of Medical Examiners (NBME) item writing and review experience. This study examines the effectiveness of this process to improve the quality of in-house examinations. Five hundred and twenty items were randomly selected from two academic years for initial comparison; 270 from 2000 to 2001, and 250 from 2001 to 2002. The first set of items represented the style, content and format when courses and tests were departmentally/discipline based, assembled by course directors, and administered separately. The latter group represented similar characteristics when courses and tests were organ-system-based, committee-reviewed and administered in an integrated examination. Items were randomized, blinded for year of origin, and rated by three NBME staff members with extensive item review experience. A five-point rating scale was used: one indicated a technically flawed item assessing recall of an isolated fact; five indicated a technically unflawed item assessing application of knowledge. To assess continued improvement, a follow-up set of 250 items from the 2002 to 2003 academic year was submitted to the same three reviewers who were not informed of the purpose or origin of this set of test items. The mean rating for items from 2000 to 2001 was 2.51 +/- 1.27; analogous values for 2001-2002 were 3.16 +/- 1.33, (t = 5.83; p < 0.0001), and in 2002-2003; 3.59 +/- 1.15 (t = 10.11; p<0.0001). Pre-established guidelines and an interdisciplinary review process resulted in improved item quality for in-house examinations.

Use of flawed multiple-choice items by the New England Journal of Medicine for continuing medical education

Physicians in the United States are required to complete a minimum number of continuing medical education (CME) credits annually. The goal of CME is to ensure that physicians maintain their knowledge and skills throughout their medical career. The New England Journal of Medicine (NEJM) provides its readers with the opportunity to obtain weekly CME credits. Deviation from established item-writing principles may result in a decrease in validity evidence for tests. This study evaluated the quality of 40 NEJM MCQs using the standard evidence-based principles of effective item writing. Each multiple-choice item reviewed had at least three item flaws, with a mean of 5.1 and a range of 3 to 7. The results of this study demonstrate that the NEJM uses flawed MCQs in its weekly CME program.

The Effects of Violating Standard Item Writing Principles on Tests and Students: The Consequences of Using Flawed Test Items on Achievement Examinations in Medical Education

The purpose of this research was to study the effects of violations of standard multiple-choice item writing principles on test characteristics, student scores, and pass-fail outcomes. Four basic science examinations, administered to year-one and year-two medical students, were randomly selected for study. Test items were classified as either standard or flawed by three independent raters, blinded to all item performance data. Flawed test questions violated one or more standard principles of effective item writing. Thirty-six to sixty-five percent of the items on the four tests were flawed. Flawed items were 0-15 percentage points more difficult than standard items measuring the same construct. Over all four examinations, 646 (53%) students passed the standard items while 575 (47%) passed the flawed items. The median passing rate difference between flawed and standard items was 3.5 percentage points, but ranged from -1 to 35 percentage points. Item flaws had little effect on test score reliability or other psychometric quality indices. Results showed that flawed multiple-choice test items, which violate well established and evidence-based principles of effective item writing, disadvantage some medical students. Item flaws introduce the systematic error of construct-irrelevant variance to assessments, thereby reducing the validity evidence for examinations and penalizing some examinees.

An inventory policy of inspection and setup cost on the integrated production-inventory system for flawed items

Almost all inventory models assume that setup cost is deterministic, and thus is not subject to control. However, in many practical situations, setup cost can be reduced at added cost, this article combines the concept of capital investment allocated to reducing the setup cost to determine an optimal integrated vendor-buyer inventory policy for flawed items in a just-in-time (JIT) manufacturing environment. The objective is to minimize the total joint annual costs incurred by the vendor and the buyer. The production process is assumed to deteriorate during processing and produces a certain number of defective items. The expected annual integrated total cost function is derived and a solution procedure is established to find out the optimal solution. Finally, the solution technique is illustrated with a numerical example available in the literature.

An optimal policy for a single-vendor single-buyer integrated production–inventory problem with process unreliability consideration

This article develops a model to determine an optimal integrated vendor–buyer inventory policy for flawed items in a just-in-time manufacturing environment. The objective is to minimize the total joint annual costs incurred by the vendor and the buyer. The production process is assumed to deteriorate during processing and produces a certain number of defective items. For practical purpose, the article is concerned with the policy in which the delivery quantity to the buyer is identical at each replenishment. The expected annual integrated total cost function is derived and a solution procedure is established to find out the optimal solution. Finally, the solution technique is illustrated with a numerical example available in the literature.

An integrated vendor-buyer cooperative inventory model for items with imperfect quality

This article develops a model to determine an optimal integrated vendor-buyer inventory policy for flawed items in a just-in-time (JIT) manufacturing environment. The aim is to minimize the total joint annual costs incurred by the vendor and the buyer. The proposed model extends the integrated vendor-buyer inventory model by accounting for imperfect quality items. The expected annual integrated total cost function is derived and an analytic solution procedure is proposed to determine the optimal policy. Finally, numerical examples are also given to illustrate the solution procedure presented in this article.

An integrated inventory model tor supplier and retailer with detective items

This article develops a model to determine an optimal integrated supplier-retailer inventory policy for flawed items in a just-in-time (JIT) manufacturing environment. This model is assumed to deteriorate during the production process and produce a certain number of defective items. The annual integrated total cost function is derived and both the ordering quantity and the number of deliveries per order are the parameters to be decided in the inventory policy. A numerical example illustrates the effectiveness of the proposed analysis approach.

Small Sample Studies to Detect Flaws in Item Translations

The number of tests being translated and adapted from 1 language and culture to others is increasing substantially. One shortcoming in current methodology for identifying flawed items due to the test translation-adaptation process is the failure to carry out empirical analyses. One important reason for not conducting empirical studies is the view that large examinee samples are required that are often not available in translation-adaptation studies. The purpose of this article was to investigate 2 simple procedures for detecting potentially flawed items with small samples: (a) conditional item p value comparisons, and (b) delta plots. Several factors were varied in this computer simulation study: sample sizes and ability distributions of the reference and focal groups, amount of differential item functioning (DIF), and the statistical characteristics of the items where DIF was found. The findings showed that the 2 simple graphical-descriptive procedures can be valuable in identifying flawed test items, especially when the size of the flaws is substantial. An application of both procedures to actual test data also supported their utility. Although this study was stimulated by questions that have arisen in the context of language translations of tests, the procedures for identifying potentially flawed items are equally applicable for identifying other potential sources of bias in the test items such as gender and race.1

Flawed Items in Computerized Adaptive Testing

Educational Testing Service A multiple‐choice test item is identified as flawed if it has no single best answer. In spite of extensive quality control procedures, the administration of flawed items to test takers is inevitable. A limited set of common strategies for dealing with flawed items in conventional testing, grounded in the principle of fairness to examinees, is reexamined in the context of adaptive testing. An additional strategy, available for adaptive testing, of retesting from a pool cleansed of flawed items, is compared to the existing strategies. Retesting was found to be no practical improvement over current strategies.

FLAWED ITEMS IN COMPUTERIZED ADAPTIVE TESTING1,2

ABSTRACTA (multiple‐choice) test item is identified as flawed if it has no single best answer. In spite of extensive quality control procedures, the administration of flawed items to test‐takers is inevitable. Common strategies for dealing with flawed items in conventional testing, grounded in the principle of fairness to test‐takers, are reexamined in the context of adaptive testing. An additional strategy, available for adaptive testing, of retesting from a pool cleansed of flawed items, is compared to the existing strategies. Retesting was found to be no practical improvement over current strategies.

Training Test-Wiseness and Flawed Item Types

This study investigated several current coaching practices used in training test-wiseness for analogy items in standardized test batteries. A three-group design was used which included a general test-taking, "encouragement" condition in addition to a no-training control group condition. The specific techniques used in training are described. Scholastic Aptitude Test (SAT) scores were obtained from university admission files to verify that no overall aptitude differences existed in the three conditions. Differences were observed for the coached group relative to the two control groups in terms of overall number of correct responses for the coached item types (analogies). No differences were found for the non-coached item types. Item difficulties for the three groups are also reported which show that several items were indeed made easier for individuals in the coached group. A qualitative analysis of the items made easier by coaching in terms of the training techniques used is given along with an analysis of the items that did not respond to coaching. Finally, a discussion of potentially flawed item types and item characteristics and suggestions for dealing with such flaws are given.

The physical basis for the roller‐coaster hazard rate curve for electronics

AbstractThis is the final installment of a series of three articles under the general topic ‘Off the bathtub onto the roller‐coaster curve’. A short version was presented at the 1988 Annual Reliability and Maintainability Symposium in the U.S.A. The first article was entitled ‘The bathtub does not hold water any more’, and the second was entitled ‘The roller‐coaster curve is in’. The three articles provide detailed discussions of the findings leading to the conclusions on the roller‐coaster characteristics for the hazard rate curve for electronics. The first article discussed the problems with the bathtub and the second explored the shape of the hazard rate curve in view of some recent observations. This article suggests some plausible physical reasons for the formation of the roller‐coaster shape. The generation of the shape starts with the basic failure mechanisms, which leads to the generally decreasing hazard rate. The humps could be caused by changing hazard conditions, wear‐out failure distribution of flawed items, distribution of flaw sizes or residue small size flaws left in the equipment because of test and inspection limitations.

Rationale for Construction of Microcomputer-Based Evaluation Subsytems

AbstractThe purpose of this study was to use the microcomputer to extend the notion of evaluation to establish and improve the reliability and validity of an evaluation instrument. A computerized testing program collected performance data of both students and the evaluation instrument at the local level. The study was conducted in two cycles. In the first cycle potentially flawed items were identified, removed from the test pool, and then examined by the instructional staff. The faulty items were revised by the instructional staff and reinserted in the test pool. In Cycle 2 of the study, the entire test pool was readministered. Of the 34 flagged items in Cycle 1, 20 items fell within the acceptable range during Cycle 2. An added benefit to these revisions was that the mean score for each of the six behavioral objectives reflected by the test pool increased, while the standard deviation for each objective decreased. In addition to the discussion of this process in terms of reliability and validity, a discu...