Abstract
BackgroundThis study investigated the impact of addressing item writing flaws, testing at low cognitive level and non-functioning distractors (< 5 % selection frequency) in multiple-choice assessment in preclinical medical education.MethodMultiple-choice questions with too high or too low difficulty (difficulty index < 0.4 or > 0.8) and insufficient discriminatory ability (point-biserial correlation < 0.2) on previous administration were identified. Items in Experimental Subgroup A underwent removal of item writing flaws along with enhancement of tested cognitive level (21 multiple-choice questions), while Experimental Subgroup B underwent replacement or removal of non-functioning distractors (11 multiple-choice questions). A control group of items (Group C) did not undergo any intervention (23 multiple-choice questions).ResultPost-intervention, the average number of functioning distractors (≥ 5 % selection frequency) per multiple-choice question increased from 0.67 to 0.81 in Subgroup A and from 0.91 to 1.09 in Subgroup B; a statistically significant increase in the number of multiple-choice questions with sufficient point-biserial correlation was also noted. No significant changes were noted in psychometric characteristics of the control group of items.ConclusionCorrection of item flaws, removal or replacement of non-functioning distractors, and enhancement of tested cognitive level positively impact the discriminatory ability of multiple-choice questions. This helps prevent construct-irrelevant variance from affecting the evidence of validity of scores obtained in multiple-choice questions.Electronic supplementary materialThe online version of this article (doi:10.1007/s40037-015-0212-x) contains supplementary material, which is available to authorized users.
Highlights
Assessment in undergraduate medical education is heavily reliant on multiple-choice questions
Masters et al [2] assessed multiple-choice assessments used in test banks accompanying selected nursing textbooks and found that around 47.3 % of the questions were written at low cognitive level of ‘plain factual recall’, and a meagre 6.5 % were written at the higher cognitive level of ‘analysis’
The experimental study presented here addressed the impact of item flaws, testing of a lower cognitive function and low distractor functioning on the quality of multiple-choice questions used in in-house exams
Summary
Assessment in undergraduate medical education is heavily reliant on multiple-choice questions. Jozefowicz et al [3] reported on the quality of in-house exams in three US medical schools using a five-point scale and reported a mean rating of 4.24 for item writers trained by the National Board of Medical Examiners and 2.03 for those without any formal training These works highlight the commonly perceived threats to the quality of multiple-choice assessments: item writing flaws and testing of lower cognitive function. This study investigated the impact of addressing item writing flaws, testing at low cognitive level and non-functioning distractors (< 5 % selection frequency) in multiple-choice assessment in preclinical medical education. Conclusion Correction of item flaws, removal or replacement of non-functioning distractors, and enhancement of tested cognitive level positively impact the discriminatory ability of multiple-choice questions This helps prevent construct-irrelevant variance from affecting the evidence of validity of scores obtained in multiple-choice questions
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
