Abstract
This article was migrated. The article was marked as recommended. Background: Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs impact validity as observed through student performance, item difficulty, and discrimination. Most previous studies have examined IWFs collectively and have shown that they have a diverse impact. The aim of the study was to determine if the effects of individual types of IWFs are systematic and predictable. Method: A cross-over study design was used. 100 pairs of MCQ items (with and without an IWF) were constructed to test 10 types of IWFs. Medical students were invited to participate in a mock examination. Paper A consisted of 50 flawed followed by 50 unflawed items. Paper B consisted of 50 unflawed followed by 50 flawed items. The effect of each of the IWFs on mean item scores, item difficulty and discrimination were examined. Results: The hypothesised effect of IWFs on mean item scores was confirmed in only 4 out of 10 cases. ' Longest choice is correct', 'Clues to the right answer (Eponymous terms)' and ' Implausible distractors' positively impacted, while 'Central idea in choices rather than stem' negatively impacted mean item scores. Other flaws had either the opposite or no statistically significant effect. IWFs did not impact item difficulty or discrimination. Conclusion: The effect of IWFs is neither systematic nor predictable. Unpredictability in assessment produces error and thus loss of validity. Therefore, IWFs should be avoided. Faculties should be encouraged to invest in item-writing workshops in order to improve MCQs. However, the cost of doing so should be carefully weighed against the benefits of developing programmes of assessment.
Highlights
Multiple choice questions (MCQs) are a common question type in medical assessment
Faculties should be encouraged to invest in itemwriting workshops in order to improve multiple choice questions (MCQs)
MCQs are susceptible to errors that may occur via item-writing flaws (IWFs), which are defined as violations of standard item-writing guidelines
Summary
Multiple choice questions (MCQs) are a common question type in medical assessment. They are familiar to educators, amenable to construction and administration by computer-based technology. (Schuwirth and van der Vleuten 2003) a frequently cited shortcoming of MCQs concerns the phenomenon of "cueing", which is the ability of a student to correctly recognise the answer from a list of options rather than spontaneously generate it. Other types of IWFs may mislead and penalise the student This may over or under-estimate student performance, introducing a source of error that compromises the validity of student scores. Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs impact validity as observed through student performance, item difficulty, and discrimination. The aim of the study was to determine if the effects of individual types of IWFs are systematic and predictable
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
