Item Analysis in Multiple Choice Questions: A Study on Question Difficulty and Authors’ Evaluation

Abstract

Introduction: Multiple choice questions (MCQs) are widely used in medical education. Difficulty Index (DIF I), Discrimination Index (DI), and Distractor Efficiency (DE) are the most commonly used item analyses. MCQ authors should anticipate the possible difficulty level in writing their questions and carefully arrange distractors. This study aims to evaluate the quality of MCQ and the predictions of MCQ authors about the difficulty levels of their questions. Methods: In this study, the DIF I, DI and DE values of 688 MCQs in the exams held in the first year of the 2021-2022 academic year of Dokuz Eylül University Faculty of Medicine were investigated. The effect of the number of Non-Functional Distractors (NFD) on DIF I and DI, and the compliance between the DIF I predictions of the question authors and the actual values was evaluated. Results: DIF I mean was 0.57±0.21, and 47.5% was at the ideal difficulty level. There was a significant difference between the estimated and the actual DIF I (p=0.002). The DI average was 0.31 ± 0.17, and the discrimination level of 43.6% was excellent. 36.8% of distractors were NFD. MCQ's difficulty and discriminatory ability were significantly different according to the number of NFDs (p=0.000 for all). Discussion: It was determined that the number of NFDs significantly affected difficulty and discriminatory ability. There was a difference between the estimates of the difficulty and the actual values. Reducing the number of options in MCQs and being more careful when crafting questions can improve the quality of the questions.