Psychometric properties of the Ethiopian national licensing exam in medicine: an analysis of multiple-choice questions using classical test theory

Abstract

Background : The Ethiopian Ministry of Health introduced medical licensure examinations to maintain high standards in medical practice and build public trust in healthcare professionals. Studies also suggested significant issues in clinical competence among Ethiopian junior doctors as well concerns regarding unlicensed practice. Given the need to ensure safe health care, we investigated the psychometric properties of the multiple-choice items comprising the Ethiopian national licensing exam (NLE). These analyses help to provide an argument for the validity and reliability of the test scores. Method : We used a cross-sectional study design to analyze data from three cohorts of undergraduate medicine licensing examinations in Ethiopia (2020-2022, N = 2,213). Using Classical Test Theory, we assessed the psychometric properties of 600 MCQ items with 2400 single best answer choices, specifically item difficulty, item discrimination, and the number of nonfunctional distractors, and scale reliability. We provide results regarding the overall test and its sub-domains. Results : Ethiopia’s undergraduate medical licensure examination demonstrated acceptable reliability (Alpha > 0.80), with significant variability in item difficulty and examinee performance. Although these results indicate a sufficiently defensible exam, our results point to issues regarding item statistics, especially a high number of nonfunctional distractors. Conclusions : This study provides first evidence regarding the psychometric soundness of the Ethiopian NLE. However, a significant number of items should be carefully reviewed and possibly revised. As the examination is relatively new, ongoing refinement to item-development and review processes is essential to improve and ensure its quality.