Multiple-Choice Questions in Small Animal Medicine: An Analysis of Cognitive Level and Structural Reliability, and the Impact of these Characteristics on Student Performance.

Abstract

Students entering the final year of the veterinary curriculum need to integrate information and problem solve. Assessments used to document competency prior to entry to the clinical environment should ideally provide a reliable measurement of these essential skills. In this study, five internal medicine specialists evaluated the cognitive grade (CG) and structural integrity of 100 multiple-choice questions (MCQs) used to assess learning by third-year students at a United States (US) veterinary school. Questions in CG 1 tested factual recall and simple understanding; those in CG 2 required interpretation and analysis; CG 3 MCQs tested problem solving. The majority (53%) of questions could be answered correctly using only recall or simple understanding (CG 1); 12% of MCQs required problem solving (CG 3). Less than half of the questions (43%) were structurally sound. Overall student performance for the 3 CGs differed significantly (92% for CG 1 vs. 84% for CG 3; p = .03. Structural integrity did not appear to impact overall performance, with a median pass rate of 90% for flawless questions versus 86% for those with poor structural integrity (p = .314). There was a moderate positive correlation between individual student outcomes for flawless CG 1 versus CG 3 questions (rs = 0.471; p = < .001), although 13% of students failed to achieve an aggregate passing score (65%) on the CG 3 questions. These findings suggest that MCQ-based assessments may not adequately evaluate intended learning outcomes and that instructors may benefit from guidance and training for this issue.