Employing a Root Cause Analysis Process to Improve Examination Quality

Abstract

Multiple-choice question (MCQ) examinations represent a primary mode of assessment used by medical schools. It can be challenging for faculty to produce content-aligned, comprehensive, and psychometrically sound MCQs. Despite best efforts, sometimes there are unexpected issues with examinations. Assessment best practices lack a systematic way to address gaps when actual and expected outcomes do not align. The authors propose using root cause analysis (RCA) to systematically review unexpected educational outcomes. Using a real-life example of a class's unexpectedly low reproduction examination scores (University of Michigan Medical School, 2015), the authors describe their RCA process, which included a system flow diagram, a fishbone diagram, and an application of the 5 Whys to understand the contributors and reasons for the lower-than-expected performance. Using this RCA approach, the authors identified multiple contributing factors that potentially led to the low examination scores. These included lack of examination quality improvement (QI) for poorly constructed items, content-question and pedagogy-assessment misalignment, and other issues related to environment and people. As a result of the RCA, the authors worked with stakeholders to address these issues and develop strategies to prevent similar systematic issues from reoccurring. For example, a more robust examination QI process was developed. Using an RCA approach in health care is grounded in practice and can be easily adapted for assessment. Because this is a novel use of RCA, there are opportunities to expand beyond the authors' initial approach for using RCA in assessment.