Toward Thoughtful Use of Shelf Exam Scores in Clerkship Assessment Systems.

Abstract

To the Editor: We agree with Dr. Schilling’s suggestion1 to proceed with caution when considering increasing the weight of the National Board of Medical Examiners subject (shelf) exam to circumvent the conjunctive use of an honors-eligibility cut score. Applying excessive weight to the shelf exam threatens the validity of the assessment system due to construct underrepresentation. That is, we could view the construct of interest in undergraduate clerkship education to be the characteristics of an entrustable professional. If we view clerkship grades as a form of tracking learner progress toward the 13 core entrustable professional activities (EPAs) for entering residency, we would contend that the shelf exam assesses at most 2 EPAs: “Prioritize a differential diagnosis following a clinical encounter” and “Recommend and interpret common diagnostic and screening tests.”2 Other forms of assessment targeting the remaining 11 EPAs are needed to ensure complete construct sampling. This is part of the reason why performance-based assessments, such as objective structured clinical examinations, and workplace-based assessments, including clinical performance evaluations, have value in this context. Further, Dr. Schilling states that a combined conjunctive and compensatory scoring system leads to “unreasonable outcomes”1 that make the assessment system unreliable. Medical education studies have borne this out, showing that conjunctive scoring leads to lower reliability of assessment.3 This is consistent with findings from the psychometrics and measurement literature.4 In an assessment system charged with making high-stakes decisions, reliability is of paramount importance. Educators must continue to circle back to this foundational idea throughout development, implementation, and utilization of assessment systems. Christopher Mattson, MDFourth-year resident, Department of Pediatrics—Medical Education Scholarship Track, University of Chicago Medical Center, Chicago, Illinois; [email protected]; ORCID: https://orcid.org/0000-0001-7315-7920.Yoon Soo Park, PhDDirector of health professions education research, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts; ORCID: https://orcid.org/0000-0001-8583-4335.