Unacknowledged Alternate Determinants of Medicine Subject Exam Performance.

Abstract

To the Editor: We read with interest Fitz and colleagues’ 2011–2014 retrospective cohort study 1 examining associations between internal medicine clerkship characteristics and NBME medicine subject exam scores. We appreciated the breadth of clerkship characteristics investigated and methodological rigor. However, the authors’ conclusions about medicine subject exam performance as an assessment of “overall medical knowledge” and clerkship start month merit further discussion. Fitz and colleagues state that the medicine subject exam assesses “overall medical knowledge” while reporting that United States Medical Licensing Examination Step 1 performance accounted for 46% of the variance in medicine subject exam performance. Step 1, a medical basic-science examination, does not assess knowledge in isolation. Medical College Admission Test performance accounts for 60% of the variance in Step 1 scores, 2 suggesting that Step 1—and by extension, the medicine subject exam—may function as assessments of students’ test-taking abilities. Further, students’ increased familiarity with the medicine subject exam format may contribute to shelf performance improvements over time. Indeed, improved familiarity with the test format can lead to construct-irrelevant differences in examination scores. 3 Second, the authors report “a significant association between academic start month and medicine subject exam performance.” This association has been previously reported 4 and may be explained in part by shifts in the timing of Step 1 that accompany reductions in preclinical curricula length. The lack of structured knowledge consolidation that Step 1 study affords may account for differences in medicine subject exam scores that appear to be associated with clerkship start month. 3 This nuance in Step 1 timing, however, was not adequately addressed in the authors’ study. Fitz and colleagues’ study examined numerous educational variables but identified Step 1 as the most significant contributor to medicine subject exam score variance. The secondary associations that the authors found may be linked to Step 1 timing or performance. With 54% of the variance in medicine subject exam performance unexplained, it is important to consider Fitz and colleagues’ analysis not as an endorsement of the medicine subject exam, but as a call to further investigate the use of this examination for clinical competency assessment.