Toward (More) Valid Comparison of Residency Applicants' Grades: Cluster Analysis of Clerkship Grade Distributions Across 135 U.S. MD-granting Medical Schools.

Abstract

Purpose: Clerkship grades are a core element of medical school assessment and are used by residency program directors to select residency applicants, despite substantial variation in grading systems and distributions across institutions. 1–4 Despite calls to abandon the practice, 5 program directors are likely to continue comparing applicants across programs using clerkship grades, especially as the United States Medical Licensing Examination Step 1 goes pass/fail. Here, we sought to identify groups of medical schools for which the internal medicine clerkship grading patterns were similar, permitting reasonable interinstitutional comparisons of individuals. Method: Medical Student Performance Evaluations from applicants in the 2019 and 2020 residency application cycles from each of the 155 Liaison Committee on Medical Education-accredited U.S. MD-granting medical schools were examined. Twenty institutions were excluded: 9 recently established institutions and 1 military institution had no residency applicants; 6 institutions did not provide a grade distribution; and 4 institutions used pass/fail grading. The grading system and percent of students falling into each grading tier were tabulated for the internal medicine core clerkship (inpatient portion) at each institution. K-means cluster analysis was used to cluster institutions by their grade distribution. The number of clusters was selected using the scree plot elbow, within sum of squares, and proportional reduction of error. Results: In total, 135 medical schools reported 59 distinct internal medicine clerkship grading systems, including 2-tier (n = 2), 3-tier (n = 11), 4-tier (n = 19), 5-tier (n = 20), and 6+-tier (n = 7) systems. The percent of students receiving the top tier grade ranged from 2% to 66% between institutions. K-means clustering segmented the 135 grade distributions into 4 clusters: cluster 1 (45% top tier–45% next tier–10% lowest tier, n = 64 schools), cluster 2 (25%–30%–45%, n = 40), cluster 3 (20%–75%–5%, n = 25), and cluster 4 (15%–25%–25%–25%–10%, n = 6). Discussion: We identified substantial variation between institutions’ internal medicine clerkship grading systems and distributions, consistent with prior work. 2–5 Others have noted that not all “honors” grades are the same 4; similarly, not all “pass” grades are the same, with even more variation in the middle and bottom tiers. However, using cluster analysis, these seemingly incomparable grading systems were reduced to 4 clusters. The grading distributions within these groups of institutions range from roughly normally distributed (cluster 4) to highly skewed (cluster 1), laying bare differences in medical school grade inflation. Despite schools employing 46 different grading systems with 4 or more grading tiers, 129 of 135 schools (96%) effectively use only 3 grading tiers. We suspect institutions use lower grading tiers on only rare occasions. Significance: While we advocate against comparing students’ clerkship grades between institutions, residency program directors are likely to continue this practice. Given this reality, we recommend only comparing residency applicants between institutions within the same cluster. Implementing standards for internal medicine clerkship grading terminology (e.g., tier 1, tier 2) and distributions (i.e., embrace the 4 observed clusters) would make secondary use for resident selection more defensible.