Medical Student Performance Evaluation Research Articles

Analysis of variability and trends in medical school clerkship grades

BackgroundMedical school clerkship grades are used to evaluate orthopedic surgery residency applicants, however, high interinstitutional variability in grade distribution calls into question the utility of clerkship grades when evaluating applicants from different medical schools. This study aims to evaluate the variability in grade distribution among medical schools and look for trends in grade distribution over recent years. MethodsApplications submitted to Rush University's orthopedic surgery residency program from 2015, 2019, and 2022 were collected from the Electronic Residency Application Service. Applications from the top 100 schools according to the 2023–2024 U.S. News and World Report Research Rankings were reviewed. The percentage of “honors” grades awarded by medical schools for the surgery and internal medicine clerkships were extracted from applicants' Medical Student Performance Evaluation letters. ResultsThe median percentage of honors given in 2022 was 36.0 % (range 10.0–82.0) for the surgery clerkship and 33.0 % (range 6.7–80.0) for the internal medicine clerkship. Honors were given 6.6 % more in the surgery clerkship in 2022 compared to 2015. There was a negative correlation between a higher (worse) U.S. News and World Report research ranking and the percentage of honors awarded in 2022 for the surgery and internal medicine clerkships. ConclusionThere is substantial interinstitutional variability in the rate that medical schools award an “honors” grade with evidence of grade inflation in the surgery clerkship. Residency programs using clerkship grades to compare applicants should do so cautiously provided the variability demonstrated in this study.

Gender Bias in Letters of Recommendation: Relevance to Urology Match Outcomes and Pursuit of Fellowship Training/Academic Career

To review applications to a single urology residency program to determine application characteristics predictive of (1) successful match into urology residency and (2) pursuit of fellowship training and/or academic practice after completion of residency. Our principal variables of interest were gender bias as assessed in letters of recommendation (LOR), personal statements, Medical Student Performance Evaluation (MSPE), race, and gender. Applications submitted to our urology residency program in the 2014 cycle were reviewed. Twenty-three variables were analyzed, including applicant demographics, application materials, and gender bias. Deidentified text from LOR, personal statements, and MSPE was evaluated for gender bias using an open-source gender bias calculator. A subanalysis of applicants who matched at a top 25 urology program was performed. Logistic regression analysis was performed to identify applicant variables associated with (1) match success and (2) fellowship training or academic employment as of September 2021. Two hundred and twenty-two completed applications were analyzed. First authorship of a published manuscript was significantly associated with greater odds of matching. Female gender and top 25 medical school attendance were both significant predictors of matching at a top 25 urology program. The number of first-author publications was associated with completion of fellowship training or current employment in an academic position. First-author publications are the most important preinterview determinant of match success and subsequent pursuit of academic practice/fellowship training. Certain applicant characteristics are associated with matching at highly ranked programs. Gender bias in application materials (including LOR) does not appear to exert a significant influence on match and early career outcomes.

A Qualitative Analysis of Program Director Perspectives on the Usefulness of Medical Student Performance Evaluations.

A Qualitative Analysis of Program Director Perspectives on the Usefulness of Medical Student Performance Evaluations.

Key factors for residency interview selection from the National Resident Matching Program: analysis of residency Program Director surveys, 2016-2020.

As the number of medical school graduates continues to outpace the available residency training positions, applying for residency in the United States has become a highly competitive process, often associated with a low rate of selection and invitation for interview. The National Resident Matching Program (NRMP) Program Director survey provides data assessing factors considered by Program Directors (PD) in selecting and inviting candidates for interview. Assessing the evolution of these factors over time is efficacious to inform and guide prospective applicants toward improving preparation for residency application. We aim to synthesize NRMP data showing factors that PDs reported and rated as important in their decision to select and invite applicants for interview. Data from residency PD surveys from 2008 to 2021were accessed, but after applying inclusion/exclusion criteria, only the data from 2016 to 2020 were reviewed and analyzed. The NRMP survey reports provided two metrics that characterized PDs' evaluation of the residency factors for interview, namely, "percent citing factor" and "average rating" on a 0 to 5 Likert-type scale. These two metrics were combined into an aggregate measure of importance (AI), and another measure of relative importance (RI) was constructed from normalizing the AI of each individual factor to the sum of the AI within each survey year. The top ranked factors were United States Medical Licensing Examination (USMLE) Step 1/Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1, Letter of Recommendation (LOR) in the specialty, Medical Student Performance Evaluation (MSPE/Dean's Letter), and USMLE Step 2 Clinical Knowledge (CK)/COMLEX Level 2 Cognitive Exam (CE) score, any failed attempt in USMLE/COMLEX, and perceived commitment to specialty. Factors rising in importance were Audition Elective/Rotation Within Your Department, Personal Statement (PS), Perceived Commitment to Specialty, Perceived Interest in Program, LOR in the Specialty, Other Life Experience, and Personal Prior Knowledge of the Applicant. Factors with declining importance were Interest in Academic Career, Awards or Special Honors in Basic Sciences, Graduate of Highly Regarded US Medical School, Awards or Special Honors in Clinical Clerkships, Lack of Gaps in Medical Education, Awards or Special Honors in Clerkship in Desired Specialty, and Consistency of Grades. Compared to the 2021PD survey, our findings show continued predictive consistency, particularly related to specialty and program commitment. The factors identified for the selection of medical school graduates for interview into a residency program reveal that PDs move toward a more integrated approach. Specifically, PDs are placing increasing emphasis on factors that border on subjective qualities more so than the more traditional, quantitative, and objective metrics. Medical students and educators need to continually apprise themselves of the NRMP data to inform students' preparation endeavors throughout medical school to strengthen their application portfolios and enhance their competitiveness for the matching process.

Comparing Machine Learning Models and Human Raters When Ranking Medical Student Performance Evaluations.

The Medical Student Performance Evaluation (MSPE), a narrative summary of each student's academic and professional performance in US medical school is long, making it challenging for residency programs evaluating large numbers of applicants. To create a rubric to assess MSPE narratives and to compare the ability of 3 commercially available machine learning models (MLMs) to rank MSPEs in order of positivity. Thirty out of a possible 120 MSPEs from the University of Central Florida class of 2020 were de-identified and subjected to manual scoring and ranking by a pair of faculty members using a new rubric based on the Accreditation Council for Graduate Medical Education competencies, and to global sentiment analysis by the MLMs. Correlation analysis was used to assess reliability and agreement between student rank orders produced by faculty and MLMs. The intraclass correlation coefficient used to assess faculty interrater reliability was 0.864 (P<.001; 95% CI 0.715-0.935) for total rubric scores and ranged from 0.402 to 0.768 for isolated subscales; faculty rank orders were also highly correlated (rs=0.758; P<.001; 95% CI 0.539-0.881). The authors report good feasibility as the rubric was easy to use and added minimal time to reading MSPEs. The MLMs correctly reported a positive sentiment for all 30 MSPE narratives, but their rank orders produced no significant correlations between different MLMs, or when compared with faculty rankings. The rubric for manual grading provided reliable overall scoring and ranking of MSPEs. The MLMs accurately detected positive sentiment in the MSPEs but were unable to provide reliable rank ordering.

Using Natural Language Processing and Machine Learning to Identify Internal Medicine-Pediatrics Residency Values in Applications.

Although holistic review has been used successfully in some residency programs to decrease bias, such review is time-consuming and unsustainable for many programs without initial prescreening. The unstructured qualitative data in residency applications, including notable experiences, letters of recommendation, personal statement, and medical student performance evaluations, require extensive time, resources, and metrics to evaluate; therefore, previous applicant screening relied heavily on quantitative metrics, which can be socioeconomically and racially biased. Using residency applications to the University of Utah internal medicine-pediatrics program from 2015 to 2019, the authors extracted relevant snippets of text from the narrative sections of applications. Expert reviewers annotated these snippets into specific values (academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion; professionalism; and adaptability) previously identified as associated with resident success. The authors prospectively applied a machine learning model (MLM) to snippets from applications from 2023, and output was compared with a manual holistic review performed without knowledge of MLM results. Overall, the MLM had a sensitivity of 0.64, specificity of 0.97, positive predictive value of 0.62, negative predictive value of 0.97, and F1 score of 0.63. The mean (SD) total number of annotations per application was significantly correlated with invited for interview status (invited: 208.6 [59.1]; not invited: 145.2 [57.2]; P < .001). In addition, 8 of the 10 individual values were significantly predictive of an applicant's invited for interview status. The authors created an MLM that can identify several values important for resident success in internal medicine-pediatrics programs with moderate sensitivity and high specificity. The authors will continue to refine the MLM by increasing the number of annotations, exploring parameter tuning and feature engineering options, and identifying which application sections have the highest correlation with invited for interview status.

Primary Care Residency Perspectives on Medical Student Performance Evaluations.

With the transition of the United States Medical Licensing Examination (USMLE) Step 1 exam to pass-fail, residency directors are exploring alternative objective approaches when selecting candidates for interviews. The Medical Student Performance Evaluations (MSPE) portion of the application may be an area where objectivity could be provided. This study explored program directors' (PDs) perspectives on the utility of the MSPE as a discriminating factor for residency candidate selection. We invited PDs of primary care residencies listed in the American Medical Association FRIEDA database to participate in a mixed-methods study assessing opinions on the MSPE, and the importance of student skills and application components when considering a candidate for interview. We obtained summary statistics for Likert-scale responses. We used inductive thematic analysis to generate themes from open-ended comments. Two hundred forty-nine PDs completed the survey (response rate=15.9%). Patient communication (83.6%) and teamwork (81.9%) were rated as very/extremely important skills, and being a graduate of a US medical school in the past 3 years (73.1%), no failures on board exams (58.2%), and MSPEs (54.8%) were rated as very/extremely important application components. Six hundred seventy-eight open-ended comments yielded themes related to desire for more transparency and standardization, importance of student attributes and activities, and other important components of applications. PDs place a high value on the MSPE but find it limited by concerns over validity, objectivity, and lack of standardization. The quality of MSPEs may be improved by using a common language of skill attainment such as the Association of American Medical Colleges' Entrustable Professional Activities and using the document to discuss students' other attributes and contributions.

An Elite Privilege: Top-Ranked Medical Schools Provide Fewer Comparative Performance Data on Their Students.

The objective of this study was to determine differences in the reporting of performance data on medical student performance evaluations (MSPEs) by medical school ranking. MSPEs from all US allopathic and osteopathic medical schools received by a single diagnostic radiology residency program during the 2021-2022 application cycle were retrospectively reviewed. Preclinical class and core clerkship grades were categorized as pass/fail or multitiered. Comparative summative assessments provided in the MSPEs were recorded. Medical schools were grouped by their US News & World Report rankings, and the proportion of reported performance metrics for each group was compared. Information from 95% of US allopathic medical schools (148 of 155) and 73% of osteopathic medical schools (27 of 37) was collected, on the basis of 1,046 applications received. For preclinical classes, multitiered grading was reported by no schools ranked in the top 10, 17% of schools ranked 11th to 50th, 52% of schools ranked 51st to 100th, and 59% of unranked schools (P < .001). For core clinical clerkships, multitiered grades were reported by 70% of the top 10 ranked schools, 90% of schools ranked 11th to 50th, 94% of those ranked 51st to 100th, and 94% of unranked schools (P= .0463). Comparative summative assessments were reported by none of the top 10 ranked schools, 56% of schools ranked 11th to 50th, 80% of those ranked 51th to 100th, and 81% of unranked schools (P < .001). Higher ranked medical schools are less likely to provide comparative assessment data on their MSPEs, which may disadvantage students from lower ranked medical schools.

Embracing our responsibility to ensure trainee competency.

Embracing our responsibility to ensure trainee competency.

Identifying Meaningful Patterns of Internal Medicine Clerkship Grading Distributions: Application of Data Science Techniques Across 135 U.S. Medical Schools.

Residency program directors use clerkship grades for high-stakes selection decisions despite substantial variability in grading systems and distributions. The authors apply clustering techniques from data science to identify groups of schools for which grading distributions were statistically similar in the internal medicine clerkship. Grading systems (e.g., honors/pass/fail) and distributions (i.e., percent of students in each grade tier) were tabulated for the internal medicine clerkship at U.S. MD-granting medical schools by manually reviewing Medical Student Performance Evaluations (MSPEs) in the 2019 and 2020 residency application cycles. Grading distributions were analyzed using k-means cluster analysis, with the optimal number of clusters selected using model fit indices. Among the 145 medical schools with available MSPE data, 64 distinct grading systems were reported. Among the 135 schools reporting a grading distribution, the median percent of students receiving the highest and lowest tier grade was 32% (range: 2%-66%) and 2% (range: 0%-91%), respectively. Four clusters was the most optimal solution (η 2 = 0.8): cluster 1 (45% [highest grade tier]-45% [middle tier]-10% [lowest tier], n = 64 [47%] schools), cluster 2 (25%-30%-45%, n = 40 [30%] schools), cluster 3 (20%-75%-5%, n = 25 [19%] schools), and cluster 4 (15%-25%-25%-25%-10%, n = 6 [4%] schools). The findings suggest internal medicine clerkship grading systems may be more comparable across institutions than previously thought. The authors will prospectively review reported clerkship grading approaches across additional specialties and are conducting a mixed-methods analysis, incorporating a sequential explanatory model, to interview stakeholder groups on the use of the patterns identified.

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail.

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations.

The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018-2022). NCs were converted into single words and characterized into word categories: ability (e.g., "talent"), standout ("best"), grindstone ("meticulous"), communal ("caring"), or agentic ("ambitious"). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval [CI] = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared with women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03; 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48, 0.92; P = .02). The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.

The Consequences of Step 2 Clinical Skills Examination Discontinuation for Medical Schools and Sustainability Plans for Clinical Skills Assessment.

Comprehensive clinical skills examinations using standardized patients are widely used to assess multiple physician competencies. However, these exams are resource intensive. With the discontinuation of the Step 2 Clinical Skills (CS) exam in 2021, how medical schools will change their approaches to comprehensive clinical skills exams is unknown. This study explores school responses to this change and future directions of comprehensive clinical skills exams using the program sustainability framework. This cross-sectional, descriptive study surveyed medical school curriculum deans at 150 Liaison Committee on Medical Education-accredited U.S. medical schools from September to October 2021. The 30-question survey included questions about medical school and participant role, current comprehensive clinical skills exams, sustainability dimensions, and challenges and future directions. Descriptive statistics were used to characterize responses, and content analysis was used to identify themes in the open-ended responses. Educators at 75 of 150 institutions (50%) responded. Sixty-three respondents (84%) reported conducting a comprehensive clinical skills exam. The comprehensive clinical skills exam assessed readiness for graduation (51 [81%]), provided feedback for students (49 [78%]), evaluated curricula (38 [60%]), provided information for medical student performance evaluation or communication with residency (10 [16%]), and assessed other factors (6 [10%]), including preparation for Step 2 CS in the past and readiness for advancement to fourth year of medical school (multiple responses were allowed). Factors facilitating sustainability included sufficient funding to continue the exam (55 [87%]) and the belief that clinical skills assessment in medical school is now more important after discontinuation of the Step 2 CS exam (55 [87%]). Challenges to sustainability included organizational capacity and limited interinstitutional collaboration. Educators remain committed to the purpose of comprehensive clinical skills exams. Adapting to changed licensing requirements while sustaining clinical skills exams enables innovation and improvement in assessment of clinical competence.

Toward (More) Valid Comparison of Residency Applicants' Grades: Cluster Analysis of Clerkship Grade Distributions Across 135 U.S. MD-granting Medical Schools.

Purpose: Clerkship grades are a core element of medical school assessment and are used by residency program directors to select residency applicants, despite substantial variation in grading systems and distributions across institutions. 1–4 Despite calls to abandon the practice, 5 program directors are likely to continue comparing applicants across programs using clerkship grades, especially as the United States Medical Licensing Examination Step 1 goes pass/fail. Here, we sought to identify groups of medical schools for which the internal medicine clerkship grading patterns were similar, permitting reasonable interinstitutional comparisons of individuals. Method: Medical Student Performance Evaluations from applicants in the 2019 and 2020 residency application cycles from each of the 155 Liaison Committee on Medical Education-accredited U.S. MD-granting medical schools were examined. Twenty institutions were excluded: 9 recently established institutions and 1 military institution had no residency applicants; 6 institutions did not provide a grade distribution; and 4 institutions used pass/fail grading. The grading system and percent of students falling into each grading tier were tabulated for the internal medicine core clerkship (inpatient portion) at each institution. K-means cluster analysis was used to cluster institutions by their grade distribution. The number of clusters was selected using the scree plot elbow, within sum of squares, and proportional reduction of error. Results: In total, 135 medical schools reported 59 distinct internal medicine clerkship grading systems, including 2-tier (n = 2), 3-tier (n = 11), 4-tier (n = 19), 5-tier (n = 20), and 6+-tier (n = 7) systems. The percent of students receiving the top tier grade ranged from 2% to 66% between institutions. K-means clustering segmented the 135 grade distributions into 4 clusters: cluster 1 (45% top tier–45% next tier–10% lowest tier, n = 64 schools), cluster 2 (25%–30%–45%, n = 40), cluster 3 (20%–75%–5%, n = 25), and cluster 4 (15%–25%–25%–25%–10%, n = 6). Discussion: We identified substantial variation between institutions’ internal medicine clerkship grading systems and distributions, consistent with prior work. 2–5 Others have noted that not all “honors” grades are the same 4; similarly, not all “pass” grades are the same, with even more variation in the middle and bottom tiers. However, using cluster analysis, these seemingly incomparable grading systems were reduced to 4 clusters. The grading distributions within these groups of institutions range from roughly normally distributed (cluster 4) to highly skewed (cluster 1), laying bare differences in medical school grade inflation. Despite schools employing 46 different grading systems with 4 or more grading tiers, 129 of 135 schools (96%) effectively use only 3 grading tiers. We suspect institutions use lower grading tiers on only rare occasions. Significance: While we advocate against comparing students’ clerkship grades between institutions, residency program directors are likely to continue this practice. Given this reality, we recommend only comparing residency applicants between institutions within the same cluster. Implementing standards for internal medicine clerkship grading terminology (e.g., tier 1, tier 2) and distributions (i.e., embrace the 4 observed clusters) would make secondary use for resident selection more defensible.

Racial Bias on the Emergency Medicine Standardized Letter of Evaluation.

Studies on components of residency applications have shown evidence of racial bias. The Standardized Letter of Evaluation (SLOE) is an assessment measure for emergency medicine (EM) residency applications and, as more specialties opt to use SLOEs in place of narrative letters of recommendation, understanding bias on standardized assessments is essential. To determine whether there is a difference in rankings on the EM SLOE between underrepresented in medicine (UIM) and non-UIM applicants, White and non-White applicants, and to examine whether differences persist after controlling for other characteristics. The sample was drawn from medical students who applied to EM residency at the study institution in 2019. We compared rankings between UIM and non-UIM students and between students of each individual race/ethnicity and White students, after controlling for United States Medical Licensing Examination Step scores, Alpha Omega Alpha status, type of school (US MD, US DO, internation medical graduate), Medical Student Performance Evaluation class percentile, affiliated program vs visiting clerkship SLOE, gender and the interaction of race/ethnicity and gender, and adjusted for students submitting multiple SLOEs, using ordinal regression. There were 1555 applicants to the study institution in 2019; 1418 (91.2%) had a SLOE and self-identified race/ethnicity. After controlling for applicant characteristics, non-UIM students were significantly more likely to be ranked higher than UIM students on "Rank Against Peers," (OR 1.46, 95% CI 1.03-2.07) and Grade (OR 1.46, 95% CI 1.05-2.04). Analysis of EM SLOEs submitted to our institution demonstrates racial bias on this standardized assessment tool, which persists after controlling for other performance predictors.

A multi-year evaluation of medical student performance on and perceptions of collaborative gross anatomy laboratory examinations.

Collaborative testing and its benefits have been reported in diverse disciplines across different types of academic institutions. However, there has been minimal research conducted on collaborative assessments in medical schools, particularly in the gross anatomy laboratory. The objectives of this study were to explore the effect of collaborative anatomy laboratory examinations on student performance and to gauge student perceptions of this assessment format. This study examined five academic years of medical students' performance on a two-stage, collaborative anatomy laboratory examination wherein each student's overall score was a weighted combination of scores from the individual and team examination. Analyses of a descriptive survey capturing students' perceptions of the assessment method were also performed. Individual examination averages increased since implementing the collaborative assessment (p < 0.001), and team examination averages were higher than individual examination averages (p < 0.001). Teams outperformed each of their team members 98% of the time. Teams had a greater than 0.90 incidence of answering a question correctly if more than one person in the group got the answer correct on the individual portion, and a 0.66 incidence of answering correctly if only one person in their group answered correctly on the individual portion. Student feedback identified the discussions and learning that took place during the team portion to be a beneficial feature of this assessment format. Students also reported that this collaborative assessment made them feel a higher level of responsibility to perform well, and that it improved their understanding of gross anatomy.

Variation in core clerkship grading reported on the Medical Student Performance Evaluation (MSPE) for orthopaedic surgery applicants: a retrospective review

Background: Heterogeneity of grading schema among medical schools complicates utility of core clerkship grades reported. The purpose of this study was to understand the variation in number and verbiage in grading schemes during third-year core clerkships for orthopaedic surgery residency applicants. Methods: Applications to a single institution’s orthopaedic surgery residency program during the 2017-2018 match cycle were reviewed. Data extracted from the Medical Student Performance Evaluation (MSPE) included medical school name, number of core clerkships, grade options/tiers, and % grade distribution in clerkships. Applicant data collected included Step 1 Score, Step 2 Clinical Knowledge (CK) Score, Alpha Omega Alpha (AOA) membership status, clerkship grades, and medical school rank. Results: A total of 858 applications from 211 medical schools were reviewed. Further analysis was performed on 142 schools, representing 721 students, that reported grading distributions. The number of grade tiers varied from two to 11, with three (26.1%), four (43.7%), and five (20.4%) tiered grading systems being the most common. One-hundred unique verbiages were identified to describe grading among all schools. Schools ranked in the top 25 distributed honors more often than schools ranked outside the top 25 (P&lt;0.001). The median for the average percentage of honors distributed by a school was 32.3%, with a total range of 2.4% to 72.6%. A significant relationship between applicant match success and medical school grading practices could not be determined (P=0.054). Conclusions: Significant differences in assigned grades by medical schools for third-year core clerkships were found. Therefore, students’ core clerkship grades should be reviewed in the context of the grade distributions at their medical schools. Level of Evidence: Level IV.

Step 1 is Pass/Fail, Now What? Can Clinical Clerkship Grades be Used as a Reliable Metric to Screen General Surgery Residency Applicants?

IntroductionFor decades, the three-digit United States Medical Licensing Exam Step 1 score has been used to competitively evaluate and compare candidates during the residency application process. Starting in 2022, however, all Step 1 scores will be converted to pass/fail. A different quantitative measure will likely gain importance in its stead, one such being clerkship performance grades. This study aims to determine the consistency of class rank and distribution of clerkship grades reported by medical schools for applicants to a general surgery program. MethodsCandidates’ Medical Student Performance Evaluation letters from 141 unique US allopathic medical schools were reviewed for student overall class rank, the number of grading tiers in each clerkship, and the percent achieving honors criteria in each clerkship from the 2020 application cycle. Comparative analysis was performed by region and medical school prestige. ResultsMost medical schools rank students using a four-tier system (e.g., fail, pass, high pass, and honors). A third of schools do not provide an overall class rank of students (34.7% of schools); this was most prevalent in the Northeast and Western regions. Schools in the Central US more often rank their students in five tiers compared to the South (P < 0.01). The percent of students that achieve the highest grading tier varies across the core clerkships (mean 37.1%, range 6.5%-78%); an average of 34.5% of students meet the highest honors tier in their Surgery clerkship. Students at US News and World Report Top 20 medical schools are more likely to receive the highest honors tier, across all core clerkships and overall class rank, than students at schools outside the Top 20 (P < 0.05). ConclusionsIn the absence of the United States Medical Licensing Exam Step 1 score, the variability in clerkship grading tiers and overall class rank will likely pose a challenge to residency programs’ ability to stratify desirable applicants. Further transparency and standardization may be required to compare students objectively and fairly from medical schools across the country.

A Retrospective Analysis of Medical Student Performance Evaluations, 2014-2020: Recommend with Reservations.

The Medical Student Performance Evaluations (MSPE) is a cornerstone of residency applications. Little is known regarding adherence to Association of American Medical Colleges (AAMC) MSPE recommendations and longitudinal changes in MSPE content. Evaluate current MSPE quality and longitudinal changes in MSPE and grading practices. Retrospective analysis. Students from all Liaison Committee on Medical Education (LCME)-accredited medical schools from which the Stanford University Internal Medicine residency program received applications between 2014-2015 and 2019-2020. Inclusion of key words to describe applicant performance and metrics thereof, including distribution among students and key word assignment explanation; inclusion of clerkship grades, grade distributions, and grade composition; and evidence of grade inflation over time. MSPE comprehensiveness varied substantially among the 149 schools analyzed. In total, 25% of schools provided complete information consistent with AAMC recommendations regarding key word/categorization of medical students and clerkship grades in 2019-2020. Seventy-seven distinct key word terms appeared across the 139 schools examined in 2019-2020. Grading practices markedly varied, with 2-83% of students receiving the top internal medicine clerkship grade depending on the year and school. Individual schools frequently changed key word and grading practices, with 33% and 18% of schools starting and/or stopping use of key words and grades, respectively. Significant grade inflation occurred over the 6-year study period, with an average 14% relative increase in the proportion of students receiving top clerkship grades. A minority of schools complies with AAMC MSPE guidelines, and MSPEs are inconsistent across time and schools. These practices may impair evaluation of students within and between schools.

Psychiatry Residency Directors' Attitudes Toward and Uses of the Medical Student Performance Evaluation and Other Potential Tools for Residency Selection.

A survey was conducted to determine US psychiatry residency directors' attitudes regarding current measures of medical student performance and their preferences for the future. A team of psychiatry medical student educators and residency program directors developed a 23-question survey. In July 2021, links to the survey were sent out to all program directors registered with the American Association of Directors of Psychiatric Residency Training. Seventy program directors out of 223 initiated the survey, resulting in a response rate of 31.4%. Forty percent of respondents reported that the most important use of the Medical Student Performance Evaluation (MSPE) is in screening out applicants for interviews, and only 26.1% reported that the MSPE in its current form could be trusted to provide a valid and reliable assessment of a student's medical school performance. Most respondents agreed that in the absence of United States Medical Licensing Examination (USMLE) step 1 numerical scores, the existing MSPE format/content requirements should be modified, use a set of ranking categories that are uniform across all medical schools, and be supplemented with additional measures of the student's character and ability specific to psychiatry. US psychiatry program directors are eager for change when it comes to the MSPE and how it reports rankings, grades, and professionalism. The transition of the USMLE step 1 score reporting to pass/fail presents an opportunity to pursue this change and for stakeholders from all medical specialties to work together toward a shared goal of an improved residency selection process.