States Medical Licensing Examination Scores Research Articles

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ±63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match

To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM). Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit. A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P<.001). As cutoff score increases, percentage of URM applicants decreases. The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

Geographic Reach of Surgery Residency Applicants During In-Person and Virtual Interviews.

Background Virtual interviews for surgery residency may improve interview opportunities for applicants from underrepresented in medicine (UIM) and lower socioeconomic backgrounds. Objective To compare the geographic reach of surgical residency applicants during in-person versus virtual interviews. Methods This study compared applicants for the 2019 (in-person) and 2020 (virtual interviews) application cycle for surgery residency. Geographic reach (GR) was defined as the distance between applicants' current location and the program. Federal Financial Institutions Examination Council's website supplied socioeconomic data using applicants' geographic locations. Applicant demographics, United States Medical Licensing Examination (USMLE) scores, and geographic distance to program were collected. Multivariable analyses examined GR with interaction terms between interview type, UIM status, and socioeconomic status, while controlling for USMLE scores. Results A total of 667 (2019) and 698 (2020) National Resident Matching Program applications were reviewed. Overall, there was no difference in GR for applicants during in-person and virtual interviews in multivariable testing. UIM status had no association with GR for in-person interviews, but virtual interviews were associated with an increased GR for UIM applicants compared to non-UIM applicants (235.17; 95% CI 28.87-441.47; P=.02). For in-person interviews, applicants living in communities with poverty levels ≥7% had less GR vs those in communities with levels <7% (-332.45; 95% CI -492.10, -172.79; P<.001), an effect not observed during virtual interviews. Conclusions There was no difference in overall GR, or the proportion of UIM applicants or those from higher poverty level communities, but virtual survey interviews during the COVID-19 pandemic were associated with increased GR for UIM and from lower socioeconomic backgrounds applicants.

A Systematic Review of the State of Preclinical Mentorship Programs in Plastic and Reconstructive Surgery.

Plastic and reconstructive surgery is consistently one of the most competitive medical specialties in the match. The recent United States Medical Licensing Examination score reporting switch to pass-fail led to a change in metrics by which applicants are evaluated by plastic surgery programs. Applicant research productivity and the demand for plastic surgery mentorship will continue to rise. Given the competitive nature of the residency match and shift in metrics emphasis after the change in STEP 1 scoring, early exposure to plastic surgery and mentoring relationships are paramount to applicant success. However, most medical students are not exposed to plastic surgery until they begin clinical rotations. A literature review of plastic surgery mentorship programs available during preclinical years was conducted to identify preclinical mentorship opportunities in plastic surgery. Sixty-eight references were identified, but only two studies met the inclusion criteria of addressing mentorship programs in preclinical years. Examination of the included studies indicated that preclinical medical students achieve self-identified goals and generate longitudinal benefits in plastic surgery by participating in early and focused mentorship programs. The limited number of studies in this review highlights a lack of available, studied preclinical mentorship programs in plastic surgery and reveals a knowledge gap concerning the creation of successful preclinical mentorship programs. Early exposure to plastic surgery, combined with the development of structured preclinical mentorship programs, can potentially replicate successful outcomes seen in other surgical subspecialties' mentorship programs while addressing the lack of formalized mentorship opportunities for preclinical students in plastic surgery.

Minimum United States Medical Licensing Examination and Comprehensive Osteopathic Medical Licensing Examination Scores Often Do Not Align.

Background Many residency programs do not accept the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) alone for osteopathic applicants. Furthermore, among those programs that do accept the COMLEX-USA, it is unknown how programs scale their minimum COMLEX-USA scores compared to their minimum United States Medical Licensing Examination (USMLE) scores. Objective Our objective was to examine the variation of relative within-program differences between minimum USMLE Step and COMLEX-USA Level scores required for consideration by United States residency programs. Methods We performed a cross-sectional analysis of the Fellowship and Residency Electronic Interactive Database Access (FREIDA) database from April 2023, including the 10 specialties with the most training spots in 2022. These specialties were internal medicine, family medicine, pediatrics, emergency medicine, psychiatry, surgery, anesthesiology, obstetrics-gynecology, orthopedic surgery, and neurology. Within-program differences were calculated by subtracting the minimum USMLE Step 1 and 2 scores from the converted minimum USMLE Step 1 and 2 scores calculated from the minimum COMLEX-USA Level 1 and 2 scores using two conversion tools. We present differences as medians with interquartile ranges (IQR). Additionally, we report the proportion of programs with greater than 10-point differences for each step (1 and 2). Results Of the 3,364 accredited programs from the examined specialties, we included 1,477 in the Step 1 analysis and 1,227 in the Step 2 analysis with complete data. The median within-program difference between the minimum Step 1 score and the predicted Step 1 score was 12.0 (IQR 2.0, 17.0) using the Barnum and colleagues' conversion tool and -1.7 (IQR -6.2, 6.3) using the Smith and colleagues' tool. The median differences for Step 2 were 2.0 (IQR -8.0, 12.0) and -6.5 (IQR -13.9, -1.5) for each tool, respectively. Using the Barnum and Smith conversion tools, 937 (63%) and 435 (29%) programs had a greater than 10-point Step 1 score difference, respectively. Similarly, for Step 2, 564 (46%) and 515 (42%) programs had a greater than 10-point difference with each conversion tool. Conclusion There is wide variation in the within-program differences between minimum USMLE and predicted minimum USMLE (from COMLEX-USA) scores. Many programs have greater than 10-point differences, which may be a source of bias in osteopathic applicant selection.

Barriers in Ophthalmology Residency Applications for Students Identifying as Underrepresented in Medicine: A San Francisco Match Analysis

There is a significant lack of ophthalmologists who self-identify as underrepresented in medicine (URiM) in the physician workforce. Prior literature has revealed bias in traditional metrics for selection relied on by resident programs such as United States Medical Licensing Examination (USMLE) scores, letters of recommendation (LOR), and induction into medical honors societies such as Alpha Omega Alpha (AOA). The purpose of this study was to elucidate race-based differences in word usage within ophthalmology residency letters of recommendation that may disproportionately affect URiM applicants. This was a retrospective, cohort study. This was a multicenter study across the Wilmer Eye Institute at Johns Hopkins, the University of California San Francisco, and the University of North Carolina at Chapel Hill. San Francisco (SF) Match applications submitted to three ophthalmology residency programs between 2018 and 2020 were reviewed. URiM status, USMLE Step 1 score, and AOA membership were recorded. Letters of recommendation were analyzed using text analysis software. T-tests and chi-squared or Fisher's exact tests were used to compare continuous and categorical variables, respectively. Frequency of word/summary term usage in letters of recommendation were the main outcome measures. Relative to non-URiM applicants, URiM applicants had lower USMLE Step 1 scores (mean difference=7.0; p<0.001). Non-URiM letters of recommendation were more likely to describe applicants as "dependable" (p=0.009) and highlight "research" (p=0.046). URiM letters were more likely to describe applicants as "warm" (p=0.02) and "caring" (p=0.02). This study identified potential barriers for URiM ophthalmology residency applicants which can help guide future interventions to increase workforce diversity.

Predictive Value of Preresidency Academic Metrics on Resident Publication Potential.

Otolaryngology residency is highly competitive, and applicant academic metrics are scrutinized. The predictive value of preresidency academic metrics on applicants' future research productivity and career aspirations remains largely undefined. Retrospective cohort study. Academic otolaryngology department, 2014 to 2015. Applicant demographics, publication history, and United States Medical Licensing Examination (USMLE) scores were downloaded from Electronic Residency Application Service archives. Publications during residency were tallied from all PubMed articles indexed between July 1, 2015 and June 30, 2020. Postresidency career paths were examined by 2 investigators (D.J.C. and L.X.Y.) using Google searches with an emphasis on program websites, Doximity, and LinkedIn profiles. Associations with publication potential and postresidency positions were evaluated with Spearman rank correlation coefficients and Kruskal-Wallis, Wilcoxon rank sum, and χ 2 tests. Of 321 applicants, 226 (70%) matched, and 205 (64%) completed residency by June 2020. Matched residents published a median of 4 (range: 0-41) manuscripts during residency. USMLE scores, Alpha Omega Alpha status, and the number of preresidency publications did not significantly correlate with publication potential during residency. The number of research experiences had a significant positive correlation with publications during residency (p < 0.001). Asian race (p = 0.002) and geographical region of residency (p < 0.001) also had significant associations with publication potential. Of the 205 graduates, 118 (58%) enrolled in fellowship. Age and female sex (74% vs 48%; p = 0.002) were the only factors significantly associated with pursuing a fellowship. In otolaryngology, not all preresidency academic metrics are associated with publication potential during residency or propensity for fellowship training. Programs should not use academic metrics alone to predict an applicant's future research productivity or career trajectory.

The Impact of Holistic Review of Urology Residency Applications on Selection for Interview During the COVID-19 Pandemic

To investigate the impact of a holistic review of urology residency applications on interview selection at our institution during the COVID-19 pandemic. In the 2019-2020 cycle, applicants were filtered by a Step 1 score of 230 and whether they applied from selected east coast medical schools. For the 2020-2021 and 2021-2022 cycles, we implemented a scoring system which focused on desirable attributes based on our program training needs and resources. We compared applicant and interviewee demographics and United States Medical Licensing Examination (USMLE) scores using descriptive statistics and 1-way analysis of variance tests. A total of 282, 300, and 367 students applied to our residency program with 50, 45, and 52 selected for interviews during the 2019-2020, 2020-2021, and 2021-2022 cycles, respectively. Compared to 2019-2020, the 2020-2021 and 2021-2022 interviewee cohorts comprised of more non-tri-state applicants (36%, 55.6%, and 46.2%, respectively). Underrepresented minority representation increased for the 2020-2021 interviewee cohort; however, this was not observed in 2021-2022 (16%, 24.4%, 15.4%, respectively). Additionally, USMLE Step 1 and 2 scores were similar between interviewee cohorts in 2019-2020, 2020-2021 and 2021-2022, respectively (Step 1: 244.2 ± 8.8, 242 ± 12.1, 242.8 ± 12.4, P = .624) (Step 2: 249.1 ± 11.5, 251.5 ± 10.5, 254.4 ± 10.8, P = .143). Utilizing a comprehensive review resulted in a geographically diverse interview pool and no significant difference in academic performance among interviewees. Holistic review provides an alternative, balanced evaluation of residency applicants which may increase diversity in urology.

Applicant characteristics and interview-day structure of shoulder and elbow surgery fellowships

Background: The purpose of this study was to determine the selection criteria and interview-day structure used by shoulder and elbow fellowship program directors. Methods: An anonymous 14-question survey was distributed to the program directors of the 31 American Shoulder and Elbow Surgeons (ASES) recognized orthopaedic shoulder and elbow fellowship programs. Surveys were sent via email, and responses were collected from March 31, 2021 to May 10, 2021. Results: All 31 fellowship directors responded (100%). The most valuable selection criteria for applicants were subjective measures: quality of interview, letters of recommendation, and quality of residency program. Objective criteria such as United States Medical Licensing Examination (USMLE) scores and Orthopaedic In-Training Examination (OITE) scores were less important. Fifteen (50%, 15/30) programs required applicants to have one to two publications, while 15 (50%, 15 of 30) required three to five publications. Programs most commonly interviewed 16 to 20 residents (38.7%). The interview day was a half day for the majority of programs (n=20; 64.5%), with most programs utilizing three (25.8%) or four (45.2%) separate interviews. Current fellows interviewed applicants in about half of the programs (51.6%). Conclusions: Shoulder and elbow fellowship program directors prioritize subjective over objective attributes when selecting applicants. The interview day is most commonly a half day with each applicant undergoing multiple interviews at each institution. Understanding the selection process used by shoulder and elbow fellowship directors provides a valuable resource for residents. Level of Evidence: Level V.

A 5-Year Analysis of the Integrated Plastic Surgery Residency Match: The Most Competitive Specialty?

The National Residency Matching Program is becoming more competitive across all medical specialties. The match rate for surgical specialties is considerably lower than the overall match rate each year. For those wishing to match into integrated plastic surgery, the magnitude of difficulty is in question. Public data from 2016 to 2020 on the number of residency programs, number of residency positions, number of applicants, United States Medical Licensing Examination (USMLE) scores, and research experiences were collected for dermatology, categorical general surgery, neurological surgery, ophthalmology, orthopedic surgery, otolaryngology, integrated plastic surgery, integrated thoracic surgery, urology, and integrated vascular surgery. In the 2020 Match, integrated plastic surgery had only 82 programs and 180 positions, and over the 2016-2020 Matches, the growth of integrated plastic surgery was a mere 6.9 positions per year. For matched 2020 applicants, integrated plastic surgery had the highest USMLE Step 1 score (249), highest USMLE Step 2 score (tied at 256), and second highest number of abstracts/presentations/publications (19.1). The limited availability of residency spots in integrated plastic surgery, in conjunction with the quality of the applicant pool, makes it one of the most competitive matches. Candidates should understand this context before applying as to not risk going unmatched.

A Call to Evaluate Manual Dexterity of Prospective Surgical Trainees

IntroductionThis study assesses the correlation between academic grades and gross and fine motor skills in prospective surgical trainees. MethodsForty-seven General Surgery Residency applicants and 32 medical students with prospective surgical interests were recruited. Manual dexterity (MD) was assessed through six tasks: O'Connor Tweezer Dexterity Test and Minnesota Manual Dexterity Test; Peg Transfer Test Fundamentals of Laparoscopic Surgery (box); Ring and Rail, Thread the Ring and Suture Sponge (da Vinci Surgical Simulator). ResultsMedical students with higher academic scores had longer completion times for the peg transfer test (P = 0.013). Individuals who played musical instruments and perceived themselves to have “Excellent” MD and motor coordination (MC) were more likely to score higher on the Thread the Ring test (P = 0.007; P = 0.009 ,respectively). Those who perceived themselves to have “Mediocre” MD and MC performed the worst on the: O'Connor Tweezer Dexterity Test (P = 0.023). ConclusionsPreliminary data suggest that MD ability correlates with neither high United States Medical Licensing Examination scores nor high academic grades; however, previous experience playing a musical instrument and high self-ratings of MD/MC may be associated with better test performance.

Integrated cardiothoracic surgery match: Trends among applicants compared with other surgical subspecialties

ObjectiveThe objective of this study was to evaluate trends, qualifications, race/ethnicity, and gender of applicants to integrated cardiothoracic (CT I-6) residency programs and compare them with other competitive surgical subspecialties. MethodsData were collected from the National Residency Matching Program, Electronic Residency Application Service, and Association of American Medical Colleges for thoracic surgery, orthopedic surgery, neurological surgery, otolaryngology (ENT), plastic surgery, and vascular surgery for 2010 t0 2020. Applicant gender, race/ethnicity, Alpha Omega Alpha (AOA) membership, United States Medical Licensing Examination scores, research productivity, and graduation from a top-40 medical school were analyzed. ResultsFrom 2010 to 2020, CT I-6 experienced growth in postgraduate year 1 positions (280.0%), total applicants (62.2%), and US senior applicants (59.2%). No growth in CT I-6 positions (38) or programs (29) occurred from 2016 to 2020. CT I-6 had the lowest match rates among total applicants (31.7%) and US seniors (41.0%) in 2020. CT I-6 had fewer female applicants compared with ENT (P < .001) and plastic surgery (P < .001), but more than orthopedic surgery (P < .001). Although most CT I-6 US applicants self-identified as White (75.0%), there were more Asian applicants compared with applicants for orthopedic surgery (P < .001), ENT (P < .001), plastic surgery (P < .001), and neurological surgery (P < .01). Matched applicants averaged the highest Step 2-Clinical Knowledge scores (255.1), AOA membership (48.5%), and graduation rates from top-40 medical schools (54.5%). ConclusionsDespite tremendous growth in positions, CT I-6 has consistently been the most difficult surgical subspecialty to match. CT I-6 has recently attracted an increasingly diverse applicant pool. For the 2019 to 2020 National Residency Matching Program Match Cycle, successful applicants had the highest Step 2-Clinical Knowledge scores, AOA membership rates, and graduation rates from a top-40 medical school among all surgical subspecialties examined.

Comment and Controversy Edited by Stephen P. Stone, MD Applying to dermatology residency without a home program: Advice to medical students in the COVID-19 pandemic and beyond.

Dermatology has historically been one of the most competitive residencies for matching. Successful candidates generally have stellar United States Medical Licensing Examination (USMLE) scores, often with significant amounts of research and excellent letters of recommendation.1,2 One factor often overlooked is the importance of having mentors in the field, as they have experience guiding successful applicants and can provide great insight for what residency programs are looking . While many mentor-mentee relationships naturally occur due to home affiliations, students who do not have home dermatology programs (NHDPs) may struggle to form these relationships and subsequently obtain proper guidance. Unfortunately, the COVID-19 pandemic has made it even more challenging for NHDPs given limited shadowing and away rotation opportunities. Wee share advice on how NHDPs can form strong mentor-mentee relationships during the COVID-19 pandemic and beyond in order to obtain the proper guidance needed to match into dermatology.

Selecting the Modern Cardiothoracic Surgery Trainee

In this issue of The Annals of Thoracic Surgery, in their review of the latest National Residency Matching Program’s report coupled with survey data from more than half of the country’s 6-year Integrated Thoracic Surgery (I-6) residency program directors, Obafemi and colleagues1 explore the ever-intensifying process of securing a training position in the field of cardiothoracic surgery. Whereas the metrics speak for themselves (average United States Medical Licensing Examination scores continue to rise, research experiences surpass those of applicants to any other specialty), excellence in these aspects on the part of a candidate can now be viewed only as a prerequisite for consideration.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences. PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded. Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation. As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

A Concordance Study of COMLEX-USA and USMLE Scores.

Under the single GME accreditation system, residency programs receive applicants from MD- and DO-granting medical schools, each of which have their own set of licensing examinations, making concordance studies increasingly relevant. Previous studies comparing Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) and United States Medical Licensing Examination (USMLE) scores have been limited in sample size and examinee composition and have yielded comparisons that may not be generalizable across all applicants. Some osteopathic medical students take USMLE in addition to COMLEX-USA, often at considerable cost and effort, with the aim of making themselves more desirable to potential residency programs. Having more reliable comparisons of COMLEX-USA and USMLE scores would allow program directors to better estimate a score on the alternate examination. To derive an accurate concordance between COMLEX-USA and USMLE scores, based on a large sample of osteopathic students who took both examinations. Five colleges of osteopathic medicine, representing various regions of the United States, participated in this study. The data included demographics and COMLEX-USA and USMLE scores from September 2015 through August 2020 for students who took both examinations. We derived the concordance between COMLEX-USA and USMLE scores using equipercentile matching. Comparisons of demographic characteristics showed only minor differences between the sample and the overall population for COMLEX-USA takers, although scores for the study sample were, on average, greater. A strong association exists between the scores on the COMLEX-USA and USMLE examinations, allowing prediction of performance on USMLE from COMLEX-USA.

Applicant Ranking Criteria in Adult Reconstruction Fellowship: Your Interview Counts When Applying for Fellowship.

BackgroundThe criteria for successful ranking in the fellowship match remains unclear. Although some data are available regarding the qualities sought after in medical students for the residency match, little information is available regarding the fellowship match. In this study, for arthroplasty applications to our institution, the interview was hypothesized to be the most important factor for ranking, with little impact from other commonly assessed variables.MethodologyAll 40 applicants who applied for fellowship were selected to interview for the 2017 Brown University Comprehensive Adult Reconstruction Fellowship and were evaluated on the interview, United States Medical Licensing Examination (USMLE) scores, letters of recommendation, personal statement, extracurricular activities, research, and caliber of undergraduate institution, medical school, and residency program. The interview score was based on a combined score of individual components of personality, program fit, and research.ResultsOf the 40 applicants who were interviewed, eight did not match. The interview score was the only statistically significant variable and had the highest correlation with ranking (r = 0.92). Moreover, extracurricular activities correlated with a higher ranking whereas USMLE Step 1/Step 2 scores had a relatively low correlation (r = 0.32 and 0.29, respectively). Recommendation letters and caliber of medical school, undergraduate education, and residency demonstrated low correlations. The personal statement and research components had the lowest correlations.ConclusionsThe combined interview score, in particular the personality and program fit components, was the most important determinant of successful ranking at our institution. Because all 40 applicants who applied for an arthroplasty fellowship at our institution were selected for an interview, there was no pre-interview selection bias that would confound these results. However, the ranking does not correlate with an applicant’s success in fellowship, and further research is required to determine the qualifications of a successful surgeon.

Does Objective Scoring of Applications for Radiology Residency Affect Diversity?

It is important to recognize if reliance on certain factors in applications affects selection of trainees from under-represented groups. Our purpose is to determine if objective scoring of radiology residency applications based on quantifiable data regarding academic performance, United States Medical Licensing Examination (USMLE) scores and research productivity affects selection of female and under-represented minority (URM) candidates. We reviewed 502 applications from three successive match cycles from United States allopathic medical students. Each application was scored for academic performance, USMLE results and research productivity determining an overall score. The scores of males were compared to females and URM were compared to non-URM candidates. USMLE cutoff scores were evaluated for disparate effects. There were 348 male, 154 female, 73 URM and 429 non-URM candidates. For male versus female applicants, there was no significant difference in mean academic performance, USMLE Step 2 Clinical Knowledge and research productivity scores. Males had higher mean USMLE Step 1 (p=0.005) and overall candidate scores (p=0.02). Between URM and non-URM candidates there was no significant difference in academic performance. Non-URM applicants had higher mean USMLE Step 1 (p=0.008), USMLE Step 2 (p=0.002), research productivity (p=0.001) and overall scores (p=0.02). Use of USMLE cutoff scores demonstrated disparate effects on female and URM candidates. Objective scoring of applications and use of USMLE cutoff scores can disadvantage candidates from underrepresented groups. Screening filters can affect the diversity of candidate pools for radiology residency.

Development and Validation of a Machine Learning-Based Decision Support Tool for Residency Applicant Screening and Review.

Residency programs face overwhelming numbers of residency applications, limiting holistic review. Artificial intelligence techniques have been proposed to address this challenge but have not been created. Here, a multidisciplinary team sought to develop and validate a machine learning (ML)-based decision support tool (DST) for residency applicant screening and review. Categorical applicant data from the 2018, 2019, and 2020 residency application cycles (n = 8,243 applicants) at one large internal medicine residency program were downloaded from the Electronic Residency Application Service and linked to the outcome measure: interview invitation by human reviewers (n = 1,235 invites). An ML model using gradient boosting was designed using training data (80% of applicants) with over 60 applicant features (e.g., demographics, experiences, academic metrics). Model performance was validated on held-out data (20% of applicants). Sensitivity analysis was conducted without United States Medical Licensing Examination (USMLE) scores. An interactive DST incorporating the ML model was designed and deployed that provided applicant- and cohort-level visualizations. The ML model areas under the receiver operating characteristic and precision recall curves were 0.95 and 0.76, respectively; these changed to 0.94 and 0.72, respectively, with removal of USMLE scores. Applicants' medical school information was an important driver of predictions-which had face validity based on the local selection process-but numerous predictors contributed. Program directors used the DST in the 2021 application cycle to select 20 applicants for interview that had been initially screened out during human review. The authors developed and validated an ML algorithm for predicting residency interview offers from numerous application elements with high performance-even when USMLE scores were removed. Model deployment in a DST highlighted its potential for screening candidates and helped quantify and mitigate biases existing in the selection process. Further work will incorporate unstructured textual data through natural language processing methods.

Proposed Solutions for Improving the Plastic Surgery Residency Application Process.

Integrated plastic surgery residency programs have historically utilized a combination of objective measures, such as United States Medical Licensing Examination scores and research experience, and subjective measures, including performance on away rotations and letters of recommendation, to select applicants to interview.1 The 2019 National Resident Matching Program applicant survey reported applicants from U.S. allopathic medical schools who matched applied to a median of 76 programs, received 21 interview offers, and attended 15 interviews.2 Due to the coronavirus disease of 2019 pandemic, in-person away rotations were canceled and interviews became virtual. These changes occurred across all specialties, and the consequences of these necessary changes are becoming apparent as we move through this interview season. The American College of Surgeons recently described the potential for a crisis in the surgery match this year secondary to the low number of virtual interview cancellations.3 The Association of American Medical Colleges also issued a letter voicing concern over the distribution of interview invitations, suggesting that students with a surplus of interviews release invites and program directors rank more applicants than prior years to successfully fill spots.4 We fear this pattern may also be occurring in the integrated plastic surgery application cycle. Due to these changes, we anticipate a select number of applicants with exceptional objective statistics will receive the majority of interview invites, while qualified applicants without a home plastic surgery program, applicants without optimal curriculum vitae, or applicants who decided on plastics later in medical school are forced to scramble for interview spots. In addition, applicants are more likely to schedule and attend more interviews due to minimal geographical or financial limitations, fewer time constraints, and increased uncertainty associated with this application cycle.5 Applicants who believe they need a higher number of interviews to successfully match this year are unlikely to cancel interviews to programs they are not interested in, preventing others from interviewing. Programs may not have students from moderately ranked schools or students without well-known plastic surgery programs on their radar, disincentivizing them to offer interviews to applicants from these schools.3 A higher than anticipated number of applications may increase the difficulty of holistically reviewing all applicants without utilizing common objective cutoffs. We will not be able to determine the effects of the changes to the application season until after the match. However, this match crisis only highlights longstanding structural defects in the application cycle, and we see this as an opportunity for change. As the 2021/2022 season may also be virtual, we recommend implementing several changes to ensure that program positions do not go unfilled: Limiting the number of programs to which applicants can apply. Limiting the number of interviews applicants can attend. Using the Electronic Residency Application Service scheduling platform to ensure that applicants and programs are following guidelines. DISCLOSURE The authors have no financial interest to declare in relation to the content of this article. No funding was received for this article.