United States Medical Licensing Examination Scores Research Articles

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ±63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

The importance of USMLE step 2 on the screening and selection of applicants for general surgery residency positions

BackgroundAs announced by the Federation of State Medical Boards (FSMB) and National Board of Medical Examiners (NBME), the United States Medical Licensing Examination (USMLE) Step 1 score reporting has transitioned to pass/fail outcomes instead of the traditional numeric score after January 26, 2022. USMLE Step 1 scores have been used widely as a crucial tool in screening and selecting applicants for residency programs. This study aims to determine the role of USMLE Step 2 in the selection of applicants for general surgery residency. MethodsA retrospective study was conducted over six recruiting cycles from 2016 to 2021. The data from 334 interviewed applicants from one general surgery residency program were assessed. Data analyzed included USMLE Step 1 and Step 2 scores, applicant gender, Alpha Omega Alpha (AOA) status, letters of recommendation (LOR), and research/publications (RS). ResultsOf the 334 interviewed applicants, 209 (62.6%) were male. The mean [SD] USMLE Step 1 and USMLE Step 2 C K (Clinical Knowledge) scores were 239.6 [±10.4] and 249.2 [±11.4], respectively. The mean (SD) LOR and RS scores were 4.24 [±0.4] and 3.9 [±0.7], respectively. A positive correlation was observed between USMLE Step 1 and USMLE Step 2 C K (Clinical Knowledge) scores (r = 0.60, p < .001), LOR scores (r = 0.24, p = .008), and AOA status (r = 0.19, p = .038). There was a negligible correlation between USMLE scores and applicant gender. ConclusionTransitioning USMLE Step 1 to pass/fail will make the initial screening and selection process of applications challenging for residency programs. In the short term, USMLE Step 2 scores, LOR, and AOA status are important as screening assessments. Valid measures to ensure appropriate, equitable, and fair assessments are needed.

Predictive Value of Preresidency Academic Metrics on Resident Publication Potential.

Otolaryngology residency is highly competitive, and applicant academic metrics are scrutinized. The predictive value of preresidency academic metrics on applicants' future research productivity and career aspirations remains largely undefined. Retrospective cohort study. Academic otolaryngology department, 2014 to 2015. Applicant demographics, publication history, and United States Medical Licensing Examination (USMLE) scores were downloaded from Electronic Residency Application Service archives. Publications during residency were tallied from all PubMed articles indexed between July 1, 2015 and June 30, 2020. Postresidency career paths were examined by 2 investigators (D.J.C. and L.X.Y.) using Google searches with an emphasis on program websites, Doximity, and LinkedIn profiles. Associations with publication potential and postresidency positions were evaluated with Spearman rank correlation coefficients and Kruskal-Wallis, Wilcoxon rank sum, and χ 2 tests. Of 321 applicants, 226 (70%) matched, and 205 (64%) completed residency by June 2020. Matched residents published a median of 4 (range: 0-41) manuscripts during residency. USMLE scores, Alpha Omega Alpha status, and the number of preresidency publications did not significantly correlate with publication potential during residency. The number of research experiences had a significant positive correlation with publications during residency (p < 0.001). Asian race (p = 0.002) and geographical region of residency (p < 0.001) also had significant associations with publication potential. Of the 205 graduates, 118 (58%) enrolled in fellowship. Age and female sex (74% vs 48%; p = 0.002) were the only factors significantly associated with pursuing a fellowship. In otolaryngology, not all preresidency academic metrics are associated with publication potential during residency or propensity for fellowship training. Programs should not use academic metrics alone to predict an applicant's future research productivity or career trajectory.

Association of an Early Interest in Orthopedic Surgery with Match Rate into Orthopedic Surgery.

Orthopedic surgery residency is considered one of the most competitive specialties in which to match. Studies examining the factors associated with a successful match have neglected whether participation in an orthopedic interest group (OIG) improves the chances of orthopedic residency match. The goal of this study was to test the hypothesis that participation in the OIG would improve matching into an orthopedic surgery residency. We performed a retrospective cohort study between May 2017 and 2019 at one state-funded medical school. All of the applicants in orthopedic surgery from 2004 to 2019 were identified and contacted for OIG membership status. The Office of Student Affairs provided academic performance data (US Medical Licensing Examination scores and third-year clinical clerkship grades), Alpha Omega Alpha and Gold Humanism Honor Society status, and demographics (race and sex) of applicants. Between 2004 and 2019, 67 students (56 OIG and 11 non-OIG) applied for orthopedic surgery residency match. The match rate for the OIG was 86% compared with 64% for the non-OIG group, resulting in an adjusted odds ratio (adjusted for academic performance) of 10.23 (95% confidence interval 1.14-92.3, <i>P =</i> 0.038). OIG membership was associated with a significantly higher rate of orthopedic surgery residency matches. The higher rate of match associated with OIG membership may be the result of opportunities to diversify a residency application. Future studies are needed to further evaluate the potential association between OIG involvement and orthopedic surgery match.

An Examination of the Associations Among USMLE Step 3 Scores and the Likelihood of Disciplinary Action in Practice.

As the last examination in the United States Medical Licensing Examination (USMLE) sequence, Step 3 provides a safeguard before physicians enter into unsupervised practice. There is, however, little validity research focusing on Step 3 scores beyond examining its associations with other educational and professional assessments thought to cover similar content. This study examines the associations between Step 3 scores and subsequent receipt of disciplinary action taken by state medical boards for problematic behavior in practice. It analyzes Step 3 total, Step 3 computer-based case simulation (CCS), and Step 3 multiple-choice question (MCQ) scores. The final sample included 275,392 board-certified physicians who graduated from MD-granting medical schools and who passed Step 3 between 2000 and 2017. Cross-classified multilevel logistic regression models were used to examine the effects of Step 3 scores on the likelihood of receiving a disciplinary action, controlling for other USMLE scores and accounting for jurisdiction and specialty. Results showed that physicians with higher Step 3 total, CCS, and MCQ scores tended to have lower chances of receiving a disciplinary action, after accounting for other USMLE scores. Specifically, a 1-standard-deviation increase in Step 3 total, CCS, and MCQ score was associated with a 23%, 11%, and 17% decrease in the odds of receiving a disciplinary action, respectively. The effect of Step 2 CK score on the likelihood of receiving a disciplinary action was statistically significant, while the effect of Step 1 score became statistically nonsignificant when other Step scores were included in the analysis. Physicians who perform better on Step 3 are less likely to receive a disciplinary action from a state medical board for problematic behavior in practice. These findings provide some validity evidence for the use of Step 3 scores when making medical licensure decisions in the United States.

Extracurricular Humanism in Medicine Initiative and Medical Student Wellness: Retrospective Study.

BackgroundHumanism in Medicine Initiative (HIMI), an extracurricular program at Ohio State University College of Medicine (OSUCOM) with 27 subgroups, fosters the humanities. Stress and burnout among first- and second-year medical students are prevalent across the United States. Solutions for stress among first- and second-year medical students have been proposed, but no gold standard exists. The relationship of humanism with stress and burnout has yet to be described in the literature.ObjectiveThis study investigates the relationship between participation in the HIMI and stress, burnout, and academic success among first- and second-year medical students.MethodsFirst- and second-year medical students enrolled at OSUCOM between August 2018 and August 2019 were recruited. Attendance in the HIMI and membership records were used to measure their participation. Curricular examination scores and those on Step 1 of United States Medical Licensing Examination (USMLE) were used to measure academic success. Stress and burnout were measured using the Maslach Burnout Inventory and the Perceived Stress Scale.ResultsIn total, 412 students were enrolled with 362 (87%) students participating in HIMI. Those with high participation were more often Black, Asian, female, or with a humanities undergraduate major compared to the overall study population. There were significant relationships between Gold Humanism Honors Society (GHHS) induction and participation of first- and second-year medical students in service- (χ21=5.8, P<.05) or leadership-focused (χ21=19.3, P<.001) HIMI groups. Medium levels of participation in the HIMI were associated with significantly higher stress. Performance on the Step 1 USMLE was not significantly associated with participation levels in the HIMI (low=233.7 vs high=238.0; P=.10).ConclusionsThe HIMI is an extracurricular program vastly utilized by first- and second-year medical students at OSUCOM and did not impact Step 1 USMLE scores. Medium participation in the HIMI was associated with higher stress, and service- and leadership-focused HIMI participation was associated with a higher level of induction to the GHHS. This study identifies areas for future studies to understand the relationship of the HIMI with stress and academic success.

Which Application Factors Are Associated With Outstanding Performance in Orthopaedic Surgery Residency?

Identifying ideal candidates for orthopaedic surgery residency is difficult. Data available for applicant selection are evolving; preclinical grades and the Alpha Omega Alpha (AOA) honors society are being phased out at some medical schools. Similarly, three-digit United States Medical Licensing Examination (USMLE) Step 1 scores have been eliminated. There is renewed interest in improving resident selection to provide a diverse, comprehensive educational opportunity that produces orthopaedic surgeons who are prepared for practice. We sought to identify whether (1) academic achievements, (2) letters of recommendation, (3) research activity, and (4) miscellaneous factors available on Electronic Residency Application Service (ERAS) applications were associated with outstanding residency performance. Ten faculty members (22% of all full-time faculty) with extensive educational involvement for at least 7 years, whose expertise covered all subspecialty departments at an urban, academic orthopaedic surgery residency program, were given an anonymous survey on the performance of the four most recent classes of residency graduates (24 residents). This survey was developed due to the lack of a validated residency outcomes tool or objective metrics for residency performance. The evaluated criteria were decided upon after discussion by a relatively large group of academic orthopaedic surgeons considering the factors most important for graduating orthopaedic residents. The faculty were selected based on their long-term knowledge of the residency, along with their diversity of specialty and backgrounds; there were no nonresponders. Faculty graded each resident on a scale from 1 to 10 (higher is better) on six criteria: surgical technical skills, research productivity, clinical knowledge, professionalism, personality, and fellowship match. The mean of the faculty ratings made by all faculty for all six criteria was calculated, producing the overall residency performance score. Factors available on each resident's ERAS application were then correlated with their overall residency performance score. Categorical ERAS factors, including AOA status, five or more honors in core clerkships, at least three exceptional letters of recommendation, collegiate athletics participation, expertise with a musical instrument, and research (6-year) track residents, were correlated with overall residency performance score via point biserial analysis. Continuous ERAS factors including USMLE Step 1 and Step 2 scores, number of publications before residency, number of research years before residency, medical school ranking, and number of volunteer experiences were correlated with overall residency performance score via Pearson correlation. USMLE Step 1 three-digit scores were evaluated despite their recent elimination because of their historic importance as a screening tool for residency interviews and for comparison to USMLE Step 2, which retains a three-digit score. Application factors with a p < 0.2 on univariate analysis (five or more honors in core clerkships, at least three exceptional letters of recommendation, research track residents) were included in a stepwise linear regression model with "overall residency performance score" as the outcome variable. All p values < 0.05 were considered significant. The mean overall residency performance score was 7.9 ± 1.2. Applicants with at least five honors grades in core clerkships had overall residency performance scores 1.2 points greater than those of their peers (95% confidence interval (CI) 0.3 to 2.0; p = 0.01, Cohen ƒ 2 = 0.2, representing a small effect size). ERAS applications including at least three exceptional letters of recommendation were associated with a 0.9-point increase in residency performance (95% CI 0.02 to 1.7; p = 0.046, Cohen ƒ 2 = 0.1, representing a small effect size). Participation in the residency research (6-year) track was associated with a 1-point improvement in residency performance (95% CI 0.1 to 1.9; p = 0.03, Cohen ƒ 2 = 0.2, again, representing a small effect size). Together, these three factors accounted for 53% of the variance in overall residency performance score observed in this study. Past clinical excellence, measured by core clerkship grades and exceptional letters of recommendation, is associated with slightly improved overall orthopaedic residency performance scores. Applicants meeting both criteria who also complete a research track residency may perform substantially better in residency than their counterparts, as these three factors accounted for half of all the variance observed in the current study. Although minimum requirements are necessary, traditionally used screening factors (such as USMLE scores, AOA status, medical school rank, and number of publications) may be of less utility in identifying successful future residents than previously thought. Level III, therapeutic study.

Pearls of wisdom for aspiring physician-scientist residency applicants and program directors.

Postgraduate physician-scientist training programs (PSTPs) enhance the experiences of physician-scientist trainees following medical school graduation. PSTPs usually span residency and fellowship training, but this varies widely by institution. Applicant competitiveness for these programs would be enhanced, and unnecessary trainee anxiety relieved, by a clear understanding of what factors define a successful PSTP matriculant. Such information would also be invaluable to PSTP directors and would allow benchmarking of their admissions processes with peer programs. We conducted a survey of PSTP directors across the US to understand the importance they placed on components of PSTP applications. Of 41 survey respondents, most were from internal medicine and pediatrics residency programs. Of all components in the application, two elements were considered very important by a majority of PSTP directors: (a) having one or more first-author publications and (b) the thesis advisor’s letter. Less weight was consistently placed on factors often considered more relevant for non-physician-scientist postgraduate applicants — such as US Medical Licensing Examination scores, awards, and leadership activities. The data presented here highlight important metrics for PSTP applicants and directors and suggest that indicators of scientific productivity and commitment to research outweigh traditional quantitative measures of medical school performance.

A Concordance Study of COMLEX-USA and USMLE Scores.

Under the single GME accreditation system, residency programs receive applicants from MD- and DO-granting medical schools, each of which have their own set of licensing examinations, making concordance studies increasingly relevant. Previous studies comparing Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) and United States Medical Licensing Examination (USMLE) scores have been limited in sample size and examinee composition and have yielded comparisons that may not be generalizable across all applicants. Some osteopathic medical students take USMLE in addition to COMLEX-USA, often at considerable cost and effort, with the aim of making themselves more desirable to potential residency programs. Having more reliable comparisons of COMLEX-USA and USMLE scores would allow program directors to better estimate a score on the alternate examination. To derive an accurate concordance between COMLEX-USA and USMLE scores, based on a large sample of osteopathic students who took both examinations. Five colleges of osteopathic medicine, representing various regions of the United States, participated in this study. The data included demographics and COMLEX-USA and USMLE scores from September 2015 through August 2020 for students who took both examinations. We derived the concordance between COMLEX-USA and USMLE scores using equipercentile matching. Comparisons of demographic characteristics showed only minor differences between the sample and the overall population for COMLEX-USA takers, although scores for the study sample were, on average, greater. A strong association exists between the scores on the COMLEX-USA and USMLE examinations, allowing prediction of performance on USMLE from COMLEX-USA.

To the Editor: Holistic Approach to Resident Selection May Eliminate Potential Biases Associated With USMLE Step 1 Pass/Fail Scoring System.

We read with profound interest the article by Ganesh Kumar et al1 highlighting trainee perspectives regarding the use of a United States Medical Licensing Examination (USMLE) pass/fail score in place of the traditional 3-digit numerical scoring system for resident selection in graduate medical education (GME) residency programs. The authors surveyed a diverse cohort of residents, fellows, and medical students, inclusive of US allopathic (MD) students, US osteopathic (DO) students, and international medical graduates (IMGs) in 2020. A total of 11 633 trainees (7254 residents/fellows and 4379 medical students), 14% of which were underrepresented in medicine (UiM), responded to the survey and provided insights on this controversial subject. The authors emphasized in their discussion that, although 35% of the respondents supported the proposed USMLE Step 1 pass/fail scoring system, 44% of the survey respondents disliked this system. US allopathic medical students and UiM trainees were the most supportive of a USMLE Step 1 pass/fail scoring system. In contrast, many non-UiM, DO, and IMG respondents felt that they would be at a disadvantage. Regarding the secondary outcome measure that a USMLE Step 1 pass/fail scoring system will reduce socioeconomic disparities in resident selection, it was fascinating to read that UiM medical students primarily supported this hypothesis. We applaud the Vanderbilt University investigators for performing this study, which provides GME leaders with vital data as we transition to a USMLE Step 1 pass/fail scoring system.In the United States, GME is at a crucial crossroads regarding resident selection. Historically, USMLE Step 1 scores have served as a benchmark for screening resident candidates, especially in highly competitive programs. Although USMLE scores may reflect some degree of inherent knowledge, the preparation required to excel on the examination may also depend on socioeconomic factors. Medical students often take multiple preparation courses, such as the PASS Program and Kaplan, to improve their content knowledge and test-taking skills for the USMLE examinations. Many of these are high cost, and in an era of rising medical school tuition costs, access may be limited to students with adequate financial resources. Students from socioeconomically disadvantaged backgrounds may have lower USMLE Step 1 scores because of reduced resources. For this and other reasons, the decision to move to a USMLE Step 1 pass/fail scoring system was made. However, as found by Ganesh Kumar et al,1 trainees who responded to this survey had varied perspectives—a significant subset of these respondents, particularly DO and IMG trainees, were skeptical of this transformation. As a result, we caution GME leaders to be thoughtful with this new binary USMLE Step 1 pass/fail scoring system, because it may introduce additional implicit biases against DO and IMG applicants. It may be helpful to remove credentials that could encourage implicit bias, such as the MD, DO, or IMG designation when reviewing an applicant.The critical question now becomes: How will residency admission committees identify top-level applicants in the era of a USMLE Step 1 pass/fail scoring system? We believe that using a meritocratic, holistic approach to residency admission, inclusive of letters of recommendation, the binary USMLE Step 1 pass/fail score, medical school transcripts, trainee experiences, personal attributes, interview performance, and barriers encountered during their education and training, can identify the best and brightest medical students for residency programs.2 Hopefully, this holistic approach to resident selection will eliminate ingrained socioeconomic disparities that may hinder UiM students from joining competitive residency programs.

The importance of personality traits for predicting clinical clerkship grades and USMLE scores

Background: Important aspects of medical school success are thought to rely on non-cognitive attributes such as personality traits. Methods: In a sample of medical students who completed their 3rd-year clerkships, the current study examined the impact of personality traits on performance on four measures of 3rd-year clerkship grades and United States Medical Licensing Examination (USMLE) Step 2CK (Clinical Knowledge) scores. Results: We found that specific personality traits, including conscientiousness, achievement-striving, morality, and self-efficacy predicted multiple outcome measures, including Step 2CK performance. However, while we observed a clear link between personality and success during the 3rd year, we also found that Step 1 scores, not specific or grouped personality traits, best predicted all measures of clerkship performance except clerkship evaluations and, as expected, best predicted Step 2CK performance. Additionally, we created a composite personality profile for our sample by averaging the high and low scores of the Big Five personality trait scales and their subscales. However, this personality composite, which included the traits agreeableness, cooperation, Conscientiousness, and cautiousness, did not predict any of our outcome measures. Conclusions: These findings indicate that personality traits individually and in combination predict 3rd year medical school success, especially when performance may rely less on intellectual prowess, and more on personality factors. Importantly, however, Step 1 scores were usually much better predictors not only of Step 2CK and shelf exam performance, but even overall clerkship grades and Objective Structured Clinical Examination (OSCE) performance. We hope these results can be used to help clerkship directors and faculty understand that personality factors impact medical school performance and that they should be considered when evaluating the more non-cognitive aspects of student achievement.

Traits of the current traditional pathway cardiothoracic surgery training pool: Results of a cross-sectional study

ObjectiveAs new paradigms for cardiothoracic surgery training emerged in recent years, the traditional 2- or 3-year pathway has persisted as an option for trainees completing general surgery residencies. Although the applicant pool for 6-year integrated cardiothoracic surgery training programs has been superficially explored, little data exist characterizing those applicants to the traditional cardiothoracic surgery training pathway and the influence of 6-year integrated expansion on the traditional applicant pool. MethodsWe reviewed materials from candidates applying to a single 2-year cardiothoracic surgery training program between 2015 and 2020. Descriptive and comparative analyses of multiple characteristics were performed over the years of the study. ResultsDuring the years 2015 through 2020, we received 571 applications, accounting for 72% of the total National Residency Matching Program applicant pool. We saw no significant trends in numbers of peer-reviewed publications or presentations. There was a minimal year-to-year increase in number of first-authored posters, 2.04 in 2015 to 2.13 in 2020 (P = .008). Online publications, book chapters, and other publications were stable throughout the study period. Applicants consistently provided an average of 3.6 letters of recommendation, 1.9 from cardiothoracic surgery faculty. Mean in-service score percentiles were stable at the 54th percentile, whereas US Medical Licensing Examination scores increased. ConclusionsDespite expansion of the 6-year integrated pathway to cardiothoracic surgery, we have seen no substantial year-to-year changes in attributes of traditional applicants. Our findings suggest that the cardiothoracic surgery applicant pool continues to be composed of a stable group of highly productive trainees. Future initiatives in candidate selection should emphasize interview strategies to highlight aspects of grit, emotional intelligence, and team dynamics.

Medical schools as gatekeepers: A survey and analysis of factors predicting dermatology residency placement

To the Editor: Dermatology is among the most competitive specialties for residency applicants. With an increasingly rigorous match process, successful applicants typically have strong US Medical Licensing Examination scores alongside extensive research experience. Additionally, 41% of matched applicants attended the top 40 ranked US medical schools (by National Institutes of Health funding), the second highest of any specialty (Supplemental Table I available via Mendeley at https://data.mendeley.com/datasets/gr8w4dg2dv/1.)1

Predictors of Success in an Orthopaedic Residency.

Orthopaedic surgery is one of the most competitive residencies to match into. Meanwhile, the average applicant's United States Medical Licensing Examination (USMLE) test scores, research involvement, and number of clinical honors increase every year. Measures such as USMLE scores, productivity in research, Alpha Omega Alpha (AΩA) honor society status, number of clinical honors, and performance on away rotations have all been cited as factors contributing to program directors choosing applicants for interviews and ranking them for their program. However, questions remain as to whether these measures translate to success on board examinations, high faculty evaluations, and designation as chief resident during orthopaedic residency. USMLE scores have been shown to correlate with Orthopaedic In-Training Examination (OITE) and American Board of Orthopaedic Surgery (ABOS) scores, while clinical grades and AΩA status correlate with faculty evaluations. Participating in research as a medical student was predictive of research productivity in residency but did not correlate with standardized testing scores or faculty evaluations. The literature has suggested ways in which measures such as personality and grit may be used in the application process and how these factors may contribute to predictors of success. However, additional research is needed to measure and define personality and grit during the application evaluation process.

Objective predictors of intern performance

BackgroundResidency programs select medical students for interviews and employment using metrics such as the United States Medical Licensing Examination (USMLE) scores, grade-point average (GPA), and class rank/quartile. It is unclear whether these metrics predict performance as an intern. This study tested the hypothesis that performance on these metrics would predict intern performance.MethodsThis single institution, retrospective cohort analysis included 244 graduates from four classes (2015–2018) who completed an Accreditation Council for Graduate Medical Education (ACGME) certified internship and were evaluated by program directors (PDs) at the end of the year. PDs provided a global assessment rating and ratings addressing ACGME competencies (response rate = 47%) with five response options: excellent = 5, very good = 4, acceptable = 3, marginal = 2, unacceptable = 1. PDs also classified interns as outstanding = 4, above average = 3, average = 2, and below average = 1 relative to other interns from the same residency program. Mean USMLE scores (Step 1 and Step 2CK), third-year GPA, class rank, and core competency ratings were compared using Welch’s ANOVA and follow-up pairwise t-tests.ResultsBetter performance on PD evaluations at the end of intern year was associated with higher USMLE Step 1 (p = 0.006), Step 2CK (p = 0.030), medical school GPA (p = 0.020) and class rank (p = 0.016). Interns rated as average had lower USMLE scores, GPA, and class rank than those rated as above average or outstanding; there were no significant differences between above average and outstanding interns. Higher rating in each of the ACGME core competencies was associated with better intern performance (p < 0.01).ConclusionsBetter performance as an intern was associated with higher USMLE scores, medical school GPA and class rank. When USMLE Step 1 reporting changes from numeric scores to pass/fail, residency programs can use other metrics to select medical students for interviews and employment.

Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training.

AIMThe study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination.METHODSIn this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05.RESULTSMean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1.CONCLUSIONSThe change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant’s future performance in residency.

To Increase Diversity, Medicine Should Rethink USMLE Scores, Psychiatrists Say

To Increase Diversity, Medicine Should Rethink USMLE Scores, Psychiatrists Say

On Step 1 Mania, USMLE Score Reporting, and Financial Conflict of Interest at the National Board of Medical Examiners.

Though intended to inform a binary decision on initial medical licensure, the United States Medical Licensing Examination (USMLE) is frequently used for screening candidates for residency positions. Some have argued that reporting results as pass/fail would honor the test's purpose while preventing inappropriate use. To date, the USMLE's sponsor organizations have declined to make such a change. In this Perspective, the authors examine the history and mission of the National Board of Medical Examiners (NBME), trace the rise of "Step 1 mania," and consider the current financial incentives for the NBME in implementing a pass/fail score-reporting policy.The NBME was founded in 1915 to address the lack of interstate reciprocity in medical licensure examination. With the creation of the USMLE in 1992, a single pathway for licensure was established, and the organization's original mission was achieved. Yet even after fulfilling its primary purpose, the NBME-classified as a nonprofit organization-has seen its revenues rise dramatically over the past 2 decades. Much of the increased revenue is derived from test products and services not required for medical licensure, with sales driven by the increasing importance of Step 1 scores in residency selection. Revenue from these products and services would likely decline if the NBME reported Step 1 results as pass/fail.A financial conflict of interest occurs when a judgment concerning a primary interest may be influenced by a secondary interest, such as financial gain. The data presented here demonstrate that the NBME has a conflict of interest in its current score-reporting policy. Possible remedies, such as disclosure, recusal, divestiture, and restructuring, are considered.

Correlations Between the USMLE Step Examinations, American College of Physicians In-Training Examination, and ABIM Internal Medicine Certification Examination.

To assess the correlations between United States Medical Licensing Examination (USMLE) performance, American College of Physicians Internal Medicine In-Training Examination (IM-ITE) performance, American Board of Internal Medicine Internal Medicine Certification Exam (IM-CE) performance, and other medical knowledge and demographic variables. The study included 9,676 postgraduate year (PGY)-1, 11,424 PGY-2, and 10,239 PGY-3 internal medicine (IM) residents from any Accreditation Council for Graduate Medical Education-accredited IM residency program who took the IM-ITE (2014 or 2015) and the IM-CE (2015-2018). USMLE scores, IM-ITE percent correct scores, and IM-CE scores were analyzed using multiple linear regression, and IM-CE pass/fail status was analyzed using multiple logistic regression, controlling for USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores; averaged medical knowledge milestones; age at IM-ITE; gender; and medical school location (United States or Canada vs international). All variables were significant predictors of passing the IM-CE with IM-ITE scores having the strongest association and USMLE Step scores being the next strongest predictors. Prediction curves for the probability of passing the IM-CE based solely on IM-ITE score for each PGY show that residents must score higher on the IM-ITE with each subsequent administration to maintain the same estimated probability of passing the IM-CE. The findings from this study should support residents and program directors in their efforts to more precisely identify and evaluate knowledge gaps for both personal learning and program improvement. While no individual USMLE Step score was as strongly predictive of IM-CE score as IM-ITE score, the combined relative contribution of all 3 USMLE Step scores was of a magnitude similar to that of IM-ITE score.

Application Factors May Not Be Predictors of Success Among General Surgery Residents as Measured by ACGME Milestones

BackgroundCan factors within the Electronic Residency Application Service application be used to predict the success of general surgery residents as measured by the Accreditation Council for Graduate Medical Education (ACGME) general surgery milestones? MethodsThis is a retrospective study of 21 residents who completed training at a single general surgery residency program. Electronic Residency Application Service applications were reviewed for objective data, such as age, US Medical Licensing Examination scores, and authorship of academic publications as well as for letters of recommendation, which were scored using a standardized grading system. These factors were correlated to resident success as measured by ACGME general surgery milestone outcomes using univariate and multivariate analyses. This study was conducted at a single academic tertiary care and level 1 trauma facility. Residents who completed general surgery residency training from the years of 2012-2018 were included in the study. ResultsThere were few correlations between application factors and resident success determined by the ACGME milestones. ConclusionsApplication factors alone do not account for ongoing growth and development throughout residency. Unlike the results presented in the literature for other surgical subspecialties, predicting general surgery resident success based on application factors is not straightforward.