United States Medical Licensing Exam Step Research Articles

Assessing ChatGPT 4.0’s test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports

While there is data assessing the test performance of artificial intelligence (AI) chatbots, including the Generative Pre-trained Transformer 4.0 (GPT 4) chatbot (ChatGPT 4.0), there is scarce data on its diagnostic accuracy of clinical cases. We assessed the large language model (LLM), ChatGPT 4.0, on its ability to answer questions from the United States Medical Licensing Exam (USMLE) Step 2, as well as its ability to generate a differential diagnosis based on corresponding clinical vignettes from published case reports. A total of 109 Step 2 Clinical Knowledge (CK) practice questions were inputted into both ChatGPT 3.5 and ChatGPT 4.0, asking ChatGPT to pick the correct answer. Compared to its previous version, ChatGPT 3.5, we found improved accuracy of ChatGPT 4.0 when answering these questions, from 47.7 to 87.2% (p = 0.035) respectively. Utilizing the topics tested on Step 2 CK questions, we additionally found 63 corresponding published case report vignettes and asked ChatGPT 4.0 to come up with its top three differential diagnosis. ChatGPT 4.0 accurately created a shortlist of differential diagnoses in 74.6% of the 63 case reports (74.6%). We analyzed ChatGPT 4.0’s confidence in its diagnosis by asking it to rank its top three differentials from most to least likely. Out of the 47 correct diagnoses, 33 were the first (70.2%) on the differential diagnosis list, 11 were second (23.4%), and three were third (6.4%). Our study shows the continued iterative improvement in ChatGPT’s ability to answer standardized USMLE questions accurately and provides insights into ChatGPT’s clinical diagnostic accuracy.

Race, But Not Gender, Is Associated With Admissions Into Orthopaedic Residency Programs.

Orthopaedic surgery is one of the most competitive but least diverse surgical specialties, with ever-increasing academic achievements (such as test scores) shown by applicants. Prior research shows that white applicants had higher United States Medical Licensing Exam (USMLE) Step 1 and Step 2 Clinical Knowledge scores as well as higher odds of Alpha Omega Alpha status compared with Black, Hispanic, and other applicant groups. Yet, it still remains unknown whether differences in application metrics by race/ethnicity sufficiently explain the underrepresentation of certain racial or ethnic minority groups in orthopaedic residency programs. In this study, we sought to determine (1) the relative weight of academic variables for admission into orthopaedic residency, and (2) whether race and gender are independently associated with admission into an orthopedic residency. The Electronic Residency Application System (ERAS) data from the Association of American Medical Colleges (AAMC) and the National Board of Medical Examiners (NBME) of first-time MD applicants (n = 8966) for orthopaedic surgery residency positions in the United States and of admitted orthopaedic residents (n = 6218) from 2005 to 2014 were reviewed. This dataset is the first and most comprehensive of its kind to date in orthopaedic surgery. Academic metrics, such as USMLE Step 1 and Step 2 Clinical Knowledge scores, number of publications, Alpha Omega Alpha status, volunteer experiences, work experience, as well as race and gender, were analyzed using hierarchical logistic regression models. The first model analyzed the association of academic metrics with admission into orthopaedic residency. In the second model, we added race and gender and controlled for metrics of academic performance. To determine how well the models simulated the actual admissions data, we computed the receiver operating characteristics (ROC) including the area under curve (AUC), which measures the model's ability to simulate which applicants were admitted or not admitted, with an AUC = 1.0 representing a perfect simulation. The odds ratio and confidence interval of each variable were computed. When only academic variables were analyzed in the first model, Alpha Omega Alpha status (odds ratio 2.12 [95% CI 1.80 to 2.50]; p < 0.001), the USMLE Step 1 score (OR 1.04 [95% CI 1.03 to 1.04]; p < 0.001), the USMLE Step 2 Clinical Knowledge score (OR 1.01 [95% CI 1.01 to 1.02]; p < 0.001), publication count (OR 1.04 [95% CI 1.03 to 1.05]; p < 0.001), and volunteer experience (OR 1.03 [95% CI 1.01 to 1.04]; p < 0.001) were associated with admissions into orthopaedics while work and research experience were not. This model yielded a good prediction of the results with an AUC of 0.755. The second model, in which the variables of race and gender were added to the academic variables, also had a good prediction of the results with an AUC of 0.759. This model indicates that applicant race, but not gender, is associated with admissions into orthopaedic residency. Applicants from Asian (OR 0.78 [95% CI 0.67 to 0.92]), Black (OR 0.63 [95% CI 0.51 to 0.77], Hispanic (OR 0.48 [95% CI 0.36 to 0.65]), or other race groups (OR 0.65 [95% CI 0.55 to 0.77]) had lower odds of admission into residency compared with white applicants. Minority applicants, but not women, have lower odds of admission into orthopaedic surgery residency, even when accounting for academic performance metrics. Changes in the residency selection processes are needed to eliminate the lower admission probability of qualified minority applicants in orthopaedic residency and to improve the diversity and inclusion of orthopaedic surgery. Changes including increasing the diversity of the selection committee, bias training, blinding applications before review, removal of metrics with history of racial disparities from an interviewer's candidate profile before an interview, and use of holistic application review (where an applicants' experiences, attributes, and academic metrics are all considered) can improve the diversity landscape in training. In addition, cultivating an environment of inclusion will be necessary to address these long-standing trends in orthopaedic surgery. Race, but not gender, is associated with the odds of acceptance into orthopaedic surgery residency despite equivalent academic metrics. Changes in residency selection processes are suggested to eliminate the lower admission probability of qualified minority applicants into orthopaedic residency and to improve the diversity and inclusion of orthopaedic surgery.

Relationships between preadmission variables and academic outcomes for postbaccalaureate students in medical school

There is currently little guidance for medical school admissions committees regarding how to weigh postbaccalaureate program grades relative to undergraduate grades. This study was designed to address this issue. Admissions data, preclerkship course performance and United States Medical Licensing Exam (USMLE) Step 1 results were analyzed over three years for University of California, San Diego (UCSD) postbaccalaureate premedical (PBPM) students (n = 25), students who participated in other postbaccalaureate programs (n = 34), and for the remainder of the medical students who did not participate in any postbaccalaureate programs (n = 329). UCSD PBPM program alumni did not significantly differ in their cumulative academic performance on exams in preclerkship courses and USMLE Step 1 pass rates compared to the rest of the class despite their significantly lower GPA, lower Biology, Chemistry, Physics and Math (BCPM) GPA, and Medical College Admissions Test (MCAT) percentiles. For students who participated in the PBPM programs, PBPM program GPA was a significant predictor of preclerkship academic performance and USMLE Step 1 performance. When assessing academic readiness of applicants who have completed postbaccalaureate programs, admissions committees might closely consider the postbaccalaureate program GPA in addition to other academic metrices such as BCPM GPA and MCAT score.

The Use of Race, Ethnicity, and Region Associations in a USMLE Step 1 Resource

Phenomenon : Mental shortcuts are commonly used in medical education to facilitate the learning and application of a large volume of information. However, the use of demographic identifiers such as race, ethnicity, region, and descent from one of these groups as mental shortcuts in association with disease can perpetuate misconceptions about the construction of these identities and reinforce stereotypes. The United States Medical Licensing Exam (USMLE) Step 1 is a major driver of pre-clinical undergraduate medical education that requires memorization of a large volume of information and application of this information to clinical vignettes. This study assesses how demographic identifiers have been used in a nearly universally used study resource for this exam. Approach : The authors analyzed First Aid for the USMLE Step 1 2020, Thirtieth Edition, a resource that contains “high yield facts” and was built and maintained based on experiences with the USMLE Step 1 for references to race, ethnicity, region, and descent from one of these groups and the distribution of skin tones used in photos. These findings were subsequently compared to the changes made in the 2021 edition of the resource. Findings : The authors found 50 references in the 2020 edition to race, ethnicity, region, and descent from one of these groups, all in relation to disease. More than half of these references had an associated heritable component. Black or African American race was disproportionately represented, comprising more than half of all racial associations (13/24). Additionally, light skin tone was used in 170/204 photos (84.2%) in the 2020 edition. In the 2021 edition, only 12/209 photos (5.7%) were new or changed. Insights : These findings highlight the trend to associate race with disease while also furthering the misconception that there are innate, heritable differences between socially constructed groups and establishing light skin tone as the norm. While some favorable changes were made to the 2021 text, further work within this resource and across medical education is required to avoid further misuse of race and challenge existing implicit biases.

Analysis of the 1990-2007 neurosurgery residency match: does applicant gender affect neurosurgery match outcome?

OBJECTIVE With nearly half of graduating US medical students being female, it is imperative to understand why females typically make up less than 20% of the neurosurgery applicant pool, a number that has changed very slowly over the past several decades. Organized neurosurgery has strongly indicated the desire to overcome the underrepresentation of women, and it is critical to explore whether females are at a disadvantage during the residency application process, one of the first steps in a neurosurgical career. To date, there are no published studies on specific applicant characteristics, including gender, that are associated with match outcome among neurosurgery resident applicants. The purpose of this study is to determine which characteristics of neurosurgery residency applicants, including gender, are associated with a successful match outcome. METHODS De-identified neurosurgical resident applicant data obtained from the San Francisco Fellowship and Residency Matching Service for the years 1990-2007 were analyzed. Applicant characteristics including gender, medical school attended, year of application, United States Medical Licensing Exam (USMLE) Step 1 score, Alpha Omega Alpha (AOA) status, and match outcome were available for study. RESULTS Of the total 3426 applicants studied, 473 (13.8%) applicants were female and 2953 (86.2%) were male. Two thousand four hundred forty-eight (71.5%) applicants successfully matched. USMLE Step 1 score was the strongest predictor of match outcome with scores > 245 having an OR of 20.84 (95% CI 10.31-42.12) compared with those scoring < 215. The mean USMLE Step 1 score for applicants who successfully matched was 233.2 and was 210.8 for those applicants who did not match (p < 0.001). Medical school rank was also associated with match outcome (p < 0.001). AOA status was not significantly associated with match outcome. Female gender was associated with significantly lower odds of matching in both simple (OR 0.59, 95% CI 0.48-0.72) and multivariate analyses (OR 0.57, 95% CI 0.34-0.94 CI). USMLE Step 1 scores were significantly lower for females compared to males with a mean score of 230.1 for males and 221.5 for females (p < 0.001). There was no significant difference in medical school ranking or AOA status when stratified by applicant gender. CONCLUSIONS The limited historical applicant data from 1990-2007 suggests that USMLE Step 1 score is the best predictor of match outcome, although applicant gender may also play a role.

Relationships of demographic variables to USMLE physician licensing exam scores: a statistical analysis on five years of medical student data.

IntroductionThe purpose of this study was to determine the associations of the demographic variables of gender, state of legal residency, student age, and undergraduate major with scores on the Medical College Admissions Test (MCAT) and the United States Medical Licensing Exam (USMLE) Step 1 and Step 2 Clinical Knowledge.MethodsThe researchers collected and analyzed exam scores and demographic student data from participants of five graduating classes of students at the University of Minnesota Medical School (N = 1,067).ResultsSignificant differences (p < 0.05) were found for traditional-aged (defined as < 25 years old at matriculation) versus nontraditional-aged students on USMLE Step 1 scores (t[1065] = 2.91, p = 0.004) and USMLE Step 2 scores (t[1061] = 4.39, p < 0.001), both in favor of traditional-aged students. Significant differences were found for males versus females on MCAT Composite scores (t[1063] = 6.53, p < 0.001) and USMLE Step 1 scores (t[1065] = 5.14, p < 0.001), both in favor of males. There were no significant differences between science and nonscience majors or between Minnesota legal residents and nonresidents.ConclusionTraditional age and male gender were associated with higher exam scores, although patterns differed between tests, whereas undergraduate major and state of legal residency were not associated with higher exam scores.

Not the Last Word: Ending The Residency Application Arms Race-Starting with the USMLE.

Not the Last Word: Ending The Residency Application Arms Race-Starting with the USMLE.

Correlation of the National Emergency Medicine M4 Clerkship Examination with USMLE Examination Performance.

IntroductionAssessment of medical students’ knowledge in clinical settings is complex yet essential to the learning process. Clinical clerkships use various types of written examinations to objectively test medical knowledge within a given discipline. Within emergency medicine (EM), a new national standardized exam was developed to test medical knowledge in this specialty. Evaluation of the psychometric properties of a new examination is an important issue to address during test development and use. Studies have shown that student performance on selected standardized exams will reveal students’ strengths and/or weaknesses, so that effective remedial efforts can be implemented. Our study sought to address these issues by examining the association of scores on the new EM national exam with other standardized exam scores.MethodsFrom August 2011 to April 2013, average National EM M4 examination scores of fourth-year medical students taken at the end of a required EM clerkship were compiled. We examined the correlation of the National EM M4 examination with the scores of initial attempts of the United States Medical Licensing Exam (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) examinations. Correlation coefficients and 95% confidence intervals of correlation coefficients are reported. We also examined the association between the national EM M4 examination score, final grades for the EM rotation, and USMLE Step 1 and Step 2 CK scores.Results133 students were included in the study and achieved a mean score of 79.5 SD 8.0 on the National EM M4 exam compared to a national mean of 79.7 SD 3.89. The mean USMLE Step 1 score was 226.8 SD 19.3. The mean USMLE Step 2 CK score was 238.5 SD 18.9. National EM M4 examination scores showed moderate correlation with both USMLE Step 1 (mean score=226.8; correlation coefficient=0.50; 95% CI [0.28–0.67]) and USMLE Step 2 CK (mean score=238.5; correlation coefficient=0.47; 95% CI [0.25–0.65]). Students scoring below the median on the national EM M4 exam also scored well below their colleagues on USMLE exams.ConclusionThe moderate correlation of the national EM M4 examination and USMLE Step 1 and Step 2 CK scores provides support for the utilization of the CDEM National EM M4 examination as an effective means of assessing medical knowledge for fourth-year medical students. Identification of students scoring lower on standardized exams allows for effective remedial efforts to be undertaken throughout the medical education process.

Standardized Training Examinations among Ophthalmology Residents and the American Board of Ophthalmology Written Qualifying Examination First Attempt: The Morsani College of Medicine Experience

Purpose The aim of the study is to demonstrate whether resident performance on the United States Medical Licensing Exam (USMLE) Step 1 and the Ophthalmology Knowledge Assessment Program (OKAP) correlates to American Board of Ophthalmology Written Qualifying Examination (ABO-WQE). Design This is a retrospective cohort study. Participants Forty-one residents who completed their postgraduate training between 1999 and 2011 from our institution were included in the study. Methods Percentiles on the USMLE Step 1, OKAP, and ABO-WQE first attempt were recorded and analyzed for possible correlations among scores across examinations using Pearson rank correlation, chi-square analysis, Fisher exact test, and Student t-test. Results The percentile score on the USMLE Step 1 was the only standardized test that had a statistically significant association with the percentile score on the ABO-WQE (Pearson rank coefficient = 0.45, p = 0.01). Not achieving a score of 20th percentile on the OKAP examination among second-year residents had the highest statistical association with failing the ABO-WQE on first attempt (p = 0.001) with an odds ratio of 20.4, and the relative risk of failing the ABO-WQE was 11.3; however, the relative risk of passing if the cutoff was achieved was only 1.8, and a positive predictive value for failing the ABO-WQE on first attempt of 0.47. Conclusions Percentile scores on the OKAP are not indicative of percentile scores on the ABO-WQE. While OKAP examinations in the second and third year appear to separate out residents at risk of failing the ABO-WQE, there is a large disparity between the risk of passing and failing which is likely due to the inadequate prognostic value of the test. It is likely OKAP examinations may be a useful tool to identify residents who are at low risk for failing the ABO-WQE on first attempt, but much less useful for identifying residents who are at high risk for failing the ABO-WQE.

Global benchmarking of medical student learning outcomes? Implementation and pilot results of the International Foundations of Medicine Clinical Sciences Exam at The University of Queensland, Australia

Objective: To report pilot results for international benchmarking of learning outcomes among 426 final year medical students at the University of Queensland (UQ), Australia.Methods: Students took the International Foundations of Medicine (IFOM) Clinical Sciences Exam (CSE) developed by the National Board of Medical Examiners, USA, as a required formative assessment. IFOM CSE comprises 160 multiple-choice questions in medicine, surgery, obstetrics, paediatrics and mental health, taken over 4.5 hours.Outcomes: Significant implementation issues; IFOM scores and benchmarking with International Comparison Group (ICG) scores and United States Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) scores; and correlation with UQ medical degree cumulative grade point average (GPA).Results: Implementation as an online exam, under university-mandated conditions was successful. Mean IFOM score was 531.3 (maximum 779-minimum 200). The UQ cohort performed better (31% scored below 500) than the ICG (55% below 500). However 49% of the UQ cohort did not meet the USMLE Step 2 CK minimum score. Correlation between IFOM scores and UQ cumulative GPA was reasonable at 0.552 (p < 0.001).Conclusions: International benchmarking is feasible and provides a variety of useful benchmarking opportunities.

Obstetrics and gynecology clerkship for males and females: similar curriculum, different outcomes?

ObjectiveTo determine if performance differences exist between male and female students on a 6-week obstetrics and gynecology (Ob/Gyn) clerkship and to evaluate potential variables that might underlie any observed variations.Study DesignFinal clerkship grades and component scores (clinical evaluations, objective structured clinical examination [OSCE], oral examination, and National Board of Medical Examiners [NBME] subject examination) from July 2007 to June 2010 were matched by student and analyzed by gender. Basic science grade point average (GPA) and initial United States Medical Licensing Exam (USMLE) Step 1 scores were used to establish students’ baseline medical knowledge. On a post-clerkship questionnaire, a subset of students reported the numbers of procedures they performed during the clerkship; students also completed online pre- and post-clerkship questionnaires reflecting their self-assessed confidence in women's health clinical skills.ResultsScores were analyzed for 136 women and 220 men. Final clerkship grades were significantly higher for females than for males (89.05 vs. 87.34, p=0.0004, η2=0.08). Specifically, females outscored males on the OSCE, oral, and NBME subject examination portions of the clerkship but not clinical evaluations. Males reported completing fewer breast examinations (p=0.001, η2=0.14). Pre-clerkship, males were significantly less confident than females in women's health clinical skills (p<0.01) but reached similar levels upon completion of the clerkship. No gender differences were detected for basic science GPA and USMLE Step 1 scores.ConclusionStudent gender is associated with final grades on an Ob/Gyn clerkship. Further research regarding these differences should be explored.

Two Perspectives on the Effects of a Curriculum Change: Student Experience and the United States Medical Licensing Examination, Step 1

Students' perceptions of curricular experience and study effort were compared for a traditional and a new integrative, interdisciplinary curriculum at a single institution. United States Medical Licensing Exam (USMLE) Step 1 scores were examined for subgroup interactions. Medical students from four cohorts completed an educational goals survey and USMLE Step 1. Analysis included subgroup performance based on admissions data. Students rated the new curriculum as more helpful in achieving educational goals. USMLE Step 1 was significantly higher for students in the lowest quartile of MCAT scores in the new compared with the previous curriculum. To understand the outcomes of a large-scale curricular intervention, interactions of curriculum and aptitude should be examined.

Study of Selected Outcomes of Medical Students Who Fail USMLE Step 1

Purpose: Passage of the United States Medical Licensing Exam (USMLE) is required to obtain a medical license in the United States. Currently the majority of US medical schools require passage of USMLE Step 1 for either promotion to the third year or graduation from medical school. Virginia Commonwealth University School of Medicine (VCUSOM) requires that students take the USMLE but does not require passing of USMLE Step 1 for promotion or graduation. This policy enabled the authors to analyze performance outcomes during clinical rotations and monitor the residency match for a group of students who failed USMLE Step 1 on the first attempt.Methods: Third year clerkship grades and residency match results were reviewed for 64 students of the graduating classes of 1999-2005 who failed Step 1 on the first attempt. An equal number of students who passed Step 1 were randomly selected from each class as a comparison group. Average clinical performance ratings, NBME subject exam scores and final third year clerkship grades for the two groups were compared. Residency match rates and specialty certification were also compared.Results: The USMLE Fail Group had more Pass and fewer Honors clerkship grades than the comparison group. Subject exam scores were significantly lower in the USMLE Fail Group in all clerkships. Clinical performance ratings were significantly lower in the Fail group in three out of six clerkships: Internal Medicine, Pediatrics, and Psychiatry. However, 82% of the USMLE Fail Group later passed USMLE Step 1 and 2. Fifty-nine of the 64 students in the USMLE Fail Group matched for a residency, whereas all of the students in the Pass Group matched for a residency.Conclusion: Students who fail USMLE Step 1 have lower final clerkship grades due in part to lower NBME subject exam scores. The majority of these students, however, successfully pass USMLE Step 1 prior to graduation, go on to graduate medical training, and become board certified in their specialty.

Is there declining interest in general surgery training?

Purpose To monitor and report the quality of categorical first-year surgery residents matched to U.S. general surgery training programs from 1996 to 2001. Methods A survey was sent to 258 program directors of accredited general surgery training programs. In this survey, data were requested regarding United States Medical Licensing Exam (USMLE) Step 1 and 2 scores, matched residents' rank list position, Alpha Omega Alpha (AOA) status, number of applications received, and interviews granted pertaining to all National Residency Matching Program (NRMP) residents matched. In addition, the USMLE and NRMP were petitioned for national data regarding Step 1 and 2 scores in all entering surgery residents and first-time USMLE takers. Results Usable survey data were received on 1241 residents. The number for each year (and percent of total matched PGY-1 residents) was as follows: 1996-196 (20.0), 1997-206 (20.4), 1998-204 (19.9), 1999-212 (21.0), 2000-212 (20.7), and 2001-211 (21.7). The mean Step 1 scores increased over time (p < 0.001), and programs with 5 or more categorical spots had higher scores than those with 4 or less (p < 0.001). The depth required to fill the rank list increased over the study period (p < 0.05). National data received from the NRMP from 1994 to 2001 (mean of 916 residents at each year) showed a similar increase in USMLE Step 1 scores when compared with our survey. The proportion of AOA students matching into general surgery has decreased from 30% in 1996 to 15% in 2001 (p < 0.001). Conclusions Over the 6 years of our study, USMLE Step 1 scores increased and the results of our survey are in agreement with the national data. However, the proportion of AOA students declined, implying the top 10% of the medical school class found general surgery training less attractive. Also, programs went deeper into their rank lists to fill, implying a shrinking pool of candidates.