Medical Licensing Examination Research Articles

Dear Program Director: An Evaluation of Implicit Bias in Letters of Recommendation for Neurosurgery Residency.

Despite comprising half of medical students, women represent only 29.6% of neurosurgery applicants and 17% of residents, suggesting a "leak" in the career pipeline for women neurosurgeons. Surveys persistently show that neurosurgery programs identify United States Medical Licensing Examination (USMLE®) Step 1 score and letters of recommendation (LORs) as the most important factors in selecting applicants to interview. A previous study in neurosurgery found no differences in LORs. However, multiple studies in other specialties have demonstrated implicit gender bias in LORs, which may influence resident selection. Our objective is to evaluate neurosurgery residency LORs for evidence of implicit gender bias. Retrospective analysis of LORs for interviewed neurosurgery applicants at a single institution during the 2014 to 2020 National Residency Matching Program (NRMP®) match cycles. Letters were evaluated using Linguistic Inquiry & Word Count (LIWC) software (Pennebaker Conglomerates), and additional applicant data were obtained from candidate applications. LIWC (Pennebaker Conglomerates) output data included custom dictionary categories and terms that were analyzed using Prism 10 and Rstudio. Two hundred eighteen applications were reviewed for a total of 827 letters. LIWC (Pennebaker Conglomerates) analysis showed significant differences in word count (331 vs 297, difference = 34, 95% CI: 9-61, P = .008). LORs for applicants who were men were more likely to mention Alpha Omega Alpha Honor Medical Society (1.17 vs 0.778, difference = 0.4, 95% CI: 0.13-0.67, P = .023). USMLE® Step 1 scores were significantly lower for women (241 vs 247, difference = 6, 95% CI: 2-10, P = .004). There was no significant difference between letters for men and women for all categories evaluated in the linguistic evaluation. LORs are vital to the neurosurgical residency application process. The data exhibit some differences between the men and women applicants but few differences in their LORs, consistent with the results of the previous neurosurgical study.

Evaluating ChatGPT-4 in medical education: an assessment of subject exam performance reveals limitations in clinical curriculum support for students

This study evaluates the proficiency of ChatGPT-4 across various medical specialties and assesses its potential as a study tool for medical students preparing for the United States Medical Licensing Examination (USMLE) Step 2 and related clinical subject exams. ChatGPT-4 answered board-level questions with 89% accuracy, but showcased significant discrepancies in performance across specialties. Although it excelled in psychiatry, neurology, and obstetrics and gynecology, it underperformed in pediatrics, emergency medicine, and family medicine. These variations may be potentially attributed to the depth and recency of training data as well as the scope of the specialties assessed. Specialties with significant interdisciplinary overlap had lower performance, suggesting complex clinical scenarios pose a challenge to the AI. In terms of the future, the overall efficacy of ChatGPT-4 indicates a promising supplemental role in medical education, but performance inconsistencies across specialties in the current version lead us to recommend that medical students use AI with caution.

Exploring the usage of learning resources by medical students in the basic science stage and their effect on academic performance

BackgroundThe United States Medical Licensing Examination (USMLE) step 1 is one of the two examinations written after completion of the first two years (basic science stage) of medical school to be eligible to apply for residency training in the USA. A huge number and types of study materials are available to prepare for the exam which might confuse students choosing a resource.We investigated learning resources being used by the third and fifth-semester medical students and their association with academic performance. We also compared learning resources and exam scores of high-performing and low-performing students.MethodsData collection was done using structured (quantitative study) and semi-structured (qualitative study) questionnaires during a face-to-face interview. This article is about the quantitative part which was designed as a correlational study. Single factor one-way analysis of variance (ANOVA), Pearson correlation coefficient test, T-test, and Fisher’s exact test were used to analyze the data.ResultsAbout half of all students used three or more commercial resources dealing with the same content. A weak negative correlation was observed between the number of commercial resources and the exam scores, especially when the number of these resources was three or more (r = -0.26). The mean exam score of textbook users was statistically significantly higher than the mean score of textbook non-users (p = 0.01). The usage of textbooks was statistically significantly higher in the cohort of top performers in comparison to the rest of the students (p = 0.006). In addition to less usage of textbooks, the mean number of review books was higher in the group of weakest students (2.84 versus 3.7; p = 0.75).ConclusionsMost students did not use professional textbooks and about half used too many commercial review resources. While the former fact was significantly associated with poor academic performance, the later fact had weak negative correlation with exam score.Pedagogical interventions are urgently needed to make the right type of learning resources available by making professional textbooks more USMLE-oriented and helping the students choose the best and right number of resources for optimum academic performance. By fulfilling the observed needs of the students in this way, they might feel empowered because of self-determination which will motivate studies.

United States Medical Licensing Examination Performance and Outcomes of Patient Care.

United States Medical Licensing Examination Performance and Outcomes of Patient Care.

Gender differences in learning and study strategies impact medical students’ preclinical and USMLE step 1 examination performance

BackgroundEvaluation of students’ learning strategies can enhance academic support. Few studies have investigated differences in learning strategies between male and female students as well as their impact on United States Medical Licensing Examination® (USMLE) Step 1 and preclinical performance.MethodsThe Learning and Study Strategies Inventory (LASSI) was administered to the classes of 2019–2024 (female (n = 350) and male (n = 262)). Students’ performance on preclinical first-year (M1) courses, preclinical second-year (M2) courses, and USMLE Step 1 was recorded. An independent t-test evaluated differences between females and males on each LASSI scale. A Pearson product moment correlation determined which LASSI scales correlated with preclinical performance and USMLE Step 1 examinations.ResultsOf the 10 LASSI scales, Anxiety, Attention, Information Processing, Selecting Main Idea, Test Strategies and Using Academic Resources showed significant differences between genders. Females reported higher levels of Anxiety (p < 0.001), which significantly influenced their performance. While males and females scored similarly in Concentration, Motivation, and Time Management, these scales were significant predictors of performance variation in females. Test Strategies was the largest contributor to performance variation for all students, regardless of gender.ConclusionGender differences in learning influence performance on STEP1. Consideration of this study’s results will allow for targeted interventions for academic success.

Policy perception, job satisfaction and intentions to remain in rural area: evidence from the National Compulsory Service Programme in China

BackgroundExploring factors that may influence general practitioners (GPs)’ intentions to remain in rural area is necessary to inform the training and placement of future medical workforce in rural area. However, little is known about how GPs’ perception towards the National Compulsory Service Programme (NCSP) and job satisfaction impact their turnover intention. This paper explores GPs’ intentions to remain in rural China and how their policy perception and job satisfaction predict the intentions.MethodsWe conducted a cross-sectional, online survey from December 2021 to February 2022 to investigate GPs’ perception towards NCSP, job satisfaction, and intentions to remain in rural area. Eligible participants were GPs who were required to provide health services as part of NCSP at township health centres of 9 provinces which could represent all NCSP GPs in China. Multinomial logistic regression analyses were performed to explore the associations between policy perceptions, job satisfaction, and intentions to remain.ResultsOf 3615 GPs included in the analysis, 442 (12.2%) would like to remain in rural area and 1266 (35.0%) were unsure. Results of the multinomial logistic regression analyses showed that compared with GPs who would leave, GPs with higher perception scores for the restriction on taking postgraduate exam (RRR: 1.93, 95% CI 1.72, 2.16) and the commitment to work for six years (RRR: 1.53, 95% CI 1.31, 1.78) were more likely to remain. In contrast, GPs who had higher perception scores for completing standardised residency training (RRR: 0.75, 95% CI 0.64, 0.88) and passing National Medical Licensing Examinations (RRR: 0.74, 95% CI 0.62, 0.87) were more likely to leave. GPs who were satisfied with the freedom of choosing work methods (RRR: 1.52, 95% CI 1.25, 1.84) and chances of promotion (RRR: 1.60, 95% CI 1.32, 1.94) were more likely to remain.ConclusionsThis study highlights the significance of policy perception and job satisfaction on GPs’ intentions to remain in rural area. Factors such as career advancement and the empowerment of GPs to build on and use their skills and abilities should be taken into account when designing rural placement programmes.

Exploring the Performance of ChatGPT Versions 3.5, 4, and 4 With Vision in the Chilean Medical Licensing Examination: Observational Study.

The deployment of OpenAI's ChatGPT-3.5 and its subsequent versions, ChatGPT-4 and ChatGPT-4 With Vision (4V; also known as "GPT-4 Turbo With Vision"), has notably influenced the medical field. Having demonstrated remarkable performance in medical examinations globally, these models show potential for educational applications. However, their effectiveness in non-English contexts, particularly in Chile's medical licensing examinations-a critical step for medical practitioners in Chile-is less explored. This gap highlights the need to evaluate ChatGPT's adaptability to diverse linguistic and cultural contexts. This study aims to evaluate the performance of ChatGPT versions 3.5, 4, and 4V in the EUNACOM (Examen Único Nacional de Conocimientos de Medicina), a major medical examination in Chile. Three official practice drills (540 questions) from the University of Chile, mirroring the EUNACOM's structure and difficulty, were used to test ChatGPT versions 3.5, 4, and 4V. The 3 ChatGPT versions were provided 3 attempts for each drill. Responses to questions during each attempt were systematically categorized and analyzed to assess their accuracy rate. All versions of ChatGPT passed the EUNACOM drills. Specifically, versions 4 and 4V outperformed version 3.5, achieving average accuracy rates of 79.32% and 78.83%, respectively, compared to 57.53% for version 3.5 (P<.001). Version 4V, however, did not outperform version 4 (P=.73), despite the additional visual capabilities. We also evaluated ChatGPT's performance in different medical areas of the EUNACOM and found that versions 4 and 4V consistently outperformed version 3.5. Across the different medical areas, version 3.5 displayed the highest accuracy in psychiatry (69.84%), while versions 4 and 4V achieved the highest accuracy in surgery (90.00% and 86.11%, respectively). Versions 3.5 and 4 had the lowest performance in internal medicine (52.74% and 75.62%, respectively), while version 4V had the lowest performance in public health (74.07%). This study reveals ChatGPT's ability to pass the EUNACOM, with distinct proficiencies across versions 3.5, 4, and 4V. Notably, advancements in artificial intelligence (AI) have not significantly led to enhancements in performance on image-based questions. The variations in proficiency across medical fields suggest the need for more nuanced AI training. Additionally, the study underscores the importance of exploring innovative approaches to using AI to augment human cognition and enhance the learning process. Such advancements have the potential to significantly influence medical education, fostering not only knowledge acquisition but also the development of critical thinking and problem-solving skills among health care professionals.

Large language models leverage external knowledge to extend clinical insight beyond language boundaries.

Large Language Models (LLMs) such as ChatGPT and Med-PaLM have excelled in various medical question-answering tasks. However, these English-centric models encounter challenges in non-English clinical settings, primarily due to limited clinical knowledge in respective languages, a consequence of imbalanced training corpora. We systematically evaluate LLMs in the Chinese medical context and develop a novel in-context learning framework to enhance their performance. The latest China National Medical Licensing Examination (CNMLE-2022) served as the benchmark. We collected 53 medical books and 381149 medical questions to construct the medical knowledge base and question bank. The proposed Knowledge and Few-shot Enhancement In-context Learning (KFE) framework leverages the in-context learning ability of LLMs to integrate diverse external clinical knowledge sources. We evaluated KFE with ChatGPT (GPT-3.5), GPT-4, Baichuan2-7B, Baichuan2-13B, and QWEN-72B in CNMLE-2022 and further investigated the effectiveness of different pathways for incorporating LLMs with medical knowledge from 7 distinct perspectives. Directly applying ChatGPT failed to qualify for the CNMLE-2022 at a score of 51. Cooperated with the KFE framework, the LLMs with varying sizes yielded consistent and significant improvements. The ChatGPT's performance surged to 70.04 and GPT-4 achieved the highest score of 82.59. This surpasses the qualification threshold (60) and exceeds the average human score of 68.70, affirming the effectiveness and robustness of the framework. It also enabled a smaller Baichuan2-13B to pass the examination, showcasing the great potential in low-resource settings. This study shed light on the optimal practices to enhance the capabilities of LLMs in non-English medical scenarios. By synergizing medical knowledge through in-context learning, LLMs can extend clinical insight beyond language barriers in healthcare, significantly reducing language-related disparities of LLM applications and ensuring global benefit in this field.

Academic performance of students in an accelerated medical pathway

ABSTRACT Accelerated medical school curricula, such as three-year programs, have gained attention in recent years but studies evaluating their impact are still scarce. This study examines the Fully Integrated Readiness for Service Training (FIRST) program, a three-year accelerated pathway, to assess its impact on students’ academic performance preparedness for residency. In this observational study, we compared the academic outcomes of FIRST program students to traditional four-year curriculum students from 2018 to 2023. We analyzed multiple metrics, including exam performance (United States Medical Licensing Examination Step scores, shelf exam scores, and pre-clinical course scores) and clinical performance scores during the application and individualization phases. Analysis of Variance was used to examine the effect of accelerated pathway program experience relative to traditional 4-year medical school curriculum on the learning outcomes. FIRST program students were on average 1.5 years younger upon graduation than their traditional peers. While FIRST program students scored slightly lower on Step 2 Clinical Knowledge (CK), they exhibited no significant differences in other exam scores or clinical performance relative to the traditional students. Notably, FIRST students achieved equivalent clinical performance ratings during critical clerkships and rotations. Our findings suggest that a three-year medical school curriculum can effectively prepare students for residency and produce graduates with comparable medical knowledge and clinical skills, offering potential benefits in terms of financial relief and personal well-being for medical students.

ChatGPT's Performance in Cardiac Arrest and Bradycardia Simulations Using the American Heart Association's Advanced Cardiovascular Life Support Guidelines: Exploratory Study.

ChatGPT is the most advanced large language model to date, with prior iterations having passed medical licensing examinations, providing clinical decision support, and improved diagnostics. Although limited, past studies of ChatGPT's performance found that artificial intelligence could pass the American Heart Association's advanced cardiovascular life support (ACLS) examinations with modifications. ChatGPT's accuracy has not been studied in more complex clinical scenarios. As heart disease and cardiac arrest remain leading causes of morbidity and mortality in the United States, finding technologies that help increase adherence to ACLS algorithms, which improves survival outcomes, is critical. This study aims to examine the accuracy of ChatGPT in following ACLS guidelines for bradycardia and cardiac arrest. We evaluated the accuracy of ChatGPT's responses to 2 simulations based on the 2020 American Heart Association ACLS guidelines with 3 primary outcomes of interest: the mean individual step accuracy, the accuracy score per simulation attempt, and the accuracy score for each algorithm. For each simulation step, ChatGPT was scored for correctness (1 point) or incorrectness (0 points). Each simulation was conducted 20 times. ChatGPT's median accuracy for each step was 85% (IQR 40%-100%) for cardiac arrest and 30% (IQR 13%-81%) for bradycardia. ChatGPT's median accuracy over 20 simulation attempts for cardiac arrest was 69% (IQR 67%-74%) and for bradycardia was 42% (IQR 33%-50%). We found that ChatGPT's outputs varied despite consistent input, the same actions were persistently missed, repetitive overemphasis hindered guidance, and erroneous medication information was presented. This study highlights the need for consistent and reliable guidance to prevent potential medical errors and optimize the application of ChatGPT to enhance its reliability and effectiveness in clinical practice.

GPT-4/4V's performance on the Japanese National Medical Licensing Examination

Background Recent advances in Artificial Intelligence (AI) are changing the medical world, and AI will likely replace many of the actions performed by medical professionals. The overall clinical ability of the AI has been evaluated by its ability to answer a text-based national medical examination. This study uniquely assesses the performance of Open AI's ChatGPT against all Japanese National Medical Licensing Examination (NMLE), including images, illustrations, and pictures. Methods We obtained the questions of the past six years of the NMLE (112th to 117th) from the Japanese Ministry of Health, Labour and Welfare website. We converted them to JavaScript Object Notation (JSON) format. We created an application programming interface (API) to output correct answers using GPT-4 for questions without images and GPT4-V(ision) or GPT4 console for questions with images. Results The percentage of image questions was 723/2400 (30.1%) over the past six years. In all years, GPT-4/4V exceeded the minimum score the examinee should score. In total, over the six years, the percentage of correct answers for basic medical knowledge questions was 665/905 (73.5%); for clinical knowledge questions, 1143/1531 (74.7%); and for image questions 497/723 (68.7%), respectively. Conclusions Regarding medical knowledge, GPT-4/4V met the minimum criteria regardless of whether the questions included images, illustrations, and pictures. Our study sheds light on the potential utility of AI in medical education.

DO seniors and IMGs have lower match probabilities than MD seniors after adjusting for specialty choice and USMLE Step 1 score.

It is unknown if US residency applicants of different educational backgrounds (US allopathic [MD], Doctor of Osteopathic Medicine [DO], and international medical graduates [IMG]) but comparable academic performance have similar match success. Our objective was to compare match probabilities between applicant types after adjusting for specialty choice and United States Medical Licensing Examination (USMLE) Step 1 scores. We performed a secondary analysis of published data in National Resident Matching Program (NRMP) reports from 2016, 2018, 2020, and 2022 for US MD seniors, DO seniors, and IMGs (US citizens and non-US citizens). We examined the 10 specialties with the most available spots in 2022. Average marginal effects from a multiple variable logistic regression model were utilized to estimate each non-MD senior applicant type's probability of matching into their preferred specialty compared to MD seniors adjusting for specialty choice, Step 1 score, and match year. Each non-MD applicant type had a lower adjusted percent difference in matching to their preferred specialty than MD seniors,-7.1 % (95 % confidence interval [CI],-11.3 to-2.9) for DO seniors,-45.6 % (-50.6 to-40.5) for US IMGs, and-56.6 % (-61.5 to-51.6) for non-US IMGs. Similarly, each non-MD applicant type had a lower adjusted percent difference in matching than MD seniors across almost all Step 1 score ranges, except for DO seniors with Step 1 scores <200 (-2.0 % [-9.5 to 5.5]). After adjusting for specialty choice, Step 1 score, and match year, non-US MD applicants had lower probabilities of matching into their preferred specialties than their US MD colleagues.

Medical students' perspective on the United States Medical Licensing Examination (USMLE) Step 1 transition to Pass/Fail

Background The transition of the United States Medical Licensing Exam: Step 1 to Pass/Fail (P/F), from scored, caused uncertainty about students’ preparedness and wellbeing related to the exam. Comparison of study behavior and results, before and after the P/F transition can provide insights for the medical curriculum and student support. Methods Data from four cohorts of second-year medical students (Class of 2022–25, N = 204) were collected from their dedicated Step 1 self-study block. Student study regiments, aggregate practice test results, Step 1 pass rates and post-block self-reported surveys were analyzed. Results Analysis of practice exam averages across the four student cohorts demonstrates a potentially slower and less rigorous start to Step 1 self-studying during the dedicated preparation block for the cohorts that took the P/F Step 1 exam format as compared to the previous cohorts that prepared for a scored Step 1 exam. Similarly, self-reported study regiments decreased in the median number of hours/day and number of weeks of study for the cohorts with P/F Step 1 exam. There was also a slight shift in the type of study resources used, between the two groups, with the scored group using more traditional board preparation resources. Conclusions The P/F transition of the Step 1 exam may lead to reduced student preparedness and may require adjustments in the resources and support provided by institutions.

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ±63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

The Impact of United States Medical Licensing Examination Step 1 Transitioning To Pass/Fail on Medical Student Perception of Research Needed to Match Into One's Preferred Specialty.

Purpose To evaluate how the transition of United States Medical Licensing Examination (USMLE) Step 1 to a pass/fail scoring influenced medical student perceptions of the importance of research required to match into their preferred residency specialty. Methods A 14-item survey was distributed by e-mail to medical students at one medical school in the southeastern United States in November of 2021. Responses were compared between medical students taking USMLE Step 1 pass/fail in the future and medical students taking USMLE Step 1 for a three-digit score. Results A total of 168 medical students responded to the survey with 98 respondents who planned on taking USMLE Step 1 pass/fail (45 first-year medical students (MS1) and 53 MS2) and 70 respondents who took USMLE Step 1 for a numerical score (37 MS3and 33 MS4). There were no differences in how each cohort scored the level of importance of research in matching into their preferred residency specialty (p=0.10); however, those taking USMLE Step 1 pass/fail believe an average of 4.6 research experiences are necessary to match into their preferred residency, compared to only 3.4 research experiences for those who took it for a numerical score (p=0.04). Conclusion No statistically significant difference in the perceived importance of research in matching into one's preferred residency specialty was found between cohorts. However, the pass/fail cohort believes they will need more research experiences to match their chosen specialty than the numerical score cohort. Results could indicate that students participate in more research and extracurricular activities to be more competitive for residency applications.

Matching Against Men: 5 Years of Residency Match Data Show Disparities Still Exist.

Despite progress toward equal representation by sex in medical practice, women remain underrepresented in many specialties. This study sought to examine the current state of gender equality among recently graduated doctors in multiple specialties. Deidentified demographics, standardized examination scores, and Match results were gathered for 829 graduates. Participants were selected from an allopathic medical school between 2016 and 2020. Nineteen students (2.29%) were excluded from the study. Descriptive statistics were calculated, and χ2 tests for independence were used to compare proportions between reported sex and specialty and program Match results. One-way analysis of variance was then performed to test for differences in US Medical Licensing Examination Step 1 and Step 2 scores between sexes. P < 0.05 was considered statistically significant. Of the 829 individuals studied, 44.6% were women. A significantly smaller proportion of women matched into the most competitive specialties, despite no significant difference in US Medical Licensing Examination Step 1 scores between sexes. Furthermore, there was an overall significant trend of women matching into more competitive programs for any given specialty. In this study, we found that men matched into more highly competitive specialties, whereas women matched into more competitive residency program locations. Further research is needed to determine why women matched into specific specialties at lower rates than their male peers and seek to understand how sex affects the narrative of specialty choice.

Racial and Ethnic Disparities in Situational Judgment Testing Among Applicants to an Anesthesiology Residency Program

Background The Computer-Based Assessment for Sampling Personal Characteristics (CASPer) is a situational judgment test (SJT) that assesses noncognitive skills like professionalism, communication, and empathy. There are no reports of the effects of race/ethnicity and sex on CASPer scores among residency applicants. Objective We examined the effects of race/ethnicity, sex, and United States vs international medical school attendance on CASPer performance. Methods Our anesthesiology residency program required all applicants for the 2021-2022 Match cycle to complete an online video and text-based SJT (CASPer). We compared these results, reported as z-scores, with self-identified race/ethnicity, sex, United States vs international medical school attendance, and United States Medical Licensing Examination (USMLE) Step 1 scores. Results Of the 1245 applicants who completed CASPer, 783 identified as male. The racial/ethnic distribution was 512 White, 412 Asian, 106 Black, 126 Hispanic, and 89 Other/No Answer. CASPer z-scores did not differ by sex. White candidates scored higher than Black (0.18 vs -0.57, P&amp;lt;.001) and Hispanic (0.18 vs -0.52, P&amp;lt;.001) candidates. Applicants attending US medical schools scored higher than those attending international medical schools (z-scores: 0.15 vs -0.68, P&amp;lt;.001). There was no correlation between CASPer z-scores and USMLE Step 1 scores. Conclusions Our results suggest that CASPer scores favor White applicants over Black and Hispanic ones and applicants attending US medical schools over those attending international medical schools.

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establish- ing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.

Program Directors' Perspectives on Residency Applications in the Post‒United States Medical Licensing Examination Step 1 Era: A Case for Standardized Letters of Recommendation?

The United States Medical Licensing Examination (USMLE) Step 1 test evolved into a key metric utilized by program directors (PDs) in assessing candidates for residency. The transition to a USMLE Step 1 binary pass/fail scoring system has resulted in a loss of an important objective assessment. With national movements toward pass/fail systems for clerkship grading and trends toward abandonment of class ranking, assessing residency applications has become increasingly challenging. The Society of Academic Urologists convened a task force to, in part, assess the perspectives of urology PDs regarding the importance of various aspects of a residency application for predicting clinical performance. An anonymous survey was disseminated to all urology PDs in the US. Perspectives on 11 potential application predictors of clinical performance and demographics were recorded. Descriptive statistics characterized PD responses. Friedman test and pairwise Wilcoxon tests were used to evaluate the relative ranks assigned to application elements by PDs. There was a 60.5% response rate (89/147). Letters of recommendation (LORs) were ranked as the most important predictor, with a mean rank of 2.39, median of 2 (IQR 1-3). Clerkship grades and USMLE Step 1 were comparable and ranked second. Medical school reputation ranked the lowest. There was significant subjective heterogeneity among categories; however, this was less so for LORs, which predominated as the most important factor among application elements (P < .001). To our knowledge, this is the largest sample size assessing PD perspectives on application factors that predict clinical performance. The second (clerkship grades) and third (USLME Step 1) most important factors moving toward binary pass/fail systems create an opportunity for actionable change to improve assessment objectivity. Our data demonstrate LORs to be the most important factor of residency applications, making a compelling argument for moving toward a standardized LOR to maximize this tool, mitigate bias, and improve interreviewer reliability.

A validity study of COMLEX-USA Level 3 with the new test design.

The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores. The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework. Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt testscores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE). All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure. This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series.