United States Medical Licensing Examination Research Articles

Out-of-Hospital Birth in Medical Education: A Thematic Analysis of Question Banks for Medical Licensing Examinations in the United States.

This study assessed the content of US Medical Licensing Examination question banks with regard to out-of-hospital births and whether the questions aligned with current evidence. Three question banks were searched for key words regarding out-of-hospital births. A thematic analysis was then utilized to analyze the results. Forty-seven questions were identified, and of these, 55% indicated a lack of inadequate, limited, or irregular prenatal care in the question stem. Systematic studies comparing prenatal care in out-of-hospital births versus hospital births are nonexistent, leading to the potential for bias and adverse outcomes. Adjustments to question stems that accurately portray current evidence are recommended.

Letters of Reference in the Current Era.

Letters of reference (LORs) are a common component of the application process for residency training programs. With the United States Medical Licensing Examination Step 1 transitioning to pass/fail grading and with the increasing use of holistic review, the potential role of LORs is rising in importance. Among some key benefits are the ability to provide a broader and more holistic view of applicants, which can include highlighting elements of experiences or skills that could be missed in their application, as well as providing a third-party assessment of the applicant external to their rotation experiences. However, LORs also face issues, including variation in quality, challenges with comparability, and risk of bias. In this article, the authors discuss the unique benefits, limitations, and best practice recommendations for LORs in academic medicine. The authors also discuss future directions, including the role of artificial intelligence, unblinded, and co-created LORs.

Dear Program Director: An Evaluation of Implicit Bias in Letters of Recommendation for Neurosurgery Residency.

Despite comprising half of medical students, women represent only 29.6% of neurosurgery applicants and 17% of residents, suggesting a "leak" in the career pipeline for women neurosurgeons. Surveys persistently show that neurosurgery programs identify United States Medical Licensing Examination (USMLE®) Step 1 score and letters of recommendation (LORs) as the most important factors in selecting applicants to interview. A previous study in neurosurgery found no differences in LORs. However, multiple studies in other specialties have demonstrated implicit gender bias in LORs, which may influence resident selection. Our objective is to evaluate neurosurgery residency LORs for evidence of implicit gender bias. Retrospective analysis of LORs for interviewed neurosurgery applicants at a single institution during the 2014 to 2020 National Residency Matching Program (NRMP®) match cycles. Letters were evaluated using Linguistic Inquiry & Word Count (LIWC) software (Pennebaker Conglomerates), and additional applicant data were obtained from candidate applications. LIWC (Pennebaker Conglomerates) output data included custom dictionary categories and terms that were analyzed using Prism 10 and Rstudio. Two hundred eighteen applications were reviewed for a total of 827 letters. LIWC (Pennebaker Conglomerates) analysis showed significant differences in word count (331 vs 297, difference = 34, 95% CI: 9-61, P = .008). LORs for applicants who were men were more likely to mention Alpha Omega Alpha Honor Medical Society (1.17 vs 0.778, difference = 0.4, 95% CI: 0.13-0.67, P = .023). USMLE® Step 1 scores were significantly lower for women (241 vs 247, difference = 6, 95% CI: 2-10, P = .004). There was no significant difference between letters for men and women for all categories evaluated in the linguistic evaluation. LORs are vital to the neurosurgical residency application process. The data exhibit some differences between the men and women applicants but few differences in their LORs, consistent with the results of the previous neurosurgical study.

Evaluating ChatGPT-4 in medical education: an assessment of subject exam performance reveals limitations in clinical curriculum support for students

This study evaluates the proficiency of ChatGPT-4 across various medical specialties and assesses its potential as a study tool for medical students preparing for the United States Medical Licensing Examination (USMLE) Step 2 and related clinical subject exams. ChatGPT-4 answered board-level questions with 89% accuracy, but showcased significant discrepancies in performance across specialties. Although it excelled in psychiatry, neurology, and obstetrics and gynecology, it underperformed in pediatrics, emergency medicine, and family medicine. These variations may be potentially attributed to the depth and recency of training data as well as the scope of the specialties assessed. Specialties with significant interdisciplinary overlap had lower performance, suggesting complex clinical scenarios pose a challenge to the AI. In terms of the future, the overall efficacy of ChatGPT-4 indicates a promising supplemental role in medical education, but performance inconsistencies across specialties in the current version lead us to recommend that medical students use AI with caution.

Exploring the usage of learning resources by medical students in the basic science stage and their effect on academic performance

BackgroundThe United States Medical Licensing Examination (USMLE) step 1 is one of the two examinations written after completion of the first two years (basic science stage) of medical school to be eligible to apply for residency training in the USA. A huge number and types of study materials are available to prepare for the exam which might confuse students choosing a resource.We investigated learning resources being used by the third and fifth-semester medical students and their association with academic performance. We also compared learning resources and exam scores of high-performing and low-performing students.MethodsData collection was done using structured (quantitative study) and semi-structured (qualitative study) questionnaires during a face-to-face interview. This article is about the quantitative part which was designed as a correlational study. Single factor one-way analysis of variance (ANOVA), Pearson correlation coefficient test, T-test, and Fisher’s exact test were used to analyze the data.ResultsAbout half of all students used three or more commercial resources dealing with the same content. A weak negative correlation was observed between the number of commercial resources and the exam scores, especially when the number of these resources was three or more (r = -0.26). The mean exam score of textbook users was statistically significantly higher than the mean score of textbook non-users (p = 0.01). The usage of textbooks was statistically significantly higher in the cohort of top performers in comparison to the rest of the students (p = 0.006). In addition to less usage of textbooks, the mean number of review books was higher in the group of weakest students (2.84 versus 3.7; p = 0.75).ConclusionsMost students did not use professional textbooks and about half used too many commercial review resources. While the former fact was significantly associated with poor academic performance, the later fact had weak negative correlation with exam score.Pedagogical interventions are urgently needed to make the right type of learning resources available by making professional textbooks more USMLE-oriented and helping the students choose the best and right number of resources for optimum academic performance. By fulfilling the observed needs of the students in this way, they might feel empowered because of self-determination which will motivate studies.

Gender differences in learning and study strategies impact medical students’ preclinical and USMLE step 1 examination performance

BackgroundEvaluation of students’ learning strategies can enhance academic support. Few studies have investigated differences in learning strategies between male and female students as well as their impact on United States Medical Licensing Examination® (USMLE) Step 1 and preclinical performance.MethodsThe Learning and Study Strategies Inventory (LASSI) was administered to the classes of 2019–2024 (female (n = 350) and male (n = 262)). Students’ performance on preclinical first-year (M1) courses, preclinical second-year (M2) courses, and USMLE Step 1 was recorded. An independent t-test evaluated differences between females and males on each LASSI scale. A Pearson product moment correlation determined which LASSI scales correlated with preclinical performance and USMLE Step 1 examinations.ResultsOf the 10 LASSI scales, Anxiety, Attention, Information Processing, Selecting Main Idea, Test Strategies and Using Academic Resources showed significant differences between genders. Females reported higher levels of Anxiety (p < 0.001), which significantly influenced their performance. While males and females scored similarly in Concentration, Motivation, and Time Management, these scales were significant predictors of performance variation in females. Test Strategies was the largest contributor to performance variation for all students, regardless of gender.ConclusionGender differences in learning influence performance on STEP1. Consideration of this study’s results will allow for targeted interventions for academic success.

Student Perspectives on the Undergraduate Medical Student Research Experience at a Single United States Allopathic Institution.

Introduction The purpose of this study was to identify student-reported institutional facilitators and barriers to successful research experiences at a single United States allopathic institution. Residency applications have increasingly become more competitive, and with the United States Medical Licensing Examination (USMLE) Step 1 exam's transition to pass/fail, factors such as research experience and outcomes may become more important to increase residency application competitiveness. This study sought to explore factors that impact successful research experiences leading to tangible outcomes for medical students at our medical school, the Joe R. & Teresa Lozano Long School of Medicine. Methods A cross-sectional survey was developed and administered via REDCap to 853 students in May 2022. Survey question domains included demographics, past and present research participation, perceived barriers/facilitators to research, tangible outcomes (e.g., publications and posters), and overall satisfaction with research comparing subjectively "best" and "worst" experiences. The Institutional Review Board (IRB) deemed this project as non-regulated research. Results We had a 24% (n = 204/853) response rate. The responses were distributed equally among the four classes. A big portion of the participants (71%, n = 59/83) identified a tangible outcome as the most important measure of success. Regarding facilitators, students identified having a mentor (89%, n = 165/184) and departmental connections (85%, n = 156/184) as the most important when looking for a project. Barriers included SMART goals (Specific, Measurable, Achievable, Relevant, and Time-Bound) lacking in 31% (n = 24/75) of worst projects, followed by a clear timeline in 29% (n = 22/76) and hours of commitment in 27% (n = 21/78). The best projects were more likely to have resulted in a publication (61% (27/44) vs. 32% (14/44)) or have a poster (64% (28/44) vs. 36% (16/44)). Conclusions Medical students are interested in participating in research, with important facilitators including mentorship and departmental connections. Modifiable variables include lack of clear timelines, well-defined roles and responsibilities, and time commitments. This information may be useful for faculty who mentor medical students or medical schools interested in designing medical student research programs.

Academic performance of students in an accelerated medical pathway

ABSTRACT Accelerated medical school curricula, such as three-year programs, have gained attention in recent years but studies evaluating their impact are still scarce. This study examines the Fully Integrated Readiness for Service Training (FIRST) program, a three-year accelerated pathway, to assess its impact on students’ academic performance preparedness for residency. In this observational study, we compared the academic outcomes of FIRST program students to traditional four-year curriculum students from 2018 to 2023. We analyzed multiple metrics, including exam performance (United States Medical Licensing Examination Step scores, shelf exam scores, and pre-clinical course scores) and clinical performance scores during the application and individualization phases. Analysis of Variance was used to examine the effect of accelerated pathway program experience relative to traditional 4-year medical school curriculum on the learning outcomes. FIRST program students were on average 1.5 years younger upon graduation than their traditional peers. While FIRST program students scored slightly lower on Step 2 Clinical Knowledge (CK), they exhibited no significant differences in other exam scores or clinical performance relative to the traditional students. Notably, FIRST students achieved equivalent clinical performance ratings during critical clerkships and rotations. Our findings suggest that a three-year medical school curriculum can effectively prepare students for residency and produce graduates with comparable medical knowledge and clinical skills, offering potential benefits in terms of financial relief and personal well-being for medical students.

DO seniors and IMGs have lower match probabilities than MD seniors after adjusting for specialty choice and USMLE Step 1 score.

It is unknown if US residency applicants of different educational backgrounds (US allopathic [MD], Doctor of Osteopathic Medicine [DO], and international medical graduates [IMG]) but comparable academic performance have similar match success. Our objective was to compare match probabilities between applicant types after adjusting for specialty choice and United States Medical Licensing Examination (USMLE) Step 1 scores. We performed a secondary analysis of published data in National Resident Matching Program (NRMP) reports from 2016, 2018, 2020, and 2022 for US MD seniors, DO seniors, and IMGs (US citizens and non-US citizens). We examined the 10 specialties with the most available spots in 2022. Average marginal effects from a multiple variable logistic regression model were utilized to estimate each non-MD senior applicant type's probability of matching into their preferred specialty compared to MD seniors adjusting for specialty choice, Step 1 score, and match year. Each non-MD applicant type had a lower adjusted percent difference in matching to their preferred specialty than MD seniors,-7.1 % (95 % confidence interval [CI],-11.3 to-2.9) for DO seniors,-45.6 % (-50.6 to-40.5) for US IMGs, and-56.6 % (-61.5 to-51.6) for non-US IMGs. Similarly, each non-MD applicant type had a lower adjusted percent difference in matching than MD seniors across almost all Step 1 score ranges, except for DO seniors with Step 1 scores <200 (-2.0 % [-9.5 to 5.5]). After adjusting for specialty choice, Step 1 score, and match year, non-US MD applicants had lower probabilities of matching into their preferred specialties than their US MD colleagues.

Medical students' perspective on the United States Medical Licensing Examination (USMLE) Step 1 transition to Pass/Fail

Background The transition of the United States Medical Licensing Exam: Step 1 to Pass/Fail (P/F), from scored, caused uncertainty about students’ preparedness and wellbeing related to the exam. Comparison of study behavior and results, before and after the P/F transition can provide insights for the medical curriculum and student support. Methods Data from four cohorts of second-year medical students (Class of 2022–25, N = 204) were collected from their dedicated Step 1 self-study block. Student study regiments, aggregate practice test results, Step 1 pass rates and post-block self-reported surveys were analyzed. Results Analysis of practice exam averages across the four student cohorts demonstrates a potentially slower and less rigorous start to Step 1 self-studying during the dedicated preparation block for the cohorts that took the P/F Step 1 exam format as compared to the previous cohorts that prepared for a scored Step 1 exam. Similarly, self-reported study regiments decreased in the median number of hours/day and number of weeks of study for the cohorts with P/F Step 1 exam. There was also a slight shift in the type of study resources used, between the two groups, with the scored group using more traditional board preparation resources. Conclusions The P/F transition of the Step 1 exam may lead to reduced student preparedness and may require adjustments in the resources and support provided by institutions.

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ±63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

The Impact of United States Medical Licensing Examination Step 1 Transitioning To Pass/Fail on Medical Student Perception of Research Needed to Match Into One's Preferred Specialty.

Purpose To evaluate how the transition of United States Medical Licensing Examination (USMLE) Step 1 to a pass/fail scoring influenced medical student perceptions of the importance of research required to match into their preferred residency specialty. Methods A 14-item survey was distributed by e-mail to medical students at one medical school in the southeastern United States in November of 2021. Responses were compared between medical students taking USMLE Step 1 pass/fail in the future and medical students taking USMLE Step 1 for a three-digit score. Results A total of 168 medical students responded to the survey with 98 respondents who planned on taking USMLE Step 1 pass/fail (45 first-year medical students (MS1) and 53 MS2) and 70 respondents who took USMLE Step 1 for a numerical score (37 MS3and 33 MS4). There were no differences in how each cohort scored the level of importance of research in matching into their preferred residency specialty (p=0.10); however, those taking USMLE Step 1 pass/fail believe an average of 4.6 research experiences are necessary to match into their preferred residency, compared to only 3.4 research experiences for those who took it for a numerical score (p=0.04). Conclusion No statistically significant difference in the perceived importance of research in matching into one's preferred residency specialty was found between cohorts. However, the pass/fail cohort believes they will need more research experiences to match their chosen specialty than the numerical score cohort. Results could indicate that students participate in more research and extracurricular activities to be more competitive for residency applications.

Matching Against Men: 5 Years of Residency Match Data Show Disparities Still Exist.

Despite progress toward equal representation by sex in medical practice, women remain underrepresented in many specialties. This study sought to examine the current state of gender equality among recently graduated doctors in multiple specialties. Deidentified demographics, standardized examination scores, and Match results were gathered for 829 graduates. Participants were selected from an allopathic medical school between 2016 and 2020. Nineteen students (2.29%) were excluded from the study. Descriptive statistics were calculated, and χ2 tests for independence were used to compare proportions between reported sex and specialty and program Match results. One-way analysis of variance was then performed to test for differences in US Medical Licensing Examination Step 1 and Step 2 scores between sexes. P < 0.05 was considered statistically significant. Of the 829 individuals studied, 44.6% were women. A significantly smaller proportion of women matched into the most competitive specialties, despite no significant difference in US Medical Licensing Examination Step 1 scores between sexes. Furthermore, there was an overall significant trend of women matching into more competitive programs for any given specialty. In this study, we found that men matched into more highly competitive specialties, whereas women matched into more competitive residency program locations. Further research is needed to determine why women matched into specific specialties at lower rates than their male peers and seek to understand how sex affects the narrative of specialty choice.

Racial and Ethnic Disparities in Situational Judgment Testing Among Applicants to an Anesthesiology Residency Program.

Background The Computer-Based Assessment for Sampling Personal Characteristics (CASPer) is a situational judgment test (SJT) that assesses noncognitive skills like professionalism, communication, and empathy. There are no reports of the effects of race/ethnicity and sex on CASPer scores among residency applicants. Objective We examined the effects of race/ethnicity, sex, and United States vs international medical school attendance on CASPer performance. Methods Our anesthesiology residency program required all applicants for the 2021-2022 Match cycle to complete an online video and text-based SJT (CASPer). We compared these results, reported as z-scores, with self-identified race/ethnicity, sex, United States vs international medical school attendance, and United States Medical Licensing Examination (USMLE) Step 1 scores. Results Of the 1245 applicants who completed CASPer, 783 identified as male. The racial/ethnic distribution was 512 White, 412 Asian, 106 Black, 126 Hispanic, and 89 Other/No Answer. CASPer z-scores did not differ by sex. White candidates scored higher than Black (0.18 vs -0.57, P<.001) and Hispanic (0.18 vs -0.52, P<.001) candidates. Applicants attending US medical schools scored higher than those attending international medical schools (z-scores: 0.15 vs -0.68, P<.001). There was no correlation between CASPer z-scores and USMLE Step 1 scores. Conclusions Our results suggest that CASPer scores favor White applicants over Black and Hispanic ones and applicants attending US medical schools over those attending international medical schools.

Program Directors' Perspectives on Residency Applications in the Post‒United States Medical Licensing Examination Step 1 Era: A Case for Standardized Letters of Recommendation?

The United States Medical Licensing Examination (USMLE) Step 1 test evolved into a key metric utilized by program directors (PDs) in assessing candidates for residency. The transition to a USMLE Step 1 binary pass/fail scoring system has resulted in a loss of an important objective assessment. With national movements toward pass/fail systems for clerkship grading and trends toward abandonment of class ranking, assessing residency applications has become increasingly challenging. The Society of Academic Urologists convened a task force to, in part, assess the perspectives of urology PDs regarding the importance of various aspects of a residency application for predicting clinical performance. An anonymous survey was disseminated to all urology PDs in the US. Perspectives on 11 potential application predictors of clinical performance and demographics were recorded. Descriptive statistics characterized PD responses. Friedman test and pairwise Wilcoxon tests were used to evaluate the relative ranks assigned to application elements by PDs. There was a 60.5% response rate (89/147). Letters of recommendation (LORs) were ranked as the most important predictor, with a mean rank of 2.39, median of 2 (IQR 1-3). Clerkship grades and USMLE Step 1 were comparable and ranked second. Medical school reputation ranked the lowest. There was significant subjective heterogeneity among categories; however, this was less so for LORs, which predominated as the most important factor among application elements (P < .001). To our knowledge, this is the largest sample size assessing PD perspectives on application factors that predict clinical performance. The second (clerkship grades) and third (USLME Step 1) most important factors moving toward binary pass/fail systems create an opportunity for actionable change to improve assessment objectivity. Our data demonstrate LORs to be the most important factor of residency applications, making a compelling argument for moving toward a standardized LOR to maximize this tool, mitigate bias, and improve interreviewer reliability.

A Comparison Between GPT-3.5, GPT-4, and GPT-4V: Can the Large Language Model (ChatGPT) Pass the Japanese Board of Orthopaedic Surgery Examination?

Introduction Recently, large-scale language models, such as ChatGPT (OpenAI, San Francisco, CA), have evolved. Thesemodels are designed to think and act like humans and possess a broad range of specialized knowledge. GPT-3.5 was reported to be at a level of passing the United States Medical Licensing Examination. Its capabilities continue to evolve, and in October 2023, GPT-4V became available as a model capable of image recognition. Therefore, it is important to know the current performance of these models because they will be soon incorporated into medical practice. We aimed to evaluate the performance of ChatGPT in the field of orthopedic surgery. Methods We used three years' worth of Japanese Board of Orthopaedic Surgery Examinations (JBOSE) conducted in 2021, 2022, and 2023. Questions and their multiple-choice answers were used in their original Japanese form, as was the official examination rubric. We inputted these questions into three versions of ChatGPT: GPT-3.5, GPT-4, and GPT-4V. For image-based questions, we inputted only textual statements for GPT-3.5 and GPT-4, and both image and textual statements for GPT-4V. As the minimum scoring rate acquired to pass is not officially disclosed, it was calculated using publicly available data. Results The estimated minimum scoring rate acquired to pass was calculated as 50.1% (43.7-53.8%). For GPT-4, even when answering all questions, including the image-based ones, the percentage of correct answers was 59% (55-61%) and GPT-4 was able to achieve the passing line. When excluding image-based questions, the score reached 67% (63-73%). For GPT-3.5, the percentage was limited to 30% (28-32%), and this version could not pass the examination. There was a significant difference in the performance between GPT-4 and GPT-3.5 (p < 0.001). For image-based questions, the percentage of correct answers was 25% in GPT-3.5, 38% in GPT-4, and 38% in GPT-4V. There was no significant difference in the performance for image-based questions between GPT-4 and GPT-4V. Conclusions ChatGPT had enough performance to pass the orthopedic specialist examination. After adding further training data such as images, ChatGPT is expected to be applied to the orthopedics field.

The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination.

Large language models (LLM), such as Chat Generative Pre-trained Transformer (ChatGPT) and Bard utilise deep learning algorithms that have been trained on a massive data set of text and code to generate human-like responses. Several studies have demonstrated satisfactory performance on postgraduate examinations, including the United States Medical Licensing Examination. We aimed to evaluate artificial intelligence performance in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination. The MRCS mock examination from Pastest, a commonly used question bank for examinees, was used to assess the performance of three LLMs: GPT-3.5, GPT 4.0 and Bard. Three hundred mock questions were input into the three LLMs, and the responses provided by the LLMs were recorded and analysed. The pass mark was set at 70%. The overall accuracies for GPT-3.5, GPT 4.0 and Bard were 67.33%, 71.67% and 65.67%, respectively (p = 0.27). The performances of GPT-3.5, GPT 4.0 and Bard in Applied Basic Sciences were 68.89%, 72.78% and 63.33% (p = 0.15), respectively. Furthermore, the three LLMs obtained correct answers in 65.00%, 70.00% and 69.17% of the Principles of Surgery in General questions (p = 0.67). There were no differences in performance in the overall and subcategories among the three LLMs. Our findings demonstrated satisfactory performance for all three LLMs in the MRCS Part A examination, with GPT 4.0 the only LLM that achieved the pass mark set.

Evaluating Urology Program Directors' Perception on Resident Application Parameters Following the Transition of USMLE Step 1 to Pass/Fail

ObjectiveTo investigate how the shift of the United States Medical Licensing Examination (USMLE) Step 1 to a Pass/Fail (P/F) scoring system impacts the perceptions of Urology Program Directors (PDs) on evaluating urology residency applicants. Methods and MaterialsA cross-sectional survey was sent to 117 PDs, including questions about program characteristics, perceptions of shelf scores and medical school rank post-transition, beliefs about the predictive value of Step 1 and Step 2 Clinical Knowledge (CK) scores for board success and residency performance, and changes in applicant parameter ranking. Results45 PDs (38% response rate) participated. Notably, 49% favored releasing quantitative clerkship grades, and 71% valued medical school rank more. Opinions on Step 1 scores' correlation with board success were split (49% agreed), and 44% endorsed Step 2 CK scores' connection to board performance. As predictors of good residents, only 9% and 22% considered Step 1 and Step 2 CK scores, respectively, indicative. Clerkship grades and Urology rotation recommendation letters-maintained significance, while research experience gained importance. Step 2 CK scores' importance rose but didn’t match Step 1 scores' previous significance. ConclusionThe transition to Pass/Fail (P/F) for USMLE Step 1 adds intricacies to urology residency selection, exposing Program Directors' uncertainties regarding clerkship grades and the relevance of medical school rank. This research underscores the dynamic nature of urology residency admissions, emphasizing the increasing importance of research in evaluating applicants and a diminishing emphasis on volunteering and leadership.

Performance of Progressive Generations of GPT on an Exam Designed for Certifying Physicians as Certified Clinical Densitometrists

Background: Artificial intelligence (AI) large language models (LLMs) such as ChatGPT have demonstrated the ability to pass standardized exams. These models are not trained for a specific task, but instead trained to predict sequences of text from large corpora of documents sourced from the internet. It has been shown that even models trained on this general task can pass exams in a variety of domain-specific fields, including the United States Medical Licensing Examination. We asked if large language models would perform as well on a much narrower subdomain tests designed for medical specialists. Furthermore, we wanted to better understand how progressive generations of GPT (generative pre-trained transformer) models may be evolving in the completeness and sophistication of their responses even while generational training remains general. In this study, we evaluated the performance of two versions of GPT (GPT 3 and 4) on their ability to pass the certification exam given to physicians to work as osteoporosis specialists and become a certified clinical densitometrists. The CCD exam has a possible score range of 150 to 400. To pass, you need a score of 300.Methods: A 100-question multiple-choice practice exam was obtained from a 3rd party exam preparation website that mimics the accredited certification tests given by the ISCD (International Society for Clinical Densitometry). The exam was administered to two versions of GPT, the free version (GPT Playground) and ChatGPT+, which are based on GPT-3 and GPT-4, respectively (OpenAI, San Francisco, CA). The systems were prompted with the exam questions verbatim. If the response was purely textual and did not specify which of the multiple-choice answers to select, the authors matched the text to the closest answer. Each exam was graded and an estimated ISCD score was provided from the exam website. In addition, each response was evaluated by a rheumatologist CCD and ranked for accuracy using a 5-level scale. The two GPT versions were compared in terms of response accuracy and length.Results: The average response length was 11.6 ±19 words for GPT-3 and 50.0±43.6 words for GPT-4. GPT-3 answered 62 questions correctly resulting in a failing ISCD score of 289. However, GPT-4 answered 82 questions correctly with a passing score of 342. GPT-3 scored highest on the “Overview of Low Bone Mass and Osteoporosis” category (72 % correct) while GPT-4 scored well above 80 % accuracy on all categories except “Imaging Technology in Bone Health” (65 % correct). Regarding subjective accuracy, GPT-3 answered 23 questions with nonsensical or totally wrong responses while GPT-4 had no responses in that category.Conclusion: If this had been an actual certification exam, GPT-4 would now have a CCD suffix to its name even after being trained using general internet knowledge. Clearly, more goes into physician training than can be captured in this exam. However, GPT algorithms may prove to be valuable physician aids in the diagnoses and monitoring of osteoporosis and other diseases.

Update on the Selection Criteria of Plastic Surgery Residents: A Survey of Program Directors and Associate Program Directors.

Plastic and reconstructive surgery (PRS) is recognized as a highly competitive specialty. Since the first assessment of resident selection criteria in 2007, PRS residency programs have adopted holistic review processes and adapted to changes such as a decline in medical schools participating in the Alpha Omega Alpha Honor Medical Society as well as the recent transition to pass/fail grading for the United States Medical Licensing Examination (USMLE) step 1 examination (Schultz et al. Plast Reconstr Surg Glob Open . 2020;8:e2892; Tadisina et al. Plast Reconstr Surg . 2017;139:330e-331e). This study was devised to evaluate current PRS residency criteria in light of these changes. An anonymous, 12-item, electronic survey was generated and distributed using Alchemer. An email was sent to 171 program directors (PDs) and associate program directors (APDs) of PRS residency programs. Survey questions were developed to collect data regarding respondent demographics and their desired criteria when assessing residency applicants. Complete responses were collected and analyzed with summary statistics and multivariate logistic regression using RStudio (version 1.3.109). In total, 44 (25.7% response rate) of the 171 PDs and APDs completed the survey. Of the 16 programs (36.4%) with a USMLE cutoff score, 7 (43.8%) reported a range of 230 to 239 and 6 (37.5%) reported a range of 240 to 249. Without a score for step 1, the majority (48.8%) of respondents believe that step 2 scores will replace step 1 scores in terms of assessment criteria, and the content of recommendation letters was selected as the criterion with the greatest increase in weight (66.7%). In addition, 27.3% of programs require a step 2 score at the time of interview. The top 3 academic criteria in order of decreasing importance were the content of recommendation letters, clinical grades, and letter writers, whereas the top 3 nonacademic criteria were subinternship performance, maturity, and interview performance. Plastic and reconstructive surgery remains a highly competitive specialty for residency applicants. Our findings suggest that Alpha Omega Alpha membership remains diminished in importance, whereas USMLE cutoff scores have increased. With recent changes in the step 1 grading system, PDs and APDs will rely more heavily on step 2 scores and the content of recommendation letters.