United States Medical Licensing Examination Research Articles

ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis.

Recent studies, including those by the National Board of Medical Examiners, have highlighted the remarkable capabilities of recent large language models (LLMs) such as ChatGPT in passing the United States Medical Licensing Examination (USMLE). However, there is a gap in detailed analysis of LLM performance in specific medical content areas, thus limiting an assessment of their potential utility in medical education. This study aimed to assess and compare the accuracy of successive ChatGPT versions (GPT-3.5, GPT-4, and GPT-4 Omni) in USMLE disciplines, clinical clerkships, and the clinical skills of diagnostics and management. This study used 750 clinical vignette-based multiple-choice questions to characterize the performance of successive ChatGPT versions (ChatGPT 3.5 [GPT-3.5], ChatGPT 4 [GPT-4], and ChatGPT 4 Omni [GPT-4o]) across USMLE disciplines, clinical clerkships, and in clinical skills (diagnostics and management). Accuracy was assessed using a standardized protocol, with statistical analyses conducted to compare the models' performances. GPT-4o achieved the highest accuracy across 750 multiple-choice questions at 90.4%, outperforming GPT-4 and GPT-3.5, which scored 81.1% and 60.0%, respectively. GPT-4o's highest performances were in social sciences (95.5%), behavioral and neuroscience (94.2%), and pharmacology (93.2%). In clinical skills, GPT-4o's diagnostic accuracy was 92.7% and management accuracy was 88.8%, significantly higher than its predecessors. Notably, both GPT-4o and GPT-4 significantly outperformed the medical student average accuracy of 59.3% (95% CI 58.3-60.3). GPT-4o's performance in USMLE disciplines, clinical clerkships, and clinical skills indicates substantial improvements over its predecessors, suggesting significant potential for the use of this technology as an educational aid for medical students. These findings underscore the need for careful consideration when integrating LLMs into medical education, emphasizing the importance of structured curricula to guide their appropriate use and the need for ongoing critical analyses to ensure their reliability and effectiveness.

Performance evaluation of ChatGPT 4.0 on cardiovascular questions from the medical knowledge self-assessment program

Abstract Background Medical trainees are increasingly using online chat-based artificial intelligence (AI) platforms as supplementary resources for board exam preparation and clinical decision support. Prior studies have evaluated the performance of AI chatbots like ChatGPT on various general standardized tests such as the United States Medical Licensing Examination (USMLE), but little is known about their performance on subspecialty-focused exam questions, particularly related to clinical management and treatment. Objective This study aims to evaluate the performance of ChatGPT version 4.0 on the cardiovascular questions from the Medical Knowledge Self-Assessment Program (MKSAP) 19, a widely used resource for board exam preparation in the United States. Methods We submitted all cardiovascular questions from MKSAP 19 to ChatGPT 4.0, covering a broad range of cardiology topics in a multiple-choice format. Performance was gauged against both the official MKSAP answer key and average trainee scores obtained from the MKSAP website. Out of 129 questions, 4 were invalidated due to post-publication data, and 18 were excluded due to reliance on visual aids, leaving 107 questions for analysis. Results ChatGPT 4.0 correctly answered 93 out of 107 questions, reflecting an 87% accuracy rate, compared to a 60% accuracy rate averaged among all human users for the same questions (p<0.0001). ChatGPT accuracy rates for each question category (e.g., heart failure, electrophysiology, etc.) are provided in the figure. On the 14 questions that ChatGPT answered incorrectly, human users averaged a 47% accuracy rate. All 14 incorrectly answered questions were related to clinical management and treatment (as opposed to diagnosis or epidemiology), and the majority (10/14) involved choosing treatment medications or choosing between medications versus interventions. Conclusion ChatGPT 4.0 surpassed the typical 70% accuracy threshold required for passing the internal medicine board certification but performed less well in answering questions about clinical management and treatment. This performance suggests that ChatGPT 4.0 could be an effective adjunct tool for cardiology education, but raises some concerns about its use in clinical decision support for trainees.

Opportunities to Improve Communication With Residency Applicants: Cross-Sectional Study of Obstetrics and Gynecology Residency Program Websites.

As part of the residency application process in the United States, many medical specialties now offer applicants the opportunity to send program signals that indicate high interest to a limited number of residency programs. To determine which residency programs to apply to, and which programs to send signals to, applicants need accurate information to determine which programs align with their future training goals. Most applicants use a program's website to review program characteristics and criteria, so describing the current state of residency program websites can inform programs of best practices. This study aims to characterize information available on obstetrics and gynecology residency program websites and to determine whether there are differences in information available between different types of residency programs. This was a cross-sectional observational study of all US obstetrics and gynecology residency program website content. The authorship group identified factors that would be useful for residency applicants around program demographics and learner trajectories; application criteria including standardized testing metrics, residency statistics, and benefits; and diversity, equity, and inclusion mission statements and values. Two authors examined all available websites from November 2011 through March 2022. Data analysis consisted of descriptive statistics and one-way ANOVA, with P<.05 considered significant. Among 290 programs, 283 (97.6%) had websites; 238 (82.1%) listed medical schools of current residents; 158 (54.5%) described residency alumni trajectories; 107 (36.9%) included guidance related to the preferred United States Medical Licensing Examination Step 1 scores; 53 (18.3%) included guidance related to the Comprehensive Osteopathic Medical Licensing Examination Level 1 scores; 185 (63.8%) included international applicant guidance; 132 (45.5%) included a program-specific mission statement; 84 (29%) included a diversity, equity, and inclusion statement; and 167 (57.6%) included program-specific media or links to program social media on their websites. University-based programs were more likely to include a variety of information compared to community-based university-affiliated and community-based programs, including medical schools of current residents (113/123, 91.9%, university-based; 85/111, 76.6%, community-based university-affiliated; 40/56, 71.4%, community-based; P<.001); alumni trajectories (90/123, 73.2%, university-based; 51/111, 45.9%, community-based university-affiliated; 17/56, 30.4%, community-based; P<.001); the United States Medical Licensing Examination Step 1 score guidance (58/123, 47.2%, university-based; 36/111, 32.4%, community-based university-affiliated; 13/56, 23.2%, community-based; P=.004); and diversity, equity, and inclusion statements (57/123, 46.3%, university-based; 19/111, 17.1%, community-based university-affiliated; 8/56, 14.3%, community-based; P<.001). There are opportunities to improve the quantity and quality of data on residency websites. From this work, we propose best practices for what information should be included on residency websites that will enable applicants to make informed decisions.

The Effect of Implicit Bias on the OB/GYN Residency Application Process.

Background Medical students from racially minoritized backgrounds receive negative evaluations more frequently, possibly from implicit bias. Objective To determine if a preference for White applicants over Asian and Black applicants exists among resident and attending reviewers of obstetrics and gynecology (OB/GYN) residency applications. Methods In 2021, we sampled academic OB/GYN physicians from 6 academic institutions to randomly receive questionnaire form A or B, each containing 4 contrived Electronic Residency Application Service profiles stratified by United States Medical Licensing Examination scores; institutions attended; and strength of recommendation letters into top-, mid-, and low-tiers. Form A applicants were White top-tier, White mid-tier without resilience, Asian mid-tier with resilience, and White low-tier. Form B applicants were Black top-tier, Asian mid-tier without resilience, White mid-tier with resilience, and White low-tier applicants. Both questionnaires' profiles were identical except for applicants' names and races. The primary outcome was participants' rankings of applicants from most to least desirable. Baseline characteristics of participants were compared with chi-square tests. Rankings were compared using generalized estimating equations to calculate the odds ratio (significance P<.05). Results One hundred and ninety-one were invited to participate; 109 participants started the questionnaire, and 103 ranked applicants (response rate 54%). Fifty-three participants completed form A, and 56 completed form B. No significant differences were found in participants' demographics. Participants frequently ranked White applicants significantly more highly than Black applicants (-3.3, 95% CI -3.5, -3.1) and Asian applicants (-3.3, 95% CI -3.5, -3.1). Conclusions Despite identical qualifications, a stronger preference for White applicants over Black or Asian applicants was found.

Exploring the 6-year Trend in Dedicated Research Years among Integrated Plastic and Reconstructive Surgery Residents.

The plastic and reconstructive surgery (PRS) match has become more competitive with an emphasis on research productivity. With the transition of the US Medical Licensing Examination step 1 examination to pass-fail grading, alternative evaluation metrics for residency applications are needed. Our study provides a landscape of the incidence of research years amongst integrated PRS residents and the potential impacts of gender and race. A retrospective study was conducted for all integrated PRS residents who matched between 2017 and 2022, using publicly available sources including residency websites, PubMed, Doximity, and social media. Demographics, educational background, and participation in a dedicated PRS research year were recorded. Among 993 residents across 85 accredited integrated PRS programs, 15.7% (156 of 993) took a research year, with a significant increase over 6 years (R 2 = 0.73, P = 0.019). The majority were men at 51.9%, and White at 53.5%, with underrepresented in medicine residents comprising 13.5%. A total of 73.1% came from a medical school with a home program, and 52.6% matched into a top 20 residency program. No significant difference was seen in the racial/ethnic or gender representation of residents who took a research year compared with all PRS applicants and active PRS residents. Research years before applying to residency are becoming more common among integrated PRS residents. Research years are especially common with students who do not have a home program and may help them gain skills and access to mentorship that improves the likelihood of matching.

Performance of OpenAI’s GPT-4 in a mock MRCS Part A Examination

Introduction: OpenAI’s latest iteration of a Large Language Model (LLM); GPT-4 (Generative Pre-trained Transformer 4) has demonstrated its proficiency against various professional examination standards like the USMLE (United States Medical Licensing Examination), FRCS (Fellowship of the Royal Colleges of Surgeons) and the United States Bar. However, GPT-4’s capability with the MRCS (Membership of the Royal College of Surgeons) Part A has not yet been investigated. Methodology: A representative MRCS Part A examination that was prepared and provided by “TeachMeSurgery” based on the MRCS Intercollegiate Curriculum was used to assess GPT-4's performance. Each question was processed via the web-based interface of ChatGPT (Chat Generative Pre-trained Transformer) Plus. Results: GPT-4 scored 87.2% on Applied Basic Sciences (157/180) and 86.7% on Principles of Surgery in General (104/120 questions), achieving an overall score of 261/300 (87%), which is above the typical passing threshold. GPT-4 scored 100% in four out of the eleven predefined curriculum areas, which included: Pharmacology, Microbiology, Data Interpretation and Audit, and The Surgical Care of Children. GPT-4’s weakest performance was in the Medico-Legal Aspects of Surgical Practice, in which it scored 33.3%. Conclusion: GPT-4 successfully passed the mock MRCS Part A without any specialised preparatory training, however further research could look at integrating the GPT-4 model to enhance a trainee surgeon’s learning and use it as an effective tool to deliver high quality &amp; efficient patient care.

Performance of ChatGPT in the In-Training Examination for Anesthesiology and Pain Medicine Residents in South Korea: Observational Study.

ChatGPT has been tested in health care, including the US Medical Licensing Examination and specialty exams, showing near-passing results. Its performance in the field of anesthesiology has been assessed using English board examination questions; however, its effectiveness in Korea remains unexplored. This study investigated the problem-solving performance of ChatGPT in the fields of anesthesiology and pain medicine in the Korean language context, highlighted advancements in artificial intelligence (AI), and explored its potential applications in medical education. We investigated the performance (number of correct answers/number of questions) of GPT-4, GPT-3.5, and CLOVA X in the fields of anesthesiology and pain medicine, using in-training examinations that have been administered to Korean anesthesiology residents over the past 5 years, with an annual composition of 100 questions. Questions containing images, diagrams, or photographs were excluded from the analysis. Furthermore, to assess the performance differences of the GPT across different languages, we conducted a comparative analysis of the GPT-4's problem-solving proficiency using both the original Korean texts and their English translations. A total of 398 questions were analyzed. GPT-4 (67.8%) demonstrated a significantly better overall performance than GPT-3.5 (37.2%) and CLOVA-X (36.7%). However, GPT-3.5 and CLOVA X did not show significant differences in their overall performance. Additionally, the GPT-4 showed superior performance on questions translated into English, indicating a language processing discrepancy (English: 75.4% vs Korean: 67.8%; difference 7.5%; 95% CI 3.1%-11.9%; P=.001). This study underscores the potential of AI tools, such as ChatGPT, in medical education and practice but emphasizes the need for cautious application and further refinement, especially in non-English medical contexts. The findings suggest that although AI advancements are promising, they require careful evaluation and development to ensure acceptable performance across diverse linguistic and professional settings.

Imbalanced specialty representation of USMLE and NBME test writers

PurposeThe United States Medical Licensing Examination (USMLE) is an examination series required for allopathic physician licensure in the United States (US). USMLE content is created and maintained by the National Board of Medical Examinations (NBME). The specialty composition of the USMLE and NBME taskforce members involved in the creation of examination content is currently unknown.MethodsUsing the 2021 USMLE and 2021 NBME Committees and Task Forces documents, we determined each member’s board-certified primary specialty and involvement in test material development committees who we dubbed “test writers”. Total active physicians by primary specialty were recorded from the 2020 Physician Specialty Data Report published by the Association of American Medical Colleges (AAMC). Descriptive statistics and chi-square analysis were used to analyze the cohorts.ResultsThe USMLE and NBME test writer primary specialty composition was found to be significantly different compared to the US active physician population (USMLE χ2[32] = 172, p < .001 and NBME χ2[32] = 200, p < .001). Only nineteen specialties were represented within USMLE test writers, with three specialties being proportionally represented. Two specialties were represented within NBME test writers. Obstetrics and Gynecology physicians were proportionally represented in USMLE but not within NBME test writers. Internal Medicine (IM) accounts for the largest percentage of all USMLE test writers (60/197, 30%) with an excess representation of 31 individuals.ConclusionsThere is an imbalance in the specialty representation of USMLE and NBME test writers compared to the US active physician population. These findings may have implications for the unbiased and accurate portrayal of topics in such national examinations; thus, future investigation is warranted.

Aptitude for Research Among Medical Students in India: Identifying the Challenges.

Introduction Understanding the reasons behind the undervaluation of research among undergraduate medical students in India is crucial for advancing medical knowledge. This cross-sectional study aims to investigate the knowledge, interest, and barriers faced by 606 Indian medical students through an online questionnaire distributed via social media platforms. Aims To identify the knowledge of research and the challenges faced by medical students to pursue research in their undergraduate education. Methods and material This was a prospective observational study carried out over a period of one month in October 2022. Study approval was taken from the Genebandhu Independent Ethics Committee (Reference Number- ECG004/2022). The responses obtained from the questionnaire were recorded in Google Sheets and transferred to Microsoft Excel (Redmond, WA, USA). Results Among the participants, 46.53% were male and 53.47% were female, primarily from the first (11.39%) and second (9.90%) years of their medical education. Notably, the majority had undertaken United States Medical Licensing Examination (USMLE) (54.95%), followed by National Eligibility cum Entrance Test (Postgraduate) (NEET-PG) (34.65%) and Professional and Linguistic Assessments Board (PLAB) (7.92%). The study revealed that 78.22% considered research pivotal to their academic trajectory, with a mean age of 23.14 years (SD=2.82) and an average research knowledge score of 2.75 (SD=1.31). The type of postgraduate exam significantly influenced enthusiasm for research activities (p<0.001), with higher enthusiasm among those preparing for exams abroad. Nonetheless, the type of medical college did not significantly affect interest in research activities (p=0.4879). Conclusion Addressing the undervaluation of research among undergraduate medical students in India is imperative. The curriculum should integrate robust support mechanisms to nurture research skills, emphasizing its importance for future medical practitioners. This could involve seminars, courses, and interactive sessions aimed at fostering research acumen among students.

Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

Large language models (LLMs) have the intrinsic potential to acquire medical knowledge. Several studies assessing LLMs on medical examinations have been published. However, there is no reported evidence on tests related to robot-assisted surgery. The aims of this study were to perform the first systematic review of LLMs on medical examinations and to establish whether ChatGPT, GPT-4, and Bard can pass the Fundamentals of Robotic Surgery (FRS) didactic test. A literature search was performed on PubMed, Web of Science, Scopus, and arXiv following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach. A total of 45 studies were analyzed. GPT-4 passed several national qualifying examinations with questions in English, Chinese, and Japanese using zero-shot and few-shot learning. Med-PaLM 2 obtained similar scores on the United States Medical Licensing Examination with more refined prompt engineering techniques. Five different 2023 releases of ChatGPT, one of GPT-4, and one of Bard were tested on FRS. Seven attempts were performed with each release. The pass score was 79.5%. ChatGPT achieved a mean score of 64.6%, 65.6%, 75.0%, 78.9%, and 72.7% respectively from the first to the fifth tested release on FRS vs 91.5% of GPT-4 and 79.5% of Bard. GPT-4 outperformed ChatGPT and Bard in all corresponding attempts with a statistically significant difference for ChatGPT (p < 0.001), but not Bard (p = 0.002). Our findings agree with other studies included in this systematic review. We highlighted the potential and challenges of LLMs to transform the education of healthcare professionals in the different stages of learning, by assisting teachers in the preparation of teaching contents, and trainees in the acquisition of knowledge, up to becoming an assessment framework of leaners.

Many Paths to the Summit: Survey of Step 1 Study Methods with Pass/Fail Scoring.

After the United States Medical Licensing Examination (USMLE) shifted the Step 1 licensure exam to pass/fail, there have been limited studies to analyze changes in student study strategies. Surveys were distributed to third- and fourth-year medical students at the University of Michigan Medical School (UMMS). The response rate was 66%. The largest proportion (18.81%) of students chose 8weeks of study time. 40.59% of students increased the length of their study period. To determine time allocated for dedicated study, 37.26% of respondents consulted near-peers who had already taken Step 1. Students also considered prior experiences with standardized tests (15.57%), personal reasons (14.62%), and conversations with school advisors (13.21%). 44.55% of students studied for 9-11h a day, and 42.57% studied for 5-8h a day. 52.69% of students scored between 70 and 80% on their final practice NBME test before their Step 1 exam. One hundred percent of respondents passed the exam. Exam non-extenders achieved higher end average practice test scores with shorter study periods. No differences in Step 1 study time or intensity were found when comparing students by intended specialty competitiveness. Our results demonstrated patterns in study strategies for the new pass/fail Step 1 exam that may prove useful for curriculum design and schedule plan for future cohorts. The online version contains supplementary material available at 10.1007/s40670-024-02072-2.

An Update on the Independent Plastic Surgery Match (2019-2022): Trends, Predictors, and Program Leaders' Perspectives.

A previous study by the authors noted a decline in independent plastic surgery residency programs and rising applicant participation. This study provides updates on match trends and influential predictors, and gathers program leaders' views on the future of the independent track. Match data (2019-2022) were obtained from the San Francisco match after American Council of Educators in Plastic Surgery approval. Variables influencing match success were analyzed, and program leaders were surveyed about desirable applicant traits and program trajectories. From 2019 to 2022, 243 of 428 applicants matched. Programs and positions declined by 10% and 9.5%, respectively. Applicants rose to 42.3%, but match rates fell from 82% to 56%. Osteopathic graduates doubled, whereas international graduates increased to 53.8%. Successful matches were associated with US allopathic medical school graduates, university-affiliated general surgery residencies, eight or more interviews, United States Medical Licensing Examination scores greater than 230, and high post graduate year (PGY)1-3 American Board of Surgery In-service Training Examination scores (PGY1-64.7%, PGY2-61.2%, PGY3-60.7%; P < 0.05). Of surveyed programs, 55.6% aimed to continue running the independent track in the next year. Conversely, 7.4% planned to discontinue in the next year, 22.2% within 2-5 years, 7.4% within the next decade, and 7.4% were unsure. Although support for the independent plastic surgery track remains, program participation diminishes as applicant interest increases, intensifying match challenges. Increasing number of interviews improves match potential. Program leaders display varied commitments, with looming plans for some programs to discontinue offering this track. Applicant evaluation pivots on strong recommendations, research, and test scores.

Factors Related to Clinical Performance in Spine Surgery Fellowship: Can We Predict Success.

The factors most important in the spine fellowship match may not ultimately correlate with quality of performance during fellowship. This study examined the spine fellow applicant metrics correlated with high application rank compared with the metrics associated with the strongest clinical performance during fellowship. Spine fellow applications at three academic institutions were retrieved from the San Francisco Match database (first available to 2021) and deidentified for application review. Application metrics pertaining to research, academics, education, extracurriculars, leadership, examinations, career interests, and letter of recommendations were extracted. Attending spine surgeons involved in spine fellow selection at their institutions were sent a survey to rank (1) fellow applicants based on their perceived candidacy and (2) the strength of performance of their previous fellows. Pearson correlation assessed the associations of application metrics with theoretical fellow rank and actual performance. A total of 37 spine fellow applications were included (Institution A: 15, Institution B: 12, Institution C: 10), rated by 14 spine surgeons (Institution A: 6, Institution B: 4, Institution C: 4). Theoretical fellow rank demonstrated a moderate positive association with overall research, residency program rank, recommendation writer H-index, US Medical Licensing Examination (USMLE) scores, and journal reviewer positions. Actual fellow performance demonstrated a moderate positive association with residency program rank, recommendation writer H-index, USMLE scores, and journal reviewer positions. Linear regressions identified journal reviewer positions (ß = 1.73, P = 0.002), Step 1 (ß = 0.09, P = 0.010) and Step 3 (ß = 0.10, P = 0.002) scores, recommendation writer H-index (ß = 0.06, P = 0.029, and ß = 0.07, P = 0.006), and overall research (ß = 0.01, P = 0.005) as predictors of theoretical rank. Recommendation writer H-index (ß = 0.21, P = 0.030) and Alpha Omega Alpha achievement (ß = 6.88, P = 0.021) predicted actual performance. Residency program reputation, USMLE scores, and a recommendation from an established spine surgeon were important in application review and performance during fellowship. Research productivity, although important during application review, was not predictive of fellow performance. III. Cohort Study.

Medical students' perspective on the United States Medical Licensing Examination (USMLE) Step 1 transition to Pass/Fail

Background The transition of the United States Medical Licensing Exam: Step 1 to Pass/Fail (P/F), from scored, caused uncertainty about students’ preparedness and wellbeing related to the exam. Comparison of study behavior and results, before and after the P/F transition can provide insights for the medical curriculum and student support. The aim of this study is to evaluate students’ perceptions of their efforts in Step 1 preparation, as compared to cohort performances to determine the impact of the transition of Step 1 to P/F on medical students. Methods Data from four cohorts of second-year medical students (Class of 2022–25, N = 204) were collected from their dedicated Step 1 self-study block. Student study regiments, aggregate practice test results, Step 1 pass rates and post-block self-reported surveys were analyzed qualitatively to compare student effort and outcomes for scores and P/F cohorts. Results Analysis of practice exam averages across the four student cohorts demonstrates a potentially slower and less rigorous start to Step 1 self-studying during the dedicated preparation block for the cohorts that took the P/F Step 1 exam format as compared to the previous cohorts that prepared for a scored Step 1 exam. Similarly, self-reported study regiments decreased in the median number of hours/day and number of weeks of study for the cohorts with P/F Step 1 exam. There was also a slight shift in the type of study resources used, between the two groups, with the scored group using more traditional board preparation resources. Conclusions The P/F transition of the Step 1 exam may lead to reduced student preparedness and may require adjustments in the resources and support provided by institutions.

The Correlation Between International Medical Graduates' Self-Study Habits and Their United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) Scores.

To secure a residency in the United States, medical students must pass the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 CK exams. This study examines the correlation between international medical graduates' (IMGs) self-study habits and their USMLE scores. A retrospective study was conducted with 51 anonymous third- and fourth-year IMGs from Saint James Medical School, IL, United States. Participants completed an online survey about their study habits and USMLE Step 1 and Step 2 CK scores. All participants were undergoing clinical clerkships at South Texas Health Hospitals in McAllen, TX. The highest mean Step 1 scores were 211.3 for completing ≥7,000 questions, 222.2 for 91-120 days of study, 209.2 for 76-100% time on practice questions, 229.7 for 16-19 hours/day of study, and 228.0 for 51-75% group study. The highest mean Step 2 CK scores were 241.0 for completing ≥6,000 questions, 239.8 for <30 days of study, 238.8 for 76-100% time on practice questions, 239.0 for 16-19 hours/day of study, and 237.5 for 26-50% group study. No significant relationship was found between study habits and passing Step 1 scores (p>0.05), but moderate correlations were found for completing ≥4,000 questions and 61-90 days of study. No significant relationship was found between study habits and the national average Step 2 CK score, but a strong correlation was found for 25-50% time on practice questions. While some study habits correlate with higher scores, no significant relationship was found between specific study habits and passing Step 1 or achieving the national average Step 2 CK score.

Evaluating academic productivity and match outcomes in neurosurgery residents from medical schools without a neurosurgery residency program: a 2016-2022 bibliometric analysis.

Transition of the United States Medical Licensing Examination Step 1 to a pass/fail structure has focused attention on medical student research in residency application. Previous studies have explored how various factors affect the neurosurgery match success, but none have focused on applicants from schools without a neurosurgery residency program. The authors compiled a list of neurosurgery residents matched from 2016 to 2022 from schools lacking a neurosurgery program. They gathered demographic and bibliometric data, focusing on academic productivity before residency. The distinction between the top 40 and non-top 40 programs used the 5-year institutional h-index (ih[5]-index) of departments. Between 2016 and 2022, the gross number of students entering neurosurgery from schools without a home program rose from 15 to 26 in 2021, declining to 23 in 2022. The range of matched applicants per school was 0 to 10. The median number of publications per resident increased from 2 in 2016 to 5 in 2022 (p < 0.001). The lowest and highest numbers of publications by applicants were 0 and 40, respectively, with 22.5% reporting no publications. The lowest and highest h-indices by applicants were 0 and 11, respectively, with nearly one-third (31.2%) possessing an h-index of 0. Applicants from schools lacking a neurosurgery residency program who matched into top 40 programs had a publication range of 0-11, with a higher median of 3 compared with those who did not (median 2, range 0-8). While no significant differences were found in publication numbers (p = 0.084), a difference in h-index was observed (p = 0.024) between the two groups. Publications significantly correlated with the h-index, with each additional publication increasing the h-index by 0.19 (p < 0.001, adjusted R2 = 0.3348). Median publication counts have increased in this cohort, but they do not distinguish top 40 program matches. Conversely, the h-index, correlating with publication quantity and journal impact factor, does.

Can Artificial Intelligence Support Infection Prevention and Control Consultations?

Background: Artificial intelligence (AI) tools have demonstrated success in US medical licensing examinations; however, their utility in infection prevention and control (IPC) remains unknown. Methods: The program of hospital epidemiology handles consultation calls and records each question and answer. Using 2022 data, we selected 31 frequently asked questions. We utilized four AI tools, including Chat GPT-3.5 and 4.0, Bing AI, and OpenEvidence, to generate answers. We predefined scales (Table 1) to capture responses by three reviewers, including two hospital epidemiologists and one infection preventionist. The mean score of ≥ 3 and ≥ 4 was considered acceptable in accuracy and completeness, respectively. We reported the percentage of responses with acceptable accuracy and completeness out of assessed questions for each category. Results: Among 31 questions, 16 were associated with isolation duration, 9 with healthcare personnel (HCP) exposure, 4 with cleaning contaminated rooms, and 2 with patient exposure. Regarding accuracy, most AI tools performed worse in questions about isolation duration, ranging between 75% and 93.8%. All AI tools, except OpenEvidence, had a 100% accuracy rate for HCP and patient exposure. All AI tools had a 100% accuracy rate for contaminated room handling. The highest overall acceptable accuracy rate was observed in Chat GPT-3.5. Regarding completeness, most AI tools performed worse in questions about isolation duration, ranging between 44% and 75%. All AI tools, except OpenEvidence, had a 100% completeness rate for contaminated rooms and patient exposure. The highest overall acceptable completeness rate was observed in Bing AI (Table 2). Conclusions: All AI tools provided reasonable answers to commonly asked IPC-related questions, although, there were variations among different tools used. AI could be used to supplement the infection control program, especially if resources are limited.

The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland.

In November 2022, OpenAI released a chatbot named ChatGPT, a product capable of processing natural language to create human-like conversational dialogue. It has generated a lot of interest, including from the scientific community and the medical science community. Recent publications have shown that ChatGPT can correctly answer questions from medical exams such as the United States Medical Licensing Examination and other specialty exams. To date, there have been no studies in which ChatGPT has been tested on specialty questions in the field of nephrology anywhere in the world. Using the ChatGPT-3.5 and -4.0 algorithms in this comparative cross-sectional study, we analysed 1560 single-answer questions from the national specialty exam in nephrology from 2017 to 2023 that were available in the Polish Medical Examination Center's question database along with answer keys. Of the 1556 questions posed to ChatGPT-4.0, correct answers were obtained with an accuracy of 69.84%, compared with ChatGPT-3.5 (45.70%, P=.0001) and with the top results of medical doctors (85.73%, P=.0001). Of the 13 tests, ChatGPT-4.0 exceeded the required ≥60% pass rate in 11 tests passed, and scored higher than the average of the human exam results. ChatGPT-3.5 was not spectacularly successful in nephrology exams. The ChatGPT-4.0 algorithm was able to pass most of the analysed nephrology specialty exams. New generations of ChatGPT achieve similar results to humans. The best results of humans are better than those of ChatGPT-4.0.

Orthopaedic Surgery Residency Program Websites: Assessing Information Gaps and Opportunities for Attracting Applicants.

The quality and volume of information on orthopaedic surgery residency program websites are helpful for informed decision making of prospective applicants. The purpose of this study was to evaluate the content on orthopaedic surgery residency program websites and identify areas for improvement. In November 2023, 203 orthopaedic surgery residency programs were reviewed using the Association of American Medical Colleges Residency Explorer tool. Fourteen nonfunctional websites were excluded, leaving n = 189. Websites were assessed for 11 variables: resources for assisting resident research (eg, biomechanics laboratory, research assistant, and advisors), residents' past institutional affiliations, current resident profiles (including hometown and interests), alumni data, fellowship matches, American Board of Orthopedic Surgery I/II pass rates, Alpha Omega Alpha Society affiliation, target United States Medical Licensing Examination/ Comprehensive Osteopathic Medical Learning Exam Step 2 scores, educational activities (journal club, grand rounds, didactics), evaluation methods, and mentorship. Website variables differed markedly among identified program types, which included community programs, university programs (UPs), and affiliated programs (APs). Disparities were noted in the disclosure of residents' previous institutions ( P = 0.001), the availability of alumni names ( P = 0.001), research resources ( P = 0.001), academic activities ( P = 0.007), assigned mentorship programs ( P = 0.006), and fellowship match information ( P = 0.027). Notably, community programs shared statistically significantly less information in these areas compared with UPs and APs ( P = 0.001). While there was a proportionate sharing of information between UPs and APs, UPs exhibited a slightly higher overall percentage of websites sharing more detailed information. The quality and comprehensiveness of information on orthopaedic surgery residency program websites vary markedly. Most of the orthopaedic surgery residency programs provided information on residents' past institutions and academic activities. However, very few provided information on designated mentorship programs or objective measures such as target Step 2 examination scores and ABOS pass rates. Improving the quality of information available on program websites will enhance transparency and consistency, enabling prospective applicants to make more informed decisions about where to apply and interview.

How Student and Faculty Perceptions Differ on the Stressors that Medical Students Face.

Medical education is notorious for the stress that students face as they strive to succeed both academically and clinically. This stress has been linked to declining academic performance and worsening mental health. To combat these negative outcomes, it is essential for medical school faculty and administration to address common stressors among medical students. No studies have addressed whether medical school faculty and students perceive stressors similarly, however. In this two-part study, data collected from medical students in 2021 to 2022 to identify their most significant sources of stress were used to create a survey that queries the frequency and intensity of these stressors. This survey was distributed to medical students and faculty at the same institution. The responses between students and faculty were compared and student data also were analyzed by academic year to observe changes in perception that accompany progression through the medical curriculum. The results showed that faculty overestimated the impact of certain stressors on medical students (eg, in-house examinations, US Medical Licensing Examination Steps 1 and 2 examinations, and patient interactions). In addition, preclinical students were more concerned with finding extracurricular activities, missing opportunities, and performing research compared with clinical students. This study demonstrated that although faculty anticipated most medical student stressors, there are significant gaps that still need to be addressed to better reduce and respond to the stress experienced by medical students.