Medical Licensing Examination Research Articles

A Comparison Between GPT-3.5, GPT-4, and GPT-4V: Can the Large Language Model (ChatGPT) Pass the Japanese Board of Orthopaedic Surgery Examination?

Introduction Recently, large-scale language models, such as ChatGPT (OpenAI, San Francisco, CA), have evolved. Thesemodels are designed to think and act like humans and possess a broad range of specialized knowledge. GPT-3.5 was reported to be at a level of passing the United States Medical Licensing Examination. Its capabilities continue to evolve, and in October 2023, GPT-4V became available as a model capable of image recognition. Therefore, it is important to know the current performance of these models because they will be soon incorporated into medical practice. We aimed to evaluate the performance of ChatGPT in the field of orthopedic surgery. Methods We used three years' worth of Japanese Board of Orthopaedic Surgery Examinations (JBOSE) conducted in 2021, 2022, and 2023. Questions and their multiple-choice answers were used in their original Japanese form, as was the official examination rubric. We inputted these questions into three versions of ChatGPT: GPT-3.5, GPT-4, and GPT-4V. For image-based questions, we inputted only textual statements for GPT-3.5 and GPT-4, and both image and textual statements for GPT-4V. As the minimum scoring rate acquired to pass is not officially disclosed, it was calculated using publicly available data. Results The estimated minimum scoring rate acquired to pass was calculated as 50.1% (43.7-53.8%). For GPT-4, even when answering all questions, including the image-based ones, the percentage of correct answers was 59% (55-61%) and GPT-4 was able to achieve the passing line. When excluding image-based questions, the score reached 67% (63-73%). For GPT-3.5, the percentage was limited to 30% (28-32%), and this version could not pass the examination. There was a significant difference in the performance between GPT-4 and GPT-3.5 (p < 0.001). For image-based questions, the percentage of correct answers was 25% in GPT-3.5, 38% in GPT-4, and 38% in GPT-4V. There was no significant difference in the performance for image-based questions between GPT-4 and GPT-4V. Conclusions ChatGPT had enough performance to pass the orthopedic specialist examination. After adding further training data such as images, ChatGPT is expected to be applied to the orthopedics field.

Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study.

Previous research applying large language models (LLMs) to medicine was focused on text-based information. Recently, multimodal variants of LLMs acquired the capability of recognizing images. We aim to evaluate the image recognition capability of generative pretrained transformer (GPT)-4V, a recent multimodal LLM developed by OpenAI, in the medical field by testing how visual information affects its performance to answer questions in the 117th Japanese National Medical Licensing Examination. We focused on 108 questions that had 1 or more images as part of a question and presented GPT-4V with the same questions under two conditions: (1) with both the question text and associated images and (2) with the question text only. We then compared the difference in accuracy between the 2 conditions using the exact McNemar test. Among the 108 questions with images, GPT-4V's accuracy was 68% (73/108) when presented with images and 72% (78/108) when presented without images (P=.36). For the 2 question categories, clinical and general, the accuracies with and those without images were 71% (70/98) versus 78% (76/98; P=.21) and 30% (3/10) versus 20% (2/10; P≥.99), respectively. The additional information from the images did not significantly improve the performance of GPT-4V in the Japanese National Medical Licensing Examination.

The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination.

Large language models (LLM), such as Chat Generative Pre-trained Transformer (ChatGPT) and Bard utilise deep learning algorithms that have been trained on a massive data set of text and code to generate human-like responses. Several studies have demonstrated satisfactory performance on postgraduate examinations, including the United States Medical Licensing Examination. We aimed to evaluate artificial intelligence performance in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination. The MRCS mock examination from Pastest, a commonly used question bank for examinees, was used to assess the performance of three LLMs: GPT-3.5, GPT 4.0 and Bard. Three hundred mock questions were input into the three LLMs, and the responses provided by the LLMs were recorded and analysed. The pass mark was set at 70%. The overall accuracies for GPT-3.5, GPT 4.0 and Bard were 67.33%, 71.67% and 65.67%, respectively (p = 0.27). The performances of GPT-3.5, GPT 4.0 and Bard in Applied Basic Sciences were 68.89%, 72.78% and 63.33% (p = 0.15), respectively. Furthermore, the three LLMs obtained correct answers in 65.00%, 70.00% and 69.17% of the Principles of Surgery in General questions (p = 0.67). There were no differences in performance in the overall and subcategories among the three LLMs. Our findings demonstrated satisfactory performance for all three LLMs in the MRCS Part A examination, with GPT 4.0 the only LLM that achieved the pass mark set.

Evaluating Urology Program Directors' Perception on Resident Application Parameters Following the Transition of USMLE Step 1 to Pass/Fail

ObjectiveTo investigate how the shift of the United States Medical Licensing Examination (USMLE) Step 1 to a Pass/Fail (P/F) scoring system impacts the perceptions of Urology Program Directors (PDs) on evaluating urology residency applicants. Methods and MaterialsA cross-sectional survey was sent to 117 PDs, including questions about program characteristics, perceptions of shelf scores and medical school rank post-transition, beliefs about the predictive value of Step 1 and Step 2 Clinical Knowledge (CK) scores for board success and residency performance, and changes in applicant parameter ranking. Results45 PDs (38% response rate) participated. Notably, 49% favored releasing quantitative clerkship grades, and 71% valued medical school rank more. Opinions on Step 1 scores' correlation with board success were split (49% agreed), and 44% endorsed Step 2 CK scores' connection to board performance. As predictors of good residents, only 9% and 22% considered Step 1 and Step 2 CK scores, respectively, indicative. Clerkship grades and Urology rotation recommendation letters-maintained significance, while research experience gained importance. Step 2 CK scores' importance rose but didn’t match Step 1 scores' previous significance. ConclusionThe transition to Pass/Fail (P/F) for USMLE Step 1 adds intricacies to urology residency selection, exposing Program Directors' uncertainties regarding clerkship grades and the relevance of medical school rank. This research underscores the dynamic nature of urology residency admissions, emphasizing the increasing importance of research in evaluating applicants and a diminishing emphasis on volunteering and leadership.

Can large language models reason about medical questions?

Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.

Performance of Progressive Generations of GPT on an Exam Designed for Certifying Physicians as Certified Clinical Densitometrists

Background: Artificial intelligence (AI) large language models (LLMs) such as ChatGPT have demonstrated the ability to pass standardized exams. These models are not trained for a specific task, but instead trained to predict sequences of text from large corpora of documents sourced from the internet. It has been shown that even models trained on this general task can pass exams in a variety of domain-specific fields, including the United States Medical Licensing Examination. We asked if large language models would perform as well on a much narrower subdomain tests designed for medical specialists. Furthermore, we wanted to better understand how progressive generations of GPT (generative pre-trained transformer) models may be evolving in the completeness and sophistication of their responses even while generational training remains general. In this study, we evaluated the performance of two versions of GPT (GPT 3 and 4) on their ability to pass the certification exam given to physicians to work as osteoporosis specialists and become a certified clinical densitometrists. The CCD exam has a possible score range of 150 to 400. To pass, you need a score of 300.Methods: A 100-question multiple-choice practice exam was obtained from a 3rd party exam preparation website that mimics the accredited certification tests given by the ISCD (International Society for Clinical Densitometry). The exam was administered to two versions of GPT, the free version (GPT Playground) and ChatGPT+, which are based on GPT-3 and GPT-4, respectively (OpenAI, San Francisco, CA). The systems were prompted with the exam questions verbatim. If the response was purely textual and did not specify which of the multiple-choice answers to select, the authors matched the text to the closest answer. Each exam was graded and an estimated ISCD score was provided from the exam website. In addition, each response was evaluated by a rheumatologist CCD and ranked for accuracy using a 5-level scale. The two GPT versions were compared in terms of response accuracy and length.Results: The average response length was 11.6 ±19 words for GPT-3 and 50.0±43.6 words for GPT-4. GPT-3 answered 62 questions correctly resulting in a failing ISCD score of 289. However, GPT-4 answered 82 questions correctly with a passing score of 342. GPT-3 scored highest on the “Overview of Low Bone Mass and Osteoporosis” category (72 % correct) while GPT-4 scored well above 80 % accuracy on all categories except “Imaging Technology in Bone Health” (65 % correct). Regarding subjective accuracy, GPT-3 answered 23 questions with nonsensical or totally wrong responses while GPT-4 had no responses in that category.Conclusion: If this had been an actual certification exam, GPT-4 would now have a CCD suffix to its name even after being trained using general internet knowledge. Clearly, more goes into physician training than can be captured in this exam. However, GPT algorithms may prove to be valuable physician aids in the diagnoses and monitoring of osteoporosis and other diseases.

Update on the Selection Criteria of Plastic Surgery Residents: A Survey of Program Directors and Associate Program Directors.

Plastic and reconstructive surgery (PRS) is recognized as a highly competitive specialty. Since the first assessment of resident selection criteria in 2007, PRS residency programs have adopted holistic review processes and adapted to changes such as a decline in medical schools participating in the Alpha Omega Alpha Honor Medical Society as well as the recent transition to pass/fail grading for the United States Medical Licensing Examination (USMLE) step 1 examination (Schultz et al. Plast Reconstr Surg Glob Open . 2020;8:e2892; Tadisina et al. Plast Reconstr Surg . 2017;139:330e-331e). This study was devised to evaluate current PRS residency criteria in light of these changes. An anonymous, 12-item, electronic survey was generated and distributed using Alchemer. An email was sent to 171 program directors (PDs) and associate program directors (APDs) of PRS residency programs. Survey questions were developed to collect data regarding respondent demographics and their desired criteria when assessing residency applicants. Complete responses were collected and analyzed with summary statistics and multivariate logistic regression using RStudio (version 1.3.109). In total, 44 (25.7% response rate) of the 171 PDs and APDs completed the survey. Of the 16 programs (36.4%) with a USMLE cutoff score, 7 (43.8%) reported a range of 230 to 239 and 6 (37.5%) reported a range of 240 to 249. Without a score for step 1, the majority (48.8%) of respondents believe that step 2 scores will replace step 1 scores in terms of assessment criteria, and the content of recommendation letters was selected as the criterion with the greatest increase in weight (66.7%). In addition, 27.3% of programs require a step 2 score at the time of interview. The top 3 academic criteria in order of decreasing importance were the content of recommendation letters, clinical grades, and letter writers, whereas the top 3 nonacademic criteria were subinternship performance, maturity, and interview performance. Plastic and reconstructive surgery remains a highly competitive specialty for residency applicants. Our findings suggest that Alpha Omega Alpha membership remains diminished in importance, whereas USMLE cutoff scores have increased. With recent changes in the step 1 grading system, PDs and APDs will rely more heavily on step 2 scores and the content of recommendation letters.

Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses

BackgroundLarge language models like ChatGPT have revolutionized the field of natural language processing with their capability to comprehend and generate textual content, showing great potential to play a role in medical education. This study aimed to quantitatively evaluate and comprehensively analysis the performance of ChatGPT on three types of national medical examinations in China, including National Medical Licensing Examination (NMLE), National Pharmacist Licensing Examination (NPLE), and National Nurse Licensing Examination (NNLE).MethodsWe collected questions from Chinese NMLE, NPLE and NNLE from year 2017 to 2021. In NMLE and NPLE, each exam consists of 4 units, while in NNLE, each exam consists of 2 units. The questions with figures, tables or chemical structure were manually identified and excluded by clinician. We applied direct instruction strategy via multiple prompts to force ChatGPT to generate the clear answer with the capability to distinguish between single-choice and multiple-choice questions.ResultsChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations over the five years. Specifically, in the NMLE, the highest recorded accuracy was 0.5467, which was attained in both 2018 and 2021. In the NPLE, the highest accuracy was 0.5599 in 2017. In the NNLE, the most impressive result was shown in 2017, with an accuracy of 0.5897, which is also the highest accuracy in our entire evaluation. ChatGPT’s performance showed no significant difference in different units, but significant difference in different question types. ChatGPT performed well in a range of subject areas, including clinical epidemiology, human parasitology, and dermatology, as well as in various medical topics such as molecules, health management and prevention, diagnosis and screening.ConclusionsThese results indicate ChatGPT failed the NMLE, NPLE and NNLE in China, spanning from year 2017 to 2021. but show great potential of large language models in medical education. In the future high-quality medical data will be required to improve the performance.

Factors associated with academic performance among medical students at a medical school in South Korea: A retrospective cohort study.

Longitudinal research has provided systematic empirical data on the short- and long-term outcomes of admissions policies, curricular innovations, and complex decisions on students' academic progress. This study aimed to investigate the academic performance of medical students and related factors using cohort database collected from a medical school. The study participants included 134 medical students who graduated from Chonnam National University Medical School in 2022. The medical school's cohort database was used to collect data on demographics, admission, academic performance, extracurricular activities, and performance on the National Korean Medical Licensing Examination (KMLE). Participating in club activities had a significant association with medical students' academic advancement delay or leave of absence during the entire course of medical school (P = 0.007). Logistic regression analysis indicated that the nationwide clinical knowledge mock examination during the fourth year of medical school was significantly associated with passing the KMLE (adjusted odds ratio 1.12, 95% confidence interval 1.02-1.22; P = 0.014). Extracurricular school activities (a non-cognitive student attribute) and a wide range of cognitive student attributes captured from the cohort database were associated with medical students' academic performance. In conclusion, this study can reinforce a strong emphasis on the inclusion of cognitive and non-cognitive information in medical school curricula and assessments in order to improve medical education programs and future postgraduate performance.

Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match

To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM). Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit. A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P<.001). As cutoff score increases, percentage of URM applicants decreases. The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

Relationship between individual and country-level socio-economic background, USMLE step scores, and demographics of international medical graduates and residency match results

PurposeTwenty five percent of practicing physicians in the US are International Medical Graduates (IMGs) – physicians who completed their medical school training outside of the United States and Canada. There are multiple studies demonstrating higher socio-economic background is associated with medical school matriculation in the US. However, despite a substantial prevalence of IMGs in the American healthcare system, studies of the association between demographics, socio-economic background, and securing a residency position in the match are lacking.MethodsWe created a survey with questions on residency match-related data and information on personal socio-economic background. An invitation to participate in the study was sent to all IMGs that applied to the included residency programs after the conclusion of the 2022 residency match. We used multivariable logistic regression to compare survey responses to the odds of securing a residency match.ResultsThe total number of survey respondents was 744 (response rate 15.1%). We found that younger age, higher United States Medical License Examination (USMLE) scores, higher-income country of origin (including the United States), fewer match attempts, applying to fewer specialties, having parents with college degree or higher, and coming from higher-than-average or lower-than-average family income were independently associated with increased odds of matching. Gender, personal income, and visa status did not demonstrate significant associations with residency match.ConclusionsResidency match is a significant expense for IMGs, especially for those from lower-income countries. International applicants from higher socio-economic backgrounds might have advantages in securing medical residency positions in the United States when controlling for other variables.

Flipping Tradition on Its Head: A Single-Institution Study on United States Medical Licensing Examination Step Order.

Given the recent change in Step 1 grading, medical students are faced with decisions surrounding the optimal Step examination order and timing. There is a current lack of guidance outside commonly used United States Medical Licensing Examination study blogs. The aim of this study was to characterize current practices in Step examination order and understand the factors influencing student examination preference. To gather data, we used a 24-item survey and electronically distributed it through three separate email requests to medical students. Participants included medical students (MS1-4) attending a single institution, interested in a surgical residency. In total, 56 students responded to the survey (response rate of 20%). Most students (63.6%) opted to take Step 2 before Step 1. Justifications included the perceived importance of Step 2 for residency applications (33.3%), the change to pass/fail grading (27.0%), the relevance of Step 2 after rotations (27.0%), and a desire to obtain a Step 2 score earlier (7.9%). Step examination order did not lead to differences in Step 2 score (P = 0.459), time used to prepare for Step 1 (P = 0.396), or time used to prepare for Step 2 (P = 0.078). Due to the perceived importance of Step 2, most medical students at our institution who are interested in pursuing a surgical residency are reversing the traditional Step examination order. As this may represent a national trend, additional work needs to be done to provide guidance to medical students and identify the optimal step examination testing strategy.

Implementation of a dermatology skin of color educational module for medical students.

Research in dermatology education highlights the lack of skin of color (SOC) instruction for medical students, leading to concerning healthcare outcomes. Because of the already limited opportunity for students to have dedicated teaching in pathophysiology, management, and treatment of dermatologic diseases in medical school, we developed an educational module that addresses these gaps. We created a one-hour virtual lecture for medical students focused on common skin diseases tested on the United States Medical Licensing Examination with visual images across all skin types. A questionnaire was administered before and after the educational module to assess outcomes comparing disease identification in lighter (Fitzpatrick scale I-III) versus darker (Fitzpatrick scale IV-VI) skin tones and to determine medical school student attitudes. An analysis of 43 examination scores before, and after attending the educational module determined rosacea, psoriasis, and basal cell carcinoma to be conditions in SOC patients that demonstrated the most significant improvement (47.3%, 54.9%, and 30.8%, respectively). Our results also highlighted worse performance outcomes for diseases in SOC in the pre-examination questionnaire. Thus, our study indicates that a concise education module focused on disease presentations inclusive of all skin types may efficiently increase students' ability to identify diseases commonly misdiagnosed in the clinical setting.

Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan.

The remarkable performance of ChatGPT, launched in November 2022, has significantly impacted the field of natural language processing, inspiring the application of large language models as supportive tools in clinical practice and research worldwide. Although GPT-3.5 recently scored high on the United States Medical Licensing Examination, its performance on medical licensing examinations of other nations, especially non-English speaking nations, has not been sufficiently evaluated. This study assessed GPT's performance on the National Medical Licensing Examination (NMLE) in Japan and compared it with the actual minimal passing rate for this exam. In particular, the performances of both the GPT-3.5 and GPT-4 models were considered for the comparative analysis. We initially used the GPT models and several prompts for 290 questions without image data from the 116th NMLE (held in February 2022 in Japan) to maximize the performance for delivering correct answers and explanations of the questions. Thereafter, we tested the performance of the best GPT model (GPT-4) with optimized prompts on a dataset of 262 questions without images from the latest 117th NMLE (held in February 2023). The best model with the optimized prompts scored 82.7% for the essential questions and 77.2% for the basic and clinical questions, both of which sufficed the minimum passing scoring rates of 80.0% and 74.6%, respectively. After an exploratory analysis of 56 incorrect answers from the model, we identified the three major factors contributing to the generation of the incorrect answers-insufficient medical knowledge, information on Japan-specific medical system and guidelines, and mathematical errors. In conclusion, GPT-4 with our optimized prompts achieved a minimum passing scoring rate in the latest 117th NMLE in Japan. Beyond its original design of answering examination questions for humans, these artificial intelligence (AI) models can serve as one of the best "sidekicks" for solving problems and addressing the unmet needs in the medical and healthcare fields.

Pure Wisdom or Potemkin Villages? A Comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 Style Questions: Quantitative Analysis.

The United States Medical Licensing Examination (USMLE) has been critical in medical education since 1992, testing various aspects of a medical student's knowledge and skills through different steps, based on their training level. Artificial intelligence (AI) tools, including chatbots like ChatGPT, are emerging technologies with potential applications in medicine. However, comprehensive studies analyzing ChatGPT's performance on USMLE Step 3 in large-scale scenarios and comparing different versions of ChatGPT are limited. This paper aimed to analyze ChatGPT's performance on USMLE Step 3 practice test questions to better elucidate the strengths and weaknesses of AI use in medical education and deduce evidence-based strategies to counteract AI cheating. A total of 2069 USMLE Step 3 practice questions were extracted from the AMBOSS study platform. After including 229 image-based questions, a total of 1840 text-based questions were further categorized and entered into ChatGPT 3.5, while a subset of 229 questions were entered into ChatGPT 4. Responses were recorded, and the accuracy of ChatGPT answers as well as its performance in different test question categories and for different difficulty levels were compared between both versions. Overall, ChatGPT 4 demonstrated a statistically significant superior performance compared to ChatGPT 3.5, achieving an accuracy of 84.7% (194/229) and 56.9% (1047/1840), respectively. A noteworthy correlation was observed between the length of test questions and the performance of ChatGPT 3.5 (ρ=-0.069; P=.003), which was absent in ChatGPT 4 (P=.87). Additionally, the difficulty of test questions, as categorized by AMBOSS hammer ratings, showed a statistically significant correlation with performance for both ChatGPT versions, with ρ=-0.289 for ChatGPT 3.5 and ρ=-0.344 for ChatGPT 4. ChatGPT 4 surpassed ChatGPT 3.5 in all levels of test question difficulty, except for the 2 highest difficulty tiers (4 and 5 hammers), where statistical significance was not reached. In this study, ChatGPT 4 demonstrated remarkable proficiency in taking the USMLE Step 3, with an accuracy rate of 84.7% (194/229), outshining ChatGPT 3.5 with an accuracy rate of 56.9% (1047/1840). Although ChatGPT 4 performed exceptionally, it encountered difficulties in questions requiring the application of theoretical concepts, particularly in cardiology and neurology. These insights are pivotal for the development of examination strategies that are resilient to AI and underline the promising role of AI in the realm of medical education and diagnostics.

Understanding the USMLE journey of Brazilian medical students and graduates: a narrative review.

Over the past decades, advances in medical technology and research have promoted the globalization of medicine, drawing medical students and physicians worldwide into seeking for better career and financial prospects, as well as enhanced clinical and research training abroad. The medical landscape in Brazil has been marked by significant challenges over the past years, particularly within the Unified Health System (SUS). Despite the expansion of medical schools and initiatives such as the Mais Médicos program arising as efforts to improve healthcare access and decrease regional inequalities in the country, such problems as under-resourced working environments, inadequate infrastructure, and unattractive financial compensation for both physician and research careers, have risen as persisting challenges. In the pursuit of improved conditions, the United States Medical Licensing Examination (USMLE) has emerged as an increasingly common pathway for those seeking to overcome these challenges. This review explores the motivations behind Brazilians pursuing the USMLE while examining structural and financial challenges within the country's medical education and SUS landscape.

Are Medical School Curricula Adapting With Their Students? A Survey on How Medical Students Study and How it Relates to USMLE Step 1 Scores.

Over the years, medical schools have evolved their curricula in response to the medical field, faculty, and students. The current study aims to examine how medical students study, what resources they most commonly use, and how it relates to United States Medical Licensing Examination (USMLE) Step 1 scores. A cross-sectional survey study of United States medical students was distributed via social media, GroupMe school chats, and school listservs from September 8, 2020, to December 12, 2020. The survey gathered data including, demographic and school information, resources students, time spent using each resource, and USMLE Step 1 scores. The survey was completed by 560 students from 102 different United States medical schools. Study guides as online resources (83.2%) were mostly used, lecture (82.5%), Anki (spaced repetition flashcards, 68.3%), and school-organized sessions (workshops, labs, and small groups) (60.7%). Of the students surveyed, 90% attended schools with a recorded lecture option. Only 54% of these students watched their lectures live. When watched online, the average watching speed was 1.75 ± 0.4× with a mode of 2× speed. In examining different medical school styles, schools with a 1.5-year preclinical curriculum had higher USMLE Step 1 scores (244.5 ± 15.6 vs 236.9 ± 16.2, P-score = .024) compared to schools that did not (1- or 2-year preclinical curriculum). Medical students seem to be using third-party resources in addition to their medical school curriculums. Because students are already broadly using these, medical schools could allow their curriculums to reference, require, or complement third-party online resources. Additionally, because of their increased cost, medical schools could consider ways to alleviate the cost on medical students as a means of equitable support. Lastly, Step 1 scores significantly correlated with schools with a 1.5-year preclinical curriculum, although the exact reasoning for this remains uncertain.

Underrepresented Applicants Post-USMLE Pass/Fail: A National Survey of Competitive Residency Directors.

To report the selection criteria important to residency program directors (PDs) and whether they believe pass/fail scoring will impact underrepresented in medicine (URM), International Medical Graduate (IMG), or osteopathic (DO) residency applicants after the United States Medical Licensing Examination (USMLE) changed Step 1 score reporting to pass/fail in January 2022. A Qualtrics survey was sent between August 2022 and January 2023 to 1141 US PDs from specialties with traditionally low residency selection rates: dermatology, ophthalmology, otolaryngology, orthopedic surgery, neurosurgery, interventional radiology, diagnostic radiology, radiation oncology, thoracic surgery, vascular surgery, plastic surgery, and urology. Contact information was obtained from AMA, FRIEDA, or program websites. We received 433 responses (38%). When asked to anticipate the difficulty student groups will face matching into their specialty, PDs reported: for URM, 24.0% increased, 46.0% unchanged, and 30.0% decreased; for DO, 49.19% increased, 44.58% unchanged, and 6.23% decreased and for IMG, 56.35% increased, 39.72% unchanged, and 3.93% decreased. When asked to rank the most important selection factors, the top two responses were Step 2 CK score and away rotation participation at their site. PDs overwhelmingly believed residency selection difficulty would either increase or remain unchanged for DO (93.77%) and IMG (96.07%). In contrast, 76.0% reported difficulty for URM students would either decrease or remain unchanged. PDs ranked Step 2 CK score and away rotation participation as the most important selection factors. Despite PDs' belief that the Step 1 pass/fail scoring system may mitigate one barrier for URM students, emphasis on Step 2 CK and away rotations place additional barriers.

Exploring the proficiency of ChatGPT-4: An evaluation of its performance in the Taiwan advanced medical licensing examination.

Taiwan is well-known for its quality healthcare system. The country's medical licensing exams offer a way to evaluate ChatGPT's medical proficiency. We analyzed exam data from February 2022, July 2022, February 2023, and July 2033. Each exam included four papers with 80 single-choice questions, grouped as descriptive or picture-based. We used ChatGPT-4 for evaluation. Incorrect answers prompted a "chain of thought" approach. Accuracy rates were calculated as percentages. ChatGPT-4's accuracy in medical exams ranged from 63.75% to 93.75% (February 2022-July 2023). The highest accuracy (93.75%) was in February 2022's Medicine Exam (3). Subjects with the highest misanswered rates were ophthalmology (28.95%), breast surgery (27.27%), plastic surgery (26.67%), orthopedics (25.00%), and general surgery (24.59%). While using "chain of thought," the "Accuracy of (CoT) prompting" ranged from 0.00% to 88.89%, and the final overall accuracy rate ranged from 90% to 98%. ChatGPT-4 succeeded in Taiwan's medical licensing exams. With the "chain of thought" prompt, it improved accuracy to over 90%.

The Impact of Signaling on the 2022 to 2023 Orthopaedic Residency Application Cycle: A Survey of Incoming Residents.

During the 2022 to 2023 orthopaedic surgery residency application cycle, "signaling" was added, allowing applicants to communicate strong interest to 30 programs of their choosing. This study's purpose was to evaluate signaling's impact on the 2022 to 2023 application cycle. A cross-sectional survey was distributed to applicants who applied to a single residency program. We evaluated how many interviews the average applicant received, what proportion of interviews came from programs they had signaled, and what percentage of applicants matched at a program they had signaled. We stratified data by American Orthopaedic Association status, United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, research items, sex, and race. This survey was distributed to 611 applicants, with 124 (20.3%) responding to the survey. Ninety-five respondents (76.6%) matched. The percentage of applicant interviews that came from signaled programs was 78.7%. Ninety-one percent of matched applicants matched at a program they signaled. Sixty-three percent of matched applicants performed an away rotation at their matched programs. Forty-five percent of applicants felt that signaling incentivized reducing the programs they applied to, and applicants gave signaling a high favorability rating of 4/5. Applicants with Alpha Omega Alpha status received more interviews per application (0.18 ± 0.11 vs. 0.10 ± 0.10, p < 0.001) and more interviews from programs they did not signal (74% of interviews from signaled programs vs. 90% of interviews from signaled programs, p < 0.001). Higher Step 1 and Step 2 scores were associated with more interviews per application (Step 1: 0.16 ± 0.12 vs. 0.12 ± 0.08, p = 0.032) (Step 2: 0.16 ± 0.11 vs. 0.12 ± 0.09, p = 0.032). Orthopaedic residency applicants received most of their interviews from programs that they signaled, with an overwhelming majority matching at signaled program. Alpha Omega Alpha status and high USMLE scores were associated with more interviews granted per application, regardless of signaling status. Signaling seems to be a favorable option for orthopaedic applicants. Data from future application cycles will help further evaluate signaling's impact on the orthopaedic match.