Board Examinations Research Articles

Role of visual information in multimodal large language model performance: an evaluation using the Japanese nuclear medicine board examination

Role of visual information in multimodal large language model performance: an evaluation using the Japanese nuclear medicine board examination

Predictive Value of Neurosurgery Applicant Metrics on Resident Academic Productivity

BACKGROUND AND OBJECTIVES: Scholarship has been critical to neurosurgery. As grades and board examinations become pass-fail, finding metrics to distinguish applicants coupled with an emphasis on research has led to growth of reported academic output among neurosurgery applicants. We aimed to evaluate applicant factors that associate with an academically productive neurosurgery resident. METHODS: Applicant characteristics were extracted from Electronic Residency Application Service archives from 2 geographically distinct neurosurgical programs for the 2014 to 2015 match cycle. Publications during residency were quantified, and residency careers were examined. Factors associated with residency publications were examined using univariate and multivariate regressions. RESULTS: A total of 228 United States (US) applicants to neurosurgery were assessed (89% of US neurosurgery applicants), with 173 matching across 93 programs. The average publication number of matched applicants was higher at 6.6 (median: 4, range: 0-43) that of than unmatched applicants (mean: 2.9, median: 1, range: 0-51). A total of 93.1% of publications were substantiated on PubMed review. Matched candidates published 19.3 manuscripts (median: 13, range: 0-120) on average during residency. On univariate analysis, factors associated with higher residency publications included taking a non–degree-granting extra year for research in medical school, consistently high clerkship grades, depth of preresidency research involvement, number of coresidents, program R25 status, and academic output of neurosurgery department leadership. After multivariate correction, the training environment played an outsized role in predicting resident academic output, with program R25 status significantly associated with resident academic output (odds ratio: 1.25, P = .012). Taking an extra research year in medical school approached but was not significant (odds ratio: 1.19, P = .099). Twelve matched international medical school graduates (IMGs) were also assessed (75% of matched IMG neurosurgery applicants). IMGs exhibited higher total publications and conference abstracts than US matched applicants and also published more during residency. CONCLUSION: Cultivating an environment that promotes research endeavors is critical for neurosurgical resident academic growth. Preresidency publication number does not predict publication potential during residency.

Elimination of the Percentile Score From the Surgical ABSITE—The Resident Perspective

This Viewpoint describes the elimination of the percentile score from the American Board of Surgery In-Training Examination (ABSITE) from the resident’s perspective.

Elimination of the Percentile Score From the Surgical ABSITE—The Program Director Perspective

This Viewpoint discusses the elimination of the percentile score from the American Board of Surgery In-Training Examination (ABSITE) from the program director’s perspective.

Elimination of the Percentile Score From the Surgical ABSITE—The Fellowship Director Perspective

This Viewpoint describes the elimination of the percentile score from the American Board of Surgery In-Training Examination (ABSITE) from the fellowship director’s perspective.

Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan.

This study aims to investigate the effects of language selection and translation quality on Generative Pre-trained Transformer-4 (GPT-4)'s response accuracy to expert-level diagnostic radiology questions. We analyzed 146 diagnostic radiology questions from the Japan Radiology Board Examination (2020-2022), with consensus answers provided by two board-certified radiologists. The questions, originally in Japanese, were translated into English by GPT-4 and DeepL and into German and Chinese by GPT-4. Responses were generated by GPT-4 five times per question set per language. Response accuracy was compared between languages using one-way ANOVA with Bonferroni correction or the Mann-Whitney U test. Scores on selected English questions translated by a professional service and GPT-4 were also compared. The impact of translation quality on GPT-4's performance was assessed by linear regression analysis. The median scores (interquartile range) for the 146 questions were 70 (68-72) (Japanese), 89 (84.5-95.5) (GPT-4 English), 64 (55.5-67) (Chinese), and 56 (46.5-67.5) (German). Significant differences were found between Japanese and English (p = 0.002) and between Japanese and German (p = 0.022). The counts of correct responses across five attempts for each question were significantly associated with the quality of translation into English (GPT-4, DeepL) and German (GPT-4). In a subset of 31 questions where English translations yielded fewer correct responses than Japanese originals, professionally translated questions yielded better scores than those translated by GPT-4 (13 versus 8 points, p = 0.0079). GPT-4 exhibits higher accuracy when responding to English-translated questions compared to original Japanese questions, a trend not observed with German or Chinese translations. Accuracy improves with higher-quality English translations, underscoring the importance of high-quality translations in improving GPT-4's response accuracy to diagnostic radiology questions in non-English languages and aiding non-native English speakers in obtaining accurate answers from large language models.

Talking and Thinking About Systems: An Interview with Nick Saville

ABSTRACT This interview with Nick Saville covers the many facets of a language testing career working in a UK examination board. While the conversation traces his career over a 35 year period, the reader ends up seeing how the field of language assessment itself has developed and grown over that period. Topics covered include the adoption of Rasch methodology in the early days, all the way to more recent topics like the use of AI. The conversation went over his writing – on systemic approaches to ensure positive assessment impact – and on his work with Cambridge Assessment and with ALTE to illustrate how those ideas have been practically carried out in different systems and contexts. Also discussed were other publications he has been involved in, particularly the Studies in Language Testing monograph series, as well as the genesis of Language Assessment Quarterly itself, which highlight Anglo and American differences, but also how different traditions have come together to bring innovation to the field of language assessment.

Editorial Comment on Can artificial intelligence pass the Japanese urology board examinations?

Editorial Comment on Can artificial intelligence pass the Japanese urology board examinations?

Associations of benzodiazepine use with cognitive ability and age-related cognitive decline.

It remains uncertain whether long-term use of benzodiazepines is associated with age-related cognitive decline, and if cognitive ability in early life is the driver of any association. This study examines the association of cognitive ability in young adulthood with later use of benzodiazepines and explores whether the use of benzodiazepines during adult life is associated with cognitive decline in late midlife. The study samples include cognitive tests on the Børge Priens Prøve (BPP) from the conscription board examination (age 19 years) from 335 513 men born 1949-1961 and data from re-examinations of 5183 men 44 years later. Cognitive decline was defined as the difference between scores at the conscription board and the re-examination. Information on purchases of benzodiazepines was obtained from the Danish National Prescription Registry, 1995-2022. Associations were analysed using Cox proportional hazards and linear regression. In total, 120 911 (36%) men purchased benzodiazepines during a follow-up of 20 years. Lower cognitive scores in young adulthood were associated with a higher risk of initiating benzodiazepines (hazard ratio [95% CI] = 0.71[0.68-0.75]). Men with the highest cumulative use of benzodiazepines had larger cognitive decline (β-coefficient [95% CI] = -1.66 [-2.09 to -1.23] BPP scores) compared with never users. Current benzodiazepine users showed a larger cognitive decline than never users (β-coefficient [95% CI] = -2.42[-3.18 to -1.66] BPP scores) and this partially explained the above association. These estimates for cognitive decline were relatively small and may lack clinical relevance. Low cognitive ability increases the risk of benzodiazepine use in adulthood and cognitive decline is more pronounced in those with the highest benzodiazepine use compared with never-use, but the difference lacks clinical significance.

Assessment of ChatGPT-4 in Family Medicine Board Examinations Using Advanced AI Learning and Analytical Methods: Observational Study.

This research explores the capabilities of ChatGPT-4 in passing the American Board of Family Medicine (ABFM) Certification Examination. Addressing a gap in existing literature, where earlier artificial intelligence (AI) models showed limitations in medical board examinations, this study evaluates the enhanced features and potential of ChatGPT-4, especially in document analysis and information synthesis. The primary goal is to assess whether ChatGPT-4, when provided with extensive preparation resources and when using sophisticated data analysis, can achieve a score equal to or above the passing threshold for the Family Medicine Board Examinations. In this study, ChatGPT-4 was embedded in a specialized subenvironment, "AI Family Medicine Board Exam Taker," designed to closely mimic the conditions of the ABFM Certification Examination. This subenvironment enabled the AI to access and analyze a range of relevant study materials, including a primary medical textbook and supplementary web-based resources. The AI was presented with a series of ABFM-type examination questions, reflecting the breadth and complexity typical of the examination. Emphasis was placed on assessing the AI's ability to interpret and respond to these questions accurately, leveraging its advanced data processing and analysis capabilities within this controlled subenvironment. In our study, ChatGPT-4's performance was quantitatively assessed on 300 practice ABFM examination questions. The AI achieved a correct response rate of 88.67% (95% CI 85.08%-92.25%) for the Custom Robot version and 87.33% (95% CI 83.57%-91.10%) for the Regular version. Statistical analysis, including the McNemar test (P=.45), indicated no significant difference in accuracy between the 2 versions. In addition, the chi-square test for error-type distribution (P=.32) revealed no significant variation in the pattern of errors across versions. These results highlight ChatGPT-4's capacity for high-level performance and consistency in responding to complex medical examination questions under controlled conditions. The study demonstrates that ChatGPT-4, particularly when equipped with specialized preparation and when operating in a tailored subenvironment, shows promising potential in handling the intricacies of medical board examinations. While its performance is comparable with the expected standards for passing the ABFM Certification Examination, further enhancements in AI technology and tailored training methods could push these capabilities to new heights. This exploration opens avenues for integrating AI tools such as ChatGPT-4 in medical education and assessment, emphasizing the importance of continuous advancement and specialized training in medical applications of AI.

The Dunning‒Kruger effect in resident predicted and actual performance on the American Board of Emergency Medicine in-training examination.

The Dunning-Kruger effect (DKE) is a cognitive bias wherein individuals who are unskilled overestimate their abilities, while those who are skilled tend to underestimate their capabilities. The purpose of this investigation is to determine if the DKE exists among American Board of Emergency Medicine (ABEM) in-training examination (ITE) participants. This is a prospective, cross-sectional survey of residents in Accreditation Council for Graduate Medical Education (ACGME)-accredited emergency medicine (EM) residency programs. All residents who took the 2022 ABEM ITE were eligible for inclusion. Residents from international programs, residents in combined training programs, and those who did not complete the voluntary post-ITE survey were excluded. Half of the residents taking the ITE were asked to predict their self-assessment of performance (percent correct), and the other half were asked to predict their performance relative to peers at the same level of training (quintile estimate). Pearson's correlation (r) was used for parametric interval data comparisons and a Spearman's coefficient (ρ) was determined for quintile-to-quintile comparisons. A total of 7568 of 8918 (84.9%) residents completed their assigned survey question. A total of 3694 residents completed self-assessment (mean predicted percentage correct 67.4% and actual 74.6%), with a strong positive correlation (Pearson's r 0.58, p<0.001). There was also a strong positive correlation (Spearman's ρ 0.53, p<0.001) for the 3874 residents who predicted their performance compared to peers. Of these, 8.5% of residents in the first (lowest) quintile and 15.7% of residents in the fifth (highest) quintile correctly predicted their performance compared to peers. EM residents demonstrated accurate self-assessment of their performance on the ABEM ITE; however, the DKE was present when comparing their self-assessments to their peers. Lower-performing residents tended to overestimate their performance, with the most significant DKE observed among the lowest-performing residents. The highest-performing residents tended to underestimate their relative performance.

Retrieval-Augmented Generation for Large Language Models in Radiology: Another Leap Forward in Board Examination Performance.

Supplemental material is available for this article. See also the editorial by Forghani in this issue.

Correction to “Association between the American Board of Emergency Medicine Oral Certifying Examination and Future State Medical Board Disciplinary Actions”

Correction to “Association between the American Board of Emergency Medicine Oral Certifying Examination and Future State Medical Board Disciplinary Actions”

Performance of Publicly Available Large Language Models on Internal Medicine Board-style Questions.

Ongoing research attempts to benchmark large language models (LLM) against physicians' fund of knowledge by assessing LLM performance on medical examinations. No prior study has assessed LLM performance on internal medicine (IM) board examination questions. Limited data exists on how knowledge supplied to the models, derived from medical texts improves LLM performance. The performance of GPT-3.5, GPT-4.0, LaMDA and Llama 2, with and without additional model input augmentation, was assessed on 240 randomly selected IM board-style questions. Questions were sourced from the Medical Knowledge Self-Assessment Program released by the American College of Physicians with each question serving as part of the LLM prompt. When available, LLMs were accessed both through their application programming interface (API) and their corresponding chatbot. Mode inputs were augmented with Harrison's Principles of Internal Medicine using the method of Retrieval Augmented Generation. LLM-generated explanations to 25 correctly answered questions were presented in a blinded fashion alongside the MKSAP explanation to an IM board-certified physician tasked with selecting the human generated response. GPT-4.0, accessed either through Bing Chat or its API, scored 77.5-80.7% outperforming GPT-3.5, human respondents, LaMDA and Llama 2 in that order. GPT-4.0 outperformed human MKSAP users on every tested IM subject with its highest and lowest percentile scores in Infectious Disease (80th) and Rheumatology (99.7th), respectively. There is a 3.2-5.3% decrease in performance of both GPT-3.5 and GPT-4.0 when accessing the LLM through its API instead of its online chatbot. There is 4.5-7.5% increase in performance of both GPT-3.5 and GPT-4.0 accessed through their APIs after additional input augmentation. The blinded reviewer correctly identified the human generated MKSAP response in 72% of the 25-question sample set. GPT-4.0 performed best on IM board-style questions outperforming human respondents. Augmenting with domain-specific information improved performance rendering Retrieval Augmented Generation a possible technique for improving accuracy in medical examination LLM responses.

Performance of ChatGPT in the In-Training Examination for Anesthesiology and Pain Medicine Residents in South Korea: Observational Study.

ChatGPT has been tested in health care, including the US Medical Licensing Examination and specialty exams, showing near-passing results. Its performance in the field of anesthesiology has been assessed using English board examination questions; however, its effectiveness in Korea remains unexplored. This study investigated the problem-solving performance of ChatGPT in the fields of anesthesiology and pain medicine in the Korean language context, highlighted advancements in artificial intelligence (AI), and explored its potential applications in medical education. We investigated the performance (number of correct answers/number of questions) of GPT-4, GPT-3.5, and CLOVA X in the fields of anesthesiology and pain medicine, using in-training examinations that have been administered to Korean anesthesiology residents over the past 5 years, with an annual composition of 100 questions. Questions containing images, diagrams, or photographs were excluded from the analysis. Furthermore, to assess the performance differences of the GPT across different languages, we conducted a comparative analysis of the GPT-4's problem-solving proficiency using both the original Korean texts and their English translations. A total of 398 questions were analyzed. GPT-4 (67.8%) demonstrated a significantly better overall performance than GPT-3.5 (37.2%) and CLOVA-X (36.7%). However, GPT-3.5 and CLOVA X did not show significant differences in their overall performance. Additionally, the GPT-4 showed superior performance on questions translated into English, indicating a language processing discrepancy (English: 75.4% vs Korean: 67.8%; difference 7.5%; 95% CI 3.1%-11.9%; P=.001). This study underscores the potential of AI tools, such as ChatGPT, in medical education and practice but emphasizes the need for cautious application and further refinement, especially in non-English medical contexts. The findings suggest that although AI advancements are promising, they require careful evaluation and development to ensure acceptable performance across diverse linguistic and professional settings.

Computer Vision Meets Large Language Models: Performance of ChatGPT 4.0 on Dermatology Boards-Style Practice Questions

Background: ChatGPT is a generative artificial intelligence that has numerous professional applications. Applications in medical education are currently being explored. ChatGPT 4.0 performance on image-based dermatology boards-style practice questions has not been assessed. Objective: The objective of this study was to determine the accuracy with which ChatGPT can answer dermatology boards examination practice questions. Methods: 150 multiple-choice questions from the popular question bank DermQbank were inputted into ChatGPT. Of these, 83 were text-only questions and 67 had associated images. These same questions were inputted into ChatGPT again in July 2024. An additional 150 questions were inputted for a total of 300 different questions where 169 were text-only and 133 had associated images. Results: Of the aggregate 300 question data, ChatGPT answered 232 questions correctly (77.3%). ChatGPT performed significantly better with text-only questions than with questions that included images (85.2% (144/169) vs 67.7% (90/133), P&lt;.001). Of image-based questions, ChatGPT performed better with clinical image questions than with dermatopathology questions (69.0% (78/133) vs. 58.8% (10/17), P=.40), but this difference was not statistically significant partially due to the sample size of the dermatopathology questions. Compared to post-graduate year 4 (PGY-4) residents, ChatGPT performed above the 46th percentile. ChatGPT agreed with the answer choice picked by the majority of question bank users 75.3% of the time. Multivariable regression demonstrated that significant predictive variables for ChatGPT answering a question correctly included the percent of dermatology trainees who answered a question correctly and whether the question was text-based (P&lt;.001and P=.004, respectively). Conclusions: ChatGPT answered 77.3% of dermatology board examination practice questions correctly, performing above the 46th percentile of PGY-4 question bank users. If using ChatGPT as a study resource for dermatology board examination preparation, residents should be judicious with exactly how they employ ChatGPT to avoid learning incorrect information.

An Exploratory Analysis of ChatGPT Compared to Human Performance With the Anesthesiology Oral Board Examination: Initial Insights and Implications.

Chat Generative Pre-Trained Transformer (ChatGPT) has been tested and has passed various high-level examinations. However, it has not been tested on an examination such as the American Board of Anesthesiology (ABA) Standardized Oral Examination (SOE). The SOE is designed to assess higher-level competencies, such as judgment, organization, adaptability to unexpected clinical changes, and presentation of information. Four anesthesiology fellows were examined on 2 sample ABA SOEs. Their answers were compared to those produced by the same questions asked to ChatGPT. All human and ChatGPT responses were transcribed, randomized by module, and then reproduced as complete examinations, using a commercially available software-based human voice replicator. Eight ABA applied examiners listened to and scored the topic and modules from 1 of the 4 versions of each of the 2 sample examinations. The ABA did not provide any support or collaboration with any authors. The anesthesiology fellow's answers were found to have a better median score than ChatGPT, for the module topics scores (P = .03). However, there was no significant difference in the median overall global module scores between the human and ChatGPT responses (P = .17). The examiners were able to identify the ChatGPT-generated answers for 23 of 24 modules (95.83%), with only 1 ChatGPT response perceived as from a human. In contrast, the examiners thought the human (fellow) responses were artificial intelligence (AI)-generated in 10 of 24 modules (41.67%). Examiner comments explained that ChatGPT generated relevant content, but were lengthy answers, which at times did not focus on the specific scenario priorities. There were no comments from the examiners regarding Chat GPT fact "hallucinations." ChatGPT generated SOE answers with comparable module ratings to anesthesiology fellows, as graded by 8 ABA oral board examiners. However, the ChatGPT answers were deemed subjectively inferior due to the length of responses and lack of focus. Future curation and training of an AI database, like ChatGPT, could produce answers more in line with ideal ABA SOE answers. This could lead to higher performance and an anesthesiology-specific trained AI useful for training and examination preparation.

Can artificial intelligence pass the Japanese urology board examinations?

Can artificial intelligence pass the Japanese urology board examinations?

Assessing knowledge about medical physics in language-generative AI with large language model: using the medical physicist exam.

This study aimed to evaluate the performance for answering the Japanese medical physicist examination and providing the benchmark of knowledge about medical physics in language-generative AI with large language model. We used questions from Japan's 2018, 2019, 2020, 2021 and 2022 medical physicist board examinations, which covered various question types, including multiple-choice questions, and mainly focused on general medicine and medical physics. ChatGPT-3.5 and ChatGPT-4.0 (OpenAI) were used. We compared the AI-based answers with the correct ones. The average accuracy rates were 42.2 ± 2.5% (ChatGPT-3.5) and 72.7 ± 2.6% (ChatGPT-4), showing that ChatGPT-4 was more accurate than ChatGPT-3.5 [all categories (except for radiation-related laws and recommendations/medical ethics): p value < 0.05]. Even with the ChatGPT model with higher accuracy, the accuracy rates were less than 60% in two categories; radiation metrology (55.6%), and radiation-related laws and recommendations/medical ethics (40.0%). These data provide the benchmark for knowledge about medical physics in ChatGPT and can be utilized as basic data for the development of various medical physics tools using ChatGPT (e.g., radiation therapy support tools with Japanese input).

Association of internship with performance on American Board of Physical Medicine and Rehabilitation certification examinations.

Physical Medicine and Rehabilitation (PM&R) residents must complete twelve months of fundamental skills training prior to beginning PM&R residency. The objective of this study is to determine if characteristics of the first post-graduate year (PGY-1) impact performance on American Board of PM&R (ABPMR) initial certification examinations. A retrospective review was conducted on a deidentified ABPMR database of physicians who completed PM&R residency and took the Part I Examination between 2008 and 2022. Physicians who completed categorical residency programs in PM&R had higher pass rates on Part I than physicians who completed advanced programs. Physicians who completed a categorical program had higher scaled scores on the Part II Examination than physicians who completed either a transitional or non-transitional advanced program but pass rates did not differ. Completing less than 3 months of training in internal medicine prior to starting PM&R was associated with lower Part I and Part II Examination scaled scores than completing 3 or more months. Physicians who completed six or more months of internal medicine had higher Part I and Part II Examination pass rates and scaled scores than physicians who completed six or more months in surgery.