Examination Questions Research Articles

Performance of trauma-trained large language models on surgical assessment questions: A new approach in resource identification

BackgroundLarge language models have successfully navigated simulated medical board examination questions. However, whether and how language models can be used in surgical education is less understood. Our study evaluates the efficacy of domain-specific large language models in curating study materials for surgical board style questions. MethodsWe developed EAST-GPT and ACS-GPT, custom large language models with domain-specific knowledge from published guidelines from the Eastern Association of the Surgery of Trauma and the American College of Surgeons Trauma Quality Programs. EAST-GPT, ACS-GPT, and an untrained GPT-4 performance were assessed trauma-related questions from Surgical Education and Self-Assessment Program (18th edition). Large language models were asked to choose answers and provide answer rationales. Rationales were assessed against an educational framework with 5 domains: accuracy, relevance, comprehensiveness, evidence-base, and clarity. ResultsNinety guidelines trained EAST-GPT and 10 trained ACS-GPT. All large language models were tested on 62 trauma questions. EAST-GPT correctly answered 76%, whereas ACS-GPT answered 68% correctly. Both models outperformed ChatGPT-4 (P < .05), which answered 45% correctly. For reasoning, EAST-GPT achieved the gratest mean scores across all 5 educational framework metrics. ACS-GPT scored lower than ChatGPT-4 in comprehensiveness and evidence-base; however, these differences were not statistically significant. ConclusionOur study presents a novel methodology in identifying test-preparation resources by training a large language model to answer board-style multiple choice questions. Both trained models outperformed ChatGPT-4, demonstrating its answers were accurate, relevant, and evidence-based. Potential implications of such AI integration into surgical education must be explored.

Performance Evaluation of the Generative Pre-trained Transformer (GPT-4) on the Family Medicine In-Training Examination.

In this study, we sought to comprehensively evaluate GPT-4 (Generative Pre-trained Transformer)'s performance on the 2022 American Board of Family Medicine's (ABFM) In-Training Examination (ITE), compared with its predecessor, GPT-3.5, and the national family residents' performance on the same examination. We utilized both quantitative and qualitative analyses. First, a quantitative analysis was employed to evaluate the model's performance metrics using zero-shot prompt (where only examination questions were provided without any additional information). After this, qualitative analysis was executed to understand the nature of the model's responses, the depth of its medical knowledge, and its ability to comprehend contextual or new information through chain-of-thoughts prompts (interactive conversation) with the model. This study demonstrated that GPT-4 made significant improvement in accuracy compared with GPT-3.5 over a 4-month interval between their respective release dates. The correct percentage with zero-shot prompt increased from 56% to 84%, which translates to a scaled score growth from 280 to 690, a 410-point increase. Most notably, further chain-of-thought investigation revealed GPT-4's ability to integrate new information and make self-correction when needed. In this study, GPT-4 has demonstrated notably high accuracy, as well as rapid reading and learning capabilities. These results are consistent with previous research indicating GPT-4's significant potential to assist in clinical decision making. Furthermore, the study highlights the essential role of physicians' critical thinking and lifelong learning skills, particularly evident through the analysis of GPT-4's incorrect responses. This emphasizes the indispensable human element in effectively implementing and using AI technologies in medical settings.

Increasing Sexual Violence Reporting and Disclosure in Higher Education Institutions: A Proposed Approach to Critically Analyze the Internal Organizational Context.

This article explores the underreporting of sexual violence (SV) in higher education, highlighting serious implications for survivors who may silently cope with its aftermath instead of accessing crucial resources. We utilize Bolman and Deal's four-frame model for organizational change to assess how internal factors within organizations may influence reporting of SV. The four frames-symbolic, structural, human resources, and political-offer a systematic analysis of the internal organizational context in higher education institutions concerning SV reporting. Our suggested approach offers concrete dimensions and probing questions for examination. Derived from a qualitative study, our recommendations align with Bolman and Deal's four-frame model, aiding in assessing the organizational environment. This approach assists stakeholders in identifying barriers/facilitators in the internal organizational context of higher education institutions, enabling effective planning for improved SV reporting/disclosure. A thorough analysis is essential for understanding factors influencing campus SV reporting. Our proposed critical analysis and recommendations serve as a starting point to identify organizational barriers/facilitators, informing the revision of SV policies and processes, including reporting. Enhanced reporting of sexual assault in higher education benefits forensic nurses and allies, like student affairs, advocacy groups, unions, SV coordinators, health centers, equity departments, human rights officers, and administration. Improved analysis of institutional and cultural contexts allows for tailored services to better meet survivors' needs. Increased reporting should lead institutions to higher service utilization, requiring careful planning for resource allocation.

An Examination of the Relationships between Pre-service Teachers’ Test Anxiety, Use of Social Networking Sites, and Cumulative Grade Point Average

This study aimed to determine situations or conditions that promote test anxiety, identify the frequent uses of social networking sites, and examine the relationship between social media use, test anxiety, and Cumulative Grade Point Average (CGPA). A test anxiety and social networking questionnaire which consisted of 49 items was administered to randomly selected pre-service teachers in a college of education at the end of the second-semester examination. A multi-stage sampling technique was employed to select 106 pre-service teachers. The data collected from the respondents was analyzed using Pearson’s Product Moment Correlation Co-efficient, mean, and frequency. In this study, it was found that taking an unannounced examination, feeling one should have done better than he/she did after an examination, waiting to see the results of an examination published, discussing with the tutor the answer one believed to be right, however, that was marked wrong and reading through the examination questions and feeling that one does not know any of the answers promote test anxiety among the pre-service teachers. The pre-service teachers used social networking sites predominantly to read news, get information regarding current social events, conduct research, share new ideas, relieve academic stress, share pictures, and view funny videos. It was found that there is a significant positive moderate correlation between test anxiety and the use of social networking sites among pre-service teachers. It was also found that, statistically, there was no relationship between test anxiety and CGPA, and CGPA and use of social networking sites.

Exploring the Potential of Artificial Intelligence Tools in Teacher Preparation Based on the Elementary School Teacher Qualification Examination: Analyzing the Cognitive Level of Examination Questions

&lt;p&gt;為解決複雜的教學問題，需培養教師具備高階認知層次的能力，而日新月異的人工智慧有望成為師資養成的助力。教師資格考試為師資生修習師培課程能力認證的門檻，分析試題中的認知層次分布及人工智慧工具（例：ChatGPT）的答題表現，以探索其於教學輔助能力的表現有其必要性。本研究選取2019年到2022年小學教師資格考試中的教育專業科目，即「教育理念與實務」、「學習者發展與適性輔導」和「課程教學與評量」三相關考科的試題，共560題，運用修訂後的Bloom認知分類六層次，並再區分為高、低層次對不同考科試題及題型進行分析。此外，研究者探查ChatGPT於相關考科的總得分，並以盒鬚圖比較ChatGPT在不同科目和題型中高、低認知層次試題得分比的分布情形。結果顯示，教師資格考試的教育專業科目中，三考科的低認知層次試題占分比都高於高認知層次試題。符應素養導向評量要求的綜合題，高認知層次占分比比較高，達81.67 % ，但選擇題的高認知層次占分比僅達6.55 % ，有提升高認知層次比例的必要。此外，ChatGPT在教育專業科目平均總分為72.73分，成功通過教師資格考試平均60分的門檻，但對於高認知層次試題的表現歧異度大。這表明ChatGPT具有成為師資養成輔助工具的潛力，但教育工作者仍需特別注意高層次認知情境的評鑑與批判。&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;&lt;p&gt;Training teachers to develop higher cognitive levels to address complex teaching challenges is necessary. Artificial intelligence (e.g., ChatGPT) holds the potential to assist in teacher education. Teacher qualification examination serves as a threshold for certifying pre-service teachers’ abilities in pursuing teacher education curricula. Analyzing the distribution of cognitive levels in examination questions and assessing the performance of artificial intelligence tools in answering these questions is necessary to explore their potential in aiding teaching assistance. This study selected 560 education professional examination questions from the elementary school teacher qualification examination between 2019 and 2022. We used the revised Bloom’s cognitive taxonomy to analyze and categorize questions into high and low cognitive levels. We investigated the ChatGPT scores in the relevant subject areas and compared the differences in score distribution between high and low cognitive levels across different subjects and question types using box plots. The results indicate that the percentage of low-order questions in all three subjects was higher than that of high-order questions in the teacher qualification examination of education professional subjects. The proportion of high-order comprehensive questions has increased to 81.67% to meet the competence-oriented assessment requirements. The percentage for multiple-choice questions remains relatively low at only 6.55%. This indicates a need to increase the proportion of high-order questions. Furthermore, ChatGPT achieved an average total score of 72.73 points in the education professional subjects, surpassing the threshold of an average score of 60 points for the teacher qualification examination. However, there was significant variability in its performance on high-order questions. This suggests that ChatGPT can potentially become an assistive tool for teacher education. However, educators must consider developing critical thinking skills in high-order cognitive contexts when using such tools.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

The Accuracy of Artificial Intelligence ChatGPT in Oncology Examination Questions

The aim of this study is to assess the accuracy of Chat Generative Pretrained Transformer (ChatGPT) in response to oncology examination questions in the setting of one-shot learning. Consecutive national radiation oncology in-service multiple-choice examinations were collected and inputted into ChatGPT 4o and ChatGPT 3.5 to determine ChatGPT’s answers. ChatGPT’s answers were then compared with the answer keys to determine whether ChatGPT correctly or incorrectly answered each question and to determine if improvements in responses were seen with the newer ChatGPT version. A total of 600 consecutive questions were inputted into ChatGPT. ChatGPT 4o answered 72.2% questions correctly, whereas 3.5 answered 53.8% questions correctly. There was a significant difference in performance by question category (P < .01). ChatGPT performed poorer with respect to knowledge of landmark studies and treatment recommendations and planning. ChatGPT is a promising technology, with the latest version showing marked improvement. Although it still has limitations, with further evolution, it may be considered a reliable resource for medical training and decision making in the oncology space.

Llama 3 Challenges Proprietary State-of-the-Art Large Language Models in Radiology Board-style Examination Questions.

Llama 3 Challenges Proprietary State-of-the-Art Large Language Models in Radiology Board-style Examination Questions.

Enhancing Assessment towards Flexible Medical Education: A Deep Dive into Preclinical Short Answer Questions through Item Analysis

Short answer questions (SAQs) and other forms of similarly structured examination methods are employed as evaluation tools to gauge the competency of medical students at various levels of study. However, item analyses for these questions have rarely been conducted. Evaluating the quality of examination questions through item analysis ensures that stakeholders, especially those undergoing learning in various flexible pathways in medical education, are provided with reliable and relevant assessments, promoting effective learning and competency development. In this study, we performed item analyses on SAQs by extracting the passing index (PI) and discrimination index (DI) for each sub-question. Data were analysed using Microsoft Excel (Microsoft Corporation, United States) and Jeffreys’s Amazing Statistics Program (JASP) (University of Amsterdam, The Netherlands). Twentyseven sub-questions from five SAQs were analysed. The DI of the sub-questions ranged from 0.043 to 0.935 with a mean of 0.449 ± 0.223, while the PI returned a range of 0.012 to 0.971 with a mean value of 0.597 ± 0.246. In conclusion, the SAQs administered during a professional examination of preclinical medical students exhibited an acceptable range and mix of PI and DI values. However, improvements must be made to the sub-questions that return poor PI and DI values.

Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.

Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations. In this study, we reviewed all studies on ChatGPT performance in medical licensing examinations up to March 2024. This review aims to contribute to the evolving discourse on artificial intelligence (AI) in medical education by providing a comprehensive analysis of the performance of ChatGPT in various environments. The insights gained from this systematic review will guide educators, policymakers, and technical experts to effectively and judiciously use AI in medical education. We searched the literature published between January 1, 2022, and March 29, 2024, by searching query strings in Web of Science, PubMed, and Scopus. Two authors screened the literature according to the inclusion and exclusion criteria, extracted data, and independently assessed the quality of the literature concerning Quality Assessment of Diagnostic Accuracy Studies-2. We conducted both qualitative and quantitative analyses. A total of 45 studies on the performance of different versions of ChatGPT in medical licensing examinations were included in this study. GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5. GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. Translating the examination questions into English improved GPT-3.5's performance but did not affect GPT-4. GPT-3.5 showed no difference in performance between examinations from English-speaking and non-English-speaking countries (P=.72), but GPT-4 performed better on examinations from English-speaking countries significantly (P=.02). Any type of prompt could significantly improve GPT-3.5's (P=.03) and GPT-4's (P<.01) performance. GPT-3.5 performed better on short-text questions than on long-text questions. The difficulty of the questions affected the performance of GPT-3.5 and GPT-4. In image-based multiple-choice questions (MCQs), ChatGPT's accuracy rate ranges from 13.1% to 100%. ChatGPT performed significantly worse on open-ended questions than on MCQs. GPT-4 demonstrates considerable potential for future use in medical education. However, due to its insufficient accuracy, inconsistent performance, and the challenges posed by differing medical policies and knowledge across countries, GPT-4 is not yet suitable for use in medical education. PROSPERO CRD42024506687; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=506687.

Large Language Models Take on Cardiothoracic Surgery: A Comparative Analysis of the Performance of Four Models on American Board of Thoracic Surgery Exam Questions in 2023.

Objectives Large language models (LLMs), for example, ChatGPT, have performed exceptionally well in various fields. Of note, their success in answering postgraduate medical examination questions has been previously reported, indicating their possible utility in surgical education and training. This study evaluated the performance of four different LLMs on the American Board of Thoracic Surgery's(ABTS) Self-Education and Self-Assessment in Thoracic Surgery (SESATS) XIII question bank to investigate the potential applications of these LLMs in the education and training of future surgeons. Methods The dataset in this study comprised 400 best-of-four questions from the SESATS XIII exam. This included 220 adult cardiac surgery questions, 140 general thoracic surgery questions, 20 congenital cardiac surgery questions, and 20 cardiothoracic critical care questions. The GPT-3.5 (OpenAI, San Francisco, CA) and GPT-4 (OpenAI) models were evaluated, as well as Med-PaLM 2 (Google Inc., Mountain View, CA) and Claude 2 (Anthropic Inc., San Francisco, CA), and their respective performances were compared. The subspecialties included were adult cardiac, general thoracic, congenital cardiac, and critical care. Questions requiring visual information, such as clinical images or radiology, were excluded. Results GPT-4 demonstrated a significant improvement over GPT-3.5 overall (87.0% vs. 51.8% of questions answered correctly, p < 0.0001). GPT-4 also exhibited consistently improved performance across all subspecialties, with accuracy rates ranging from 70.0% to 90.0%, compared to 35.0% to 60.0% for GPT-3.5. When using the GPT-4 model, ChatGPT performed significantly better on the adult cardiac and general thoracic subspecialties (p < 0.0001). Conclusions Large language models, such as ChatGPT with the GPT-4 model, demonstrate impressive skill in understanding complex cardiothoracic surgical clinical information, achieving an overall accuracy rate of nearly 90.0% on the SESATS question bank. Our study shows significant improvement between successive GPT iterations. As LLM technology continues to evolve, its potential use in surgical education, training, and continuous medical education is anticipated to enhance patient outcomes and safety in the future.

The Utility of Wearable Cameras in Developing Examination Questions and Answers on Physical Examinations: Preliminary Study.

To assess the utility of wearable cameras in medical examinations, we created a physician-view video-based examination question and explanation, and the survey results indicated that these cameras can enhance the evaluation and educational capabilities of medical examinations.

Pelatihan Bahasa Inggris Reading Comprehension “It’s All About Reading Skill” Di SMAN Karya Sakti Muara Kelingi Musi Rawas

SMAN Karya Sakti is one of the educational units at high school level in Karya Sakti, Kec. Muara Kelingi, Kab. Musi Rawas, South Sumatra. In carrying out its activities, SMAN Karya Sakti is under the auspices of the Ministry of Education and Culture with B accreditation. SMAN Karya Sakti is located at Jalan Tapak Libok Blok A, Karya Sakti, Kec. Muara Kelingi, Kab. Musi Rawas, South Sumatra. The aim of this Community Service activity is to provide English language training on reading comprehension which is useful for students in answering ANBK exam questions. The method used in this service is through socialization, incidental training for 1 day. In the process of implementing this activity, the team will provide material as well as tips and tricks on how to easily answer reading skills questions so that it is hoped that students will have adequate abilities in answering questions about reading comprehension both on the ANBK exam and other tests. The expected output target is that students can predict the answers to National Examination questions and answer them correctly and briefly. They also have skills in Reading Skills to analyze the answers to the questions presented

Performance of three artificial intelligence (AI)-based large language models in standardized testing; implications for AI-assisted dental education.

The emerging rise in novel computer technologies and automated data analytics has the potential to change the course of dental education. In line with our long-term goal of harnessing the power of AI to augment didactic teaching, the objective of this study was to quantify and compare the accuracy of responses provided by ChatGPT (GPT-4 and GPT-3.5) and Google Gemini, the three primary large language models (LLMs), to human graduate students (control group) to the annual in-service examination questions posed by the American Academy of Periodontology (AAP). Under a comparative cross-sectional study design, a corpus of 1312 questions from the annual in-service examination of AAP administered between 2020 and 2023 were presented to the LLMs. Their responses were analyzed using chi-square tests, and the performance was juxtaposed to the scores of periodontal residents from corresponding years, as the human control group. Additionally, two sub-analyses were performed: one on the performance of the LLMs on each section of the exam; and in answering the most difficult questions. ChatGPT-4 (total average: 79.57%) outperformed all human control groups as well as GPT-3.5 and Google Gemini in all exam years (p < .001). This chatbot showed an accuracy range between 78.80% and 80.98% across the various exam years. Gemini consistently recorded superior performance with scores of 70.65% (p = .01), 73.29% (p = .02), 75.73% (p < .01), and 72.18% (p = .0008) for the exams from 2020 to 2023 compared to ChatGPT-3.5, which achieved 62.5%, 68.24%, 69.83%, and 59.27% respectively. Google Gemini (72.86%) surpassed the average scores achieved by first- (63.48% ± 31.67) and second-year residents (66.25% ± 31.61) when all exam years combined. However, it could not surpass that of third-year residents (69.06% ± 30.45). Within the confines of this analysis, ChatGPT-4 exhibited a robust capability in answering AAP in-service exam questions in terms of accuracy and reliability while Gemini and ChatGPT-3.5 showed a weaker performance. These findings underscore the potential of deploying LLMs as an educational tool in periodontics and oral implantology domains. However, the current limitations of these models such as inability to effectively process image-based inquiries, the propensity for generating inconsistent responses to the same prompts, and achieving high (80% by GPT-4) but not absolute accuracy rates should be considered. An objective comparison of their capability versus their capacity is required to further develop this field of study.

Evaluating the application of ChatGPT in China’s residency training education: An exploratory study

Objective The purpose of this study was to assess the utility of information generated by ChatGPT for residency education in China. Methods We designed a three-step survey to evaluate the performance of ChatGPT in China’s residency training education including residency final examination questions, patient cases, and resident satisfaction scores. First, 204 questions from the residency final exam were input into ChatGPT's interface to obtain the percentage of correct answers. Next, ChatGPT was asked to generate 20 clinical cases, which were subsequently evaluated by three instructors using a pre-designed Likert scale with 5 points. The quality of the cases was assessed based on criteria including clarity, relevance, logicality, credibility, and comprehensiveness. Finally, interaction sessions between 31 third-year residents and ChatGPT were conducted. Residents’ perceptions of ChatGPT's feedback were assessed using a Likert scale, focusing on aspects such as ease of use, accuracy and completeness of responses, and its effectiveness in enhancing understanding of medical knowledge. Results Our results showed ChatGPT-3.5 correctly answered 45.1% of exam questions. In the virtual patient cases, ChatGPT received mean ratings of 4.57 ± 0.50, 4.68 ± 0.47, 4.77 ± 0.46, 4.60 ± 0.53, and 3.95 ± 0.59 points for clarity, relevance, logicality, credibility, and comprehensiveness from clinical instructors, respectively. Among training residents, ChatGPT scored 4.48 ± 0.70, 4.00 ± 0.82 and 4.61 ± 0.50 points for ease of use, accuracy and completeness, and usefulness, respectively. Conclusion Our findings demonstrate ChatGPT's immense potential for personalized Chinese medical education.

Differential Item Functioning of the region-based national examination equipment

This research aims to detect Differential Item Functioning (DIF) in the 2014/2015 National Examination Questions in mathematics of junior high schools and equivalent-level schools in the Yogyakarta region as a reference group and the South Kalimantan region as a focus group using the Likelihood Ratio Test (LRT) method, Area Measure Raju, and Lord. A sensitivity analysis was carried out to determine which method was the most sensitive. The data consisted of 5,465 National Examination papers of the students from the two regions, who worked on type A questions. A sample of 1,000 exam papers for each region was established using the simple random sampling (SRS) technique, which was conducted to avoid the effect of sample size. The results of the research showed that by using the LRT method the researchers found 36 items had significant DIF detection, 32 items were significant for Raju Area, and all items had significant DIF detection using Lord. Lord Method is the most sensitive because this method can detect the most DIF items.

Enhancing Assessment Practices: A Critical Review of Examination Question Papers at University of Sargodha

The construction of the question paper is the key phase in examination system that leads towards the attainment of the goal of assessment which ultimately directs us to make better decision making. Thus, evaluation of the question paper plays an indispensable and prominent role in this regard. To find out the extent of content coverage of the question paper and to perform item analysis to estimate the quality of the questions, 10 question papers, and 489 answer scripts were selected through the Multistage Sampling technique. Question paper review format and focus group discussion were used as a tool. Quantitative analysis disclosed that question paper did not include 52% of the entire syllabus while approximately all items were in “good” or “acceptable” range regarding item analysis. The minimal strata of the question showed very weak discrimination. Variation was also found in attempting different questions ranging from 16% to 93%. Focus Group Discussion disclosed that a specific portion of the syllabus was not taught or ignored by the concerned teacher.

Use of Multiple-Choice Items in Summative Examinations: Questionnaire Survey Among German Undergraduate Dental Training Programs.

Multiple-choice examinations are frequently used in German dental schools. However, details regarding the used item types and applied scoring methods are lacking. This study aims to gain insight into the current use of multiple-choice items (ie, questions) in summative examinations in German undergraduate dental training programs. A paper-based 10-item questionnaire regarding the used assessment methods, multiple-choice item types, and applied scoring methods was designed. The pilot-tested questionnaire was mailed to the deans of studies and to the heads of the Department of Operative/Restorative Dentistry at all 30 dental schools in Germany in February 2023. Statistical analysis was performed using the Fisher exact test (P<.05). The response rate amounted to 90% (27/30 dental schools). All respondent dental schools used multiple-choice examinations for summative assessments. Examinations were delivered electronically by 70% (19/27) of the dental schools. Almost all dental schools used single-choice Type A items (24/27, 89%), which accounted for the largest number of items in approximately half of the dental schools (13/27, 48%). Further item types (eg, conventional multiple-select items, Multiple-True-False, and Pick-N) were only used by fewer dental schools (≤67%, up to 18 out of 27 dental schools). For the multiple-select item types, the applied scoring methods varied considerably (ie, awarding [intermediate] partial credit and requirements for partial credit). Dental schools with the possibility of electronic examinations used multiple-select items slightly more often (14/19, 74% vs 4/8, 50%). However, this difference was statistically not significant (P=.38). Dental schools used items either individually or as key feature problems consisting of a clinical case scenario followed by a number of items focusing on critical treatment steps (15/27, 56%). Not a single school used alternative testing methods (eg, answer-until-correct). A formal item review process was established at about half of the dental schools (15/27, 56%). Summative assessment methods among German dental schools vary widely. Especially, a large variability regarding the use and scoring of multiple-select multiple-choice items was found.

ChatENT: Augmented Large Language Model for Expert Knowledge Retrieval in Otolaryngology-Head and Neck Surgery.

The recent surge in popularity of large language models (LLMs), such as ChatGPT, has showcased their proficiency in medical examinations and potential applications in health care. However, LLMs possess inherent limitations, including inconsistent accuracy, specific prompting requirements, and the risk of generating harmful hallucinations. A domain-specific model might address these limitations effectively. Developmental design. Virtual. Otolaryngology-head and neck surgery (OHNS) relevant data were systematically gathered from open-access Internet sources and indexed into a knowledge database. We leveraged Retrieval-Augmented Language Modeling to recall this information and utilized it for pretraining, which was then integrated into ChatGPT4.0, creating an OHNS-specific knowledge question & answer platform known as ChatENT. The model is further tested on different types of questions. ChatENT showed enhanced performance in the analysis and interpretation of OHNS information, outperforming ChatGPT4.0 in both the Canadian Royal College OHNS sample examination questions challenge and the US board practice questions challenge, with a 58.4% and 26.0% error reduction, respectively. ChatENT generated fewer hallucinations and demonstrated greater consistency. To the best of our knowledge, ChatENT is the first specialty-specific knowledge retrieval artificial intelligence inthe medical field that utilizes the latest LLM. It appears to have considerable promise in areas such as medical education, patient education, and clinical decision support. The model has demonstrated the capacity to overcome the limitations of existing LLMs, thereby signaling a future of more precise, safe, and user-friendly applications in the realm of OHNS and other medical fields.

Performance of Large Language Models on Medical Oncology Examination Questions

Large language models (LLMs) recently developed an unprecedented ability to answer questions. Studies of LLMs from other fields may not generalize to medical oncology, a high-stakes clinical setting requiring rapid integration of new information. To evaluate the accuracy and safety of LLM answers on medical oncology examination questions. This cross-sectional study was conducted between May 28 and October 11, 2023. The American Society of Clinical Oncology (ASCO) Oncology Self-Assessment Series on ASCO Connection, the European Society of Medical Oncology (ESMO) Examination Trial questions, and an original set of board-style medical oncology multiple-choice questions were presented to 8 LLMs. The primary outcome was the percentage of correct answers. Medical oncologists evaluated the explanations provided by the best LLM for accuracy, classified the types of errors, and estimated the likelihood and extent of potential clinical harm. Proprietary LLM 2 correctly answered 125 of 147 questions (85.0%; 95% CI, 78.2%-90.4%; P < .001 vs random answering). Proprietary LLM 2 outperformed an earlier version, proprietary LLM 1, which correctly answered 89 of 147 questions (60.5%; 95% CI, 52.2%-68.5%; P < .001), and the best open-source LLM, Mixtral-8x7B-v0.1, which correctly answered 87 of 147 questions (59.2%; 95% CI, 50.0%-66.4%; P < .001). The explanations provided by proprietary LLM 2 contained no or minor errors for 138 of 147 questions (93.9%; 95% CI, 88.7%-97.2%). Incorrect responses were most commonly associated with errors in information retrieval, particularly with recent publications, followed by erroneous reasoning and reading comprehension. If acted upon in clinical practice, 18 of 22 incorrect answers (81.8%; 95% CI, 59.7%-94.8%) would have a medium or high likelihood of moderate to severe harm. In this cross-sectional study of the performance of LLMs on medical oncology examination questions, the best LLM answered questions with remarkable performance, although errors raised safety concerns. These results demonstrated an opportunity to develop and evaluate LLMs to improve health care clinician experiences and patient care, considering the potential impact on capabilities and safety.

Out-of-Hospital Birth in Medical Education: A Thematic Analysis of Question Banks for Medical Licensing Examinations in the United States.

This study assessed the content of US Medical Licensing Examination question banks with regard to out-of-hospital births and whether the questions aligned with current evidence. Three question banks were searched for key words regarding out-of-hospital births. A thematic analysis was then utilized to analyze the results. Forty-seven questions were identified, and of these, 55% indicated a lack of inadequate, limited, or irregular prenatal care in the question stem. Systematic studies comparing prenatal care in out-of-hospital births versus hospital births are nonexistent, leading to the potential for bias and adverse outcomes. Adjustments to question stems that accurately portray current evidence are recommended.