Generative Pre-trained Transformer Research Articles

Conversational LLM Chatbot ChatGPT-4 for Colonoscopy Boston Bowel Preparation Scoring: An Artificial Intelligence-to-Head Concordance Analysis

Background/objectives:To date, no studies have evaluated Chat Generative Pre-Trained Transformer (ChatGPT) as a large language model chatbot in optical applications for digestive endoscopy images. This study aimed to weigh the performance of ChatGPT-4 in assessing bowel preparation (BP) quality for colonoscopy. Methods: ChatGPT-4 analysed 663 anonymised endoscopic images, scoring each according to the Boston BP scale (BBPS). Expert physicians scored the same images subsequently. Results: ChatGPT-4 deemed 369 frames (62.9%) to be adequately prepared (i.e., BBPS > 1) compared to 524 frames (89.3%) assessed by human assessors. The agreement was slight (κ: 0.099, p = 0.0001). The raw human BBPS score was higher at 3 (2–3) than that of ChatGPT-4 at 2 (1–3), demonstrating moderate concordance (W: 0.554, p = 0.036). Conclusions: ChatGPT-4 demonstrates some potential in assessing BP on colonoscopy images, but further refinement is still needed.

Transformative Assessment Review

The rapid advancement of generative artificial intelligence (AI) in education and the workforce necessitates that students acquire critical competencies, including digital literacy, data integrity, and ethical AI use, to navigate an increasingly AI-driven world. Equally, academic staff must be upskilled to effectively guide students in the ethical and practical applications of AI. This project examines how a private higher education (HE) institute has modified its Quality Assessment Framework (QAF) to incorporate an AI competence dimension, improve HE assessment and foster students’ competencies. This new dimension complements the existing QAF elements—Intellectual Quality, Significance, and Student Support to ensure a holistic approach in preparing students for AI-integrated academic and professional environments. This project’s framework and innovative tools, such as the enhanced QAF and an AI-driven Generative Pre-trained Transformer (GPT) assessment review tool, support educators in adapting curriculum and assessment design, and empower them to integrate AI competencies seamlessly into the curriculum. The development of these tools drew upon established frameworks, particularly the QAF (Gore et al., 2009) and the AI Assessment Scale (AIAS) by Perkins et al. (2024), which categorises AI usage across five levels, guides assessment reviews and ensures consistent standards for AI integration in student work. A trial, conducted across selected disciplines at the institute, involved academic staff as assessment reviewers and utilised the GPT tool to streamline the assessment review process by aiding in feedback provision, coding, and suggestions for improvement. Evaluation of the tool includes qualitative and quantitative methodologies, gathering academic feedback on usability, clarity, and effectiveness, as well as comparative studies to assess review time and quality before and after GPT integration. Usability testing evaluates workflow compatibility, while academic performance data provide insights into the tool’s impact on student outcomes. Initial results on the tool’s success in enhancing assessment alignment with institutional goals and fostering a comprehensive understanding of AI competence were also examined. Developing and implementing these tools has presented challenges, including the complexity of aligning them with diverse curriculum needs across disciplines, the time investment required for staff training, and the necessity of continuous updates to keep pace with AI technology. Addressing these challenges has been crucial to ensuring the effectiveness and sustainability of these solutions. This digital poster provides insights into the project, covering its development process, trial outcomes, challenges encountered, and future directions for integrating AI competence in higher education assessments.

Performance of ChatGPT in emergency medicine residency exams in Qatar: A comparative analysis with resident physicians

Introduction: The inclusion of artificial intelligence (AI) in the healthcare sector has transformed medical practices by introducing innovative techniques for medical education, diagnosis, and treatment strategies. In medical education, the potential of AI to enhance learning and assessment methods is being increasingly recognized. This study aims to evaluate the performance of OpenAI’s Chat Generative Pre-Trained Transformer (ChatGPT) in emergency medicine (EM) residency examinations in Qatar and compare it with the performance of resident physicians. Methods: A retrospective descriptive study with a mixed-methods design was conducted in August 2023. EM residents’ examination scores were collected and compared with the performance of ChatGPT on the same examinations. The examinations consisted of multiple-choice questions (MCQs) from the same faculty responsible for Qatari Board EM examinations. ChatGPT’s performance on these examinations was analyzed and compared with residents across various postgraduate years (PGY). Results: The study included 238 emergency department residents from PGY1 to PGY4 and compared their performances with ChatGPT. ChatGPT scored consistently higher than resident groups in all examination categories. However, a notable decline in passing rates was observed among senior residents, indicating a potential misalignment between examination performance and practical competencies. Another likely reason can be the impact of the COVID-19 pandemic on their learning experience, knowledge acquisition, and consolidation. Conclusion: ChatGPT demonstrated significant proficiency in the theoretical knowledge of EM, outperforming resident physicians in examination settings. This finding suggests the potential of AI as a supplementary tool in medical education.

Global insights: ChatGPT's influence on academic and research writing, creativity, and plagiarism policies

IntroductionThe current study explored the influence of Chat Generative Pre-Trained Transformer (ChatGPT) on the concepts, parameters, policies, and practices of creativity and plagiarism in academic and research writing.MethodsData were collected from 10 researchers from 10 different countries (Australia, China, the UK, Brazil, Pakistan, Bangladesh, Iran, Nigeria, Trinidad and Tobago, and Turkiye) using semi-structured interviews. NVivo was employed for data analysis.ResultsBased on the responses, five themes about the influence of ChatGPT on academic and research writing were generated, i.e., opportunity, human assistance, thought-provoking, time-saving, and negative attitude. Although the researchers were mostly positive about it, some feared it would degrade their writing skills and lead to plagiarism. Many of them believed that ChatGPT would redefine the concepts, parameters, and practices of creativity and plagiarism.DiscussionCreativity may no longer be restricted to the ability to write, but also to use ChatGPT or other large language models (LLMs) to write creatively. Some suggested that machine-generated text might be accepted as the new norm; however, using it without proper acknowledgment would be considered plagiarism. The researchers recommended allowing ChatGPT for academic and research writing; however, they strongly advised it to be regulated with limited use and proper acknowledgment.

Artificial intelligence as a modality to enhance the readability of neurosurgical literature for patients.

In this study the authors assessed the ability of Chat Generative Pretrained Transformer (ChatGPT) 3.5 and ChatGPT4 to generate readable and accurate summaries of published neurosurgical literature. Abstracts published in journal issues released between June 2023 and August 2023 (n = 150) were randomly selected from the top 5 ranked neurosurgical journals according to Google Scholar. ChatGPT models were instructed to generate a readable layperson summary of the original abstract from a statistically validated prompt. Readability results and grade-level indicators (RR-GLIs) scores were calculated for GPT3.5- and GPT4-generated summaries and original abstracts. Two physicians independently rated the accuracy of ChatGPT-generated layperson summaries to assess scientific validity. One-way ANOVA followed by pairwise t-test with Bonferroni correction were performed to compare readability scores. Cohen's kappa was used to assess interrater agreement between the two rater physicians. Analysis of 150 original abstracts showed a statistically significant difference for all RR-GLIs between the ChatGPT-generated summaries and original abstracts. The readability scores are formatted as follows (original abstract mean, GPT3.5 summary mean, GPT4 summary mean, p value): Flesch-Kincaid reading grade (12.55, 7.80, 7.70, p < 0.0001); Gunning fog score (15.46, 10.00, 9.00, p < 0.0001); Simple Measure of Gobbledygook (SMOG) index (11.30, 7.13, 6.60, p < 0.0001); Coleman-Liau index (14.67, 11.32, 10.26, p < 0.0001); automated readability index (10.87, 8.50, 7.75, p < 0.0001); and Flesch-Kincaid reading ease (33.29, 68.45, 69.55, p < 0.0001). GPT4-generated summaries demonstrated higher RR-GLIs than GPT3.5-generated summaries in the following categories: Gunning fog score (0.0003); SMOG index (0.027); Coleman-Liau index (< 0.0001); sentences (< 0.0001); complex words (< 0.0001); and % complex words (0.0035). A total of 68.4% and 84.2% of GPT3.5- and GPT4-generated summaries, respectively, maintained moderate scientific accuracy according to the two physician-reviewers. The findings demonstrate promising potential for application of the ChatGPT in patient education. GPT4 is an accessible tool that can be an immediate solution to enhancing the readability of current neurosurgical literature. Layperson summaries generated by GPT4 would be a valuable addition to a neurosurgical journal and would be likely to improve comprehension for patients using internet resources like PubMed.

Assessing the need for coronary angiography in high-risk non-ST-elevation acute coronary syndrome patients using artificial intelligence and computed tomography.

This study aimed to evaluate the efficacy of the Chat Generative Pre-trained Transformer (ChatGPT) in guiding the need for invasive coronary angiography (ICA) in high-risk non-ST-elevation (NSTE) acute coronary syndrome (ACS) patients based on both standard clinical data and coronary computed tomography angiography (CCTA) findings. This investigation is a sub-study of a larger prospective multicentric double blinded project where high-risk NSTE-ACS patients underwent CCTA prior to ICA to compare coronary lesion by both modalities. ChatGPT analyzed clinical vignettes containing patient data, electrocardiograms, troponin levels, and CCTA results to determine the necessity of ICA. The AI's recommendations were then compared to actual ICA findings to assess its decision-making accuracy. In total, 86 patients (age: 62 ± 13 years old, female 27%) were included. ChatGPT recommended against ICA for 19 patients, 16 of whom indeed had no significant findings. For 67 patients, ChatGPT advised proceeding with ICA, and a significant lesion was confirmed in 58 of them. Consequently, ChatGPT's overall accuracy stood at 86%, with a sensitivity of 95% (95% confidence interval (CI) 0.76-0.92) and a specificity of 64% (95% CI 0.62-0.94). The model's negative predictive value was 84% (95% CI 0.44-0.79), and its positive predictive value was 87% 95% CI 0.86-0.97). Preliminary evidence suggests that ChatGPT can effectively assist in making ICA decisions for high-risk NSTE-ACS patients, potentially reducing unnecessary procedures. However, the study underscores the importance of data accuracy and calls for larger, more diverse investigations to refine artificial intelligence's role in clinical decision-making.

Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools.

Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5,213 case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to three generative pretrained transformer (GPT) models. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.

Unravelling ChatGPT's potential in summarising qualitative in-depth interviews.

Qualitative research can be laborious and time consuming, presenting a challenge for practitioners and policymakers seeking rapid, actionable results. Data collection, transcription and analysis are the main contributors to the resource-intensive nature. OpenAI's Chat Generative Pre-trained Transformer (ChatGPT), have demonstrated potential to aid in data analysis. Our study aimed to compare themes generated by ChatGPT (3.5 and 4.0) with traditional human analysis from in-depth interviews. Three transcripts from an evaluation study to understand patients' experiences at a community eye clinic were used. Transcripts were first analysed by an independent researcher. Next, specific aims, instructions and de-identified transcripts were uploaded to ChatGPT 3.5 and ChatGPT 4.0. Concordance in the themes was calculated as the number of themes generated by ChatGPT divided by the number of themes generated by the researcher. The number of unrelated subthemes and time taken by both ChatGPT were also described. The average time taken per transcript was 11.5 min, 11.9 min and 240 min for ChatGPT 3.5, ChatGPT 4.0 and researcher respectively. Six themes were identified by the researcher: (i) clinic's accessibility, (ii) patients' awareness, (iii) trust and satisfaction, (iv) patients' expectations, (v) willingness to return and (vi) explanation of the clinic by referral source. Concordance for ChatGPT 3.5 and 4.0 ranged from 66 to 100%. Preliminary results showed that ChatGPT significantly reduced analysis time with moderate to good concordance compared with current practice. This highlighted the potential adoption of ChatGPT to facilitate rapid preliminary analysis. However, regrouping of subthemes will still need to be conducted by a researcher.

Utility of Generative Artificial Intelligence for Patient Care Counseling for Mandibular Fractures.

To determine the readability and accuracy of information regarding mandible fractures generated by Chat Generative Pre-trained Transformer (ChatGPT) versions 3.5 and 4o. Patients are increasingly turning to generative artificial intelligence to answer medical queries. To date, the accuracy and readability of responses regarding mandible fractures have not been assessed. Twenty patient questions regarding mandible fractures were developed by querying AlsoAsked (https://alsoasked.com), SearchResponse (https://searchresponse.io), and Answer the Public (https://answerthepublic.com/). Questions were posed to ChatGPT 3.5 and 4o. Readability was assessed by calculating the Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, number of sentences, and percentage of complex words. Accuracy was assessed by a board-certified facial plastic and reconstructive otolaryngologist using a 5-point Likert Scale. No significant differences were observed between the two versions for readability or accuracy. Readability was above recommended levels for patient education materials. Accuracy was low, and a majority of responses were deemed inappropriate for patient use with multiple inaccuracies and/or missing information. ChatGPT produced responses written at a high level inappropriate for the average patient, in addition to containing several inaccurate statements. Patients and clinicians should be aware of the limitations of generative artificial intelligence when seeking medical information regarding mandible fractures.

High performance P300 spellers using GPT2 word prediction with cross-subject training

ABSTRACT Amyotrophic lateral sclerosis (ALS) severely impairs patients’ ability to communicate, often leading to a decline in their quality of life within a few years of diagnosis. The P300 speller brain–computer interface (BCI) offers an alternative communication method by interpreting a subject’s EEG response to flashing characters presented on a grid interface. This paper addresses the common speed limitations encountered in training efficient P300-based multi-subject classifiers by introducing innovative ‘across-subject’ classifiers. We leverage a combination of the second-generation Generative Pre-Trained Transformer (GPT2) and Dijkstra’s algorithm to optimize stimuli and suggest word completion choices based on subjects’ typing history. Additionally, we employ a multi-layered smoothing technique to accommodate out-of-vocabulary (OOV) words. Through extensive simulations employing random sampling of EEG data from multiple subjects, we demonstrate significant speed enhancements in typing passages containing rare and OOV words. These optimizations result in approximately 10 % improvement in character-level typing speed and up to 40 % improvement in multi-word prediction. We demonstrate that augmenting standard row/column highlighting techniques with layered word prediction yields close-to-optimal performance. Furthermore, we explore both ‘within-subject’ and ‘across-subject’ training techniques, showing that speed improvements are consistent across both approaches.

Can GPT-4 learn to analyse moves in research article abstracts?

Abstract One of the most powerful and enduring ideas in written discourse analysis is that genres can be described in terms of the moves which structure a writer’s purpose. Considerable research has sought to identify these distinct communicative acts, but analyses have been beset by problems of subjectivity, reliability, and the time-consuming need for multiple coders to confirm analyses. In this article, we employ the affordances of Generative Pre-trained Transformer 4 (GPT-4) to automate the annotation process by using natural language prompts. Focusing on abstracts from articles in four applied linguistics journals, we devise prompts which enable the model to identify moves effectively. The annotated outputs of these prompts were evaluated by two assessors with a third addressing disagreements. The results show that an eight-shot prompt was more effective than one using two, confirming that the inclusion of examples illustrating areas of variability can enhance GPT-4’s ability to recognize multiple moves in a single sentence and reduce bias related to textual position. We suggest that GPT-4 offers considerable potential in automating this annotation process, when human actors with domain-specific linguistic expertise inform the prompting process.

Ontology-assisted GPT-based building performance simulation and assessment: Implementation of multizone airflow simulation

Building performance simulation (BPS) is crucial for building performance assessments across its lifecycle. However, the complexity of buildings and the iterative nature of simulation poses challenges, leading to high costs and low values. Previous studies focused on simplification, but did not fully utilize advanced simulation engines. Despite recent advancements, there is a lack of research on leveraging artificial intelligence (AI), specifically generative pre-trained transformer (GPT), for BPS. Therefore, this study proposes a GPT-based BPS system, enhancing simulation efficiency and value by integrating simulation engines and advanced data analytics in the GPT environment. The ontology for GPT-based BPS is also developed to enable comprehensive, reliable, informative BPS environments. Based on this framework, case studies were conducted for GPT-based multizone airflow network simulation in a high-rise residential building using CONTAM software. They demonstrate GPT’s capabilities in retrieving simulation data, visualizing results with data mining, answering questions based on building knowledge, checking compliance with design guidelines, and proposing design alternatives. Finally, this study emphasizes expert interventions with ontological engineering informatics to utilize strictly structured BPS engines.

Enhancing rumor detection with data augmentation and generative pre-trained transformer

The advent of social networks has facilitated the rapid dissemination of false information, including rumors, leading to significant societal and individual damages. Extensive research has been dedicated to rumor detection, ranging from machine learning techniques to neural networks. However, the existing methods could not learn the deep concepts of the rumor text to detect the rumor. In addition, imbalanced datasets in the rumor domain reduce the effectiveness of these algorithms. This study addresses this challenge by leveraging the Generative Pre-trained Transformer 2 (GPT-2) model to generate rumor-like texts, thus creating a balanced dataset. Subsequently, a novel approach for classifying rumor texts is proposed by modifying the GPT-2 model. We compare our results with state-of-art machine learning and deep learning methods as well as pre-trained models on the PHEME, Twitter15, and Twitter16 datasets. Our findings demonstrate that the proposed model, implementing advanced artificial intelligence techniques, has improved accuracy and F-measure in the application of detecting rumors compared to previous methods.

Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans

The introduction of Artificial Intelligence (AI) generative language models such as GPT (Generative Pre-trained Transformer) and conversational tools such as ChatGPT has triggered a revolution that can transform how text is generated. This has many implications, for example, as AI-generated text becomes a significant fraction of the text, would this affect the language capabilities of readers and also the training of newer AI tools? Would it affect the evolution of languages? Focusing on one specific aspect of the language: words; will the use of tools such as ChatGPT increase or reduce the vocabulary used or the lexical diversity? This has implications for words, as those not included in AI-generated content will tend to be less and less popular and may eventually be lost. In this work, we perform an initial comparison of the vocabulary and lexical diversity of ChatGPT and humans when performing the same tasks. In more detail, two datasets containing the answers to different types of questions answered by ChatGPT and humans, and a third dataset in which ChatGPT paraphrases sentences and questions are used. The analysis shows that ChatGPT-3.5 tends to use fewer distinct words and lower diversity than humans while ChatGPT-4 has a similar lexical diversity as humans and in some cases even larger. These results are very preliminary and additional datasets and ChatGPT configurations have to be evaluated to extract more general conclusions. Therefore, further research is needed to understand how the use of ChatGPT and more broadly generative AI tools will affect the vocabulary and lexical diversity in different types of text and languages.

Metadata schema for virtual building models in digital twins: VB schema implemented in GPT-based applications

A virtual building model (VBM) is a virtual entity that represents the physical behavior of a target building mathematically within a digital twin environment. The creation and synchronization of a VBM are achieved by utilizing various interrelated virtual sub-models, including behavior, correction, and distance models. To achieve continuous digital twinning, it is essential to manage the VBM with virtual sub-models. However, these metadata schemas have limitations in describing VBMs representing operational building behaviors within the concept of building digital twins (DTs). Therefore, this study proposes a novel metadata schema, termed the virtual building model metadata schema (VB schema), to represent and manage VBMs in DT-built environments. The VB schema is established according to the mathematical and semantic ontology of the in-situ modeling and calibration approach for constructing and correcting virtual models during building operations, and it is linked to physical entities, data, and applications within DTs. Specifically, it involves: (1) determining classes for operational data and virtual models; (2) establishing relationships for interactions between model and data entities, between model classes, between model and physical entities, and between model and applications; (3) defining properties for each class of models; and (4) extending into the exiting metadata schema of Brick. To demonstrate the proposed VB schema, a virtual model describing supply pressure behaviors in a central heating system was developed and represented using the VB schema for DT-enabled building operations. Additionally, the VB schema was utilized for implementing generative pre-trained transformer (GPT)-based DT applications, which highlights its benefits in enhancing ontology comprehension of DTs in the context of VBMs, improving autonomous problem-solving capabilities in real building systems, and providing better interpretation of application results compared to cases where only the Brick schema was used. The VB schema is expected to enable continuous and autonomous in-situ management of VBMs for intelligent building services within the DT.

Exploring Korean Medicine professions' perspectives on the applicability of ChatGPT in facial palsy practice: A web-based survey

IntroductionSince November 2022, when Chat generative pre-trained transformer (ChatGPT) was released, studies have been conducted to explore its potential use in various text-based areas. In the field of Korean Medicine (KM), studies evaluating the feasibility of ChatGPT are mainly focused on the educational domain. This study aimed to 1) identify the current level of ChatGPT-generated responses in answering questions from patients and KM practitioners, specifically exemplified by facial palsy and 2) explore the potential applicability of ChatGPT in KM practice through a perception survey of experts. MethodsThis study evaluates the applicability of ChatGPT concerning facial palsy through a survey of board-certified acupuncturists. The survey comprises two parts: assessing response quality for patients and for KM doctors in clinical practice. Ethical approval was obtained, and a 36-item questionnaire was administered online to 33 participants. Responses were analyzed for quality, relevance, and applicability using a combination of Likert scales and a standardized assessment tool. ResultsAmong 33 eligible participants, 30 board-certified acupuncturists agreed to participate (response rate: 90.9%). Survey results showed that the applicability of ChatGPT was lower for specialized KM practitioner inquiries than for general patient inquiries. Although the responses were considered useful to the readers (part 1: 96.7%, part 2: 63.3%) and understandable (part 1: 66.7%, part 2: 60%), the proportion of positive evaluations was relatively low in the domains assessing reliability and sufficiency. ConclusionsThough ChatGPT is generally viewed positively for its potential utility, its direct application in KM clinical practice treating facial palsy at its current level seems challenging. To improve reliability, the information generated by ChatGPT should be critically reviewed by qualified expert medical personnel. To enhance sufficiency, further training of the artificial intelligence model with additional KM information is required.

Assessing the Role of the Generative Pretrained Transformer (GPT) in Alzheimer's Disease Management: Comparative Study of Neurologist- and Artificial Intelligence-Generated Responses.

Alzheimer's disease (AD) is a progressive neurodegenerative disorder posing challenges to patients, caregivers, and society. Accessible and accurate information is crucial for effective AD management. This study aimed to evaluate the accuracy, comprehensibility, clarity, and usefulness of the Generative Pretrained Transformer's (GPT) answers concerning the management and caregiving of patients with AD. In total, 14 questions related to the prevention, treatment, and care of AD were identified and posed to GPT-3.5 and GPT-4 in Chinese and English, respectively, and 4 respondent neurologists were asked to answer them. We generated 8 sets of responses (total 112) and randomly coded them in answer sheets. Next, 5 evaluator neurologists and 5 family members of patients were asked to rate the 112 responses using separate 5-point Likert scales. We evaluated the quality of the responses using a set of 8 questions rated on a 5-point Likert scale. To gauge comprehensibility and participant satisfaction, we included 3 questions dedicated to each aspect within the same set of 8 questions. As of April 10, 2023, the 5 evaluator neurologists and 5 family members of patients with AD rated the 112 responses: GPT-3.5: n=28, 25%, responses; GPT-4: n=28, 25%, responses; respondent neurologists: 56 (50%) responses. The top 5 (4.5%) responses rated by evaluator neurologists had 4 (80%) GPT (GPT-3.5+GPT-4) responses and 1 (20%) respondent neurologist's response. For the top 5 (4.5%) responses rated by patients' family members, all but the third response were GPT responses. Based on the evaluation by neurologists, the neurologist-generated responses achieved a mean score of 3.9 (SD 0.7), while the GPT-generated responses scored significantly higher (mean 4.4, SD 0.6; P<.001). Language and model analyses revealed no significant differences in response quality between the GPT-3.5 and GPT-4 models (GPT-3.5: mean 4.3, SD 0.7; GPT-4: mean 4.4, SD 0.5; P=.51). However, English responses outperformed Chinese responses in terms of comprehensibility (Chinese responses: mean 4.1, SD 0.7; English responses: mean 4.6, SD 0.5; P=.005) and participant satisfaction (Chinese responses: mean 4.2, SD 0.8; English responses: mean 4.5, SD 0.5; P=.04). According to the evaluator neurologists' review, Chinese responses had a mean score of 4.4 (SD 0.6), whereas English responses had a mean score of 4.5 (SD 0.5; P=.002). As for the family members of patients with AD, no significant differences were observed between GPT and neurologists, GPT-3.5 and GPT-4, or Chinese and English responses. GPT can provide patient education materials on AD for patients, their families and caregivers, nurses, and neurologists. This capability can contribute to the effective health care management of patients with AD, leading to enhanced patient outcomes.

Accuracy and Completeness of Bard and Chat-GPT 4 Responses for Questions Derived from the International Consensus Statement on Endoscopic Skull-Base Surgery 2019

AbstractArtificial intelligence large language models (LLMs), such as Chat Generative Pre-Trained Transformer 4 (Chat-GPT) by OpenAI and Bard by Google, emerged in 2022 as tools for answering questions, providing information, and offering suggestions to the layperson. These LLMs impact how information is disseminated and it is essential to compare their answers to experts in the corresponding field. The International Consensus Statement on Endoscopic Skull-Base Surgery 2019 (ICAR:SB) is a multidisciplinary international collaboration that critically evaluated and graded the current literature. Objectives Evaluate the accuracy and completeness of Chat-GPT and Bard responses to questions derived from the ICAR:SB policy statements. Design Thirty-four questions were created based on ICAR:SB policy statements and input into Chat-GPT and Bard. Two rhinologists and two neurosurgeons graded the accuracy and completeness of LLM responses, using a 5-point Likert scale. The Wilcoxon rank-sum and Kruskal–Wallis tests were used for analysis. Setting Online. Participants None. Outcomes Compare the mean accuracy and completeness scores between (1) responses generated by Chat-GPT versus Bard and (2) rhinologists versus neurosurgeons. Results Using the Wilcoxon rank-sum test, there were statistically significant differences in (1) accuracy (p &lt; 0.001) and completeness (p &lt; 0.001) of Chat-GPT compared with Bard; and (2) accuracy (p &lt; 0.001) and completeness (p &lt; 0.001) ratings between rhinologists and neurosurgeons. Conclusion Chat-GPT responses are overall more accurate and complete compared with Bard, but both are very accurate and complete. Overall, rhinologists graded lower than neurosurgeons. Further research is needed to better understand the full potential of LLMs.

Evaluating large language models in theory of mind tasks

Eleven large language models (LLMs) were assessed using 40 bespoke false-belief tasks, considered a gold standard in testing theory of mind (ToM) in humans. Each task included a false-belief scenario, three closely matched true-belief control scenarios, and the reversed versions of all four. An LLM had to solve all eight scenarios to solve a single task. Older models solved no tasks; Generative Pre-trained Transformer (GPT)-3-davinci-003 (from November 2022) and ChatGPT-3.5-turbo (from March 2023) solved 20% of the tasks; ChatGPT-4 (from June 2023) solved 75% of the tasks, matching the performance of 6-y-old children observed in past studies. We explore the potential interpretation of these results, including the intriguing possibility that ToM-like ability, previously considered unique to humans, may have emerged as an unintended by-product of LLMs' improving language skills. Regardless of how we interpret these outcomes, they signify the advent of more powerful and socially skilled AI-with profound positive and negative implications.

A guideline-informed language model for paediatric cardiology demonstrates high performance in answering complex medical questions

Abstract Background Paediatric cardiology presents unique challenges with its diverse and complex cases, limited evidence base, and the necessity for multi-expert involvement in decision-making processes. In this context, the introduction of generative pre-trained transformer (GPT) based large language models (LLMs) offers a potential avenue for the provision of complex information and clinical decision support. Purpose This study evaluates the quality of three different GPT LLMs in answering complex medical questions, including a state-of-the-art preview model that incorporates the German paediatric cardiology guidelines. Methods Seven paediatric cardiologists and paediatric cardiac surgeons generated 72 questions, including complex questions and medical cases with associated questions. The questions were categorized by difficulty and required knowledge (factual and experience-based or mostly experience-based). We prompted the questions to three LLMs: GPT 3.5, GPT 4 and a GPT 4 turbo preview. The GPT 4 turbo preview was customized by incorporating all guidelines from the German Society for Paediatric Cardiology by a retrieval function. Employing one complex instruction for all questions, we prompted the LLMs to provide precise and detailed expert-level responses. The responses from each model were evaluated by experts based on relevance, factual accuracy, severity of possible harm, completeness, superfluous content, and age-related appropriateness from 0 (very bad) to 7 (very good). Differences were calculated using the Kruskal-Wallis-test in SPSS Version 28. Results Our findings indicated a good performance of all models regarding the dimensions tested. The figures show the average ratings (Figure 1, Figure 2A) and highlight significant differences after Bonferroni correction in bold (Figure 2B). The GPT 4 turbo preview, including the retrieval of guidelines, provided significantly more relevant (average rating [AR] 5.94, meaning mostly relevant), accurate (AR 5.6, meaning between somewhat and mostly accurate) and complete (AR 5, meaning fairly complete) answers compared to GPT 3.5 and GPT 4. In terms of difficulty levels or the type of questions, there was no significant difference in rating. Relevance ratings were slightly better in factual questions (AR 5.7) than in those requiring more experience-based knowledge (AR 5,3). Although GPT4 had higher average scores compared to GPT 3.5 in all dimensions except superfluous content, the differences in rating were not statistically significant. All models had relevant difficulties considering the age-related aspects of the questions (AR 4.06-4.45, p=0.455). Conclusion This study highlights the potential and limitations of AI language models in addressing complex medical questions in fields characterized by highly individualized decision-making scenarios. The findings advocate for the development of more specialized AI tools in medicine, tailored to specific medical fields and patient age groups.Fig 1:Average ratings of LLMsFig 2:Rating differences between LLMs