Generative Pre-trained Transformer Research Articles

A Guide to Network Meta-Analysis Using Generative AI and No-Code Tools

Network meta-analysis (NMA), an increasingly appealing method of statistical analysis, is superior to traditional analysis methods in terms of being able to compare multiple medical treatment methods in one analysis run. In recent years, the prevalence of NMA in the medical literature has increased significantly, while advances in NMA-related statistical methods and software tools continue to improve the effectiveness of this approach. Various commercial and free statistical software packages, some of which employ generative artificial intelligence (GAI) to generate code, have been developed for NMA, leading to numerous innovative developments. In this paper, the use of generative AI for writing R programming language scripts and the netmeta package for performing NMA are introduced. Also, the web-based tool ShinyNMA is introduced. ShinyNMA allows users to conduct NMA using an intuitive "clickable" interface accessible via a standard web browser, with visual charts employed to present results. In the first section, we detail the netmeta package documentation and use ChatGPT (chat generative pre-trained transformer) to write the R scripts necessary to perform NMA with the netmeta package. In the second section, a user interface is developed using the Shiny package to create a ShinyNMA tool. This tool provides a no-code option for users unfamiliar with programming to conduct NMA statistical analysis and plotting. With appropriate prompts, ChatGPT can produce R scripts capable of performing NMA. Using this approach, an NMA analysis tool is developed that meets the research objectives, and potential applications are demonstrated using sample data. Using generative AI and existing statistical packages or no-code tools is expected to make conducting NMA analysis significantly easier for researchers. Moreover, greater access to results generated by NMA analyses will enable decision-makers to review analysis results intuitively in real-time, enhancing the importance of statistical analysis in medical decision-making. Furthermore, enabling non-specialists to conduct clinically meaningful systematic reviews may be expected to sustainably improve analytical capabilities and produce higher-quality evidence.

ChatGPT-4 extraction of heart failure symptoms and signs from electronic health records

BackgroundNatural language processing (NLP) can facilitate research utilizing data from electronic health records (EHRs). Large language models can potentially improve NLP applications leveraging EHR notes. The objective of this study was to assess the performance of zero-shot learning using Chat Generative Pre-trained Transformer 4 (ChatGPT-4) for extraction of symptoms and signs, and compare its performance to baseline machine learning and rule-based methods developed using annotated data. Methods and resultsFrom unstructured clinical notes of the national EHR data on the Veterans healthcare system, we extracted 1999 text snippets containing relevant keywords for heart failure symptoms and signs, which were then annotated by two clinicians. We also created 102 synthetic snippets that were semantically similar to snippets randomly selected from the original 1999 snippets. The authors applied zero-shot learning, using two different forms of prompt engineering in a symptom and sign extraction task with ChatGPT-4, utilizing the synthetic snippets. For comparison, baseline models using machine learning and rule-based methods were trained using the original 1999 annotated text snippets, and then used to classify the 102 synthetic snippets.The best zero-shot learning application achieved 90.6 % precision, 100 % recall, and 95 % F1 score, outperforming the best baseline method, which achieved 54.9 % precision, 82.4 % recall, and 65.5 % F1 score. Prompt style and temperature settings influenced zero-shot learning performance. ConclusionsZero-shot learning utilizing ChatGPT-4 significantly outperformed traditional machine learning and rule-based NLP. Prompt type and temperature settings affected zero-shot learning performance. These findings suggest a more efficient means of symptoms and signs extraction than traditional machine learning and rule-based methods.

Medication Extraction and Drug Interaction Chatbot: Generative Pretrained Transformer-Powered Chatbot for Drug-Drug Interaction

ObjectiveTo assist individuals, particularly cancer patients or those with complex comorbidities, in quickly identifying potentially contraindicated medications when taking multiple drugs simultaneously. Patients and MethodsIn this study, we introduce the Medication Extraction and Drug Interaction Chatbot (MEDIC), an artificial intelligence system that integrates optical character recognition and Chat generative pretrained transformer through the Langchain framework. Medication Extraction and Drug Interaction Chatbot starts by receiving 2 drug bag images from the patient. It uses optical character recognition and text similarity techniques to extract drug names from the images. The extracted drug names are then processed through Chat generative pretrained transformer and Langchain to provide the user with information about drug contraindications. The MEDIC responds to the user with clear and concise sentences to ensure the information is easily understandable. This research was conducted from July 2022 to April 2024. ResultsThis streamlined process enhances the accuracy of drug-drug interaction detection, providing a crucial tool for health care professionals and patients to improve medication safety. The proposed system was validated through rigorous evaluation using real-world data, reporting high accuracy in drug-drug interaction identification and highlighting its potential to benefit medication management practices considerably. ConclusionBy implementing MEDIC, contraindicated medications can be identified using only medication packaging, and users can be alerted to potential drug adverse effects, thereby contributing to advancements in patient care in clinical settings.

Increasing Generative Artificial Intelligence Competency among Students Enrolled in Doctoral Nursing Research Coursework.

Generative artificial intelligence (AI) tools may soon be integrated into health care practice and research. Nurses in leadership roles, many of whom are doctorally prepared, will need to determine whether and how to integrate them in a safe and useful way. This study aimed to develop and evaluate a brief intervention to increase PhD nursing students' knowledge of appropriate applications for using generative AI tools in health care. We created didactic lectures and laboratory-based activities to introduce generative AI to students enrolled in a nursing PhD data science and visualization course. Students were provided with a subscription to Chat Generative Pretrained Transformer (ChatGPT) 4.0, a general-purpose generative AI tool, for use in and outside the class. During the didactic portion, we described generative AI and its current and potential future applications in health care, including examples of appropriate and inappropriate applications. In the laboratory sessions, students were given three tasks representing different use cases of generative AI in health care practice and research (clinical decision support, patient decision support, and scientific communication) and asked to engage with ChatGPT on each. Students (n = 10) independently wrote a brief reflection for each task evaluating safety (accuracy, hallucinations) and usability (ease of use, usefulness, and intention to use in the future). Reflections were analyzed using directed content analysis. Students were able to identify the strengths and limitations of ChatGPT in completing all three tasks and developed opinions on whether they would feel comfortable using ChatGPT for similar tasks in the future. All of them reported increasing their self-rated competency in generative AI by one to two points on a five-point rating scale. This brief educational intervention supported doctoral nursing students in understanding the appropriate uses of ChatGPT, which may support their ability to appraise and use these tools in their future work.

ChatGPT Virtual Assistant for Breast Reconstruction: Assessing Preferences for a Traditional Chatbot versus a Human AI VideoBot.

Recent advancements in artificial intelligence (AI) have reshaped telehealth, with AI chatbots like Chat Generative Pretrained Transformer (ChatGPT) showing promise in various medical applications. ChatGPT is capable of offering basic patient education on procedures in plastic and reconstructive surgery (PRS), yet the preference between human AI VideoBots and traditional chatbots in plastic and reconstructive surgery remains unexplored. We developed a VideoBot by integrating ChatGPT with Synthesia, a human AI avatar video platform. The VideoBot was then integrated into Tolstoy to create an interactive experience that answered four of the most asked questions related to breast reconstruction. We used Zapier to develop a ChatGPT-integrated chatbot. A 16-item survey adapted from the 2005 validated measurement of online trust by Corritore et al was distributed online to female participants via Amazon Mechanical Turk. A total of 396 responses were gathered. Participants were 18 to 64 years old. Perceptions of truthfulness, believability, content expertise, ease of use, and safety were similar between the VideoBot and chatbot. Most participants preferred the VideoBot compared with the traditional chatbot (63.5% versus 28.1%), as they found it more captivating than the text-based chatbot. Of the participants, 77% would have preferred to see someone who they identified with in terms of gender and race. Both the VideoBot and text-based chatbot show comparable effectiveness, usability, and trust. Nonetheless, the VideoBot's human-like qualities enhance interactivity. Future research should explore the impact of race and gender concordance in telehealth to provide a more personalized experience for patients.

Evaluating Compliance of Randomized Controlled Trial Abstracts in Plastic Surgery Journals with CONSORT Guidelines Using GPT-4 AI

Background: The quality of reporting in randomized controlled trials (RCTs) is crucial for accurate interpretation and synthesis of evidence. The Consolidated Standards of Reporting Trials (CONSORT) guidelines provide a standardized framework for reporting RCT abstracts. This study aimed to evaluate the adherence of RCT abstracts published in three major plastic surgery journals to the CONSORT tool guideline for reporting abstracts, utilizing Generative Pre-trained Transformer 4 artificial intelligence (GPT-4 AI) technology. Methods: Abstracts of RCTs published between 2010 and 2023 were collected. The GPT-4 AI model was utilized to assess the abstracts based on the CONSORT criteria. Descriptive statistics were used to report the compliance scores and identify areas where abstracts lacked compliance. Results: Of the initially identified 500 abstracts, a total of 371 RCT abstracts met the inclusion criteria and were analyzed. The mean CONSORT score was 10.05 (±2.22), with a median score of 10.72. Specific areas where abstracts lacked compliance included trial design (39.6%), participant details (28.8%), intervention descriptions (15.6%), randomization process (25.3%), and the number of participants analyzed (33.4%). Trial registration (18.3%) and funding information (15.1%) were also frequently missing. Conclusions: Our study’s innovative use of the GPT-4 AI model for analysis demonstrated the potential of AI technology in streamlining and enhancing the evaluation of research compliance. We advocate for heightened awareness and more rigorous application of CONSORT guidelines among authors, reviewers, and journal editors. Emphasizing the role of AI technology in the evaluative process can further improve the reporting quality of future RCTs in plastic surgery, contributing to more reliable and transparent research in the field.

Strengths, Weaknesses, Opportunities, and Threats Analysis of Integrating ChatGPT Into Nursing Education

Artificial intelligence (AI) is driving global change, and the implementation of generative AI in higher education is inevitable. AI language models such as the chat generative pre-trained transformer (ChatGPT) hold the potential to revolutionize the delivery of nursing education in the future. Nurse educators play a crucial role in preparing nursing students for a future technology-integrated healthcare system. While the technology has limitations and potential biases, the emergence of ChatGPT presents both opportunities and challenges. It is critical for faculty to be familiar with the capabilities and limitations of this model to foster effective, ethical, and responsible utilization of AI technology while preparing students in advance for the dynamic and rapidly advancing landscape of nursing and healthcare. Therefore, this article was written to present a strengths, weaknesses, opportunities, and threats (SWOT) analysis of integrating ChatGPT into nursing education, providing a guide for implementing ChatGPT in nursing education and offering a well-rounded assessment to help nurse educators make informed decisions.

Artificial intelligence chatbots as sources of patient education material for cataract surgery: ChatGPT-4 versus Google Bard

ObjectiveTo conduct a head-to-head comparative analysis of cataract surgery patient education material generated by Chat Generative Pre-trained Transformer (ChatGPT-4) and Google Bard.Methods and analysis98 frequently asked questions on cataract surgery...

Evaluating ChatGPT-4’s correctness in patient-focused informing and awareness for atrial fibrillation

BackgroundAs artificial intelligence (AI) and large language models (LLMs) continue to evolve, their application in healthcare is expanding. OpenAI’s Chat Generative Pre-trained Transformer – version 4 (GPT-4) represents the latest advancement in this technology, capable of engaging in complex dialogues and providing information. ObjectiveThis study explores the correctness of GPT-4 in informing patients about atrial fibrillation. MethodsThis cross-sectional, observational study involved GPT-4 in responding to a structured set of 108 questions across 10 categories related to atrial fibrillation. These categories included basic information, treatment options, lifestyle adjustments, and more, reflecting common patient inquiries. The model's responses were evaluated by a panel of three cardiologists based on accuracy, comprehensiveness, clarity, relevance to clinical practice, and patient safety. The total correctness of GPT-4 was quantitatively assessed through scores assigned in each category, and statistical analysis was conducted to identify significant differences in performance across categories. ResultsGPT-4 provided correct and relevant answers with considerable variability across categories. It excelled in "Lifestyle Adjustments" and "Daily Life and Management" with perfect and near-perfect scores but struggled with "Miscellaneous Concerns" scoring lower. Statistical analysis confirmed significant differences in total scores across categories (p=0.020). ConclusionOur results suggest that, while GPT-4 is reliable in categories with structured and direct queries, it shows limitations when handling complex medical queries that require in-depth explanations or clinical judgment. GPT-4 demonstrates promising potential as a tool for patient informing in atrial fibrillation, particularly in straightforward informing content.

Performance of OpenAI’s GPT-4 in a mock MRCS Part A Examination

Introduction: OpenAI’s latest iteration of a Large Language Model (LLM); GPT-4 (Generative Pre-trained Transformer 4) has demonstrated its proficiency against various professional examination standards like the USMLE (United States Medical Licensing Examination), FRCS (Fellowship of the Royal Colleges of Surgeons) and the United States Bar. However, GPT-4’s capability with the MRCS (Membership of the Royal College of Surgeons) Part A has not yet been investigated. Methodology: A representative MRCS Part A examination that was prepared and provided by “TeachMeSurgery” based on the MRCS Intercollegiate Curriculum was used to assess GPT-4's performance. Each question was processed via the web-based interface of ChatGPT (Chat Generative Pre-trained Transformer) Plus. Results: GPT-4 scored 87.2% on Applied Basic Sciences (157/180) and 86.7% on Principles of Surgery in General (104/120 questions), achieving an overall score of 261/300 (87%), which is above the typical passing threshold. GPT-4 scored 100% in four out of the eleven predefined curriculum areas, which included: Pharmacology, Microbiology, Data Interpretation and Audit, and The Surgical Care of Children. GPT-4’s weakest performance was in the Medico-Legal Aspects of Surgical Practice, in which it scored 33.3%. Conclusion: GPT-4 successfully passed the mock MRCS Part A without any specialised preparatory training, however further research could look at integrating the GPT-4 model to enhance a trainee surgeon’s learning and use it as an effective tool to deliver high quality & efficient patient care.

Synergizing Generative Pre-Trained Transformer (GPT) Chatbots in a Process-Based Writing Paradigm to Enhance University Students’ Writing Skills

Background: The combination of the process-based writing framework and GPT-based chatbots establishes a dynamic and interactive environment, leading students through the stages of planning, drafting, revising, and editing. This collaborative approach not only elevates writing skills but also cultivates critical thinking and self-reflection, thereby transforming the writing process into a more effective and engaging learning experience. Despite the potential synergy to revolutionize the writing process, there exists a surprising lack of research within the educational domain exploring the impact of this innovative approach. Purpose: This study investigates the influence of a GPT-based chatbot within a process-based writing framework on university EFL students’ writing skills, specifically focusing on components such as organization, content, coherence-cohesion, logical connection, and argumentation. Method: Employing a sequential mixed methods type of research with a pre- and post-test design, 30 university EFL students were selected via purposive sampling technique. They engaged in 10 sessions that incorporated GPT-based chatbots within a process-based writing framework. Data collections were through pre-and post- writing tests, writing quizzes, and semi-structured interviews. Results: The results highlighted substantial improvements in participants’ writing performance, evident through a noteworthy increase in post-writing test scores (x̄=17.03) in comparison to pre-writing test scores (x̄=9.13). The study identified a progressive enhancement in four out of five writing components - organization, content, coherence-cohesion, and argumentation - across the 1st to 4th quizzes. However, the ‘logical connection’ component experienced a temporary decline during the 2nd and 3rd writing quizzes, rebounding significantly in the 4th quiz. Notably, the most improved writing components were ‘content’ and ‘argumentation,’ while the component related to ‘logical connection’ exhibited the least improved one. Qualitative findings further underscored participants’ acknowledgment of the effectiveness of the strategy in facilitating their writing tasks. Conclusion: The integration of chatbots within a writing framework was concluded as a facilitative pedagogical approach, fostering a dynamic, personalized, and effective learning experience, contributing to the multifaceted improvement of their writing skills. As educators and practitioners consider innovative approaches, this study provides a compelling case for the effective utilization of GPT-based chatbots in fostering language proficiency and a more engaging learning experience.

Sensitivity Analysis of Word Importance using GPT Model: A Ranking XAI Approach with Attention Weights and KL Divergence

This paper delves into the intricate realm of generative Artificial Intelligence (AI) models, specifically focusing on transformers like GPT (Generative Pre-trained Transformer). Despite their remarkable capabilities, these models pose challenges in terms of interpretability and accountability, owing to their complex architectures and vast training data. This paper employs a model to investigate the importance of words within the corpus, employing sensitivity analysis techniques. Specifically, attention weights are used to measure the impact of individual words on the model's predictions. The paper proposes a novel approach to rank the importance of words by leveraging attention weights and conducting sensitivity analysis across the dataset. To quantify the discrepancies between model-generated outputs and ground truth, the Kullback-Leibler (KL) divergence is employed. This divergence measure aids in evaluating how well the model captures the underlying distribution of words in the corpus. By integrating KL divergence into the sensitivity analysis, the study aims to provide a more comprehensive understanding of word importance.

Comparison of NLP machine learning models with human physicians for ASA Physical Status classification.

The American Society of Anesthesiologist's Physical Status (ASA-PS) classification system assesses comorbidities before sedation and analgesia, but inconsistencies among raters have hindered its objective use. This study aimed to develop natural language processing (NLP) models to classify ASA-PS using pre-anesthesia evaluation summaries, comparing their performance to human physicians. Data from 717,389 surgical cases in a tertiary hospital (October 2004-May 2023) was split into training, tuning, and test datasets. Board-certified anesthesiologists created reference labels for tuning and test datasets. The NLP models, including ClinicalBigBird, BioClinicalBERT, and Generative Pretrained Transformer 4, were validated against anesthesiologists. The ClinicalBigBird model achieved an area under the receiver operating characteristic curve of 0.915. It outperformed board-certified anesthesiologists with a specificity of 0.901 vs. 0.897, precision of 0.732 vs. 0.715, and F1-score of 0.716 vs. 0.713 (all p <0.01). This approach will facilitate automatic and objective ASA-PS classification, thereby streamlining the clinical workflow.

The Influence and Application of ChatGPT in the Teaching Reflection of International Chinese Teachers

The appearance and use of Chat Generative Pre-Trained Transformer (ChatGPT) has a great impact on the field of international Chinese education. The research on the combination of Artificial Intelligence (AI) and Teaching Chinese as a Foreign Language has attracted extensive attention from scholars, but there are relatively few studies on the combination of AI and teaching reflection of international Chinese teachers. This paper is to analyze the influence and application of ChatGPT in the teaching reflection of international Chinese teachers. It is concluded that ChatGPT plays a certain role in the reflection which is about teachers, students, and teaching media and environment. Based on this conclusion, the paper proposes the following suggestions: Through ChatGPT, these teachers could ensure that their attention continues to focus on information related to students' knowledge growth, and they can also provide individualized language practice and learning resources for students according to their language background. But at the same time, teachers need to pay attention to the maintenance of professional level and the improvement of human-AI interactive negotiation competence (HAINC).

Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study.

Generative artificial intelligence (GenAI) shows promise in automating key tasks involved in conducting systematic literature reviews (SLRs), including screening, bias assessment and data extraction. This potential automation is increasingly relevant as pharmaceutical developers face challenging requirements for timely and precise SLRs using the population, intervention, comparator and outcome (PICO) framework, such as those under the impending European Union (EU) Health Technology Assessment Regulation 2021/2282 (HTAR). This proof-of-concept study aimed to evaluate the feasibility, accuracy and efficiency of using GenAI for mass extraction of PICOs from PubMed abstracts. Abstracts were retrieved from PubMed using a search string targeting randomised controlled trials. A PubMed clinical study 'specific/narrow' filter was also applied. Retrieved abstracts were processed using the OpenAI Batch application programming interface (API), which allowed parallel processing and interaction with Generative Pre-trained Transformer 4 Omni (GPT-4o) via custom Python scripts. PICO elements were extracted using a zero-shot prompting strategy. Results were stored in CSV files and subsequently imported into a PostgreSQL database. The PubMed search returned 682,667 abstracts. PICOs from all abstracts were extracted in < 3 h, with an average processing time of 200 s per 1000 abstracts. A total of 395,992,770 tokens were processed, with an average of 580 tokens per abstract. The total cost was $3390. On the basis of a random sample of 350 abstracts, human verification confirmed that GPT-4o accurately and comprehensively extracted 342 (98%) of all PICOs, with only outcome elements rarely missed. Using GenAI to extract PICOs from clinical study abstracts could fundamentally transform the way SLRs are conducted. By enabling pharmaceutical developers to anticipate PICO requirements, this approach allows for proactive preparation for the EU HTAR process, or other health technology assessments (HTAs), streamlining efficiency and reducing the burden of meeting these requirements.

Advancement of Generative Pre-trained Transformer Chatbots in Answering Clinical Questions in the Practical Rhinoplasty Guideline.

The Generative Pre-trained Transformer (GPT) series, which includes ChatGPT, is an artificial large language model that provides human-like text dialogue. This study aimed to evaluate the performance of artificial intelligence chatbots in answering clinical questions based on practical rhinoplasty guidelines. Clinical questions (CQs) developed from the guidelines were used as question sources. For each question, we asked GPT-4 and GPT-3.5 (ChatGPT), developed by OpenAI, to provide answers for the CQs, Policy Level, Aggregate Evidence Quality, Level of Confidence in Evidence, and References. We compared the performance of the two types of artificial intelligence (AI) chatbots. A total of 10 questions were included in the final analysis, and the AI chatbots correctly answered 90.0% of these. GPT-4 demonstrated a lower accuracy rate than GPT-3.5 in answering CQs, although without statistically significant difference (86.0% vs. 94.0%; p = 0.05), whereas GPT-4 showed significantly higher accuracy for the level of confidence in Evidence than GPT-3.5 (52.0% vs. 28.0%; p < 0.01). No statistical differences were observed in Policy Level, Aggregate Evidence Quality, and Reference Match. In addition, GPT-4 rated significantly higher in presenting existing references than GPT-3.5 (36.9% vs. 24.1%; p = 0.01). The overall performance of GPT-4 was similar to that of GPT-3.5. However, GPT-4 provided existing references at a higher rate than GPT-3.5. GPT-4 has the potential to provide a more accurate reference in professional fields, including rhinoplasty. This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

The Performance of a Customized Generative Pre-trained Transformer on the American Society for Surgery of the Hand Self-Assessment Examination.

Multimodal large language models (MLLMs), such as OpenAI's ChatGPT (San Francisco, CA), have the potential to improve medical care delivery and education, although important shortcomings in accuracy and image interpretation have been noted. The aim of this study was to assess the multimodal performance of a ChatGPT model customized with hand surgery-specific knowledge. A customized generative pre-trained transformer (GPT) was trained using peer-reviewed literaturerecommended by the American Society for Surgery of the Hand (ASSH). Questions were taken from the ASSH 2022 Self-Assessment Examination (SAE). GPT-4 and the customized GPT were asked text-based multiple-choice questions. The customized GPT was also asked image-containingquestions, both with and without access to the image(s) associated with each question. A total of 192 questions were included. The customized GPT responded to the 119 text-only questions with greater accuracy than GPT-4 (107 (89.9%) versus 91 (76.5%), P = 0.001). Human examinees answered 87.3% (IQR: 71.6-93.7%) of the same text-based questions correctly. Of the 73 questions with images, the customized GPT answered 55 (75.3%)questions correctly, which dropped to 51 (69.9%) when the images were withheld (P = 0.317). The human examinees answered 87.2% (IQR: 79.4-95.4%) of image-based questions correctly. Our findings suggest significant improvements in ChatGPT's ability to answer text-based hand surgery questions with hand-specific training. ChatGPT is still limited in its ability to interpret images to answer questions related to hand conditions. These data show hand surgeons can create customized GPT models to provide tailored answers to specific questions, which may serve as the foundation for educational and clinical tools.

Enhancing chatbot performance for imaging recommendations: Leveraging GPT-4 and context-awareness for trustworthy clinical guidance

PurposeTo investigate if GPT-4 improves the accuracy, consistency, and trustworthiness of a context-aware chatbot to provide personalized imaging recommendations from American College of Radiology (ACR) appropriateness criteria documents using semantic similarity processing: In addition, we sought to enable auditability of the output by revealing the information source the decision relies on. Material and MethodsWe refined an existing chatbot that incorporated specialized knowledge of the ACR guidelines by upgrading GPT-3.5-Turbo to its successor GPT-4 by OpenAI, using the latest version of LlamaIndex, and improving the prompting strategy. This chatbot was compared to the previous version, generic GPT-3.5-Turbo and GPT-4, and general radiologists regarding the performance in applying the ACR appropriateness guidelines. ResultsThe refined context-aware chatbot performed superior to the previous version using GPT-3.5-Turbo, generic chatbots GPT-3.5-Turbo and GPT-4, and general radiologists in providing “usually or may be appropriate” recommendations according to the ACR guidelines (all p < 0.001). It also outperformed GPT-3.5-Turbo and general radiologists in respect to “usually appropriate” recommendations (both p < 0.001). Moreover, the consistency in correct answers was higher with 78 % consistent correct “usually appropriate” answers and 94 % for “usually or may be appropriate” recommendations. In all cases, the same source documents were chosen, ensuring transparency. ConclusionOur study demonstrates the significance of context awareness in ensuring the use of appropriate knowledge and proposes a strategy to enhance trust in chatbot-based outputs to provide transparency. The improvements in accuracy, consistency, and source transparency address trust issues and enhance the clinical decision support process.Abbreviations: ACR, American College of Radiology; accGPT, appropriateness criteria context aware GPT; accGPT-4, appropriateness criteria context aware GPT using GPT-4; GPT, generative pre-trained transformer; LLM, Large Language Model.

Developing Future Teachers’ Academic Writing and Critical Thinking Skills Using ChatGPT

Preparing future teachers to meet the demands of an increasingly dynamic world is a paramount challenge. Academic writing and critical thinking skills influence foreign language teachers’ professional growth, shaping their students’ cognitive development. This paper explores the integration of ChatGPT (generative pre-trained transformer) as a tool to augment future foreign language teachers’ academic writing and critical thinking skills. Through engagement with ChatGPT, future foreign language teachers have the opportunity to interact with a large language model that can provide targeted feedback, prompt thought-provoking discussions, and assist in refining professionally oriented competencies. This study examines the impact of ChatGPT on skills development through a comprehensive analysis of empirical data, pedagogical frameworks, and student experiences. A qualitative study conducted with 35 students enrolled in a foreign languages master’s program utilized focus group interviews and thematic analysis. The findings highlight ChatGPT’s limitations in content generation while also recognizing its potential to reduce research labor and assist in formatting and referencing tasks. The study underscores the importance of informed usage of ChatGPT, with students and educators recognizing the necessity of verifying information and maintaining academic integrity.

Can we trust LLMs to help us? An examination of the potential use of GPT-4 in generating quality literature reviews

Purpose This study aims to examine the ability of Generative Pre-trained Transformer 4 (GPT-4), one of the most powerful large language models, to generate a literature review for peer-reviewed journal publications. The objective is to determine whether business scholars can rely on GPT-4’s assistance with literature reviews and how the nature of human–artificial intelligence (AI) interaction may affect the quality of the reviews generated by GPT-4. Design/methodology/approach A survey of 30 experienced researchers was conducted to assess the quality of the literature reviews generated by GPT-4 in comparison with a human-authored literature review published in a Social Science Citation Index (SSCI) journal. The data collected were then analyzed with analysis of variance to ascertain whether we may trust GPT-4’s assistance in writing literature reviews. Findings The statistical analysis reveals that when a highly structured approach being used, GPT-4 can generate a high-quality review comparable to that found in an SSCI journal publication. However, when a less structured approach is used, the generated review lacks comprehensive understating and critical analysis, and is unable to identify literature gaps for future research, although it performed well in adequate synthesis and quality writing. The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction. Research limitations/implications The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction. Nonetheless, cautions should be taken due to the limitations of this study discussed in the text. Originality/value By breaking down the specific tasks of a literature review and using a quantitative rather than qualitative assessment method, this study provides robust and more objective findings about the ability of GPT-4 to assist us with a very important research task. The findings of this study should enhance our understanding of how GPT-4 may change our research endeavor and how we may take a full advantage of the advancement in AI technology in the future research.