Medical Questions Research Articles

Capable exam-taker and question-generator: the dual role of generative AI in medical education assessment

Abstract Objectives Artificial intelligence (AI) is being increasingly used in medical education. This narrative review presents a comprehensive analysis of generative AI tools’ performance in answering and generating medical exam questions, thereby providing a broader perspective on AI’s strengths and limitations in the medical education context. Methods The Scopus database was searched for studies on generative AI in medical examinations from 2022 to 2024. Duplicates were removed, and relevant full texts were retrieved following inclusion and exclusion criteria. Narrative analysis and descriptive statistics were used to analyze the contents of the included studies. Results A total of 70 studies were included for analysis. The results showed that AI tools’ performance varied when answering different types of questions and different specialty questions, with best average accuracy in psychiatry, and were influenced by prompts. With well-crafted prompts, AI models can efficiently produce high-quality examination questions. Conclusion Generative AI possesses the ability to answer and produce medical questions using carefully designed prompts. Its potential use in medical assessment is vast, ranging from detecting question error, aiding in exam preparation, facilitating formative assessments, to supporting personalized learning. However, it’s crucial for educators to always double-check the AI’s responses to maintain accuracy and prevent the spread of misinformation.

Large Language Models lack essential metacognition for reliable medical reasoning

Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to recognize their knowledge limitations and provided confident answers even when correct options were absent. In this work, we show that current models exhibit a critical disconnect between perceived and actual capabilities in medical reasoning, posing significant risks in clinical settings. Our findings emphasize the need for more robust evaluation frameworks that incorporate metacognitive abilities, essential for developing reliable Large Language Model enhanced clinical decision support systems.

Patient- and clinician-based evaluation of large language models for patient education in prostate cancer radiotherapy.

This study aims to evaluate the capabilities and limitations of large language models (LLMs) for providing patient education for men undergoing radiotherapy for localized prostate cancer, incorporating assessments from both clinicians and patients. Six questions about definitive radiotherapy for prostate cancer were designed based on common patient inquiries. These questions were presented to different LLMs [ChatGPT‑4, ChatGPT-4o (both OpenAI Inc., San Francisco, CA, USA), Gemini (Google LLC, Mountain View, CA, USA), Copilot (Microsoft Corp., Redmond, WA, USA), and Claude (Anthropic PBC, San Francisco, CA, USA)] via the respective web interfaces. Responses were evaluated for readability using the Flesch Reading Ease Index. Five radiation oncologists assessed the responses for relevance, correctness, and completeness using afive-point Likert scale. Additionally, 35prostate cancer patients evaluated the responses from ChatGPT‑4 for comprehensibility, accuracy, relevance, trustworthiness, and overall informativeness. The Flesch Reading Ease Index indicated that the responses from all LLMs were relatively difficult to understand. All LLMs provided answers that clinicians found to be generally relevant and correct. The answers from ChatGPT‑4, ChatGPT-4o, and Claude AI were also found to be complete. However, we found significant differences between the performance of different LLMs regarding relevance and completeness. Some answers lacked detail or contained inaccuracies. Patients perceived the information as easy to understand and relevant, with most expressing confidence in the information and awillingness to use ChatGPT‑4 for future medical questions. ChatGPT-4's responses helped patients feel better informed, despite the initially standardized information provided. Overall, LLMs show promise as atool for patient education in prostate cancer radiotherapy. While improvements are needed in terms of accuracy and readability, positive feedback from clinicians and patients suggests that LLMs can enhance patient understanding and engagement. Further research is essential to fully realize the potential of artificial intelligence in patient education.

Can ChatGPT 4.0 reliably answer patient frequently asked questions about boxer's fractures?

Patients are increasingly turning to the internet, and recently artificial intelligence engines (e.g., ChatGPT), for answers to common medical questions. Regarding orthopedic hand surgery, recent literature has focused on ChatGPT's ability to answer patient frequently asked questions (FAQs) regarding subjects such as carpal tunnel syndrome, distal radius fractures, and more. The present study seeks to determine how accurately ChatGPT can answer patient FAQs surrounding simple fracture patterns such as fifth metacarpal neck fractures. Internet queries were used to identify the ten most FAQs regarding boxer's fractures based on information from five trusted healthcare institutions. These ten questions were posed to ChatGPT 4.0, and the chatbot's responses were recorded. Two fellowship trained orthopedic hand surgeons and one orthopedic hand surgery fellow then graded ChatGPT's responses on an alphabetical grading scale (i.e., A-F); additional commentary was then provided for each response. Descriptive statistics were used to report question, grader, and overall ChatGPT response grades. ChatGPT achieved a cumulative grade of a B, indicating that the chatbot can provide adequate responses with only minor need for clarification when answering FAQs for boxer's fractures. Individual graders provided comparable overall grades of B, B, and B+ respectively. ChatGPT deferred to a medical professional in 7/10 responses. General questions were graded at an A-. Management questions were graded at a C+. Overall, with a grade of B, ChatGPT 4.0 provides adequate-to- complete responses as it pertains to patient FAQs surrounding boxer's fractures.

Toward expert-level medical question answering with large language models.

Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.

Generative Artificial Intelligence Responses to Common Patient-Centric Hand and Wrist Surgery Questions: A Quality and Usability Analysis.

Background: Due to the rapid evolution of generative artificial intelligence (AI) and its implications on patient education, there is a pressing need to evaluate AI responses to patients' medical questions. This study assessed the quality and usability of responses received from two prominent AI platforms to common patient-centric hand and wrist surgery questions. Methods: Twelve commonly encountered hand and wrist surgery patient questions were inputted twice into both Gemini and ChatGPT, generating 48 responses. Each response underwent a content analysis, followed by assessment for quality and usability with three scoring tools: DISCERN, Suitability Assessment of Materials (SAM) and the AI Response Metric (AIRM). Statistical analyses compared the features and scores of the outputs when stratified by platform, question type and response order. Results: Responses earned mean overall scores of 55.7 ('good'), 57.2% ('adequate') and 4.4 for DISCERN, SAM and AIRM, respectively. No responses provided citations. Wrist question responses had significantly higher DISCERN (p < 0.01) and AIRM (p = 0.02) scores compared to hand responses. Second responses had significantly higher AIRM (p < 0.01), but similar DISCERN (p = 0.76) and SAM (p = 0.11), scores compared to the first responses. Gemini's DISCERN (p = 0.04) and SAM (p < 0.01) scores were significantly higher than ChatGPT's corresponding metrics. Conclusions: Although responses are generally 'good' and 'adequate', there is variable quality with respect to platform used, type of question and response order. Given the diversity of publicly available AI platforms, it is important to understand the quality and usability of information patients may encounter during their search for answers to common hand and wrist surgery questions. Level of Evidence: Level IV (Therapeutic).

UnICLAM: Contrastive representation learning with adversarial masking for unified and interpretable Medical Vision Question Answering

UnICLAM: Contrastive representation learning with adversarial masking for unified and interpretable Medical Vision Question Answering

KoSEL: Knowledge subgraph enhanced large language model for medical question answering

KoSEL: Knowledge subgraph enhanced large language model for medical question answering

Enhancing Medical Question Answering Systems: Tagging and Addressing Wrongly answered Queries through Transfer Learning

Enhancing Medical Question Answering Systems: Tagging and Addressing Wrongly answered Queries through Transfer Learning

Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework.

The success of large language models (LLMs) in general areas have sparked a wave of research into their applications in the medical field. However, enhancing the medical professionalism of these models remains a major challenge. This study proposed a novel model training theoretical framework, the M-KAT framework, which integrated domain-specific training methods for LLMs with the unique characteristics of the medical discipline. This framework aimed to improve the medical professionalism of the models from three perspectives: general knowledge acquisition, specialized skill development, and alignment with clinical thinking. This study summarized the outcomes of medical LLMs across four tasks: clinical diagnosis and treatment, medical question answering, medical research, and health management. Using the M-KAT framework, we analyzed the contribution to enhancement of professionalism of models through different training stages. At the same time, for some of the potential risks associated with medical LLMs, targeted solutions can be achieved through pre-training, SFT, and model alignment based on cultivated professional capabilities. Additionally, this study identified main directions for future research on medical LLMs: advancing professional evaluation datasets and metrics tailored to the needs of medical tasks, conducting in-depth studies on medical multimodal large language models (MLLMs) capable of integrating diverse data types, and exploring the forms of medical agents and multi-agent frameworks that can interact with real healthcare environments and support clinical decision-making. It is hoped that predictions of work can provide a reference for subsequent research.

"You Have to Do This When You're Healthy": An Introduction to Library Involvement in Programming on End-of-Life Issues

While the library literature shows that librarians work to address information needs about end-of-life issues, no systematic research on library involvement in programming about these issues yet exists. A preliminary study was conducted of library staff whose libraries have been involved in end-of-life programming or had considered this programming between 2018 and 2023. The study consisted of an online survey, followed by an interview with respondents whose libraries had been involved in this programming. The study sample (N=13) included public and academic libraries around the United States and in communities of various sizes. Reported activities included speaker series, Death Cafes, instruction on obituary writing, support for palliative and pastoral care groups by creating information resources, and a virtual training for public library staff about meeting patrons’ needs for end-of-life health information. Topics covered medical, financial, legal, emotional, and spiritual questions related to death and dying. Respondents described a variety of partners as valuable contributors to programs’ success. Some programs incorporated methods to help manage program participants’ anxieties about the subject matter. Respondents reported positive experiences with these programs and positive feedback from participants, but possible effects of volunteer bias and the small sample size prevent conclusions about how typical this is. More research is necessary to understand patterns among these programs, ways to protect the emotional well-being of those involved, and strategies to tailor programs to best meet library patrons’ needs for information on end-of-life issues.

Feasibility of a Pharmabuddy Care Service for patients with Parkinson’s disease

BackgroundPharmaceutical care for patients with Parkinson’s disease (PD) is complex. Specialized pharmaceutical care provided by a dedicated pharmacy team member (pharmabuddy) for these patients may reduce medication-related problems. The feasibility of this service for PD patients is unknown.ObjectiveTo evaluate the feasibility of a Pharmabuddy Care Service (PCS) for PD patients in primary care pharmacies.MethodsPharmabuddies who offered PCS were invited to fill in a questionnaire to evaluate the feasibility of PCS. Patients received a patient questionnaire and were invited for an interview. Patient records provided information on medication-related problems and interventions. Feasibility was evaluated conform four domains of Bowen’s Framework. First, acceptability included patients’ satisfaction ratings and experiences, Pharmabuddy impression on start and continuation of the service. Second, demand included use by patients and provision by pharmabuddies. Third, implementation/practicality: implementation indicators and barriers and facilitators from patient- and pharmabuddy perspective and fourth limited efficacy: effect of the PCS on PD symptoms, medication related problems identified and interventions from patient records.ResultsTwenty-three (59%) patients completed the questionnaire, 9 were interviewed, 12 (67%) pharmabuddies responded. Acceptability was high among patients (mean 9.5 (SD 1.3) out of 10), 6 (50%) pharmacies (still) provided PCS. Demand: 56% of patients had 1–2 contact moments, 28% two or more. Ten pharmacies provided up to 14 patients per pharmacy with PCS, one up to 24. Implementation/practicality: important barriers were time constraints and perception of other healthcare professionals. Positive reactions from patients encouraged pharmabuddies to carry out their PCS-activities. Patients were positive about pharmabuddy’s listening competency while knowledge could be improved. Limited efficacy: from patient records, 89 interventions were made in response to 93 (median 3 per patient (range 1–16)) medication related problems or questions, with 20 (87%) patients reporting a positive effect from their interaction with a pharmabuddy.ConclusionThis study shows PCS is highly appreciated by responding patients and can be feasible for primary care pharmacies. Several implementation issues are still present. Future studies should focus on quantifiable effects of PCS services as well as resource and perception hurdles.

Comparative Analysis of Denoising Methods to Improve Image Quality for Medical Visual Question Answering

Medical Visual Question Answering (MedVQA) is a dual research field that intersects medical imaging and natural language processing, for better interpretability and accessibility of medical image data.Medical image quality is paramount for accurate diagnostics and subsequent medical visual question answering (MedVQA) tasks. This research focuses on applying and then analyzing results of different denoising methods on VQA-RAD, Medical VQA dataset to enhance quality of images. This study explores effectiveness of different traditional and deep learning based methods to reduce noise within medical images, thereby improving the accuracy and reliability of MedVQA task. We applied different traditional denoising filtering methods such as, gaussian filter, median filter, average filter, bilateral filter and convolutional autoencoder (CAE) based on deep learning on a VQA-RAD dataset to compare effectiveness of each denoising methods to improve image quality. Through comprehensive experiments and evaluations, this paper demonstrates that the convolutional autoencoder is potentially enhancing quality of medical images with an emphasis on preserving essential diagnostic information while suppressing unwanted noise with compare to other traditional denoising filters. The denoised images are then employed as input to improve accuracy for MedVQA tasks. The results of this research will help in optimizing medical imaging pipelines, ultimately benefiting clinical decision-making and healthcare outcomes.

Evaluating Bard Gemini Pro and GPT-4 Vision Against Student Performance in Medical Visual Question Answering: Comparative Case Study.

The rapid development of large language models (LLMs) such as OpenAI's ChatGPT has significantly impacted medical research and education. These models have shown potential in fields ranging from radiological imaging interpretation to medical licensing examination assistance. Recently, LLMs have been enhanced with image recognition capabilities. This study aims to critically examine the effectiveness of these LLMs in medical diagnostics and training by assessing their accuracy and utility in answering image-based questions from medical licensing examinations. This study analyzed 1070 image-based multiple-choice questions from the AMBOSS learning platform, divided into 605 in English and 465 in German. Customized prompts in both languages directed the models to interpret medical images and provide the most likely diagnosis. Student performance data were obtained from AMBOSS, including metrics such as the "student passed mean" and "majority vote." Statistical analysis was conducted using Python (Python Software Foundation), with key libraries for data manipulation and visualization. GPT-4 1106 Vision Preview (OpenAI) outperformed Bard Gemini Pro (Google), correctly answering 56.9% (609/1070) of questions compared to Bard's 44.6% (477/1070), a statistically significant difference (χ2₁=32.1, P<.001). However, GPT-4 1106 left 16.1% (172/1070) of questions unanswered, significantly higher than Bard's 4.1% (44/1070; χ2₁=83.1, P<.001). When considering only answered questions, GPT-4 1106's accuracy increased to 67.8% (609/898), surpassing both Bard (477/1026, 46.5%; χ2₁=87.7, P<.001) and the student passed mean of 63% (674/1070, SE 1.48%; χ2₁=4.8, P=.03). Language-specific analysis revealed both models performed better in German than English, with GPT-4 1106 showing greater accuracy in German (282/465, 60.65% vs 327/605, 54.1%; χ2₁=4.4, P=.04) and Bard Gemini Pro exhibiting a similar trend (255/465, 54.8% vs 222/605, 36.7%; χ2₁=34.3, P<.001). The student majority vote achieved an overall accuracy of 94.5% (1011/1070), significantly outperforming both artificial intelligence models (GPT-4 1106: χ2₁=408.5, P<.001; Bard Gemini Pro: χ2₁=626.6, P<.001). Our study shows that GPT-4 1106 Vision Preview and Bard Gemini Pro have potential in medical visual question-answering tasks and to serve as a support for students. However, their performance varies depending on the language used, with a preference for German. They also have limitations in responding to non-English content. The accuracy rates, particularly when compared to student responses, highlight the potential of these models in medical education, yet the need for further optimization and understanding of their limitations in diverse linguistic contexts remains critical.

AI in Healthcare: A Survey on Medical Question Answering System

The new era of question-answering systems in information retrieval has the potential to open up previously uncharted territory soon. Information retrieval (IR) is the area of competence for Question Answering Systems. It is useful in several fields, including personal help, healthcare, finance, and education. Instead of giving document lists as retrieval systems often do, the question-answering job aims to recover brief passages of text that answer a question. The fundamental concept of the question-answering system is to emphasize human-machine interaction. It is a method for retrieving accurate answers to questions posed by people. The three separate components that make up Question Answering are Question Classification, Question Pre-processing, and Answer Retrieval. This paper provides an overview of the Question Answering system along with the usage of three basic modules with their detailed description. This study aims to present a thorough review of the state of AI-driven medical question-answering technology.

Is Artificial Intelligence the Future of Radiology? Accuracy of ChatGPT in Radiologic Diagnosis of Upper Extremity Bony Pathology.

Artificial intelligence (AI) is a promising tool to aid in diagnostic accuracy and patient communication. Prior literature has shown that ChatGPT answers medical questions and can accurately diagnose surgical conditions. The purpose of this study was to determine the accuracy of ChatGPT 4.0 in evaluating radiologic imaging of common orthopedic upper extremity bony pathologies, including identifying the imaging modality and diagnostic accuracy. Diagnostic imaging was sourced from an open-source radiology database for 6 common upper extremity bony pathologies: distal radius fracture (DRF), metacarpal fracture (MFX), carpometacarpal osteoarthritis (CMC), humerus fracture (HFX), scaphoid fracture (SFX), and scaphoid nonunion (SN). X-ray, computed tomography (CT), and magnetic resonance imaging (MRI) modalities were included. Fifty images were randomly selected from each pathology where possible. Images were uploaded to ChatGPT 4.0 and queried for imaging modality, laterality, and diagnosis. Each image query was completed in a new ChatGPT search tab. Multinomial linear regression was used to identify variations in ChatGPT's diagnostic accuracy across imaging modalities and medical conditions. Overall, ChatGPT provided a diagnosis for 52% of images, with accuracy ranging from 0% to 55%. Diagnostic accuracy was significantly lower for SFX and MFX relative to HFX. ChatGPT was significantly less likely to provide a diagnosis for MRI relative to CT. Diagnostic accuracy ranged from 0% to 40% with regard to imaging modality (x-ray, CT, MRI) though this difference was not statistically significant. ChatGPT's accuracy varied significantly between conditions and imaging modalities, though its iterative learning capabilities suggest potential for future diagnostic utility within hand surgery.

Identifying Markov Chain Models from Time-to-Event Data: An Algebraic Approach.

Many biological and medical questions can be modeled using time-to-event data in finite-state Markov chains, with the phase-type distribution describing intervals between events. We solve the inverse problem: given a phase-type distribution, can we identify the transition rate parameters of the underlying Markov chain? For a specific class of solvable Markov models, we show this problem has a unique solution up to finite symmetry transformations, and we outline a recursive method for computing symbolic solutions for these models across any number of states. Using the Thomas decomposition technique from computer algebra, we further provide symbolic solutions for any model. Interestingly, different models with the same state count but distinct transition graphs can yield identical phase-type distributions. To distinguish among these, we propose additional properties beyond just the time to the next event. We demonstrate the method's applicability by inferring transcriptional regulation models from single-cell transcription imaging data.

Counterfactual Causal-Effect Intervention for Interpretable Medical Visual Question Answering.

Medical Visual Question Answering (VQA-Med) is a challenging task that involves answering clinical questions related to medical images. However, most current VQA-Med methods ignore the causal correlation between specific lesion or abnormality features and answers, while also failing to provide accurate explanations for their decisions. To explore the interpretability of VQA-Med, this paper proposes a novel CCIS-MVQA model for VQA-Med based on a counterfactual causal-effect intervention strategy. This model consists of the modified ResNet for image feature extraction, a GloVe decoder for question feature extraction, a bilinear attention network for vision and language feature fusion, and an interpretability generator for producing the interpretability and prediction results. The proposed CCIS-MVQA introduces a layer-wise relevance propagation method to automatically generate counterfactual samples. Additionally, CCIS-MVQA applies counterfactual causal reasoning throughout the training phase to enhance interpretability and generalization. Extensive experiments on three benchmark datasets show that the proposed CCIS-MVQA model outperforms the state-of-the-art methods. Enough visualization results are produced to analyze the interpretability and performance of CCIS-MVQA.

Enhancing the Graduate Medical Education Clinical Learning Environment Through Community of Practice Forums.

An optimal clinical learning environment (CLE) is associated with improved learning and patient care outcomes. Significant concerns exist about the state of the CLE in graduate medical education (GME). Research suggests that GME programming falls short in interpersonal aspects of training that promote trainee engagement and psychological safety. Furthermore, published educational interventions in the CLE lack adequate theoretical backing to inform a rational approach to interventions in the CLE that could address these important problems.The authors apply the 2002 work of Etienne Wenger on communities of practice (COP) to address these GME CLE concerns. To distinguish this COP intervention from earlier theoretical work on COPs, the authors refer to this management concept as "COP forums." COP forums favorably influence the GME CLE through effects that complement experiential learning in patient care. COP forums support trainee psychological safety, mentorship from near peers, and opportunities to innovate-effects that can serve as a counterbalance to the time pressures, hierarchy, and compliance culture often experienced in the clinical environment. Deliverables of COP forums, including practice innovation and trainee self-efficacy, can favorably impact organization-wide performance and engagement.This article describes the historical position of COP forums in the evolution of COP theory and outlines the basic structure and function of COP forums. It contrasts COP forums to other COP-related concepts to explain their relevance to the GME CLE. Examples of innovative GME COP forums illustrate the structure and function of these interventions. Finally, the authors call for more research on the impact of COP forums on the GME CLE. To avoid confusion, such scholarship must account for the ongoing evolution of the larger COP framework and target specific dimensions of the theory most pertinent to the medical education research question at hand.

ChatGPT (GPT-4) versus doctors on complex cases of the Swedish family medicine specialist examination: an observational comparative study

BackgroundRecent breakthroughs in artificial intelligence research include the development of generative pretrained transformers (GPT). ChatGPT has been shown to perform well when answering several sets of medical multiple-choice questions. However,...