Question Bank Research Articles

Question Paper Generation using Bloom’s Taxonomy

The "Question Paper Generator using Bloom's Taxonomy" is a high-tech, AI-based tool designed to make the generation of varied and well-balanced assessments easier for instructors. A wide variety of input sources, including syllabus documents, banks of questions, and even historical exams, are analysed and classified by NLP and machine learning into Bloom's Taxonomy cognitive levels: Remember, Understand, Apply, Analyse, Evaluate, and Create. The AI model is actually placed in an intuitive web application with which application with which educators can readily update such parameters as total marks, question difficulty, and the distribution of cognitive levels. This tool provides a logical interface where users can create assessments centered on standards for education and elicit thinking at different cognitive dimensions. Additionally, the web application feeds back a graphic report that plots through Bloom's Taxonomy in such a way that cognitive distribution is just and transparent. With the infusion of AI in a web- based setting, the tool brings forth practical and accessible solutions for educators in automating question papers creation to enrich their assessments. This application, therefore, holds best practices in instructional design through efficiency and scalability in the approach towards educational evaluation.

The performance of AI in medical examinations: an exploration of ChatGPT in ultrasound medical education

ObjectiveThis study aims to evaluate the accuracy of ChatGPT in the context of China’s Intermediate Professional Technical Qualification Examination for Ultrasound Medicine, exploring its potential role in ultrasound medical education.MethodsA total of 100 questions, comprising 70 single-choice and 30 multiple-choice questions, were selected from the examination’s question bank. These questions were categorized into four groups: basic knowledge, relevant clinical knowledge, professional knowledge, and professional practice. ChatGPT versions 3.5 and 4.0 were tested, and accuracy was measured based on the proportion of correct answers for each version.ResultsChatGPT 3.5 achieved an accuracy of 35.7% for single-choice and 30.0% for multiple-choice questions, while version 4.0 improved to 61.4 and 50.0%, respectively. Both versions performed better in basic knowledge questions but showed limitations in professional practice-related questions. Version 4.0 demonstrated significant improvements across all categories compared to version 3.5, but it still underperformed when compared to resident doctors in certain areas.ConclusionWhile ChatGPT did not meet the passing criteria for the Intermediate Professional Technical Qualification Examination in Ultrasound Medicine, its strong performance in basic medical knowledge suggests potential as a supplementary tool in medical education. However, its limitations in addressing professional practice tasks need to be addressed.

Comparing the dental knowledge of large language models.

Introduction With the advancement of artificial intelligence, large language models (LLMs) have emerged as technology that can generate human-like text across various domains. They hold vast potential in the dental field, able to be integrated into clinical dentistry, administrative dentistry, and for student and patient education. However, the successful integration of LLMs into dentistry is reliant on the dental knowledge of the models used, as inaccuracies can lead to significant risks in patient care and education.Aims We are the first to compare different LLMs on their dental knowledge through testing the accuracy of different model responses to Integrated National Board Dental Examination (INBDE) questions.Methods We include closed-source and open-source models and analysed responses to both 'patient box' style board questions and more traditional, textual-based, multiple-choice questions.Results For the entire INBDE question bank, ChatGPT-4 had the highest dental knowledge, with an accuracy of 75.88%, followed by Claude-2.1 with 66.38% and then Mistral-Medium at 54.77%. There was a statistically significant difference in performance across all models.Conclusion Our results highlight the high potential of LLM integration into the dental field, the importance of which LLM is chosen when developing new technologies, and the limitations that must be overcome before unsupervised clinical integration can be adopted.

Dynamic sequential cross-sectional scanning increases detection rate of congenital heart disease in sonographers: a prenatal ultrasound training program

BackgroundPrenatal ultrasound is the preferred modality for diagnosing fetal congenital heart disease. Given issues of physician proficiency and hospital distribution, we propose a dynamic sequential cross-sectional scanning (SCS) to explore the feasibility of cardiac screening by sonographers with less than 5 years of experience in ultrasound.Materials and methodsTwenty residents were randomly divided into two groups, receiving training in the American Institute of Ultrasound in Medicine (AIUM) fetal echocardiography and the SCS method. According to the needs of training, the professional staff developed the theoretical knowledge question bank, the CHD ultrasonic video disease bank, and the assessment scale. Trainees completed the pre-training examination, theory and skill operation training, and post-training assessment. For the two groups, the theoretical knowledge, skill operation and disease diagnosis were analyzed statistically before and after training.ResultsAfter training, the trainees in both groups had significantly improved knowledge and diagnostic abilities, their diagnostic thinking about CHD was clear, and they could identify major or even all structural abnormalities and make a definite diagnosis. In terms of skill operation, both groups could complete all required scanning within the specified time. The scanning time of the SCS group was significantly lower than that of the AIUM group, and the effect of the receptor site in the AIUM group was significantly higher than that in the SCS group.ConclusionSCS can be used as a new rapid fetal cardiac scanning method and try to popularize among echocardiographer.

Evaluation of a Large Language Model on the American Academy of Pediatrics' PREP Emergency Medicine Question Bank

Background Large language models (LLMs), including ChatGPT (Chat Generative Pretrained Transformer), a popular, publicly available LLM, represent an important innovation in the application of artificial intelligence. These systems generate relevant content by identifying patterns in large text datasets based on user input across various topics. We sought to evaluate the performance of ChatGPT in practice test questions designed to assess knowledge competency for pediatric emergency medicine (PEM). Methods We evaluated the performance of ChatGPT for PEM board certification using a popular question bank used for board certification in PEM published between 2022 and 2024. Clinicians assessed performance of ChatGPT by inputting prompts and recording the software's responses, asking each question over 3 separate iterations. We calculated correct answer percentages (defined as correct in at least 2/3 iterations) and assessed for agreement between the iterations using Fleiss' κ. Results We included 215 questions over the 3 study years. ChatGPT responded correctly to 161 of PREP EM questions over 3 years (74.5%; 95% confidence interval, 68.5%–80.5%), which was similar within each study year (75.0%, 71.8%, and 77.8% for study years 2022, 2023, and 2024, respectively). Among correct responses, most were answered correctly on all 3 iterations (137/161, 85.1%). Performance varied by topic, with the highest scores in research and medical specialties and lower in procedures and toxicology. Fleiss' κ across the 3 iterations was 0.71, indicating substantial agreement. Conclusion ChatGPT provided correct answers to PEM responses in three-quarters of cases, over the recommended minimum of 65% provided by the question publisher for passing. Responses by ChatGPT included detailed explanations, suggesting potential use for medical education. We identified limitations in specific topics and image interpretation. These results demonstrate opportunities for LLMs to enhance both the education and clinical practice of PEM.

Comprehensive review of resources and strategy for dermatology board certification exams in the United States.

This article offers an up-to-date overview of resources tailored for dermatology residents in the United States preparing for their board certification exams. We have assembled a comprehensive list of exam preparation resources categorized by cost, alongside recommendations from attending and resident physicians regarding the most effective resources for each exam. Although examinations are only one facet of the training involved in becoming a competent dermatologist, these exams help residents focus on learning crucial aspects of medicine. This review aims to furnish a concise summary of available resources for dermatology residents, organized by cost. The average cost of a paid dermatology question bank resource in this review was approximately $279, reflecting the need for review resources for board exam preparation that can fit a resident physician's budget or are free to use.

Integrating Clinical Reasoning Into Medical Students' First Weeks of Education Improves Understanding of Cranial Nerve Anatomy.

Clinical reasoning is essential to the practice of medicine.Such reasoning involves analytical (deductive) and non-analytical (recall) processes. Non-analytical reasoning is taught extensively in medical schools, and it dominates medical students' time as they review question banks and lecture notes, watch videos online, and memorize flashcards, algorithms, and illness scripts. However, few opportunities are provided in the curriculum to develop students' clinical reasoning skills, and when they are, the diverse levels of innate reasoning ability among students often lead to significant learning disparity. To address this deficiency, a pilot module on cranial nerve anatomy was developed to foster analytical clinical reasoning in an individualized manner. It was hypothesized that this module would not only introduce the foundations of an essential medical skill but also improve overall student understanding of the subject and reduce learning disparities among students. A comparative study was conducted using thismodule in one group and adidactic module in the other, employing pre- and post-testing measures. Results indicated a 26% improvement in average scores following the analytical module, whereas the control module showed no significant improvement. In addition, the disparity between students improving or not improving following the intervention was reduced, with 74% ofstudentsimproving after the reasoning module andonly33% of students improving after the didactic module. A novel cranial nerve educational module introduced analytical reasoning in medical students' first few weeks of education, facilitating the learning of complex anatomy and reducing learning disparity between students.

The Educational Benefits of Plastic Surgery Rotations for Off-Service Residents.

With increasing specialization among surgical divisions, a well-rounded education during a surgical residency is often accomplished by rotating among different subspecialties. Inclusion of specific rotations in the resident curriculum can be considered as a cost-benefit calculation balancing the value of exposure to a subspecialty versus the opportunity cost of potential learning from another rotation. We find that often these decisions are based on anecdotal feedback. Our goal is to supplement these reports with a quantifiable metric of learning achieved on the plastic surgery rotation. Our hypothesis in this prospective study was that residents would demonstrate improved performance on a post-rotation test after their 1-month rotation on plastic surgery compared to the pre-rotation test. A question bank was developed to reflect institutional curriculum objectives and clinical scenarios commonly seen on the service. The questions were developed, validated and vetted in collaboration with medical educators and attending plastic surgeons yielding 20 questions available for use. Postgraduate year 1 residents were given a 10-question test before and after their plastic surgery rotation. A one-tailed paired t-test was used to assess improvement between the pre-rotation test and the post-rotation test. A total of 378 tests were administered with 228 (60%) pre- and post-rotation tests completed meeting inclusion criteria. Average percentage of correct answers for the pre-rotation test was 29% and 88% for the post-rotation test showing a differential improvement of 58% (p < 0.001). Surgical trainee time is a limited commodity. Each clinical rotation needs proven consistent benefit for the trainees. We developed a questionnaire that documents the improvement in clinical knowledge after a one-month rotation on plastic surgery relative to before. The test results were consistent even when comparing trainees who did the rotation early versus late in the PGY-1 year. Clinical exposure reinforces and solidifies specialty learning.

ELECTRONIC ASSESSMENT ANXIETY SCALE: DEVELOPMENT, VALIDITY AND RELIABILITY

The widespread availability of internet access in daily life has resulted in a greater acceptance of online assessment methods. E-assessment platforms offer various features such as randomizing questions and answers, utilizing extensive question banks, setting time limits, and managing access during online exams. Electronic assessment enables real-time monitoring, customization, and scalability of feedback, benefiting students, academic staff, and administrative personnel. However, students encounter specific challenges in the electronic assessment environments. These challenges include limited control over test settings and the isolated nature of taking exams without peers. Furthermore, the technological proficiency of both instructors and students, along with resource constraints (computers, mobile devices, internet), can impede the effective utilization of these assessment tools. Technical issues like slow internet connection or disconnections can have significant consequences, especially in online exams, posing difficulties for corrections. The main goal of this study is to develop a Likert-type scale capable of measuring anxiety related to technical issues, social isolation, and the test interface experienced in e-assessment contexts. The study consists of two separate groups: the first group comprising 359 participants and the second group consisting of 356 participants. Both groups include undergraduate and pedagogical formation certificate program students from a university in the Eastern Anatolia Region of Turkiye. Construct validity was assessed through exploratory and confirmatory factor analyses. Item parameters were examined using item analysis based on classical test theory. As a result of the study, a two-factor scale structure comprising 21 items measuring social and technical anxiety was developed. These two dimensions account for 59.89% of the total variance. The Cronbach’s alpha coefficient for the entire scale was 0.93, the McDonald’s omega coefficient was 0.93, and the construct reliability was 0.99.

Exploration of medical students’ approach to progress test preparation

Introduction: Medical students are expected to develop competencies in applying clinical knowledge. Progress testing is an assessment for learning and is intended to assist in knowledge acquisition and to promote ongoing recall. This study explored student preparation for progress tests (PTs), relationships between approach and performance, and patterns that could assist with targeting learning support. Methods: A cross-sectional survey exploring study approach and learning context, comprising multi-choice and open-ended responses, was administered to students during their clinical years (n = 297). Quantitative data was analysed using descriptive statistics and comparative analysis, and qualitative data was categorised. Results: One hundred twenty-nine students responded (43.4%), with most demonstrating a stable PT performance over time. Twenty-two students had dynamic changes in their aggregate grades. Poor early PT performance was associated with developing an improvement strategy (χ2 6.954, p = 0.008). Students never falling below satisfactory were less likely to have a strategy (χ2 10.084, p = 0.001). All poorly performing students were using practice question banks, but this was not true for students who scored satisfactorily. The need for pastoral care was associated with poorer performance (χ2 4.701, p = 0.030). Conclusions: Student approaches to PT preparation are diverse, however there are correlations between preparation factors and performance outcomes. Students with poor early performance demonstrated improved results over time that may be attributed to targeted support and preparation strategies. Widespread use of practice questions may not be sufficient to impact results without additional strategies. External impacts to performance were common, and barriers to accessing pastoral care were evident. The feedback dashboard was underutilised, suggesting a need to improve feedback literacy and ensure this is fit for purpose.

Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o.

This study evaluates the performance of AI models, ChatGPT-4o and Google Gemini, in answering rheumatology board-level questions, comparing their effectiveness, reliability, and applicability in clinical practice. A cross-sectional study was conducted using 420 rheumatology questions from the BoardVitals question bank, excluding 27 visual data questions. Both artificial intelligence models categorized the questions according to difficulty (easy, medium, hard) and answered them. In addition, the reliability of the answers was assessed by asking the questions a second time. The accuracy, reliability, and difficulty categorization of the AI models' response to the questions were analyzed. ChatGPT-4o answered 86.9% of the questions correctly, significantly outperforming Google Gemini's 60.2% accuracy (p < 0.001). When the questions were asked a second time, the success rate was 86.7% for ChatGPT-4o and 60.5% for Google Gemini. Both models mainly categorized questions as medium difficulty. ChatGPT-4o showed higher accuracy in various rheumatology subfields, notably in Basic and Clinical Science (p = 0.028), Osteoarthritis (p = 0.023), and Rheumatoid Arthritis (p < 0.001). ChatGPT-4o significantly outperformed Google Gemini in rheumatology board-level questions. This demonstrates the success of ChatGPT-4o in situations requiring complex and specialized knowledge related to rheumatological diseases. The performance of both AI models decreased as the question difficulty increased. This study demonstrates the potential of AI in clinical applications and suggests that its use as a tool to assist clinicians may improve healthcare efficiency in the future. Future studies using real clinical scenarios and real board questions are recommended. Key Points •ChatGPT-4o significantly outperformed Google Gemini in answering rheumatology board-level questions, achieving 86.9% accuracy compared to Google Gemini's 60.2%. •For both AI models, the correct answer rate decreased as the question difficulty increased. •The study demonstrates the potential for AI models to be used in clinical practice as a tool to assist clinicians and improve healthcare efficiency.

Information that matters: Exploring information needs of people affected by algorithmic decisions

Every AI system that makes decisions about people has a group of stakeholders that are personally affected by these decisions. However, explanations of AI systems rarely address the information needs of this stakeholder group, who often are AI novices. This creates a gap between conveyed information and information that matters to those who are impacted by the system’s decisions, such as domain experts and decision subjects. To address this, we present the “XAI Novice Question Bank”, an extension of the XAI Question Bank (Liao et al., 2020) containing a catalog of information needs from AI novices in two use cases: employment prediction and health monitoring. The catalog covers the categories of data, system context, system usage, and system specifications. We gathered information needs through task based interviews where participants asked questions about two AI systems to decide on their adoption and received verbal explanations in response. Our analysis showed that participants’ confidence increased after receiving explanations but that their understanding faced challenges. These included difficulties in locating information and in assessing their own understanding, as well as attempts to outsource understanding. Additionally, participants’ prior perceptions of the systems’ risks and benefits influenced their information needs. Participants who perceived high risks sought explanations about the intentions behind a system’s deployment, while those who perceived low risks rather asked about the system’s operation. Our work aims to support the inclusion of AI novices in explainability efforts by highlighting their information needs, aims, and challenges. We summarize our findings as five key implications that can inform the design of future explanations for lay stakeholder audiences.

The Comparative Performance of Large Language Models on the Hand Surgery Self-Assessment Examination.

Generative artificial intelligence (AI) models have emerged as capable of producing human-like responses and have showcased their potential in general medical specialties. This study explores the performance of AI systems on the American Society for Surgery of the Hand (ASSH) Self-Assessment Exams (SAE). ChatGPT 4.0 and Bing AI were evaluated on a set of multiple-choice questions drawn from the ASSH SAE online question bank spanning 5 years (2019-2023). Each system was evaluated with 999 questions. Images and video links were inserted into question prompts to allow for complete AI interpretation. The performance of both systems was standardized using the May 2023 version of ChatGPT 4.0 and Microsoft Bing AI, both of which had web browsing and image capabilities. ChatGPT 4.0 scored an average of 66.5% on the ASSH questions. Bing AI scored higher, with an average of 75.3%. Bing AI outperformed ChatGPT 4.0 by an average of 8.8%. As a benchmark, a minimum passing score of 50% was required for continuing medical education credit. Both ChatGPT 4.0 and Bing AI had poorer performance on video-type and image-type questions on analysis of variance testing. Responses from both models contained elements from sources such as PubMed, Journal of Hand Surgery, and American Academy of Orthopedic Surgeons. ChatGPT 4.0 with browsing and Bing AI can both be anticipated to achieve passing scores on the ASSH SAE. Generative AI, with its ability to provide logical responses and literature citations, presents a convincing argument for use as an interactive learning aid and educational tool.

Artificial intelligence large language model scores highly on focused practice designation in metabolic and bariatric surgery board practice questions.

Artificial intelligence models such as ChatGPT (Open AI) have performed well on the exams of various medical and surgical fields. It is not yet known how ChatGPT performs on similar metabolic and bariatric surgery (MBS) questions. Assess the performance of ChatGPT on Focused Practice Designation in Metabolic and Bariatric Surgery board-style questions. United States. Questions obtained from the largest commercially available bank of FPD-MBS practice questions were entered into ChatGPT-4, as is, without prior training. We assessed the overall percentage correct as well as the percentage correct within each of the five American Board of Surgery (ABS) question categories. One-way ANOVA was used to determine if the frequency of correct answers differed between categories. Out of 255 questions, ChatGPT-4 correctly answered 189 (74.1%). Between the five question categories there was no difference between the frequency of correct answers (p = 0.22). It did not matter if questions were entered individually or in groups of up to 10. Without prior training, ChatGPT-4 scored highly when evaluated on the largest practice question bank for the FPD-MBS exam.

A Review of FRCOphth Part 1 Question Banks

A Review of FRCOphth Part 1 Question Banks

Capabilities of GPT-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering

BackgroundEvidence on the performance of Generative Pre-trained Transformer 4 (GPT-4), a large language model (LLM), in the ophthalmology question-answering domain is needed.MethodsWe tested GPT-4 on two 260-question multiple choice question...

Comparison of Gemini Advanced and ChatGPT 4.0's Performances on the Ophthalmology Resident Ophthalmic Knowledge Assessment Program (OKAP) Examination Review Question Banks.

Background With advancements in natural language processing, tools such as Chat Generative Pre-Trained Transformers (ChatGPT) version 4.0 and Google Bard's Gemini Advanced are being increasingly evaluated for their potential in various medical applications. The objective of this study was to systematically assess the performance of these language learning models (LLMs) on both image and non-image-based questions within the specialized field of Ophthalmology. We used a review question bank for the Ophthalmic Knowledge Assessment Program (OKAP) used by ophthalmology residents nationally to prepare for the Ophthalmology Board Exam to assess the accuracy and performance of ChatGPT and Gemini Advanced. Methodology A total of 260 randomly generated multiple-choice questions from the OphthoQuestions question bank were run through ChatGPT and Gemini Advanced. A simulated 260-question OKAP examination was created at random from the bank. Question-specific data were analyzed, including overall percent correct, subspecialty accuracy, whether the question was "high yield," difficulty (1-4), and question type (e.g., image, text). To compare the performance of ChatGPT-4 and Gemini on the difficulty of questions, we utilized the standard deviation of user answer choices to determine question difficulty. In this study, a statistical analysis of Google Sheets was conducted using two-tailed t-tests with unequal variance to compare the performance of ChatGPT-4.0 and Google's Gemini Advanced across various question types, subspecialties, and difficulty levels. Results In total, 259 of the 260 questions were included in the study as one question used a video that any form of ChatGPT could not interpret as of May 1, 2024. For text-only questions, ChatGPT-4.0.0 correctly answered 57.14% (148/259, p < 0.018), and Gemini Advanced correctly answered 46.72% (121/259, p < 0.018). Both versions answered most questions without a prompt and would have received a below-average score on the OKAP. Moreover, there were 27 questions utilizing a secondary prompt in ChatGPT-4.0 compared to 67 questions in Gemini Advanced. ChatGPT-4.0 performed 68.99% on easier questions (<2 on a scale from 1-4) and 44.96% on harder questions (>2 on a scale from 1-4). On the other hand, Gemini Advanced performed 49.61% on easier questions (<2 on a scale from 1-4) and 44.19% on harder questions (>2 on a scale from 1-4). There was a statistically significant difference in accuracy between ChatGPT-4.0 and Gemini Advanced for easy (p < 0.0015) but not for hard (p < 0.55) questions. For image-only questions, ChatGPT-4.0 correctly answered 39.58% (19/48, p < 0.013), and Gemini Advanced correctly answered 33.33% (16/48, p < 0.022), resulting in a statistically insignificant difference in accuracy between ChatGPT-4.0 and Gemini Advanced (p < 0.530). A comparison against text-only and image-based questions demonstrated a statistically significant difference in accuracy for both ChatGPT-4.0 (p < 0.013) and Gemini Advanced (p < 0.022). Conclusions This study provides evidence that ChatGPT-4.0 performs better on the OKAP-style exams and is improved compared to Gemini Advanced within the context of ophthalmic multiple-choice questions. This may show an opportunity for increased worth for ChatGPT in ophthalmic medical education. While showing promise within medical education, caution should be used as a more detailed evaluation of reliability is needed.

Computer Vision Meets Large Language Models: Performance of ChatGPT 4.0 on Dermatology Boards-Style Practice Questions

Background: ChatGPT is a generative artificial intelligence that has numerous professional applications. Applications in medical education are currently being explored. ChatGPT 4.0 performance on image-based dermatology boards-style practice questions has not been assessed. Objective: The objective of this study was to determine the accuracy with which ChatGPT can answer dermatology boards examination practice questions. Methods: 150 multiple-choice questions from the popular question bank DermQbank were inputted into ChatGPT. Of these, 83 were text-only questions and 67 had associated images. These same questions were inputted into ChatGPT again in July 2024. An additional 150 questions were inputted for a total of 300 different questions where 169 were text-only and 133 had associated images. Results: Of the aggregate 300 question data, ChatGPT answered 232 questions correctly (77.3%). ChatGPT performed significantly better with text-only questions than with questions that included images (85.2% (144/169) vs 67.7% (90/133), P&lt;.001). Of image-based questions, ChatGPT performed better with clinical image questions than with dermatopathology questions (69.0% (78/133) vs. 58.8% (10/17), P=.40), but this difference was not statistically significant partially due to the sample size of the dermatopathology questions. Compared to post-graduate year 4 (PGY-4) residents, ChatGPT performed above the 46th percentile. ChatGPT agreed with the answer choice picked by the majority of question bank users 75.3% of the time. Multivariable regression demonstrated that significant predictive variables for ChatGPT answering a question correctly included the percent of dermatology trainees who answered a question correctly and whether the question was text-based (P&lt;.001and P=.004, respectively). Conclusions: ChatGPT answered 77.3% of dermatology board examination practice questions correctly, performing above the 46th percentile of PGY-4 question bank users. If using ChatGPT as a study resource for dermatology board examination preparation, residents should be judicious with exactly how they employ ChatGPT to avoid learning incorrect information.

GPT-4o vs. Human Candidates: Performance Analysis in the Polish Final Dentistry Examination.

Background This study aims to evaluate the performance of OpenAI's GPT-4o in the Polish Final Dentistry Examination (LDEK) and compare it with human candidates' results. The LDEK is a standardized test essential for dental graduates in Poland to obtain their professional license. With artificial intelligence (AI) becoming increasingly integrated into medical and dental education, it is important to assess AI's capabilities in such high-stakes examinations. Materials and methods The study was conducted from August 1 to August 15, 2024, using the Spring 2023 LDEK exam. The exam comprised 200 multiple-choice questions, each with one correct answer among five options. Questions spanned various dental disciplines, including Conservative Dentistry with Endodontics, Pediatric Dentistry, Dental Surgery, Prosthetic Dentistry, Periodontology, Orthodontics, Emergency Medicine, Bioethics and Medical Law, Medical Certification, and Public Health. The exam organizers withdrew one question. GPT-4o was tested on these questions without access to the publicly available question bank. The AI model's responses were recorded, and each answer's confidence level was assessed. Correct answers were determined based on the official key provided by the Center for Medical Education (CEM) in Łódź, Poland. Statistical analyses, including Pearson's chi-square test and the Mann-Whitney U test, were performed to evaluate the accuracy and confidence of ChatGPT's answers across different dental fields. Results GPT-4o correctly answered 141 out of 199 valid questions (70.85%) and incorrectly answered 58 (29.15%). The AI performed better in fields like Conservative Dentistry with Endodontics (71.74%) and Prosthetic Dentistry (80%) but showed lower accuracy in Pediatric Dentistry (62.07%) and Orthodontics (52.63%). A statistically significant difference was observed between ChatGPT's performance on clinical case-based questions (36.36% accuracy) and other factual questions (72.87% accuracy), with a p-value of 0.025. Confidence levels also varied significantly between correct and incorrect answers, with a p-value of 0.0208. Conclusions GPT-4o's performance in the LDEK suggests it has potential as a supplementary educational tool in dentistry. However, the AI's limited clinical reasoning abilities, especially in complex scenarios, reveal a substantial gap between AI and human expertise. While ChatGPT demonstrates strong performance in factual recall, it cannot yet match the critical thinking and clinical judgment exhibited by human candidates.

Artificial intelligence in orthopaedic education: A comparative analysis of ChatGPT and Bing AI's Orthopaedic In‐Training Examination performance

AbstractBackgroundThis study evaluated the performance of generative artificial intelligence (AI) models on the Orthopaedic In‐Training Examination (OITE), an annual exam administered to U.S. orthopaedic residency programs.MethodsChatGPT 3.5 and Bing AI GPT 4.0 were evaluated on standardised sets of multiple‐choice questions drawn from the American Academy of Orthopaedic Surgeons OITE online question bank spanning 5 years (2018–2022). A total of 1165 questions were posed to each AI system. The performance of both systems was standardised using the latest versions of ChatGPT 3.5 and Bing AI GPT 4.0. Historical data of resident scores taken from the annual OITE technical reports was used as a comparison.ResultsAcross the five datasets, ChatGPT 3.5 scored an average of 55.0% on the OITE questions. Bing AI GPT 4.0 scored higher with an average of 80.0%. In comparison, the average performance of orthopaedic residents in national accredited programs was 62.1%. Bing AI GPT 4.0 outperformed ChatGPT 3.5 and Accreditation Council for Graduate Medical Education examinees, and analysis of variance analysis demonstrated p &lt; 0.001 among groups. The best performance was by Bing AI GPT 4.0 on OITE 2020.ConclusionGenerative AI can provide a logical context across answer responses through its in‐depth information searches and citation of resources. This combination presents a convincing argument for the possible uses of AI in medical education as an interactive learning aid.