Evaluating generative pretraining transformer reliability in addressing dental trauma: A cross-sectional observational study on avulsion and intrusion

Abstract

Listen

Translate article

Abstract Introduction: The advancement of artificial intelligence (AI) has revolutionized digital communication, enhancing interactions between humans and computers. This study explores the application of Chat Generative Pretrained Transformer 3.5 (ChatGPT-3.5), in providing accurate information on dental trauma. Materials and Methods: Utilizing a dataset of 45 self-generated questions across three topics, general dental trauma, avulsion, and intrusion, ChatGPT-3.5 generated responses that were subsequently evaluated by five endodontic experts, each with over a decade of experience. The evaluators used a Likert scale to assess the quality of the AI-generated answers, synthesizing reliable scientific evidence and clinical expertise to ensure a thorough analysis. The data obtained from the evaluators’ scores were organized and analyzed using Microsoft Excel software and the Statistical Package for the Social Sciences version 25. For each question, descriptive statistics including the median and interquartile range were computed. Results: The study found that ChatGPT provided reliable information across the three assessed dental topics. Avulsion was rated the highest (4.40 ± 0.717), significantly outperforming general dental trauma (3.97 ± 0.885) (P = 0.005). Intrusion received a rating of 4.13 ± 0.794, showing no significant difference compared to the other topics. Most evaluator scores fell into the “Good” (44.0%) and “Very Good” (38.7%) categories. This indicates a generally positive appraisal of ChatGPT’s performance, with a fair agreement among evaluators, evidenced by a combined Fleiss’s kappa coefficient of 0.324. However, there was variability, particularly with Evaluator 4’s scores differing significantly from those of evaluators 1 and 2. Conclusions: ChatGPT’s responses on general dental trauma, avulsion, and intrusion were generally rated positively, with avulsion responses deemed the most reliable. The study underscores the need for continuous evaluation to maintain the accuracy, reliability, and safety of AI-generated content in endodontics, suggesting AI should serve as a supplementary tool rather than a primary information source.