P464 Could Chat Generative Pre-Trained Transformer (ChatGPT) be an AI-provided friend of the patients for frequently asked questions concerning their IBD management? An evidence- and guidelines-controlled textual analysis of AI-provided outputs

Abstract

Abstract Background Artificial intelligence (AI) is gradually and potentially entering the everyday lives of patients with Inflammatory Bowel Disease (IBD). Systems such as Chat Generative Pre-Trained Transformer (ChatGPT), based on large language models (LLM), are now within reach. It is necessary to weigh whether these LLM systems can be real generators of medical information and whether the latter is generated based on credible databases and evidence. This study evaluated and analysed whether the outputs of ChatGPT to common questions from IBD patients can provide credible and scientifically reliable outputs. Methods IBD-expert physicians retrieved a list of ten IBD frequently asked questions in their clinical practice. The ten with the highest frequency (Q1-10) were collected from the total number of questions and then input on ChatGPT (see Table) on three different days (18th, 19th, and 20th August 2023), and each output generated by the chatbot was categorised as O1, O2, and O3. The same research team evaluated the AI-generated responses by ChatGPT for each question by objectively comparing them with the available evidence (provided by meta-analyses, systematic reviews and ECCO guidelines). Results Q1 O1-3 were reliable, with a correct definition of the absence of definitive therapy for IBD. In Q2 O1-3, ChatGPT failed to clarify the lack of solid evidence for nutritional therapy in IBD. Q3 O1-3 provided corrected guidance on repeat endoscopic examinations for IBD. Q4 O1-3 did not provide reliable answers on removing topical therapy in ulcerative colitis. In Q5 O1-3 good advice was given to the IBD patient on how to plan a pregnancy. Q6 O1-3 brought out the lack of ChatGPT update (stopped in September 2021), limiting the range of new oral small molecule therapies available for the IBD patient. In Q7 O1-3 ChatGPT correctly outlined the need for individual risk stratification for endoscopic surveillance of colorectal cancer. In Q8 and Q9 O1-3 ChatGPT correctly weighed the risk of infection/cancer during biologic therapy and how heritability is only one piece of the pathogenetic puzzle of IBD. Finally, in Q10 he correctly elucidated the possibility of biological therapy preventing post-operative recurrence of Crohn's disease. Conclusion ChatGPT is a potentially useful complementary tool for communicating with IBD patients. It is necessary to improve the degree to which it is updated and sources on appropriate databases (e.g., Scopus, Web of Science, and MEDLINE). Ultimately, there are still significant limitations. Much of the latest findings, in fact, are excluded from ChatGPT evaluation. A portion of the outputs provided are partially incorrect or not fully detailed and not yet ready to be released to the patient without a physician filter.