Assessing the performance of ChatGPT's responses to questions related to epilepsy: A cross-sectional study on natural language processing and medical information retrieval

Abstract

BackgroundEpilepsy is a neurological condition marked by frequent seizures and various cognitive and psychological effects. Reliable information is essential for effective treatment. Natural language processing models like ChatGPT are increasingly used in healthcare for information access and data analysis, making it crucial to assess their accuracy. ObjectiveThis study aimed to investigate the accuracy of ChatGPT in providing educational information related to epilepsy. MethodsWe compared the answers from ChatGPT-4 and ChatGPT-3.5 to 57 common epilepsy questions based on the Korean Epilepsy Society's "Epilepsy Patient and Caregiver Guide." Two epileptologists reviewed the responses, with a third serving as an arbiter in cases of disagreement. ResultsOut of 57 questions, 40 responses from ChatGPT-4 had "sufficient educational value," 16 were "correct but inadequate," and one was "mixed with correct and incorrect" information. No answers were entirely incorrect. GPT-4 generally outperformed GPT-3.5 and was often on par with or better than the official guide. ConclusionsChatGPT-4 shows promise as a tool for delivering reliable epilepsy-related information and could help alleviate the educational burden on healthcare professionals. Further research is needed to explore the benefits and limitations of using such models in medical contexts.