Accuracy and Completeness of Bard and Chat-GPT 4 Responses for Questions Derived from the International Consensus Statement on Endoscopic Skull-Base Surgery 2019

Abstract

AbstractArtificial intelligence large language models (LLMs), such as Chat Generative Pre-Trained Transformer 4 (Chat-GPT) by OpenAI and Bard by Google, emerged in 2022 as tools for answering questions, providing information, and offering suggestions to the layperson. These LLMs impact how information is disseminated and it is essential to compare their answers to experts in the corresponding field. The International Consensus Statement on Endoscopic Skull-Base Surgery 2019 (ICAR:SB) is a multidisciplinary international collaboration that critically evaluated and graded the current literature. Objectives Evaluate the accuracy and completeness of Chat-GPT and Bard responses to questions derived from the ICAR:SB policy statements. Design Thirty-four questions were created based on ICAR:SB policy statements and input into Chat-GPT and Bard. Two rhinologists and two neurosurgeons graded the accuracy and completeness of LLM responses, using a 5-point Likert scale. The Wilcoxon rank-sum and Kruskal–Wallis tests were used for analysis. Setting Online. Participants None. Outcomes Compare the mean accuracy and completeness scores between (1) responses generated by Chat-GPT versus Bard and (2) rhinologists versus neurosurgeons. Results Using the Wilcoxon rank-sum test, there were statistically significant differences in (1) accuracy (p < 0.001) and completeness (p < 0.001) of Chat-GPT compared with Bard; and (2) accuracy (p < 0.001) and completeness (p < 0.001) ratings between rhinologists and neurosurgeons. Conclusion Chat-GPT responses are overall more accurate and complete compared with Bard, but both are very accurate and complete. Overall, rhinologists graded lower than neurosurgeons. Further research is needed to better understand the full potential of LLMs.