Evaluating the Accuracy and Completeness of Artificial Intelligence Responses Against Established Otology Guidelines

Abstract

Background: The incorporation of artificial intelligence (AI), especially large language models like Generative Pretrained Transformer 4 (GPT-4), into medical practice is a burgeoning field of interest. This research evaluates the applicability of GPT-4 in otology by analyzing its responses to queries based on otologic clinical practice guidelines. Methods: Key guidelines from otology were selected, and corresponding questions were formulated to examine GPT-4’s interpretation and response accuracy. Two independent reviewers assessed the AI-generated answers for accuracy and completeness, using a structured Likert scale. A re-evaluation was conducted to evaluate the reproducibility of the results. Results: The analysis showed a high accuracy level (mean score: 4.75 of 5) and completeness (mean score: 2.88 of 3) in GPT-4’s responses. The interrater agreement, as indicated by Cohen κ, was substantial. GPT-4 consistently advised on individualized treatment plans and professional consultation, particularly for guidelines with weaker evidence, demonstrating its cautious approach to handling medical information. Conclusion: GPT-4 exhibits promising potential as an auxiliary tool in otology, providing accurate and comprehensive information. However, its role should be viewed as supplementary, with emphasis on continual updates and careful monitoring to align with evolving medical knowledge. Future studies are recommended to further explore the depth of AI application in diverse clinical scenarios and its real-time impact on clinical outcomes.