Evaluation of AI-generated responses by different artificial intelligence chatbots to the clinical decision-making case-based questions in oral and maxillofacial surgery

Abstract

ObjectivesThis study aims to evaluate the correctness of the generated answers by Google Bard, GPT-3.5, GPT-4, Claude-Instant, and Bing chatbots to decision-making clinical questions in the oral and maxillofacial surgery (OMFS) area. Study DesignA group of three board-certified oral and maxillofacial surgeons designed a questionnaire with 50 case-based questions in multiple-choice and open-ended formats. Answers of chatbots to multiple-choice questions were examined against the chosen option by three referees. The chatbots' answers to the open-ended questions were evaluated based on the modified global quality scale. A p-value under 0.05 was considered significant. ResultsBard, GPT-3.5, GPT-4, Claude-Instant, and Bing answered 34%, 36%, 38%, 38%, and 26% of the questions correctly, respectively. In open-ended questions, GPT-4 scored the most answers evaluated as grades “4” or “5,” and Bing scored the most answers evaluated as grades “1” or “2”. There were no statistically significant differences between the five chatbots in responding to the open-ended (P = 0.275) and multiple-choice (P = 0.699) questions. ConclusionConsidering the major inaccuracies in the responses of chatbots, despite their relatively good performance in answering open-ended questions, this technology yet cannot be trusted as a consultant for clinicians in decision-making situations. Clinical SignificanceThese results can affect the way that OMFS is practiced. By providing clinicians with access to AI-generated responses to clinical decision-making questions, we can help them make more informed and accurate decisions, which can lead to better patient outcomes.