Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis.

Abstract

This study aims to evaluate the accuracy of ChatGPT in answering allergic rhinitis (AR) and chronic rhinosinusitis (CRS) related questions. This is a cross-sectional study. Each question was inputted as a separate, independent prompt. Responses to AR (n = 189) and CRS (n = 242) related questions, generated by GPT-3.5 and GPT-4, were independently graded for accuracy by 2 senior rhinology professors, with disagreements adjudicated by a third reviewer. Overall, ChatGPT demonstrated a satisfactory performance, accurately answering over 80% of questions across all categories. Specifically, GPT-4.0's accuracy in responding to AR-related questions significantly exceeded that of GPT-3.5, but distinction not evident in CRS-related questions. Patient-originated questions had a significantly higher accuracy compared to doctor-originated questions when utilizing GPT-4.0 to respond to AR-related questions. This discrepancy was not observed with GPT-3.5 or in the context of CRS-related questions. Across different types of content, ChatGPT excelled in covering basic knowledge, prevention, and emotion for AR and CRS. However, it experienced challenges when addressing questions about recent advancements, a trend consistent across both GPT-3.5 and GPT-4.0 iterations. Importantly, the accuracy of responses remained unaffected when questions were posed in Chinese. Our findings suggest ChatGPT's capability to convey accurate information for AR and CRS patients, and offer insights into its performance across various domains, guiding its utilization and improvement.