Agreement Analysis on Postoperative Care Between ChatGPT and a High-Volume Foot and Ankle Surgeon

Abstract

Introduction/Purpose: ChatGPT is an Artificial intelligence (AI) algorithm based on a user-friendly interface that does not requires advanced programming skills. Since its release to the public, it has made AI more accessible. As a result, the Patient refers to chat GTP for medical inquiries. This study compares the postoperative (PO) information provided by chatGPT to those of a high- volume foot and ankle orthopedics surgeon. Methods: The study includes 251 patients treated for end-stage osteoarthritis with total ankle arthroplasty. Postoperative emails containing inquiries about their PO status were uploaded to chatGTP. We then evaluated the agreement in simple (SA) and complex (CA) answers. The SA abbreviated the answer provided into "yes" or "no" Its agreement analysis was made using Cohens Kappa. In contrast, CA contained detailed information, and answers were classified into complete agreement, partial agreement, or complete disagreement. Additionally, in partial agreement answers, we calculated the percentage of agreement. Finally, we measured the cases where the surgeon added additional information unrelated to the inquiry. Results: There was only a slight agreement in the SA category (K=0.08). In the CA category, we found 27 percent of complete agreement; in 52 percent, we found complete disagreement; in 20 percent, we found only partial agreement. In 50 % of the cases, the surgeon added information unrelated to the question. Conclusion: We found limited agreement between chat GPT and the primary surgeon regarding postoperative information. Indicating that chat, GPT is currently an unreliable source for postoperative management. Table 2 Distribution of agreement percentage between a High-volume foot and ankle surgeon and ChatGPT response to postoperative inquiries from patients with total ankle arthroplasty in those with partial agreement