GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like Questions

Abstract

Artificial intelligence (AI) and machine learning (ML) have transformed healthcare, with applications in various specialized fields. Neurosurgery can benefit from AI in surgical planning, predicting patient outcomes, and analyzing neuroimaging data. GPT-4, an updated language model with additional training parameters, has exhibited exceptional performance on standardized exams. This study examines GPT-4's competence in neurosurgery through board-style questions, comparing its performance with medical students and residents, to explore its potential in clinical decision-making. GPT-4's performance was examined on 643 Congress of Neurological Surgeons (CNS) Self-Assessment Neurosurgery Exam (SANS) board-style questions from various neurosurgery subspecialties. Of these, 477 were text-based and 166 contained images. GPT-4 refused to answer 52 questions that contained no text. The remaining 591 questions were inputted into GPT-4, and its performance was evaluated based on first-time responses. Raw scores were analyzed across subspecialties and question types, then compared to previous findings on ChatGPT performance against SANS users, medical students, and neurosurgery residents. GPT-4 attempted 91.9% of CNS SANS questions and achieved 76.6% accuracy. The model's accuracy increased to 79.0% for text-only questions. GPT-4 outperformed ChatGPT (p<0.001) and scored highest in Pain/Peripheral Nerve (84%) and lowest in Spine (73%) categories. It exceeded the performance of medical students (26.3%), neurosurgery residents (61.5%), and the national average of SANS users (69.3%) across all categories. GPT-4 outperformed medical students, neurosurgery residents, and the national average of SANS users. The model's accuracy suggests potential applications in educational settings and clinical decision-making, enhancing provider efficiency and improving patient care.