“Applications of Multimodal Generative AI in a Real-World Retina Clinic Setting”

Abstract

Purpose: Evaluate a large language model, GPT4 with vision (GPT-4V), for diagnosing vitreoretinal diseases in real-world ophthalmology settings. Methods: A retrospective cross-sectional study at Bascom Palmer Eye Clinic, analyzing patient data from January 2010 to March 2023, assesses GPT-4V’s performance on retinal image analysis and ICD-10 coding across two patient groups: simpler cases (Group A) and complex cases (Group B) requiring more in-depth analysis. Diagnostic accuracy was assessed through open-ended (OEQ) and multiple-choice questions (MCQs) independently verified by three retina specialists. Results: In 256 eyes from 143 patients, GPT4-V demonstrated a 13.7% accuracy for OEQs and 31.3% for MCQs, with ICD-10 code accuracies at 5.5% and 31.3% respectively. Accurately diagnosed posterior vitreous detachment, non-exudative age-related macular degeneration, and retinal detachment. ICD-10 coding was most accurate for non-exudative age-related macular degeneration, central retinal vein occlusion, and macular hole in EOQs, and for posterior vitreous detachment, non-exudative age-related macular degeneration, and retinal detachment in MCQs. No significant difference in diagnostic or coding accuracy was found in Groups A and B. Conclusion: GPT-4V has potential in clinical care and record-keeping, particularly with standardized questions. Its effectiveness in open-ended scenarios is limited, indicating a significant limitation in providing complex medical advice.