Accuracy of a Large Language Model as a new tool for optometry education

Abstract

ABSTRACT Clinical relevance The unsupervised introduction of certain Artificial Intelligence tools in optometry education may challenge the proper acquisition of accurate clinical knowledge and skills proficiency. Background Large Language Models like ChatGPT (Generative Pretrained Transformer) are increasingly being used by researchers and students for work and academic assignments. The authoritative and conversationally correct language provided by these tools may mask their inherent limitations when presented with specific scientific and clinical queries. Methods Three sets of 10 queries related to contact lenses & anterior eye, low vision and binocular vision & vision therapy were presented to ChatGPT, with instructions to provide five relevant references to support each response. Three experts and 53 undergraduate and post-graduate students graded from 0 to 10 the accuracy of the responses, and the references were evaluated for precision and relevance. Students graded from 0 to 10 the potential usefulness of ChatGPT for their academic coursework. Results Median scores were 7, 8 and 6 (experts) and 8, 9 and 7.5 (students) for the contact lenses & anterior eye, low vision and binocular vision & vision therapy categories, respectively. Responses to more specific queries were awarded lower scores by both experts (ρ = –0.612; P < 0.001) and students (ρ = –0.578; P = 0.001). Of 150 references, 24% were accurate and 19.3% relevant. Students graded the usefulness of ChatGPT with 7.5 (2 to 9), 7 (3 to 9) and 8.5 (3 to 10) for contact lenses & anterior eye, low vision and binocular vision & vision therapy, respectively. Conclusion Careful expert appraisal of the responses and, particularly, of the references provided by ChatGPT is required in research and academic settings. As the use of these tools becomes widespread, it is essential to take proactive steps to address their limitations and ensure their responsible use.