In recent years artificial intelligence (AI), as anew segment of computer science, has also become increasingly more important in medicine. The aim of this project was to investigate whether the current version of ChatGPT (ChatGPT4.0) is able to answer open questions that could be asked in the context of aGerman board examination in ophthalmology. After excluding image-based questions, 10questions from 15different chapters/topics were selected from the textbook 1000 questions in ophthalmology (1000 Fragen Augenheilkunde 2ndedition, 2014). ChatGPT was instructed by means of aso-called prompt to assume the role of a board certified ophthalmologist and to concentrate on the essentials when answering. Ahuman expert with considerable expertise in the respective topic, evaluated the answers regarding their correctness, relevance and internal coherence. Additionally, the overall performance was rated by school grades and assessed whether the answers would have been sufficient to pass the ophthalmology board examination. The ChatGPT would have passed the board examination in 12out of 15topics. The overall performance, however, was limited with only 53.3% completely correct answers. While the correctness of the results in the different topics was highly variable (uveitis and lens/cataract 100%; optics and refraction 20%), the answers always had ahigh thematic fit (70%) and internal coherence (71%). The fact that ChatGPT4.0 would have passed the specialist examination in 12out of 15topics is remarkable considering the fact that this AI was not specifically trained for medical questions; however, there is aconsiderable performance variability between the topics, with some serious shortcomings that currently rule out its safe use in clinical practice.
Read full abstract