Can ChatGPT Aid Clinicians in Educating Patients on the Surgical Management of Glaucoma?

Abstract

ChatGPT can help health care providers automate the quality assessment of online health information, but it does not produce easier-to-understand responses compared with existing online health information. To compare the readability of ChatGPT-generated health information about glaucoma surgery to existing material online and to evaluate ChatGPT's ability to analyze the quality of information found online about glaucoma surgery. ChatGPT was asked to create patient handouts on glaucoma surgery using 7 independent prompts, aiming to generate sixth grade level reading material. Existing patient-targeted online health information about glaucoma surgery was selected from the top 50 search results of 3 search engines, excluding advertisements, blog posts, information intended for health professionals, irrelevant content, and duplicate links. Four validated tools were used to assess readability, and the readability of the ChatGPT-generated material was compared with the readability of existing online information. The DISCERN instrument was used for the quality assessment of online materials. The DISCERN instrument was also programmed to use ChatGPT to evaluate its ability to analyze quality. R software and descriptive statistics were used for data analysis. Thirty-five webpages were included. There was no difference between the reading level of online webpages (12th grade) and the reading level of ChatGPT-generated responses (11th grade), despite the ChatGPT prompts asking for simple language and a sixth grade reading level. The quality of health content was "fair," with only 5 resources receiving an "excellent" score. ChatGPT scored the quality of health resources with high precision ( r =0.725). Patient-targeted information on glaucoma surgery is beyond the reading level of the average patient, therefore at risk of not being understood, and is of subpar quality, per DISCERN tool scoring. ChatGPT did not generate documents at a lower reading level as prompted, but this tool can aid in automating the time-consuming and subjective process of quality assessment.