Abstract
Background: This study evaluates the ability of six popular chatbots; ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity, to provide reliable answers to questions concerning keratoconus. Methods: Chatbots responses were assessed using mDISCERN (range: 15-75) and Global Quality Score (GQS) (range: 1-5) metrics. Readability was evaluated using nine validated readability assessments. We also addressed the quality and accountability of websites from which the questions originated. Results: We analyzed 20 websites, 65% "Private practice or independent user" and 35% "Official patient education materials". The mean JAMA benchmark score was 1.40 ± 0.91 (0-4 points), indicating low accountability. Reliability, measured using mDISCERN, ranged from 42.91 ± 3.15 (ChatGPT-3.5) to 46.95 ± 3.53 (Copilot). The most frequent question was "What is keratoconus?" with 70% of websites providing relevant information. This received the highest mDISCERN score (49.30 ± 4.91) and a relatively high GQS score (3.40 ± 0.56) with an Automated Readability Level Calculator score of 13.17 ± 2.13. Moderate positive correlations were determined between the website numbers and both mDISCERN (r = 0.265, p = 0.25) and GQS (r = 0.453, p = 0.05) scores. The quality of information, assessed using the GQS, ranged from 3.02 ± 0.55 (ChatGPT-3.5) to 3.31 ± 0.64 (Gemini) (p = 0.34). The differences between the texts were statistically significant. Gemini emerged as the easiest to read, while ChatGPT-3.5 and Perplexity were the most difficult. Based on mDISCERN scores, Gemini and Copilot exhibited the highest percentage of responses in the "good" range (51-62 points). For the GQS, the Gemini model exhibited the highest percentage of responses in the "good" quality range with 40% of its responses scoring 4-5. Conclusions: While all chatbots performed well, Gemini and Copilot showed better reliability and quality. However, their readability often exceeded recommended levels. Continuous improvements are essential to match information with patients' health literacy for effective use in ophthalmology.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
