The Promise and the Challenge: Large Language Models for Patient Education - Are We There Yet?

Abstract

This study aimed to evaluate the potential of large language models (LLMs) for delivering patient education materials. Two LLMs, Gemini and ChatGPT 3.5, were analysed for their ability to provide clear and understandable information on the topic of blepharitis, a common eye condition. The understandability and actionability of the information provided by the LLMs in response to a set of questions were evaluated using PEMAT, a standardised tool for assessing educational materials. The responses included the important aspects of blepharitis, yet the Flesch-Kincaid readability scores were below the suggested range of 60-70 for patient education materials. Gemini received a score of 38.75, whereas ChatGPT 3.5 earned 26.35, suggesting that the content might be too intricate for the target audience.. These findings suggest that while LLMs have the potential to be informative resources, their current readability levels may limit their effectiveness in providing accessible health information to patients. Further research is needed to explore methods for adapting LLM outputs to ensure clear and concise communication suitable for patient education.