Flesch-Kincaid Research Articles

Comparative Performance of the Leading Large Language Models in Answering Complex Rhinoplasty Consultation Questions.

Background: Various large language models (LLMs) can provide human-level medical discussions, but they have not been compared regarding rhinoplasty knowledge. Objective: To compare the leading LLMs in answering complex rhinoplasty consultation questions as evaluated by plastic surgeons. Methods: Ten open-ended rhinoplasty consultation questions were presented to ChatGPT-4o, Google Gemini, Claude, and Meta-AI LLMs. The responses were randomized and ranked by seven rhinoplasty-specializing plastic surgeons (1 = worst, 4 = best) considering their quality. Textual readability was analyzed via Flesch Reading Ease (FRE) and Flesch-Kincaid Grade (FKG). Results: Claude provided the top answers for seven questions while ChatGPT provided the top answers for three questions. In overall collective scoring, Claude provided the best answers with 224 points, followed by ChatGPT's 200, Meta's 138, and Gemini's 138 scores. Claude (mean score/question 3.20 ± 1.00) significantly outperformed all the other models (p < 0.05), while ChatGPT (mean score/question 2.86 ± 0.94) outperformed Meta and Gemini. Meta and Gemini performed similarly. Meta had a significantly lower FKG than Claude and ChatGPT and a significantly lower FRE than ChatGPT. Conclusion: According to ratings by seven rhinoplasty-specializing surgeons, Claude provided the best answers for a set of complex rhinoplasty consultation questions, followed by ChatGPT. Future studies are warranted to continue comparing these models as they evolve.

Pediatric Supracondylar Humerus and Diaphyseal Femur Fractures: A Comparative Analysis of Chat Generative Pretrained Transformer and Google Gemini Recommendations Versus American Academy of Orthopaedic Surgeons Clinical Practice Guidelines.

Artificial intelligence (AI) chatbots, including chat generative pretrained transformer (ChatGPT) and Google Gemini, have significantly increased access to medical information. However, in pediatric orthopaedics, no study has evaluated the accuracy of AI chatbots compared with evidence-based recommendations, including the American Academy of Orthopaedic Surgeons clinical practice guidelines (AAOS CPGs). The aims of this study were to compare responses by ChatGPT-4.0, ChatGPT-3.5, and Google Gemini with AAOS CPG recommendations on pediatric supracondylar humerus and diaphyseal femur fractures regarding accuracy, supplementary and incomplete response patterns, and readability. ChatGPT-4.0, ChatGPT-3.5, and Google Gemini were prompted by questions created from 13 evidence-based recommendations (6 from the 2011 AAOS CPG on pediatric supracondylar humerus fractures; 7 from the 2020 AAOS CPG on pediatric diaphyseal femur fractures). Responses were anonymized and independently evaluated by 2 pediatric orthopaedic attending surgeons. Supplementary responses were, in addition, evaluated on whether no, some, or many modifications were necessary. Readability metrics (response length, Flesch-Kincaid reading level, Flesch Reading Ease, Gunning Fog Index) were compared. Cohen Kappa interrater reliability (κ) was calculated. χ2 analyses and single-factor analysis of variance were utilized to compare categorical and continuous variables, respectively. Statistical significance was set with P <0.05. ChatGPT-4.0, ChatGPT-3.5, and Google Gemini were accurate in 11/13, 9/13, and 11/13, supplementary in 13/13, 11/13, and 13/13, and incomplete in 3/13, 4/13, and 4/13 recommendations, respectively. Of 37 supplementary responses, 17 (45.9%), 19 (51.4%), and 1 (2.7%) required no, some, and many modifications, respectively. There were no significant differences in accuracy (P = 0.533), supplementary responses (P = 0.121), necessary modifications (P = 0.580), and incomplete responses (P = 0.881). Overall κ was moderate at 0.55. ChatGPT-3.5 provided shorter responses (P = 0.002), but Google Gemini was more readable in terms of Flesch-Kincaid Grade Level (P = 0.002), Flesch Reading Ease (P < 0.001), and Gunning Fog Index (P = 0.021). While AI chatbots provided responses with reasonable accuracy, most supplemental information required modification and had complex readability. Improvements are necessary before AI chatbots can be reliably used for patient education. Level IV.

Tailoring glaucoma education using large language models: Addressing health disparities in patient comprehension.

This study evaluates the efficacy of GPT-4, a Large Language Model, in simplifying medical literature for enhancing patient comprehension in glaucoma care. GPT-4 was used to transform published abstracts from 3 glaucoma journals (n = 62) and patient education materials (Patient Educational Model [PEMs], n = 9) to a 5th-grade reading level. GPT-4 was also prompted to generate de novo educational outputs at 6 different education levels (5th Grade, 8th Grade, High School, Associate's, Bachelor's and Doctorate). Readability of both transformed and de novo materials was quantified using Flesch Kincaid Grade Level (FKGL) and Flesch Reading Ease (FKRE) Score. Latent semantic analysis (LSA) using cosine similarity was applied to assess content consistency in transformed materials. The transformation of abstracts resulted in FKGL decreasing by an average of 3.21 points (30%, P < .001) and FKRE increasing by 28.6 points (66%, P < .001). For PEMs, FKGL decreased by 2.38 points (28%, P = .0272) and FKRE increased by 12.14 points (19%, P = .0459). LSA revealed high semantic consistency, with an average cosine similarity of 0.861 across all abstracts and 0.937 for PEMs, signifying topical themes were quantitatively shown to be consistent. This study shows that GPT-4 effectively simplifies medical information about glaucoma, making it more accessible while maintaining textual content. The improved readability scores for both transformed materials and GPT-4 generated content demonstrate its usefulness in patient education across different educational levels.

Patient- and clinician-based evaluation of large language models for patient education in prostate cancer radiotherapy.

This study aims to evaluate the capabilities and limitations of large language models (LLMs) for providing patient education for men undergoing radiotherapy for localized prostate cancer, incorporating assessments from both clinicians and patients. Six questions about definitive radiotherapy for prostate cancer were designed based on common patient inquiries. These questions were presented to different LLMs [ChatGPT‑4, ChatGPT-4o (both OpenAI Inc., San Francisco, CA, USA), Gemini (Google LLC, Mountain View, CA, USA), Copilot (Microsoft Corp., Redmond, WA, USA), and Claude (Anthropic PBC, San Francisco, CA, USA)] via the respective web interfaces. Responses were evaluated for readability using the Flesch Reading Ease Index. Five radiation oncologists assessed the responses for relevance, correctness, and completeness using afive-point Likert scale. Additionally, 35prostate cancer patients evaluated the responses from ChatGPT‑4 for comprehensibility, accuracy, relevance, trustworthiness, and overall informativeness. The Flesch Reading Ease Index indicated that the responses from all LLMs were relatively difficult to understand. All LLMs provided answers that clinicians found to be generally relevant and correct. The answers from ChatGPT‑4, ChatGPT-4o, and Claude AI were also found to be complete. However, we found significant differences between the performance of different LLMs regarding relevance and completeness. Some answers lacked detail or contained inaccuracies. Patients perceived the information as easy to understand and relevant, with most expressing confidence in the information and awillingness to use ChatGPT‑4 for future medical questions. ChatGPT-4's responses helped patients feel better informed, despite the initially standardized information provided. Overall, LLMs show promise as atool for patient education in prostate cancer radiotherapy. While improvements are needed in terms of accuracy and readability, positive feedback from clinicians and patients suggests that LLMs can enhance patient understanding and engagement. Further research is essential to fully realize the potential of artificial intelligence in patient education.

Proficiency, Clarity, and Objectivity of Large Language Models Versus Specialists' Knowledge on COVID-19 Impacts in Pregnancy: A Cross-Sectional Pilot Study.

The COVID-19 pandemic has significantly strained healthcare systems globally, leading to an overwhelming influx of patients and exacerbating resource limitations. Concurrently, an "infodemic" of misinformation, particularly prevalent in women's health, has emerged. This challenge has been pivotal for healthcare providers, especially gynecologists and obstetricians, in managing pregnant women's health. The pandemic heightened risks for pregnant women from COVID-19, necessitating balanced advice from specialists on vaccine safety versus known risks. Additionally, the advent of generative Artificial Intelligence (AI), such as large language models (LLMs), offers promising support in healthcare. However, they necessitate rigorous testing. To assess LLMs' proficiency, clarity, and objectivity regarding COVID-19 impacts in pregnancy. This study evaluates four major AI prototypes (ChatGPT-3.5, ChatGPT-4, Microsoft Copilot, and Google Bard) using zero-shot prompts in a questionnaire validated among 159 Israeli gynecologists and obstetricians. The questionnaire assesses proficiency in providing accurate information on COVID-19 in relation to pregnancy. Text-mining, sentiment analysis, and readability (Flesch-Kincaid grade level and Flesch Reading Ease Score) were also conducted. In terms of LLMs' knowledge, ChatGPT-4 and Microsoft Copilot each scored 97% (n=32/33), Google Bard 94% (n=31/33), and ChatGPT-3.5 82% (n=27/33). ChatGPT-4 incorrectly stated an increased risk of miscarriage due to COVID-19. Google Bard and Microsoft Copilot had minor inaccuracies concerning COVID-19 transmission and complications. At the sentiment analysis, Microsoft Copilot achieved the least negative score (-4), followed by ChatGPT-4 (-6) and Google Bard ( -7), while ChatGPT-3.5 obtained the most negative score (-12). Finally, concerning the readability analysis, Flesch-Kincaid Grade Level and Flesch Reading Ease Score showed that Microsoft Copilot was the most accessible at 9.9 and 49, followed by ChatGPT-4 at 12.4 and 37.1, while ChatGPT-3.5 (12.9 and 35.6) and Google Bard (12.9 and 35.8) generated particularly complex responses. The study highlights varying knowledge levels of LLMs in relation to COVID-19 and pregnancy. ChatGPT-3.5 showed the least knowledge and alignment with scientific evidence. Readability and complexity analyses suggest that each AI's approach was tailored to specific audiences, with ChatGPT versions being more suitable for specialized readers and Microsoft Copilot for the general public. Sentiment analysis revealed notable variations in the way LLMs communicated critical information, underscoring the essential role of neutral and objective healthcare communication in ensuring that pregnant women, particularly vulnerable during the COVID-19 pandemic, receive accurate and reassuring guidance. Overall, ChatGPT-4, Microsoft Copilot, and Google Bard generally provided accurate, updated information on COVID-19 and vaccines in maternal and fetal health, aligning with health guidelines. The study demonstrated the potential role of AI in supplementing healthcare knowledge, with a need for continuous updating and verification of AI knowledge bases. The choice of AI tool should consider the target audience and required information detail level.

Improving readability in AI-generated medical information on fragility fractures: the role of prompt wording on ChatGPT's responses.

Large language models like ChatGPT can enhance patients' understanding of medical information, making health decisions more accessible. Complex terms, such as "fragility fracture," can confuse patients, so presenting its medical content in plain language is crucial. This study explored whether conversational prompts improve readability and understanding compared to keyword-based prompts when generating patient-centered health information on fragility fractures. The 32 most frequently searched keywords related to "fragility fracture" and "osteoporotic fracture" were identified using Google Trends. From this set, 24 keywords were selected based on relevance and entered sequentially into ChatGPT. Each keyword was tested with two prompt types: (1) plain language with keywords embedded and (2) keywords alone. The readability and comprehensibility of the AI-generated responses were assessed using the Flesch-Kincaid reading ease (FKRE) and Flesch-Kincaid grade level (FKGL), respectively. The scores of the responses were compared using the Mann-Whitney U test. The FKRE scores indicated significantly higher readability with plain language prompts (median 34.35) compared to keyword-only prompts (median 23.60). Similarly, the FKGL indicated a lower grade level for plain language prompts (median 12.05) versus keyword-only (median 14.50), with both differences achieving statistical significance. Our findings suggest that using conversational prompts can enhance the readability of AI-generated medical information on fragility fractures. Clinicians and content creators should consider this approach when using AI for patient education to optimize comprehension.

Evaluation of Multi-Lingual Simplifications of IR Procedural Reports Using GPT-4.

This study assesses the feasibility of Large Language Models like GPT-4 (OpenAI, San Francisco, CA, USA) to summarize interventional radiology (IR) procedural reports to improve layperson understanding and translate medical texts into multiple languages. 200 reports from eight categories were summarized using GPT-4. Readability was assessed with Flesch-Kincaid Reading Level (FKRL) and Flesch Reading Ease Score (FRES). Accuracy was assessed by 8 IRs. Summaries were translated into Spanish, Korean, Chinese, and Swahili and their accuracy were assessed by eight bilingual IRs. The original reports' FKRL of 10.7 and FRES of 41.9 improved to 7.0, and 73.0, respectively. Summaries were mostly accurate, with minimal misinformation. Translations introduced an increase in number of misinformation but no significant increase in critically wrong information. Layperson comprehension scores improved significantly from 2.5 to 4.3 out of 5 after summarization. Overall, GPT-4 enhanced report readability and comprehension, suggesting potential for broader application in improving patient communication.

Enhancing patient education on the role of tibial osteotomy in the management of knee osteoarthritis using a customized ChatGPT: a readability and quality assessment

IntroductionKnee osteoarthritis (OA) significantly impacts the quality of life of those afflicted, with many patients eventually requiring surgical intervention. While Total Knee Arthroplasty (TKA) is common, it may not be suitable for younger patients with unicompartmental OA, who might benefit more from High Tibial Osteotomy (HTO). Effective patient education is crucial for informed decision-making, yet most online health information has been found to be too complex for the average patient to understand. AI tools like ChatGPT may offer a solution, but their outputs often exceed the public's literacy level. This study assessed whether a customised ChatGPT could be utilized to improve readability and source accuracy in patient education on Knee OA and tibial osteotomy.MethodsCommonly asked questions about HTO were gathered using Google's “People Also Asked” feature and formatted to an 8th-grade reading level. Two ChatGPT-4 models were compared: a native version and a fine-tuned model (“The Knee Guide”) optimized for readability and source citation through Instruction-Based Fine-Tuning (IBFT) and Reinforcement Learning from Human Feedback (RLHF). The responses were evaluated for quality using the DISCERN criteria and readability using the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL).ResultsThe native ChatGPT-4 model scored a mean DISCERN score of 38.41 (range 25–46), indicating poor quality, while “The Knee Guide” scored 45.9 (range 33–66), indicating moderate quality. Cronbach's Alpha was 0.86, indicating good interrater reliability. “The Knee Guide” achieved better readability with a mean FKGL of 8.2 (range 5–10.7, ±1.42) and a mean FRES of 60 (range 47–76, ±7.83), compared to the native model's FKGL of 13.9 (range 11–16, ±1.39) and FRES of 32 (range 14–47, ±8.3). These differences were statistically significant (p &amp;lt; 0.001).ConclusionsFine-tuning ChatGPT significantly improved the readability and quality of HTO-related information. “The Knee Guide” demonstrated the potential of customized AI tools in enhancing patient education by making complex medical information more accessible and understandable.

Investigating the role of large language models on questions about refractive surgery.

Large language models (LLMs) are becoming increasingly popular and are playing an important role in providing accurate clinical information to both patients and physicians. This study aimed to investigate the effectiveness of ChatGPT-4.0, Google Gemini, and Microsoft Copilot LLMs for responding to patient questions regarding refractive surgery. The LLMs' responses to 25 questions about refractive surgery, which are frequently asked by patients, were evaluated by two ophthalmologists using a 5-point Likert scale, with scores ranging from 1 to 5. Furthermore, the DISCERN scale was used to assess the reliability of the language models' responses, whereas the Flesch Reading Ease and Flesch-Kincaid Grade Level indices were used to evaluate readability. Significant differences were found among all three LLMs in the Likert scores (p=0.022). Pairwise comparisons revealed that ChatGPT-4.0's Likert score was significantly higher than that of Microsoft Copilot, while no significant difference was found when compared to Google Gemini (p=0.005 and p=0.087, respectively). In terms of reliability, ChatGPT-4.0 stood out, receiving the highest DISCERN scores among the three LLMs. However, in terms of readability, ChatGPT-4.0 received the lowest score. ChatGPT-4.0's responses to inquiries regarding refractive surgery were more intricate for patients compared to other language models; however, the information provided was more dependable and accurate.

Osteoporosis Knowledge Assessment Tool – Tamil (OKAT-T) in Postmenopausal Women: A Validity and Reliability Study.

IntroductionOsteoporosis is a silent disease that is more prevalent among postmenopausal women (PMW) due to hormonal transition. Various toolkits, including the Osteoporosis Knowledge Assessment Tool (OKAT), were available for the knowledge assessment. The Osteoporosis-related knowledge is crucial for preventing osteoporosis, but there is no validated, reliable questionnaire in Tamil to measure this knowledge. AimTo validate the Tamil version of the OKAT (OKAT-T) and its psychometric properties to measure the knowledge of osteoporosis for use in postmenopausal women. MethodologyThis cross-sectional study was done in two phases, as translation of OKAT in Tamil and validation of OKAT-T among 430 postmenopausal women in both rural and urban regions. Reliability was examined by the Flesch reading ease (FKRE&G) and McNemar’s test, along with difficulty index, item discrimination and item-total correlations, inter-item consistency (Cronbach’s alpha coefficient). ResultThe overall mean age of the 430 PMW was 59.53 ± 9.83 years. The results showed a good and satisfactory face validity and FKRE&G score (55.9). The Cronbach’s alpha for the overall scale was calculated as 0.85 and considered to be good and satisfactory. As per the difficulty index, 19 items had a 0.3 to 0.7, implying that the questionnaire was easy to understand and satisfactory. Similarly, a test-retest was assessed, which was statistically significant for only six items out of 26, showing that the tool has stable reliability. ConclusionThe management of chronic disorders such as osteoporosis has become more challenging for patients especially among PMW and healthcare professionals due to the increasing life expectancy and urbanization. The use of an Osteoporosis Knowledge Assessment tool that has been tailored to people’s understanding and developed in the local language can raise awareness levels about osteoporosis, encourage the adoption of osteoprotective strategies, and provide guidance on treatment options.

Enhancing Patient Comprehension of Glomerular Disease Treatments Using ChatGPT.

Background/Objectives: It is often challenging for patients to understand treatment options, their mechanisms of action, and the potential side effects of each treatment option for glomerular disorders. This study explored the ability of ChatGPT to simplify these treatment options to enhance patient understanding. Methods: GPT-4 was queried on sixty-seven glomerular disorders using two distinct queries for a general explanation and an explanation adjusted for an 8th grade level or lower. Accuracy was rated on a scale of 1 (incorrect) to 5 (correct and comprehensive). Readability was measured using the average of the Flesch-Kincaid Grade (FKG) and SMOG indices, along with the Flesch Reading Ease (FRE) score. The understandability score (%) was determined using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P). Results: GPT-4's general explanations had an average readability level of 12.85 ± 0.93, corresponding to the upper end of high school. When tailored for patients at or below an 8th-grade level, the readability improved to a middle school level of 8.44 ± 0.72. The FRE and PEMAT-P scores also reflected improved readability and understandability, increasing from 25.73 ± 6.98 to 60.75 ± 4.56 and from 60.7% to 76.8% (p < 0.0001 for both), respectively. The accuracy of GPT-4's tailored explanations was significantly lower compared to the general explanations (3.99 ± 0.39 versus 4.56 ± 0.66, p < 0.0001). Conclusions: ChatGPT shows significant potential for enhancing the readability and understandability of glomerular disorder therapies for patients, but at a cost of reduced comprehensiveness. Further research is needed to refine the performance, evaluate the real-world impact, and ensure the ethical use of ChatGPT in healthcare settings.

Artificial Doctors: Performance of Chatbots as a Tool for Patient Education on Keratoconus.

We aimed to compare the answers given by ChatGPT, Bard, and Copilot and that obtained from the American Academy of Ophthalmology (AAO) website to patient-written questions related to keratoconus in terms of accuracy, understandability, actionability, and readability to find out whether chatbots can be used in patient education. Twenty patient-written questions obtained from the AAO website related to keratoconus were asked to ChatGPT, Bard, and Copilot. Two ophthalmologists independently assessed the answers obtained from chatbots and the AAO website in terms of accuracy, understandability, and actionability according to the Structure of Observed Learning Outcome taxonomy, Patient Education Materials Assessment Tool-Understandability, and Patient Education Materials Assessment Tool-Actionability tests, respectively. The answers were also compared for readability according to the Flesch Reading Ease scores obtained through the website. Bard had significantly higher scores compared with ChatGPT-3.5, Copilot, and AAO website according to Structure of Observed Learning Outcome taxonomy and Patient Education Materials Assessment Tool-Understandability (P<0.001 for each), whereas there was no significant difference between the other groups. Bard and ChatGPT achieved significantly higher scores than the AAO website according to the Patient Education Materials Assessment Tool-Actionability scale (P=0.001). The AAO website achieved significantly higher scores than the Bard on the Flesch Reading Ease scale, whereas there was no significant difference between the other groups (P=0.017). Chatbots are promising to provide accurate, understandable, and actionable answers. Chatbots can be a valuable aid in the education of patients with keratoconus under clinician supervision. In this way, unnecessary hospital visits can be prevented, and the burden on the health care system can be alleviated, while patient awareness can be raised.

Quality and readability of online information on Keratoconus in Portugal and Brazil

Introduction Keratoconus patients turn to the internet for answers to their disease expectations. Webpages are not filtered or submitted to evaluation before getting published. We aim to evaluate the quality and readability of the online information regarding keratoconus in Portugal and Brazil. Methods Two independent ophthalmologists and one ophthalmologist supervisor evaluated 30 Portuguese and 30 Brazilian websites by order of appearance in Google with the word “Queratocone” and “Ceratocone”, respectively. Two quality scores were used: a quality index of consumer health information (DISCERN) and the Journal of the American Medical Association (JAMA) benchmark. Readability was evaluated with 3 scores: FleschKincaid Reading Ease (FRE), FleschKincaid Grade (FKG) and Automated Readability Index (ARI). Results Sites from private hospitals or clinics were the most prevalent in both countries, followed by health platform sites. Final JAMA benchmark was 1.13 ± 1.18 in Portugal and 1.07 ± 1.00 in Brazil. Final DISCERN was 34.07 ± 11.71 in Portugal and 38.17 ± 10.51 in Brazil. FRE and FKG scores denoted “difficult to read” and “college school level” in both countries; ARI denoted “professor” level in Portugal and “college student” level in Brazil needed to understand the text, a statistically significant difference. There was no correlation between Google ranking and quality and readability scores. Discussion and Conclusions The information on keratoconus available online to Portuguese-speaking patients is of poor quality and difficult to interpret. Ophthalmologists have a shared responsibility to tackle this challenge through multifold efforts. Educating our patients on how to find reputable websites can help them navigate their life with keratoconus.

Large Language Models May Help Patients Understand Peer-Reviewed Scientific Articles About Ophthalmology: Development and Usability Study.

Adequate health literacy has been shown to be important for the general health of a population. To address this, it is recommended that patient-targeted medical information is written at a sixth-grade reading level. To make well-informed decisions about their health, patients may want to interact directly with peer-reviewed open access scientific articles. However, studies have shown that such text is often written with highly complex language above the levels that can be comprehended by the general population. Previously, we have published on the use of large language models (LLMs) in easing the readability of patient-targeted health information on the internet. In this study, we continue to explore the advantages of LLMs in patient education. This study aimed to explore the use of LLMs, specifically ChatGPT (OpenAI), to enhance the readability of peer-reviewed scientific articles in the field of ophthalmology. A total of 12 open access, peer-reviewed papers published by the senior authors of this study (ET and RA) were selected. Readability was assessed using the Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook tests. ChatGPT 4.0 was asked "I will give you the text of a peer-reviewed scientific paper. Considering that the recommended readability of the text is 6th grade, can you simplify the following text so that a layperson reading this text can fully comprehend it? - Insert Manuscript Text -". Appropriateness was evaluated by the 2 uveitis-trained ophthalmologists. Statistical analysis was performed in Microsoft Excel. ChatGPT significantly lowered the readability and length of the selected papers from 15th to 7th grade (P<.001) while generating responses that were deemed appropriate by expert ophthalmologists. LLMs show promise in improving health literacy by enhancing the accessibility of peer-reviewed scientific articles and allowing the general population to interact directly with medical literature.

Readability of cerebrovascular diseases online educational material from major cerebrovascular organizations

IntroductionThe internet is an essential resource for patients and their loved ones to understand their medical conditions, and professional medical organizations have taken great strides to develop educational material targeting...

Acceptability and readability of ChatGPT-4 based responses for frequently asked questions about strabismus and amblyopia.

To evaluate the compatibility and readability of ChatGPT-4 in providing responses to common inquiries about strabismus and amblyopia. A series of commonly asked questions were compiled, covering topics such as the definition, prevalence, diagnostic approaches, surgical and non-surgical treatment alternatives, postoperative guidelines, surgery-related risks, and visual prognosis associated with strabismus and amblyopia. Each question was asked three times on the online ChatGPT-4 platform both in English and French, with data collection conducted on February 18, 2024. The responses generated by ChatGPT-4 were evaluated by two independent pediatric ophthalmologists, who classified them as "acceptable," "unacceptable," or "incomplete." Additionally, an online readability assessment tool called "readable" was utilized for readability analysis. The majority of responses, totaling 97% of the questions regarding strabismus and amblyopia, consistently met the criteria for acceptability. Only 3% of responses were classified as incomplete, with no instances of unacceptable responses observed. The average Flesch-Kincaid Grade Level and Flesch Reading Ease Score were calculated as 14.53±1.8 and 23.63±8.2, respectively. Furthermore, the means for all readability indices, including the Coleman-Liau index, the Gunning Fog index, and the SMOG index, were found to be 15.75±1.4, 16.96±2.4, and 16.05±1.6, respectively. ChatGPT-4 consistently produced acceptable responses to the majority of the questions asked (97%). Nevertheless, the readability of these responses proved challenging for the average layperson, requiring a college-level education for comprehension. Further improvements, particularly in terms of readability, are necessary to enhance the advisory capacity of this AI software in providing eye and health-related guidance for patients, physicians, and the general public.

Readability of Online Pediatric Orthopaedic Trauma Patient Education Materials.

The importance of web resources for educating orthopaedic trauma patients is well recognized. Yet these resources often exceed the sixth-grade reading level and are too complex for most patients to understand. This study examines the readability of the American Academy of Orthopaedic Surgeons' (AAOS) pediatric trauma-related educational content compared with similar materials from the Pediatric Orthopaedic Society of North America (POSNA). Eleven AAOS (https://www.orthoinfo.org/) and nine POSNA (https://orthokids.org) pediatric trauma education articles were included. Articles' readability was assessed by two independent examiners using (1) the Flesch-Kincaid Grade Level (FKGL) and the Flesch Reading Ease (FRE) algorithms. The FKGL mean was compared with the sixth-grade level and the average American reading level using a one-sample t-test. A two-sample t-test evaluated the differences in readability between the AAOS and POSNA materials. A total of 15 unique articles were included. Excellent agreement (>90%) was noted between reviewers for FKGL and FRE. The average (SD) FKGL for AAOS and POSNA articles were 8.5 (0.8) and 9.0 (1.5), respectively, and the FRE scores were 61.2 (3.8) for AAOS and 61.7 (7.7) for POSNA. Only one AAOS article met the sixth-grade reading level. The average readability levels of the AAOS and POSNA articles were markedly higher than the recommended sixth-grade level (P < 0.001). No notable difference was found in FKGL (P = 0.47) or FRE (P = 0.89) when comparing AAOS and POSNA articles. This study indicates that the readability of the AAOS pediatric trauma and POSNA trauma-related articles is well above that which is recommended for the general public, and this has remained so for more than a decade. Both POSNA and AAOS trauma education materials have high readability levels, potentially hindering patient understanding. Improving the readability of these widely used trauma education resources is overdue and will likely improve patient comprehension.

Quality and Readability of Hearing Aid-Related Websites in English and Spanish.

To determine the readability and quality of both English and Spanish Web sites for the topic of hearing aids. Cross-sectional Web site analysis. Various online search engines. The term "hearing aid" was queried across four popular search engines. The first resulted 75 English Web sites and first resulted 75 Spanish Web sites were extracted for data collection. Web sites that met the inclusion criteria were stratified by the presence of a Health on the Net Code (HONCode) certificate. Articles were then compiled to be independently reviewed by experts on hearing aids, using the DISCERN criteria, which allowed assessment of the quality of the Web sites. Readability was assessed by calculating the Flesch Reading Ease Score in English and the Fernandez Huerta Formula in Spanish. Readability and quality were both analyzed, comparing scores to their respective language and cross-comparing. There were 37 English Web sites and 30 Spanish Web sites that met inclusion criteria. When analyzing readability, English Web sites were determined to be significantly more difficult to read (average = 55.37, standard deviation [SD] = 7.73, 95% confidence interval [CI] = 52.9-57.9) than the Spanish Web site counterparts (average = 58.64, SD = 5.26, 95% CI = 56.8-60.5, p = 0.035). For quality, Spanish Web sites (average = 38, SD = 9.7, 95% CI = 34.5-41.5) were determined to be of significantly higher quality than English Web sites (average = 32.16, SD = 10.60, 95% CI = 29.7-34.6). Additionally, there was a significant difference between the non-HONCode English Web sites versus the non-HONCode Spanish Web sites (p = 0.0081), signifying that Spanish non-HONCode certified Web sites were less reliable than non-HONCode certified English Web sites. The present study highlights the importance and necessity of providing quality, readable materials to patients seeking information regarding hearing aids. This study shows that both English and Spanish Web sites are written at a level that is much higher than the American Medical Association (AMA)-recommended sixth-grade reading level, and no Web site included in this study fell at or below the AMA-recommended sixth-grade reading level. English and Spanish Web sites also lacked consistency and quality, as evidenced by their wide variability in DISCERN scores. Specifically, Hispanic patients are more likely to suffer long-term consequences of their health care due to low levels of health literacy. It is important to bridge this gap by providing adequate reading materials. It is especially important to provide evidence-based claims that are directly supported by experts in the field.

ChatGPT as a patient education tool in colorectal cancer-An in-depth assessment of efficacy, quality and readability.

Artificial intelligence (AI) chatbots such as Chat Generative Pretrained Transformer-4 (ChatGPT-4) have made significant strides in generating human-like responses. Trained on an extensive corpus of medical literature, ChatGPT-4 has the potential to augment patient education materials. These chatbots may be beneficial to populations considering a diagnosis of colorectal cancer (CRC). However, the accuracy and quality of patient education materials are crucial for informed decision-making. Given workforce demands impacting holistic care, AI chatbots can bridge gaps in CRC information, reaching wider demographics and crossing language barriers. However, rigorous evaluation is essential to ensure accuracy, quality and readability. Therefore, this study aims to evaluate the efficacy, quality and readability of answers generated by ChatGPT-4 on CRC, utilizing patient-style question prompts. To evaluate ChatGPT-4, eight CRC-related questions were derived using peer-reviewed literature and Google Trends. Eight colorectal surgeons evaluated AI responses for accuracy, safety, appropriateness, actionability and effectiveness. Quality was assessed using validated tools: the Patient Education Materials Assessment Tool (PEMAT-AI), modified DISCERN (DISCERN-AI) and Global Quality Score (GQS). A number of readability assessments were measured including Flesch Reading Ease (FRE) and the Gunning Fog Index (GFI). The responses were generally accurate (median 4.00), safe (4.25), appropriate (4.00), actionable (4.00) and effective (4.00). Quality assessments rated PEMAT-AI as 'very good' (71.43), DISCERN-AI as 'fair' (12.00) and GQS as 'high' (4.00). Readability scores indicated difficulty (FRE 47.00, GFI 12.40), suggesting a higher educational level was required. This study concludes that ChatGPT-4 is capable of providing safe but nonspecific medical information, suggesting its potential as a patient education aid. However, enhancements in readability through contextual prompting and fine-tuning techniques are required before considering implementation into clinical practice.

Leveraging large language models to improve patient education on dry eye disease.

Dry eye disease (DED) is an exceedingly common diagnosis in patients, yet recent analyses have demonstrated patient education materials (PEMs) on DED to be of low quality and readability. Our study evaluated the utility and performance of three large language models (LLMs) in enhancing and generating new patient education materials (PEMs) on dry eye disease (DED). We evaluated PEMs generated by ChatGPT-3.5, ChatGPT-4, Gemini Advanced, using three separate prompts. Prompts A and B requested they generate PEMs on DED, with Prompt B specifying a 6th-grade reading level, using the SMOG (Simple Measure of Gobbledygook) readability formula. Prompt C asked for a rewrite of existing PEMs at a 6th-grade reading level. Each PEM was assessed on readability (SMOG, FKGL: Flesch-Kincaid Grade Level), quality (PEMAT: Patient Education Materials Assessment Tool, DISCERN), and accuracy (Likert Misinformation scale). All LLM-generated PEMs in response to Prompt A and B were of high quality (median DISCERN = 4), understandable (PEMAT understandability ≥70%) and accurate (Likert Score=1). LLM-generated PEMs were not actionable (PEMAT Actionability <70%). ChatGPT-4 and Gemini Advanced rewrote existing PEMs (Prompt C) from a baseline readability level (FKGL: 8.0 ± 2.4, SMOG: 7.9 ± 1.7) to targeted 6th-grade reading level; rewrites contained little to no misinformation (median Likert misinformation=1 (range: 1-2)). However, only ChatGPT-4 rewrote PEMs while maintaining high quality and reliability (median DISCERN = 4). LLMs (notably ChatGPT-4) were able to generate and rewrite PEMs on DED that were readable, accurate, and high quality. Our study underscores the value of leveraging LLMs as supplementary tools to improving PEMs.