Simple Measure Of Gobbledygook Research Articles

Assessing the readability, reliability, and quality of artificial intelligence chatbot responses to the 100 most searched queries about cardiopulmonary resuscitation: An observational study.

This study aimed to evaluate the readability, reliability, and quality of responses by 4 selected artificial intelligence (AI)-based large language model (LLM) chatbots to questions related to cardiopulmonary resuscitation (CPR). This was a cross-sectional study. Responses to the 100 most frequently asked questions about CPR by 4 selected chatbots (ChatGPT-3.5 [Open AI], Google Bard [Google AI], Google Gemini [Google AI], and Perplexity [Perplexity AI]) were analyzed for readability, reliability, and quality. The chatbots were asked the following question: "What are the 100 most frequently asked questions about cardio pulmonary resuscitation?" in English. Each of the 100 queries derived from the responses was individually posed to the 4 chatbots. The 400 responses or patient education materials (PEM) from the chatbots were assessed for quality and reliability using the modified DISCERN Questionnaire, Journal of the American Medical Association and Global Quality Score. Readability assessment utilized 2 different calculators, which computed readability scores independently using metrics such as Flesch Reading Ease Score, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Gunning Fog Readability and Automated Readability Index. Analyzed 100 responses from each of the 4 chatbots. When the readability values of the median results obtained from Calculators 1 and 2 were compared with the 6th-grade reading level, there was a highly significant difference between the groups (P < .001). Compared to all formulas, the readability level of the responses was above 6th grade. It can be seen that the order of readability from easy to difficult is Bard, Perplexity, Gemini, and ChatGPT-3.5. The readability of the text content provided by all 4 chatbots was found to be above the 6th-grade level. We believe that enhancing the quality, reliability, and readability of PEMs will lead to easier understanding by readers and more accurate performance of CPR. So, patients who receive bystander CPR may experience an increased likelihood of survival.

Quality, Reliability, Readability, and Accountability of Online Information on Leukocoria.

To evaluate the quality, reliability, and readability of online patient educational materials on leukocoria. In this cross-sectional study, the Google search engine was searched for the terms "leukocoria" and "white pupil." The first 50 search outcomes were evaluated for each search term based on predefined inclusion criteria, excluding duplicates, peer-reviewed papers, forum posts, paywalled content, and multimedia links. Sources were categorized as "institutional" or "private." Three independent raters assessed each website for quality and reliability using DISCERN, Health on the Net Code of Conduct (HONcode), and JAMA criteria. Readability was evaluated using seven formulas: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG) Index, Automated Readability Index (ARI), Linsear Write (LW), Gunning Fog Index (GFI), and Coleman-Liau Index (CLI). A total of 51 websites were included. Quality, assessed by the DISCERN tool, showed a median score of 4, denoting moderate to high quality, with no significant differences between institutional and private sites or search terms. HONcode scores indicated variable reliability and trustworthiness (median: 10, range: 3 to 16), with institutional sites excelling in financial disclosure and ad differentiation. Additionally, institutional and private sites performed well in reliability and accountability, as measured by the JAMA Benchmark criteria (median: 3; range: 1 to 4). Readability, averaging an 11.3 ± 3.7 grade level, did not differ significantly between site types or search terms, consistently falling short of the recommended sixth-grade level for patient educational materials. The patient educational materials on leukocoria demonstrated moderate to high quality, commendable reliability, and accountability. However, the readability scores were above the recommended level for the layperson. [J Pediatr Ophthalmol Strabismus. 20XX;X(X):XX-XX.].

Performance of an online chat-based artificial intelligence interface for patient education on atrial fibrillation ablation

Abstract Background Chat-based artificial intelligence (AI) web interfaces that aim to mimic human conversation have increasing utilization in healthcare to help with simple tasks such as scheduling appointments, and even more complex tasks such as providing patient educational responses to COVID-19 questions as done by the World Health Organization.1 Chat-based AI has also been shown to provide accurate responses to cardiovascular disease prevention questions.2 Its ability to provide patient education for more complex treatments like atrial fibrillation (AF) ablation has not been explored. Purpose To evaluate the quality of a popular chat-based AI program’s answers to patient questions about AF ablation. Methods Twenty commonly asked questions ("prompts") regarding AF ablation were entered into ChatGPT (Chat Generative Pre-trained Transformer), a large language model-based AI program (Fig. 1). Prompts were written in plain language; technical terms were avoided except for "radiofrequency", "cryoablation" and "pulsed field ablation" (PFA). SMOG readability calculator was used to assess responses for difficulty and grade-level, as healthcare organizations recommend ≤ 8th-grade level complexity for patient information. Response content was graded by 3 experienced cardiac electrophysiologists as "reasonable", "missing important elements/some inaccuracies" or "misleading/inappropriate". Responses are presented in mean +/- standard deviation and percentages. Results Responses averaged 118±67 words (Fig. 1). Of 20 responses, 17 (85%) were deemed reasonable, 3 (15%) missing important elements/some inaccuracies and none inappropriate or misleading; 16 (80%) emphasized discussion of issues with the healthcare team (Fig. 2). Responses missing important elements/some inaccuracies were those about risks/complications of ablation [missing phrenic nerve palsy, atrioesophageal fistula (AEF), potential need for emergent cardiac surgery or pacemaker, death], concerning symptoms post-procedure (missing symptoms of hematoma, AEF, stroke), and that PFA is not yet approved for use in all regions. Average reading grade level of responses was 13.8 (college level or "professional"): 17 (85%) responses were 12th grade level, 11 (55%) were college-level or higher, and 6 (30%) were college-graduate level ("extremely difficult"). None were ≤ 8th grade level (Fig. 2). Conclusions A majority of ChatGPT responses to common patient questions about AF ablation had reasonable content quality that frequently emphasized the importance of discussion with the healthcare team. However, responses to more difficult questions regarding risks, symptoms of potential complications, or newer technology missed important details; more than half of responses required college-level reading skills. While use of Chat-AI for patient education on EP topics appears promising, patients should be advised to use caution. Further AI training to improve content and readability should be explored.

Assessment of the Arabic patient-centered online information about orthodontic pain: A quality and readability assessment.

This study assesses the quality and readability of Arabic online information about orthodontic pain. With the increasing reliance on the internet for health information, especially among Arabic speakers, it's critical to ensure the accuracy and comprehensiveness of available content. Our methodology involved a systematic search using the Arabic term for (Orthodontic Pain) in Google, Bing, and Yahoo. This search yielded 193,856 results, from which 74 websites were selected based on predefined criteria, excluding duplicates, scientific papers, and non-Arabic content. For quality assessment, we used the DISCERN instrument, the Journal of the American Medical Association (JAMA) benchmarks, and the Health on the Net (HON) code. Readability was evaluated using the Simplified Measure of Gobbledygook (SMOG), Flesch Reading Ease Score (FRES), and Flesch-Kincaid Grade Level (FKGL) scores. Results indicated that none of the websites received the HONcode seal. The DISCERN assessment showed median total scores of 14.96 (± 5.65), with low overall quality ratings. In JAMA benchmarks, currency was the most achieved aspect, observed in 45 websites (60.81%), but none met all four criteria simultaneously. Readability scores suggested that the content was generally understandable, with a median FKGL score of 6.98 and a median SMOG score of 3.98, indicating middle school-level readability. This study reveals a significant gap in the quality of Arabic online resources on orthodontic pain, highlighting the need for improved standards and reliability. Most websites failed to meet established quality criteria, underscoring the necessity for more accurate and trustworthy health information for Arabic-speaking patients.

Readability and quality assessment of online patient education materials for spinal and epidural anesthesia.

Guidelines recommend that health-related information for patients should be written at or below the sixth-grade level. We sought to evaluate the readability level and quality of online patient education materials regarding epidural and spinal anesthesia. We evaluated webpages with content written specifically regarding either spinal or epidural anesthesia, identified using 11 relevant search terms, with seven commonly used readability formulas: Flesh-Kincaid Grade Level (FKGL), Gunning Fox Index (GFI), Coleman-Liau Index (CLI), Automated Readability Index (ARI), Simple Measure of Gobbledygook (SMOG), Flesch Reading Ease (FRE), and New Dale-Chall (NDC). Two evaluators assessed the quality of the reading materials using the Brief DISCERN tool. We analyzed 261 webpages. The mean (standard deviation) readability scores were: FKGL = 8.8 (1.9), GFI = 11.2 (2.2), CLI = 10.3 (1.9), ARI = 8.1 (2.2), SMOG = 11.6 (1.6), FRE = 55.7 (10.8), and NDC = 5.4 (1.0). The mean grade level was higher than the recommended sixth-grade level when calculated with six of the seven readability formulas. The average Brief DISCERN score was 16.0. Readability levels of online patient education materials pertaining to epidural and spinal anesthesia are higher than recommended. When we evaluated the quality of the information using a validated tool, the materials were found to be just below the threshold of what is considered good quality. Authors of educational materials should provide not only readable but also good-quality information to enhance patient understanding.

A Cross-Sectional Analysis of the Readability of Online Information Regarding Hip Osteoarthritis.

Introduction Osteoarthritis (OA) is an age-related degenerative joint disease. There is a 25% risk of symptomatic hip OA in patients who live up to 85 years of age. It can impair a person's daily activities and increase their reliance on healthcare services. It is primarily managed with education, weight loss and exercise, supplemented with pharmacological interventions. Poor health literacy is associated with negative treatment outcomes and patient dissatisfaction. A literature search found there are no previously published studies examining the readability of online information about hip OA. Objectives To assess the readability of healthcare websites regarding hip OA. Methods The terms "hip pain", "hip osteoarthritis", "hip arthritis", and "hip OA" were searched on Google and Bing. Of 240 websites initially considered, 74 unique websites underwent evaluation using the WebFX online readability software (WebFX®, Harrisburg, USA). Readability was determined using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Reading Grade Level (FKGL), Gunning Fog Index (GFI), Simple Measure of Gobbledygook (SMOG), Coleman-Liau Index (CLI), and Automated Readability Index (ARI). In line with recommended guidelines and previous studies, FRES >65 or a grade level score of sixth grade and under was considered acceptable. Results The average FRES was 56.74±8.18 (range 29.5-79.4). Only nine (12.16%) websites had a FRES score >65. The average FKGL score was 7.62±1.69 (range 4.2-12.9). Only seven (9.46%) websites were written at or below a sixth-grade level according to the FKGL score. The average GFI score was 9.20±2.09 (range 5.6-16.5). Only one (1.35%) website was written at or below a sixth-grade level according to the GFI score.The average SMOG score was 7.29±1.41 (range 5.4-12.0). Only eight (10.81%) websites were written at or below a sixth-grade level according to the SMOG score. The average CLI score was 13.86±1.75 (range 9.6-19.7). All 36 websites were written above a sixth-grade level according to the CLI score. The average ARI score was 6.91±2.06 (range 3.1-14.0). Twenty-eight(37.84%) websites were written at or below a sixth-grade level according to the ARI score. One-sample t-tests showed that FRES (p<0.001, CI -10.2 to -6.37), FKGL (p<0.001, CI 1.23 to 2.01), GFI (p<0.001, CI 2.72 to 3.69), SMOG (p<0.001, CI 0.97 to 1.62), CLI (p<0.001, CI 7.46 to 8.27), and ARI (p<0.001, CI 0.43 to 1.39) scores were significantly different from the accepted standard. One-way analysis of variance (ANOVA) testing of FRES scores (p=0.009) and CLI scores (p=0.009) showed a significant difference between categories. Post hoc testing showed a significant difference between academic and non-profit categories for FRES scores (p=0.010, CI -15.17 to -1.47) and CLI scores (p=0.008, CI 0.35 to 3.29). Conclusions Most websites regarding hip OA are written above recommended reading levels, hence exceeding the comprehension levels of the average patient. Readability of these resources must be improved to improve patient access to online healthcare information which can lead to improved patient understanding of their own condition and treatment outcomes.

Accuracy, readability, and understandability of large language models for prostate cancer information to the public.

Generative Pretrained Model (GPT) chatbots have gained popularity since the public release of ChatGPT. Studies have evaluated the ability of different GPT models to provide information about medical conditions. To date, no study has assessed the quality of ChatGPT outputs to prostate cancer related questions from both the physician and public perspective while optimizing outputs for patient consumption. Nine prostate cancer-related questions, identified through Google Trends (Global), were categorized into diagnosis, treatment, and postoperative follow-up. These questions were processed using ChatGPT 3.5, and the responses were recorded. Subsequently, these responses were re-inputted into ChatGPT to create simplified summaries understandable at a sixth-grade level. Readability of both the original ChatGPT responses and the layperson summaries was evaluated using validated readability tools. A survey was conducted among urology providers (urologists and urologists in training) to rate the original ChatGPT responses for accuracy, completeness, and clarity using a 5-point Likert scale. Furthermore, two independent reviewers evaluated the layperson summaries on correctness trifecta: accuracy, completeness, and decision-making sufficiency. Public assessment of the simplified summaries' clarity and understandability was carried out through Amazon Mechanical Turk (MTurk). Participants rated the clarity and demonstrated their understanding through a multiple-choice question. GPT-generated output was deemed correct by 71.7% to 94.3% of raters (36 urologists, 17 urology residents) across 9 scenarios. GPT-generated simplified layperson summaries of this output was rated as accurate in 8 of 9 (88.9%) scenarios and sufficient for a patient to make a decision in 8 of 9 (88.9%) scenarios. Mean readability of layperson summaries was higher than original GPT outputs ([original ChatGPT v. simplified ChatGPT, mean (SD), p-value] Flesch Reading Ease: 36.5(9.1) v. 70.2(11.2), <0.0001; Gunning Fog: 15.8(1.7) v. 9.5(2.0), p < 0.0001; Flesch Grade Level: 12.8(1.2) v. 7.4(1.7), p < 0.0001; Coleman Liau: 13.7(2.1) v. 8.6(2.4), 0.0002; Smog index: 11.8(1.2) v. 6.7(1.8), <0.0001; Automated Readability Index: 13.1(1.4) v. 7.5(2.1), p < 0.0001). MTurk workers (n = 514) rated the layperson summaries as correct (89.5-95.7%) and correctly understood the content (63.0-87.4%). GPT shows promise for correct patient education for prostate cancer-related contents, but the technology is not designed for delivering patients information. Prompting the model to respond with accuracy, completeness, clarity and readability may enhance its utility when used for GPT-powered medical chatbots.

Pregnancy Loss: A Comparative Analysis of Information Delivery Between ACOG FAQs and ChatGPT [ID 2683594

INTRODUCTION: Pregnancy loss is a sensitive topic, and clear, reliable information is crucial. This study compares the readability and accuracy of ChatGPT's responses to ACOG's frequently asked questions (FAQ) sheets on the subject. METHODS: Within the five FAQ sheets under “Pregnancy Loss,” 66 questions were assessed using ChatGPT-3.5. Readability scores, ie, Flesch–Kincaid Reading Ease, Flesch–Kincaid Grade Level, Gunning Fog Score, Smog Index, Coleman Liau Index, and Automated Readability Index were computed for each response. The quality of responses were also graded by two maternal–fetal medicine specialists using a 1–4 scale, where 1 represents a comprehensive response and 4 indicates an incorrect response. Statistical analysis utilized a two-tailed t-test. A weighted Cohen’s kappa coefficient evaluated interrater reliability. RESULTS: ACOG attained better readability scores over ChatGPT across all six metrics with statistical significance (P&lt;.001). Among grading, ACOG also outperformed ChatGPT in quality of information with a summative mean of 1.53 (SD 0.60) versus 1.85 (SD 0.90) with statistical significance (P&lt;.001). Grading for ACOG possessed a Cohen's kappa coefficient of 0.493, implying moderate agreement between graders. Grading for ChatGPT possessed a Cohen's kappa coefficient of 0.744, implying substantial agreement between graders. CONCLUSION: The responses by ACOG to frequently asked questions on pregnancy loss were both clearer and more comprehensive than those of ChatGPT with consistent agreement between graders. Although ChatGPT can be a convenient source of medical queries, further improvements ought to be made to these large-language models to provide clear and accurate information regarding pregnancy loss to patients.

Evaluation of online text-based information resources of gynaecological cancer symptoms.

Gynaecological cancer symptoms are often vague and non-specific. Quality health information is central to timely cancer diagnosis and treatment. The aim of this study was to identify and evaluate the quality of online text-based patient information resources regarding gynaecological cancer symptoms. A targeted website search and Google search were conducted to identify health information resources published by the Australian government and non-government health organisations. Resources were classified by topic (gynaecological health, gynaecological cancers, cancer, general health); assessed for reading level (Simple Measure of Gobbledygook, SMOG) and difficulty (Flesch Reading Ease, FRE); understandability and actionability (Patient Education Materials Assessment Tool, PEMAT, 0-100), whereby higher scores indicate better understandability/actionability. Seven criteria were used to assess cultural inclusivity specific for Aboriginal and Torres Strait Islander people; resources which met 3-5 items were deemed to be moderately inclusive and 6+ items as inclusive. A total of 109 resources were identified and 76% provided information on symptoms in the context of gynaecological cancers. The average readability was equivalent to a grade 10 reading level on the SMOG and classified as 'difficult to read' on the FRE. The mean PEMAT scores were 95% (range 58-100) for understandability and 13% (range 0-80) for actionability. Five resources were evaluated as being moderately culturally inclusive. No resource met all the benchmarks. This study highlights the inadequate quality of online resources available on pre-diagnosis gynaecological cancer symptom information. Resources should be revised in line with the recommended standards for readability, understandability and actionability and to meet the needs of a culturally diverse population.

An Analysis of the Readability of Online Sarcoidosis Resources.

Introduction Sarcoidosis is an inflammatory disease characterized by the formation of noncaseating granulomas in multiple organ systems. The presentation can vary widely; although some patients with sarcoidosis can be asymptomatic, sarcoidosis can also present in others with symptomatic multiorgan system involvement. Considering the potential severity of the disease, patients need to be well-informed about sarcoidosis to better manage their health. This study aims to assess the readability levels of online resources about sarcoidosis. Methods We conducted a retrospective cross-sectional study. The term "sarcoidosis" was searched online using both Google and Bing to find websites written in English. Each website was categorized by type: academic, commercial, government, nonprofit, and physician. The readability scores for each website were calculated using six different readability tests: the Flesch-Kincaid reading ease (FKRE), Flesch-Kincaid grade level (FKGL), Gunning fog score (GFS), Simple Measure of Gobbledygook (SMOG), automated readability index (ARI), and Coleman-Liau index (CLI). FKRE gives a score that corresponds to the difficulty of the text, while the remaining tests give a score that corresponds to a grade level in terms of reading ability. A one-sample t-test was used to compare all test scores with the national recommended standard of a sixth-grade reading level. Our null hypothesis was that the readability scores of the websites searched would not differ statistically significantly from the sixth-grade reading level and that there would be no significant differences across website categories. To evaluate the difference between the categories of websites, ANOVA testing was used. Results Thirty-four websites were analyzed. Each of the six readability tests for the websites had an average score, which corresponded to being significantly harder to read than the nationally recommended sixth-grade reading level (p<0.001). None of the mean readability scores showed a statistically significant difference across the five different website categories. Conclusions This is the first study, to our knowledge, to examine the readability of online English resources on sarcoidosis and calculate standardized readability scores for them. It implies that the online English material for sarcoidosis is above the health literacy recommended reading levels for patients. There is a need to simplify the material to be easier to read for patients.

Using Large Language Models to Generate Educational Materials on Childhood Glaucoma

PurposeTo evaluate the quality, readability, and accuracy of large language model (LLM) generated patient education materials (PEMs) on childhood glaucoma, and their ability to improve existing online information's readability. DesignCross-sectional comparative study. MethodsWe evaluated responses of ChatGPT-3.5, ChatGPT-4, and Bard to three separate prompts requesting they write PEMs on “childhood glaucoma.” Prompt A required PEMs be “easily understandable by the average American.” Prompt B required PEMs be written “at a 6th-grade level using Simple Measure of Gobbledygook (SMOG) readability formula.” We then compared responses’ quality (DISCERN questionnaire, Patient Education Materials Assessment Tool (PEMAT)), readability (SMOG, Flesch–Kincaid Grading Level (FKGL)), and accuracy (Likert Misinformation scale). To assess the improvement of readability for existing online information, Prompt C requested LLM rewrite 20 resources from a Google search of keyword “childhood glaucoma” to the American Medical Association-recommended “6th-grade level.” Rewrites were compared on key metrics such as readability, complex words (≥3 syllables), and sentence count. ResultsAll 3 LLM generated PEMs that were of high quality, understandability, and accuracy (DISCERN≥4, ≥70% PEMAT understandability, Misinformation score=1). Prompt B responses were more readable than Prompt A responses for all 3 LLM (p≤0.001). ChatGPT-4 generated the most readable PEMs compared to ChatGPT-3.5 and Bard (p≤0.001). Although Prompt C responses showed consistent reduction of mean SMOG and FKGL scores, only ChatGPT-4 achieved the specified 6th-grade reading level (4.8 ± 0.8 and 3.7 ± 1.9, respectively). ConclusionLLMs can serve as strong supplementary tools in generating high quality, accurate, and novel PEMs, and improving the readability of existing PEMs on childhood glaucoma.

Readability and Quality of Online Patient Education Materials Concerning Posterior Cruciate Ligament Reconstruction.

Objective This study aimed to assess the quality of online patient educational materials regarding posterior cruciate ligament (PCL) reconstruction. Methods We performed a search of the top-50 results on Google® (terms: "posterior cruciate ligament reconstruction,""PCL reconstruction,""posterior cruciate ligament surgery," and "PCL surgery") and subsequently filtered to rule out duplicated/inaccessible websites or those containing only videos (67 websites included). Readability was assessed using six formulas: Flesch-Kincaid Reading Ease (FRE), Flesch-Kincaid Grade Level (FKG), Gunning Fog Score (GF), Simple Measure of Gobbledygook (SMOG) Index, Coleman-Liau Index (CLI), Automated Readability Index (ARI); quality was assessed using the JAMA benchmark criteria and recording the presence of the HONcode seal. Results The mean FRE was 49.3 (SD 11.2) and the mean FKG level was 8.09. These results were confirmed by the other readability formulae (average: GF 8.9; SMOG Index 7.3; CLI 14.7; ARI 6.5). A HONcode seal was available for 7.4 % of websites. The average JAMA score was 1.3. Conclusion The reading level of online patient materials concerning PCL reconstruction is too high for the average reader, requiring high comprehension skills. Practice implications Online medical information has been shown to influence patient healthcare decision processes. Patient-oriented educational materials should be clear and easy to understand.

Readability of online patient education materials for shoulder instability surgery in English and Spanish

Online patient education materials (OPEMs) exist to inform patient medical decisions, yet the average adult in the United States reads at an eighth-grade level and 50% of Medicaid patients read at or below a fifth-grade level. To appropriately meet US health literacy needs, the American Medical Association and National Institutes of Health recommend that patient education materials not exceed a sixth-grade level. The purpose of this study was to assess and compare the readability of English and Spanish online patient education materials pertaining to shoulder instability surgery. Google searches of the terms "shoulder instability surgery" and "cirugía de inestabilidad de hombro'' were conducted to include 25 eligible online patient education materials OPEMs per language. English OPEM readability was calculated using Flesch-Kincaid Grade Level, Flesch Reading Ease, Flesch Reading Ease Grade Level, Gunning-Fog Index, Coleman-Liau Index, and Simple Measure of Gobbledygook. Spanish OPEM readability was assessed using Fernandez-Huerta Index (FHI) (the Spanish equivalent of Flesch Reading Ease), FHI Grade Level, Gutiérrez de Polini's Fórmula de comprensibilidad, and INFLESZ. Readability index analysis revealed that the mean Flesch Reading Ease of English online patient education materials was significantly lower than the mean FHI of Spanish online patient education materials. English materials were also found to be written at a significantly higher grade level than Spanish materials. Shoulder instability surgery online patient education materials in both English and Spanish are written at higher reading levels than recommended by the American Medical Association and National Institutes of Health, though Spanish online patient education materials were more readable on average.

How good is ChatGPT at answering patients’ questions related to early detection of oral (mouth) cancer?

ObjectivesTo examine the quality, reliability, readability, and usefulness of ChatGPT in promoting oral cancer early detection. Study design108 patients-oriented questions about oral cancer early detection were compiled from expert panel, professional societies, and web-based tools. Questions were categorized into four topic domains and ChatGPT 3.5 was asked each question independently. ChatGPT answers were evaluated regarding quality, readability, actionability, and usefulness using. Two experienced reviewers independently assessed each response. ResultsQuestions related to clinical appearance constituted 36.1% (n=39) of the total questions. ChatGPT provided “very useful” responses to the majority of questions (75%; n=81). The mean Global Quality Score was 4.24 ±1.3 out of 5. The mean reliability score was 23.17 ± 9.87 out of 25. The mean understandability score was 76.6% ±25.9 out of 100, while the mean actionability score was 47.3% ± 18.9 out of 100. The mean FKS reading ease score was 38.4%± 29.9, while the mean SMOG index readability score was 11.65 ± 8.4. No misleading information was identified among ChatGPT responses. ConclusionChatGPT is an attractive and potentially useful resource for informing patients about early detection of oral cancer. Nevertheless, concerns do exist about readability and actionability of the offered information.

The quality and readability of patient information provided by ChatGPT: can AI reliably explain common ENT operations?

Access to high-quality and comprehensible patient information is crucial. However, information provided by increasingly prevalent Artificial Intelligence tools has not been thoroughly investigated. This study assesses the quality and readability of information from ChatGPT regarding three index ENT operations: tonsillectomy, adenoidectomy, and grommets. We asked ChatGPT standard and simplified questions. Readability was calculated using Flesch-Kincaid Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI) and Simple Measure of Gobbledygook (SMOG) scores. We assessed quality using the DISCERN instrument and compared these with ENT UK patient leaflets. ChatGPT readability was poor, with mean FRES of 38.9 and 55.1 pre- and post-simplification, respectively. Simplified information from ChatGPT was 43.6% more readable (FRES) but scored 11.6% lower for quality. ENT UK patient information readability and quality was consistently higher. ChatGPT can simplify information at the expense of quality, resulting in shorter answers with important omissions. Limitations in knowledge and insight curb its reliability for healthcare information. Patients should use reputable sources from professional organisations alongside clear communication with their clinicians for well-informed consent and making decisions.

The Emerging Role of AI in Patient Education: A Comparative Analysis of the Accuracy of Large Language Models for Pelvic Organ Prolapse

Introduction: This study aimed to evaluate the accuracy, completeness, precision, and readability of outputs generated by three large language models (LLMs); these are GPT by OpenAI, BARD by Google, and Bing by Microsoft, in comparison to patient education material on pelvic organ prolapse (POP) provided by the Royal College of Obstetricians and Gynecologists (RCOG). Methods: A total of 15 questions were retrieved from the RCOG website and input into the three LLMs. Two independent reviewers evaluated the outputs for accuracy, completeness, and precision. Readability was assessed using the Simplified Measure of Gobbledygook (SMOG) score and the Flesch-Kincaid Grade Level (FKGL) score. Results: Significant differences were observed in completeness and precision metrics. ChatGPT ranked highest in completeness (66.7%), while Bing led in precision (100%). No significant differences were observed in accuracy across all models. In terms of readability, ChatGPT exhibited higher difficulty than BARD, Bing, and the original RCOG answers. Conclusion: While all models displayed a variable degree of correctness, ChatGPT excelled in completeness, significantly surpassing BARD and Bing. However, Bing led in precision, providing the most relevant and concise answers. Regarding readability, ChatGPT exhibited higher difficulty. We observed that while all LLMs showed varying degrees of correctness in answering RCOG questions on patient information for POP, ChatGPT was the most comprehensive, but its answers were harder to read. Bing, on the other hand, was the most precise. The findings highlight the potential of LLMs in health information dissemination and the need for careful interpretation of their outputs.

Quality, Reliability, and Readability of Online Information on Idiopathic Intracranial Hypertension.

To evaluate the quality & reliability, technical quality, and readability of patient-targeted online information on idiopathic intracranial hypertension. In this cross-sectional study, we searched Google and Bing search engines for the terms "idiopathic intracranial hypertension" and "pseudotumor cerebri." We evaluated the first 50 search outcomes for each search term. Peer-reviewed articles, targeted advertisements, book chapters, videos, personal blogs, websites intended for healthcare professionals, and non-English websites were excluded. Websites were classified as institutional or private. Two independent reviewers assessed each website for 1) quality and reliability using DISCERN, Health on the Net Code of Conduct (HONcode), and JAMA criteria and 2) technical quality based on 11 criteria. Readability was evaluated using 6 measures (Flesch-Kincaid Reading Ease score and grade level, Gunning Fog, Simple Measure of Gobbledygook (Smog) index, Coleman-Liau index, and automated readability index). Sixty-three websites (37 institutional) were included. The mean scores for the DISCERN, HONcode, and JAMA criteria were 3.6 ± 1 (1-4.8; 1: worse, 5: best), 10.3 ± 2.8 (2-16; 0: worse, 16: best), and 2 ± 1.3 (0-4; 0: worse, 4: best), respectively. The mean technical quality score was 0.8 ± 0.1 (0.5-1). The mean Flesch-Kincaid grade level score was 8.9 ± 1.8 (3.3-13.3). For Flesch-Kincaid grade level, 47 (74.6%) websites scored a grade level of 10 or less. For Gunning Fog Score, 35 websites (55.6%) scored from 7.3 to 10.8. For the Smog Index, 46 (73%) websites scored 5.7-8. The mean Coleman Liau Index was 16 ± 2.1 (9.6-22.2). For the automated readability index, 30 (50.7%) websites scored less than the eighth grade. No statistically significant difference was present between institutional and private websites except for JAMA, which scored higher in private websites (2.4 vs 1.7, P = 0.03). Our findings suggest that online information on idiopathic intracranial hypertension/pseudotumor cerebri for patients generally demonstrates moderate to good quality and reliability. Although most websites maintain credible technical quality, the readability exceeds recommended levels for the average American reader. This may impede understanding, emphasizing the need for future improvements.

A health literacy analysis of online patient-directed educational materials about mycobacterium avium complex

IntroductionMycobacterium avium complex (MAC) is a complex lung infection requiring multi-disciplinary approach and management. Due to limited clinician-patient interactions, clinicians may refer patients to online resources to learn about the diagnosis, prognosis, and treatment of MAC. The American Medical Association (AMA) recommends educational materials be written at a sixth-grade reading level and the National Institutes of Health (NIH) recommends that patient education materials be written at an eighth-grade reading level; however, several evaluations found these materials inaccessible due to high literacy levels. To date, there has never been a health literacy assessment of MAC patient education materials. The study aims to assess the health literacy of online patient education materials about MAC. MethodsThe patient education materials were evaluated for readability, actionability, understandability and clarity. Readability was assessed through the Flesch-Kincaid Grade Level Scale (FkGL), SMOG Index, Coleman Liau Index (CLI), Gunning Fog Index (GFI), and Automated Readability Index (ARI). Actionability and understandability was evaluated using the Patient Education Materials Assessment Tool (PEMAT). The Centers for Disease Control (CDC) Clear Communication Index (CCI) was used to assess clarity. ResultsTen patient education resources were evaluated: CDC, Cleveland Clinic, Mayo Clinic, JAMA, American Thoracic Society (ATS), National Jewish Health, UpToDate, CHEST, WebMD, and Medline. The mean readability scores were as follows: FkGL (9.76), SMOG index (9.82), CLI (13.54), GFI (11.66), ARI (9.15). Four patient education materials were written at a sixth-grade reading level and eight patient education materials were written at an eighth-grade reading level. The majority of the materials received a passing score for understandability but failed to achieve a passing score for actionability. Cleveland Clinic, JAMA, and ATS all received a passing clarity score, indicating that they are easy to read. No patient education materials were available on UpToDate. ConclusionMost patient education materials scored poorly for actionability and clarity while scoring highly for readability and understandability. This study should serve as a guide for clinicians interested in offering online education materials to their patients. Increasing access to readable MAC educational materials should be a priority for those working at the intersection of public health, clinical care, and communications.

Application of generative language models to orthopaedic practice

ObjectiveTo explore whether large language models (LLMs) Generated Pre-trained Transformer (GPT)-3 and ChatGPT can write clinical letters and predict management plans for common orthopaedic scenarios.DesignFifteen scenarios were generated and ChatGPT...

Evaluating the Use of ChatGPT to Accurately Simplify Patient-centered Information about Breast Cancer Prevention and Screening.

Purpose To evaluate the use of ChatGPT as a tool to simplify answers to common questions about breast cancer prevention and screening. Materials and Methods In this retrospective, exploratory study, ChatGPT was requested to simplify responses to 25 questions about breast cancer to a sixth-grade reading level in March and August 2023. Simplified responses were evaluated for clinical appropriateness. All original and simplified responses were assessed for reading ease on the Flesch Reading Ease Index and for readability on five scales: Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, Automated Readability Index, and the Simple Measure of Gobbledygook (ie, SMOG) Index. Mean reading ease, readability, and word count were compared between original and simplified responses using paired t tests. McNemar test was used to compare the proportion of responses with adequate reading ease (score of 60 or greater) and readability (sixth-grade level). Results ChatGPT improved mean reading ease (original responses, 46 vs simplified responses, 70; P < .001) and readability (original, grade 13 vs simplified, grade 8.9; P < .001) and decreased word count (original, 193 vs simplified, 173; P < .001). Ninety-two percent (23 of 25) of simplified responses were considered clinically appropriate. All 25 (100%) simplified responses met criteria for adequate reading ease, compared with only two of 25 original responses (P < .001). Two of the 25 simplified responses (8%) met criteria for adequate readability. Conclusion ChatGPT simplified answers to common breast cancer screening and prevention questions by improving the readability by four grade levels, though the potential to produce incorrect information necessitates physician oversight when using this tool. Keywords: Mammography, Screening, Informatics, Breast, Education, Health Policy and Practice, Oncology, Technology Assessment Supplemental material is available for this article. © RSNA, 2023.