Mean Flesch-Kincaid Grade Level Research Articles

Exploring AI-generated content and professional guidelines in cancer symptom management: A comparative analysis between ChatGPT and NCCN guidelines.

e13610 Background: Artificial intelligence (AI)-driven tools, like ChatGPT, have become widely-available sources for online health information. Limited research has explored the congruity between AI-generated content and professional treatment guidelines. This study seeks to compare recommendations for cancer-related symptoms generated from ChatGPT with guidelines from National Comprehensive Cancer Network (NCCN), a provider focused source requiring users to log in or register to access its recommendations. Utilizing a provider-focused source like NCCN serves as a benchmark to assess whether ChatGPT recommendations align with the standards typically endorsed by clinicians. Methods: We extracted treatment recommendations from four NCCN Supportive Care webpages (Cancer Pain, Antiemesis, Cancer-Related Fatigue, and Distress Management) and five subsections of the NCCN Palliative Care webpage (dyspnea, constipation, diarrhea, sleep disturbances, and anorexia/cachexia). We then entered "How can I reduce my cancer-related [symptom]" into ChatGPT 3.5 and extracted its recommendations. We calculated and compared word count and Flesch-Kincaid Grade Level readability for each NCCN and ChatGPT section. We completed a comparative content analysis focusing on recommendations for medications, consultations, and non-pharmacological strategies. Results: Across the nine NCCN Supportive Care and Palliative Care webpages, the mean word count was 2393.8 (SD=2601.4) and the mean Flesch-Kincaid Grade Level was 17.3 (SD=1.4) vs 382.4 (SD=29.6) and 11.6 (SD=0.8) for ChatGPT. The mean percent agreement between NCCN and ChatGPT recommendations was 44.6% (range 14.3%-81.8%). ChatGPT and NCCN guidelines shared fewer than half of their symptom-related recommendations in all but one section, fatigue. NCCN offered specific medication recommendations across all sections. ChatGPT's recommendations lacked the specificity observed in NCCN's guidelines including often not suggesting any medications. ChatGPT recommended specific medications in the shortness of breath and diarrhea sections that were not recommended by NCCN. NCCN's guidelines often did not include recommendations related to spirituality or palliative care consults, areas that ChatGPT addressed. Conclusions: While ChatGPT provides concise, accessible supportive care advice including many non-medical support recommendations, discrepancies with guidelines raise concerns for patient-facing symptom management recommendations. Overall, AI-generated content like ChatGPT can assist in providing preliminary information but should be researched in conjunction with comprehensive, evidence-based guidance provided by healthcare professionals. Healthcare providers should work with AI developers to ensure data sources are high-quality and accurate.

ChatGPT Provides Satisfactory but Occasionally Inaccurate Answers to Common Patient Hip Arthroscopy Questions

PurposeTo assess the ability of ChatGPT to answer common patient questions regarding hip arthroscopy, and to analyze the accuracy and appropriateness of its responses. MethodsTen questions were selected from well-known patient education websites, and ChatGPT (version 3.5) responses to these questions were graded by two fellowship-trained hip preservation surgeons. Responses were analyzed, compared to the current literature, and graded from A to D (A being the highest, and D being the lowest) in a grading scale based on the accuracy and completeness of the response. If the grading differed between the two surgeons, a consensus was reached. Inter-rater agreement was calculated. The readability of responses was also assessed using the Flesch-Kincaid Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). ResultsResponses received the following consensus grades: A (50%, n=5), B (30%, n=3), C (10%, n=1), D (10%, n=1) (Table 2). Inter-rater agreement based on initial individual grading was 30%. The mean FRES was 28.2 (SD± 9.2), corresponding to a college graduate level, ranging from 11.7 to 42.5. The mean FKGL was 14.4 (SD±1.8), ranging from 12.1 to 18, indicating a college student reading level. ConclusionChatGPT can answer common patient questions regarding hip arthroscopy with satisfactory accuracy graded by two high-volume hip arthroscopists, however, incorrect information was identified in more than one instance. Caution must be observed when using ChatGPT for patient education related to hip arthroscopy. Clinical RelevanceGiven the increasing number of hip arthroscopies being performed annually, ChatGPT has the potential to aid physicians in educating their patients about this procedure and address any questions they may have.

Evaluation of Online Shoulder Instability-Related Patient Education Materials

Background: Younger patients are more likely than older patients to experience shoulder instability and to rely on online educational resources. Although the Internet has increased patient access to medical information, this may not translate to increased health literacy. Purpose: We sought to analyze the quality and readability of online information on shoulder instability. Methods: We conducted a Google search using 6 terms related to shoulder instability. We collected the first 20 non-sponsored results for each term. Readability was evaluated using the Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), and Gunning Fox Index (GFI) instruments. Quality was assessed using a Quality Grading Sheet (QGS) and the validated DISCERN instrument. Results: A total of 64 of 120 patient educational materials (PEMs) met the inclusion criteria. The mean FKGL, FRE, and GFI scores were 9.45 ± 0.552, 50.51 ± 3.4, and 11.5 ± 0.6, respectively. The mean DISCERN score and QGS rating were 33.09 ± 2.02 and 10.52 ± 1.28, respectively. While 49 (76.6%) articles discussed operative treatment for persistent shoulder instability, only 4 (6.3%) mentioned risks associated with surgery. Non-institutional sources had higher DISCERN scores than those from medical institutions. Conclusions: This review of online shoulder instability-related PEMs suggests that many do not meet current recommendations, with an average quality rating of “poor” and a mean ninth-grade reading level. Surgeons should be aware of the relative paucity of information on the risks and outcomes associated with operative treatment of shoulder instability contained in these PEMs.

ChatGPT-3.5 Responses Require Advanced Readability for the General Population and May Not Effectively Supplement Patient-Related Information Provided by the Treating Surgeon Regarding Common Questions About Rotator Cuff Repair

The purpose of this study was to investigate the accuracy of ChatGPT's responses to frequently asked questions (FAQ) prior to rotator cuff repair surgery. The ten most common FAQs related to rotator cuff repair were compiled from four institution websites. Questions were then input into ChatGPT-3.5 in one session. The provided ChatGPT-3.5 responses were analyzed by two orthopedic surgeons for reliability, quality, and readability using The Journal of the American Medical Association (JAMA) Benchmark criteria, the DISCERN score, and the Flesch-Kincaid Grade Level. The JAMA Benchmark criteria score was 0, indicating the absence of reliable source material citations. The mean Flesch-Kincaid Grade Level was 13.4 (range, 11.2-15.0). The mean DISCERN score was 43.4 (range, 36-51) indicating that the quality of the responses overall was considered fair. All responses cited making final decision-making to be made with the treating physician. ChatGPT-3.5 provided substandard patient-related information in alignment with recommendations from the treating surgeon regarding common questions around rotator cuff repair surgery. Additionally, the responses lacked reliable source material citations and the readability of the responses was relatively advanced with a complex language style. The findings of this study suggest that ChatGPT-3.5 may not effectively supplement patient-related information in the context of recommendations provided by the treating surgeon prior to rotator cuff repair surgery.

Readability of Trauma-related Patient Education Materials From the American Academy of Orthopaedic Surgeons and Orthopaedic Trauma Association Websites.

Web-based resources serve as a fundamental educational platform for orthopaedic trauma patients; however, they are frequently written above the recommended sixth-grade reading level, and previous studies have demonstrated this for the American Academy of Orthopaedic Surgeons (AAOS) web-based articles. In this study, we perform an updated assessment of the readability of AAOS trauma-related educational articles as compared with injury-matched education materials developed by the Orthopaedic Trauma Association (OTA). All 46 AAOS trauma-related web-based ( https://www.orthoinfo.org/ ) patient education articles were analyzed for readability. Two independent reviewers used (1) the Flesch-Kincaid Grade Level (FKGL) and (2) the Flesch Reading Ease (FRE) algorithms to calculate the readability level. Mean readability scores were compared across body part categories. A one-sample t -test was done to compare mean FKGL with the recommended sixth-grade readability level and the average American adult reading level. A two-sample t -test was used to compare the readability scores of the AAOS trauma-related articles with those of the OTA. The average (SD) FKGL and FRE for the AAOS articles were 8.9 (0.74) and 57.2 (5.8), respectively. All articles were written above the sixth-grade reading level. The average readability of the AAOS articles was significantly greater than the recommended sixth-grade reading level ( P < 0.001). The average FKGL and FRE for all AAOS articles were significantly higher compared with those of the OTA articles (8.9 ± 0.74 versus 8.1 ± 1.14, P < 0.001 and 57.2 ± 5.8 versus 65.6 ± 6.6, P < 0.001, respectively). Excellent agreement was observed between raters for the FKGL 0.956 (95% confidence interval, 0.922 to 0.975) and FRE 0.993 (95% confidence interval, 0.987 to 0.996). Our findings suggest that after almost a decade, the readability of the AAOS trauma-related articles remains unchanged. The AAOS and OTA trauma patient education materials have high readability levels and may be too difficult for patient comprehension. A need remains to improve the readability of these commonly used trauma education materials.

Readability of Neurosurgical Patient Education Resources by the American Association of Neurological Surgeons

The importance of patient education has become increasingly apparent in recent years. A prominent patient education tool in neurosurgery is the Neurosurgical Conditions and Treatments page provided by the American Association of Neurological Surgeons (AANS). This study aimed to investigate the readability of this resource page as many new articles have been incorporated in the past decade. Flesch-Kincaid Grade Level and Flesch-Kincaid Reading Ease scores were calculated for each article. One-way analysis of variance and Scheffé, Tukey-Kramer, Bonferroni, Fisher least significant difference, and Dunnett test post hoc analyses were conducted to determine differences between each category with respect to their mean grade level and reading ease scores. Overall mean (SD) Flesch-Kincaid Reading Ease score was 40.2 (12.24), and overall mean (SD) Flesch-Kincaid Grade Level score was 7.48 (1.26). Significant differences were found between mean reading ease scores between each categorization by the AANS (P= 0.014). No significant differences were found between mean grade level score for each categorization (analysis of variance, P= 0.154). As compared to a single previous investigation conducted 10 years ago, the readability of articles has changed modestly, and the reading grade level remains well above the recommendations by the American Medical Association and National Institutes of Health. The 6 new articles introduced in the past decade have demonstrated similar readability, presenting a persistent challenge in the realm of patient education in neurosurgery.

ChatGPT-4 Generates More Accurate and Complete Responses to Common Patient Questions About Anterior Cruciate Ligament Reconstruction Than Google’s Search Engine

To replicate a patient's internet search to evaluate ChatGPT's appropriateness in answering common patient questions about anterior cruciate ligament reconstruction compared with a Google web search. A Google web search was performed by searching the term "anterior cruciate ligament reconstruction." The top 20 frequently asked questions and responses were recorded. The prompt "What are the 20 most popular patient questions related to 'anterior cruciate ligament reconstruction?'" was input into ChatGPT and questions and responses were recorded. Questions were classified based on the Rothwell system and responses assessed via Flesch-Kincaid Grade Level, correctness, and completeness were for both Google web search and ChatGPT. Three of 20 (15%) questions were similar between Google web search and ChatGPT. The most common question types among the Google web search were value (8/20, 40%), fact (7/20, 35%), and policy (5/20, 25%). The most common question types amongst the ChatGPT search were fact (12/20, 60%), policy (6/20, 30%), and value (2/20, 10%). Mean Flesch-Kincaid Grade Level for Google web search responses was significantly lower (11.8 ± 3.8 vs 14.3 ± 2.2; P= .003) than for ChatGPT responses. The mean correctness for Google web search question answers was 1.47 ± 0.5, and mean completeness was 1.36 ± 0.5. Mean correctness for ChatGPT answers was 1.8 ± 0.4 and mean completeness was 1.9 ± 0.3, which were both significantly greater than Google web search answers (P= .03 and P= .0003). ChatGPT-4 generated more accurate and complete responses to common patient questions about anterior cruciate ligament reconstruction than Google's search engine. The use of artificial intelligence such as ChatGPT is expanding. It is important to understand the quality of information as well as how the results of ChatGPT queries compare with those from Google web searches.

Analysis of the Most Popular Online Ankle Fracture-Related Patient Education Materials.

Given the increasing accessibility of Internet access, it is critical to ensure that the informational material available online for patient education is both accurate and readable to promote a greater degree of health literacy. This study sought to investigate the quality and readability of the most popular online resources for ankle fractures. After conducting a Google search using 6 terms related to ankle fractures, we collected the first 20 nonsponsored results for each term. Readability was evaluated using the Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), and Gunning Fog Index (GFI) instruments. Quality was evaluated using custom created Ankle Fracture Index (AFI). A total of 46 of 120 articles met the inclusion criteria. The mean FKGL, FRE, and GFI scores were 8.4 ± 0.5, 57.5 ± 3.2, and 10.5 ± 0.5, respectively. The average AFI score was 15.4 ± 1.4, corresponding to an "acceptable" quality rating. Almost 70% of articles (n = 32) were written at or below the recommended eighth-grade reading level. Most articles discussed the need for imaging in diagnosis and treatment planning while neglecting to discuss the risks of surgery or potential future operations. We found that online patient-facing materials on ankle fractures demonstrated an eighth-grade average reading grade level and an acceptable quality on content analysis. Further work should surround increasing information regarding risk factors, complications for surgery, and long-term recovery while ensuring that readability levels remain below at least the eighth-grade level.

Quality, Reliability, and Readability of Online Information on Idiopathic Intracranial Hypertension.

To evaluate the quality & reliability, technical quality, and readability of patient-targeted online information on idiopathic intracranial hypertension. In this cross-sectional study, we searched Google and Bing search engines for the terms "idiopathic intracranial hypertension" and "pseudotumor cerebri." We evaluated the first 50 search outcomes for each search term. Peer-reviewed articles, targeted advertisements, book chapters, videos, personal blogs, websites intended for healthcare professionals, and non-English websites were excluded. Websites were classified as institutional or private. Two independent reviewers assessed each website for 1) quality and reliability using DISCERN, Health on the Net Code of Conduct (HONcode), and JAMA criteria and 2) technical quality based on 11 criteria. Readability was evaluated using 6 measures (Flesch-Kincaid Reading Ease score and grade level, Gunning Fog, Simple Measure of Gobbledygook (Smog) index, Coleman-Liau index, and automated readability index). Sixty-three websites (37 institutional) were included. The mean scores for the DISCERN, HONcode, and JAMA criteria were 3.6 ± 1 (1-4.8; 1: worse, 5: best), 10.3 ± 2.8 (2-16; 0: worse, 16: best), and 2 ± 1.3 (0-4; 0: worse, 4: best), respectively. The mean technical quality score was 0.8 ± 0.1 (0.5-1). The mean Flesch-Kincaid grade level score was 8.9 ± 1.8 (3.3-13.3). For Flesch-Kincaid grade level, 47 (74.6%) websites scored a grade level of 10 or less. For Gunning Fog Score, 35 websites (55.6%) scored from 7.3 to 10.8. For the Smog Index, 46 (73%) websites scored 5.7-8. The mean Coleman Liau Index was 16 ± 2.1 (9.6-22.2). For the automated readability index, 30 (50.7%) websites scored less than the eighth grade. No statistically significant difference was present between institutional and private websites except for JAMA, which scored higher in private websites (2.4 vs 1.7, P = 0.03). Our findings suggest that online information on idiopathic intracranial hypertension/pseudotumor cerebri for patients generally demonstrates moderate to good quality and reliability. Although most websites maintain credible technical quality, the readability exceeds recommended levels for the average American reader. This may impede understanding, emphasizing the need for future improvements.

Alarm Bells or Just Smoke: An Evaluation of the Potential for Cheating with ChatGPT on Criminal Justice Student Papers

OpenAI’s ChatGPT is an advanced large language model AI that has caused both excitement and concern in academia. TurnItIn and similar software programs are widely used in higher education to detect potential plagiarism. However, the extent to which such software can identify papers produced by ChatGPT remains unclear. The current study partially addressed this question by submitting five versions of short essays about criminological topics generated from ChatGPT to the TurnItIn software. Overall, the results indicated that TurnItIn adequately detected that the essays were not original works (mean percent plagiarized score = 31%). The analyses further illustrated that ChatGPT wrote at an exceedingly high level (mean Flesch-Kincaid Grade Level = 15.1) atypical of essays in higher education. Consequently, it appears that detection of AI-generated writing may be easier than initially assumed although the technology is bound to improve. Accordingly, possible strategies for approaching AI in higher education are proposed.

Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and Format

By law, patients have immediate access to discharge notes in their medical records. Technical language and abbreviations make notes difficult to read and understand for a typical patient. Large language models (LLMs [eg, GPT-4]) have the potential to transform these notes into patient-friendly language and format. To determine whether an LLM can transform discharge summaries into a format that is more readable and understandable. This cross-sectional study evaluated a sample of the discharge summaries of adult patients discharged from the General Internal Medicine service at NYU (New York University) Langone Health from June 1 to 30, 2023. Patients discharged as deceased were excluded. All discharge summaries were processed by the LLM between July 26 and August 5, 2023. A secure Health Insurance Portability and Accountability Act-compliant platform, Microsoft Azure OpenAI, was used to transform these discharge summaries into a patient-friendly format between July 26 and August 5, 2023. Outcomes included readability as measured by Flesch-Kincaid Grade Level and understandability using Patient Education Materials Assessment Tool (PEMAT) scores. Readability and understandability of the original discharge summaries were compared with the transformed, patient-friendly discharge summaries created through the LLM. As balancing metrics, accuracy and completeness of the patient-friendly version were measured. Discharge summaries of 50 patients (31 female [62.0%] and 19 male [38.0%]) were included. The median patient age was 65.5 (IQR, 59.0-77.5) years. Mean (SD) Flesch-Kincaid Grade Level was significantly lower in the patient-friendly discharge summaries (6.2 [0.5] vs 11.0 [1.5]; P < .001). PEMAT understandability scores were significantly higher for patient-friendly discharge summaries (81% vs 13%; P < .001). Two physicians reviewed each patient-friendly discharge summary for accuracy on a 6-point scale, with 54 of 100 reviews (54.0%) giving the best possible rating of 6. Summaries were rated entirely complete in 56 reviews (56.0%). Eighteen reviews noted safety concerns, mostly involving omissions, but also several inaccurate statements (termed hallucinations). The findings of this cross-sectional study of 50 discharge summaries suggest that LLMs can be used to translate discharge summaries into patient-friendly language and formats that are significantly more readable and understandable than discharge summaries as they appear in electronic health records. However, implementation will require improvements in accuracy, completeness, and safety. Given the safety concerns, initial implementation will require physician review.

Readability of postoperative discharge instructions is associated with complication rate in companion animals undergoing sterilisation.

Readability of client communications is a commonly overlooked topic in veterinary medical education. In human medicine, it has been advised that the readability of patient materials should be at USA schooling sixth-grade level or below. We hypothesised that student written discharge instructions would be of an inappropriate readability level, and discharges scored with higher reading grade levels would be associated with more complications. The cohort comprised 149 dogs and cats presenting for sterilisation. The readability of discharge instructions was assessed using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) formulas. Records were examined for evidence of postoperative complications. The mean FRE score of the discharge instructions was 61.97, with 30.87% being classified as 'difficult' or 'fairly difficult', 60.4% as 'standard' and 8.72% as 'fairly easy'. The mean FKGL was 8.64, with 98% being above reading level 6. Overall, there was an association between FKGL and complication occurrence (p = 0.005). Stratification by species demonstrated FRE and FKGL to be associated with complication occurrence in dogs (FRE score, p = 0.038; FKGL, p = 0.002), but not cats (FRE score, p = 0.964; FKGL, p = 0.679). Due to the retrospective nature of the study, there were difficulties associated with extracting relevant complication information from the medical records. Only 2% of owner-directed discharge instructions were written at readability levels aligning with the recommendations set forth in the human guidelines.

Assessment of the Readability of the Online Patient Education Materials of Intensive and Critical Care Societies.

This study aimed to evaluate the readability of patient education materials (PEMs) on websites of intensive and critical care societies. Websites of intensive and critical care societies, which are members of The World Federation of Intensive and Critical Care and The European Society of Intensive Care Medicine. Cross-sectional observational, internet-based, website, PEMs, readability study. The readability of the PEMs available on societies' sites was evaluated. The readability formulas used were the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and Gunning Fog (GFOG). One hundred twenty-seven PEM from 11 different societies were included in our study. In the readability analysis of PEM, the FRES was 58.10 (48.85-63.77) (difficult), the mean FKGL and SMOG were 10.19 (8.93-11.72) and 11.10 (10.11-11.87) years, respectively, and the mean GFOG score was 12.73 (11.37-14.15) (very difficult). All readability formula results were significantly higher than the recommended sixth-grade level ( p < 0.001). All PEMs were above the sixth-grade level when the societies were evaluated individually according to all readability results ( p < 0.05). Compared with the sixth-grade level recommended by the American Medical Association and the National Institutes of Health, the readability of PEMs in intensive and critical care societies is relatively high. PEMs in intensive and critical care societies should be prepared with attention to recommendations on readability.

Otosclerosis online: a critical analysis of quality, reliability, readability and content of otosclerosis information.

Several therapeutic options are usually discussed for otosclerosis management. Patients seek medical advice from an ENT specialist but are also increasingly using the internet for medical issues. This study intends to assess readability and quality of websites with information on otosclerosis. This is a cross-sectional study performed in a tertiary care centre. The results of the first two pages of a Google search with the keyword "otosclerosis" were reviewed by two independent investigators. Readability was assessed with the Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES) and Gunning Fog Index. For quality and reliability assessment, the 16-item DISCERN instrument was used. Spearman's coefficient was used for correlations, and multivariate analyses of variance were used to assess differences. Inter-rater agreement was evaluated with concordance correlation coefficient. 18 websites were included. Two websites (11.0%) were authored by academic institutions, 5/18 (28%) by government agencies, 6/18 (33%) by professional organisations and 5/18 (28%) were medical information websites. The mean DISCERN score of the 18 websites was 40.8 ± 6.7/80 (range 28.7-51.7), corresponding to "fair" quality. The mean FRES score was 43.27 ± 10.6, and the mean FKGL was 11.43 ± 2.30, corresponding to "difficult to read". The mean Gunning Fog index was 12.90 ± 2.19 (range 9.81-18.20), corresponding to a "college freshman" level. This study shows that internet information on otosclerosis has an overall low readability, while the quality is heterogeneous and varies from "poor" to "good". Efforts should be made to improve the readability of otosclerosis websites.

How readable and quality are online patient education materials about Helicobacter pylori?: Assessment of the readability, quality and reliability.

This study aimed to examine the readability, reliability, quality, and content of patient education materials (PEM) on the Internet about "Helicobacter pylori (H pylori)." A search was conducted on March 14, 2023, using the keyword "H pylori" in the Google search engine. The readability of PEMs was assessed using the Flesch reading ease score, FKGL, simple measure of gobbledygook, and gunning fog readability formulas. The reliability and quality of the websites were determined using the Journal of American Medical Association score, health on the net foundation code of conduct, global quality score, and DISCERN score. A total of 93 patient education websites were included in the study. In the readability analysis of PEMs, we determined that the Flesch reading ease score was 49,73 (47,46-52,00) (difficult), the mean Flesch-Kincaid grade level and simple measure of gobbledygook were 9,69 (9,26-10,12) and 9,28 (8,96-9,61) years, respectively, and the mean gunning fog score was 12,47 (12,03-12,91) (very difficult). Most of the evaluated patient educational materials were commercial websites (n = 50, 53.8%). It was found that 16.1% of the websites were of high quality according to global quality score, 30.1% were HON code certified, and 23.7% of the websites were highly reliable according to Journal of American Medical Association scores. There was no statistically significant difference between website typologies and readability (P > .05). However, there was a statistically significant difference between website typologies and quality and reliability scores (P < .005). Compared to the sixth grade level recommended by the American Medical Association and National Institutes of Health, the readability of H pylori-related internet-based PEMs is quite high. On the other hand, the reliability and quality of the PEMs were determined as moderate to poor. PEMs for issues threatening public health should be prepared with attention to recommendations on readability.

The Use of Large Language Models to Generate Education Materials about Uveitis

To assess large language models in generating readable uveitis information and in improving the readability of online health information. Evaluation of technology. Not applicable. ChatGPT and Bard were asked the following prompts: (prompt A) "considering that the average American reads at a 6th grade level, using the Flesch-Kincaid Grade Level (FKGL) formula, can you write patient-targeted health information on uveitis of around 6th grade level?" and (prompt B) "can you write patient-targeted health information on uveitis that is easy to understand by an average American?" Additionally, ChatGPT and Bard were asked the following prompt from the first-page results of Google when the term "uveitis" was searched: "Considering that the average American reads at a 6th grade level, using the FKGL formula, can you rewrite the following text to 6th grade level: [insert text]." The readability of each response was analyzed and compared using several metrics described below. The FKGL is a highly validated readability assessment tool that assigns a grade level to a given text, the total number of words, sentences, syllables, and complex words. Complex words were defined as those with > 2 syllables. ChatGPT and Bard generated responses with lower FKGL scores (i.e., easier to understand) in response to prompt A compared with prompt B. This was only significant for ChatGPT (P < 0.0001). The mean FKGL of responses to ChatGPT (6.3 ± 1.2) was significantly lower (P < 0.0001) than Bard 10.5 ± 0.8. ChatGPT responses also contained less complex words than Bard (P < 0.0001). Online health information on uveitis had a mean grade level of 11.0 ± 1.4. ChatGPT lowered the FKGL to 8.0 ± 1.0 (P < 0.0001) when asked to rewrite the content. Bard was not able to do so (mean FKGL of 11.1 ± 1.6). ChatGPT can aid clinicians in producing easier-to-understand health information on uveitis for patients compared with already-existing content. It can also help with reducing the difficulty of the language used for uveitis health information targeted for patients. Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Readability of the Orthopaedic Trauma Association Patient Education Tool.

Online educational materials have become a fundamental resource for orthopaedic surgery patients; however, they are frequently written at a high level of reading ability and are too complicated for some patients to understand. The aim of this study was to appraise the readability of the Orthopaedic Trauma Association (OTA) patient education materials. All forty-one articles on the OTA patient education website (https://ota.org/for-patients) were analyzed for readability. Readability scores were calculated by two independent reviewers using the (1) Flesch-Kincaid Grade Level (FKGL) and the (2) Flesch Reading Ease (FRE) algorithms. Mean readability scores were compared across anatomic categories. One-sample t-test was done to compare mean FKGL with the recommended 6th-grade readability level and the average American adult reading level. The average (SD) FKGL of the 41 OTA articles was 8.15 (1.14). The average (SD) FRE for the OTA patient education materials was 65.5 (6.60). Four (11%) of the articles were at or below a 6th-grade reading level. The average readability of the OTA articles was significantly higher than the recommended 6th-grade level (p < 0.001, 95% CI [7.79 - 8.51]). The average readability of the OTA articles was not significantly different from the average 8th-grade reading skill level of U.S. adults (p = 0.41, 95% CI [7.79 - 8.51]). Our findings suggest that despite the majority of the OTA patient education materials having readability levels suitable for the average US adult, these reading materials are still above the recommended 6th-grade level and may be too difficult for patient comprehension.

Health literacy in rotator cuff repair: a quantitative assessment of the understandability of online patient education material

The American Medical Association and National Institutes of Health recommend online health information be written at a 6th grade or lower reading level for clear understanding. While syntax reading grade level has previously been utilized, those analyses do not determine whether readers are processing key information (understandability) or identifying available actions to take (actionability). The Patient Education Materials Assessment Tool (PEMAT-P) is a method to measure the understandability and actionability of online patient education materials. The purpose of this study was to evaluate online resources regarding rotator cuff repair utilizing measures of readability, understandability, and actionability. The search term "rotator cuff surgery" was used in two independent online searches to obtain the top 50 search results. The readability of included resources was quantified using valid objective algorithms: Flesch-Kincaid Grade-Level, Simple Measure of Gobbledygook grade, Coleman-Liau Index, and Gunning Fog Index. The PEMAT-P form was used to assess actionability and understandability. A total of 49 unique websites were identified to meet our inclusion criteria and were included in our analysis. The mean Flesch-Kincaid Grade Level graded materials at a 10.6 (approximately a 10th grade reading level), with only two websites offering materials at a 6th grade reading level or below. The remaining readability studies graded the mean reading level at high school or greater, with the Gunning Fog Index scoring at a collegiate reading level. Mean understandability and actionability scores were 64.6% and 29.5%, respectively, falling below the 70% PEMAT score threshold for both scales. Fourteen (28.6%) websites were above the threshold for understandability, while no website (0%) scored above the 70% threshold for actionability. When comparing source categories, commercial health publishers provided websites that scored higher in understandability (P<.05), while private practice materials scored higher in actionability (P<.05). Resources published by academic institutions or organizations scored lower in both understandability and actionability than private practice and commercial health publishers (P<.05). No readability, understandability, or actionability score was significantly associated with search result rank. Overall, online patient education materials related to rotator cuff surgery scored poorly with respect to readability, understandability, and actionability. Only two (4.1%) of the patient education websites scored at the American Medical Association and National Institutes of Health recommended reading level. Fourteen (28.6%) scored above the 70% PEMAT score for understandability; however, no website met the threshold for actionability.

Evaluation of an Artificial Intelligence Chatbot for Delivery of IR Patient Education Material: A Comparison with Societal Website Content

PurposeTo assess the accuracy, completeness, and readability of patient educational material produced by a machine learning model and compare the output to that provided by a societal website. Materials and MethodsContent from the Society of Interventional Radiology Patient Center website was retrieved, categorized, and organized into discrete questions. These questions were entered into the ChatGPT platform, and the output was analyzed for word and sentence counts, readability using multiple validated scales, factual correctness, and suitability for patient education using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) instrument. ResultsA total of 21,154 words were analyzed, including 7,917 words from the website and 13,377 words representing the total output of the ChatGPT platform across 22 text passages. Compared to the societal website, output from the ChatGPT platform was longer and more difficult to read on 4 of 5 readability scales. The ChatGPT output was incorrect for 12 (11.5%) of 104 questions. When reviewed using the PEMAT-P tool, the ChatGPT content scored lower than the website material. Content from both the website and ChatGPT were significantly above the recommended fifth or sixth grade level for patient education, with a mean Flesch-Kincaid grade level of 11.1 (±1.3) for the website and 11.9 (±1.6) for the ChatGPT content. ConclusionsThe ChatGPT platform may produce incomplete or inaccurate patient educational content, and providers should be familiar with the limitations of the system in its current form. Opportunities may exist to fine-tune existing large language models, which could be optimized for the delivery of patient educational content.

Readability of patient education materials for bariatric surgery.

Bariatric surgery is a successful treatment for obesity, but barriers to surgery exist, including low health literacy. National organizations recommend patient education materials (PEM) not exceed a sixth-grade reading level. Difficult to comprehend PEM can exacerbate barriers to bariatric surgery, especially in the Deep South where high obesity and low literacy rates exist. This study aimed to assess and compare the readability of webpages and electronic medical record (EMR) bariatric surgery PEM from one institution. Readability of online bariatric surgery and standardized perioperative EMR PEM were analyzed and compared. Text readability was assessed by validated instruments: Flesch Reading Ease Formula (FRE), Flesch Kincaid Grade Level (FKGL), Gunning Fog (GF), Coleman-Liau Index (CL), Simple Measure of Gobbledygook (SMOG), Automated Readability Index (ARI), and Linsear Write Formula (LWF). Mean readability scores were calculated with standard deviations and compared using unpaired t-tests. 32 webpages and seven EMR education documents were analyzed. Webpages were overall "difficult to read" compared to "standard/average" readability EMR materials (mean FRE 50.5 ± 18.3 vs. 67.4 ± 4.2, p = 0.023). All webpages were at or above high school reading level: mean FKGL 11.8 ± 4.4, GF 14.0 ± 3.9, CL 9.5 ± 3.2, SMOG 11.0 ± 3.2, ARI 11.7 ± 5.1, and LWF 14.9 ± 6.6. Webpages with highest reading levels were nutrition information and lowest were patient testimonials. EMR materials were sixth to ninth grade reading level: FKGL 6.2 ± 0.8, GF 9.3 ± 1.4, CL 9.7 ± 0.9, SMOG 7.1 ± 0.8, ARI 6.1 ± 1.0, and LWF 5.9 ± 0.8. Surgeon curated bariatric surgery webpages have advanced reading levels above recommended thresholds compared to standardized PEM from an EMR. This readability gap may unintentionally contribute to barriers to surgery and affect postoperative outcomes. Streamlined efforts are needed to create materials that are easier to read and comply with recommendations.