Flesch-Kincaid Reading Grade Level Research Articles

Evaluación de la fiabilidad y legibilidad de las respuestas de los chatbots como recurso de información al paciente para las exploraciones PET-TC más communes

PurposeThis study aimed to evaluate the reliability and readability of responses generated by two popular AI-chatbots, ‘ChatGPT-4.0’ and ‘Google Gemini’, to potential patient questions about PET/CT scans. Materials and methodsThirty potential questions for each of [18F]FDG and [68Ga]Ga-DOTA-SSTR PET/CT, and twenty-nine potential questions for [68Ga]Ga-PSMA PET/CT were asked separately to ChatGPT-4 and Gemini in May 2024. The responses were evaluated for reliability and readability using the modified DISCERN (mDISCERN) scale, Flesch Reading Ease (FRE), Gunning Fog Index (GFI), and Flesch-Kincaid Reading Grade Level (FKRGL). The inter-rater reliability of mDISCERN scores provided by three raters (ChatGPT-4, Gemini, and a nuclear medicine physician) for the responses was assessed. ResultsThe median [min-max] mDISCERN scores reviewed by the physician for responses about FDG, PSMA and DOTA PET/CT scans were 3.5 [2–4], 3 [3–4], 3 [3–4] for ChatPT-4 and 4 [2–5], 4 [2–5], 3.5 [3–5] for Gemini, respectively. The mDISCERN scores assessed using ChatGPT-4 for answers about FDG, PSMA, and DOTA-SSTR PET/CT scans were 3.5 [3–5], 3 [3–4], 3 [2–3] for ChatGPT-4, and 4 [3–5], 4 [3–5], 4 [3–5] for Gemini, respectively. The mDISCERN scores evaluated using Gemini for responses FDG, PSMA, and DOTA-SSTR PET/CTs were 3 [2–4], 2 [2–4], 3 [2–4] for ChatGPT-4, and 3 [2–5], 3 [1–5], 3 [2–5] for Gemini, respectively. The inter-rater reliability correlation coefficient of mDISCERN scores for ChatGPT-4 responses about FDG, PSMA, and DOTA-SSTR PET/CT scans were 0.629 (95% CI = 0,32−0,812), 0.707 (95% CI = 0.458−0.853) and 0.738 (95% CI = 0.519−0.866), respectively (p < 0.001). The correlation coefficient of mDISCERN scores for Gemini responses about FDG, PSMA, and DOTA-SSTR PET/CT scans were 0.824 (95% CI = 0.677−0.910), 0.881 (95% CI = 0.78−0.94) and 0.847 (95% CI = 0.719−0.922), respectively (p < 0.001). The mDISCERN scores assessed by ChatGPT-4, Gemini, and the physician showed that the chatbots' responses about all PET/CT scans had moderate to good statistical agreement according to the inter-rater reliability correlation coefficient (p < 0,001). There was a statistically significant difference in all readability scores (FKRGL, GFI, and FRE) of ChatGPT-4 and Gemini responses about PET/CT scans (p < 0,001). Gemini responses were shorter and had better readability scores than ChatGPT-4 responses. ConclusionThere was an acceptable level of agreement between raters for the mDISCERN score, indicating agreement with the overall reliability of the responses. However, the information provided by AI-chatbots cannot be easily read by the public.

Accessing information provided via artificial intelligence regarding reverse and anatomic total shoulder arthroplasty

BackgroundThe purpose of this study is to analyze the quality, accuracy, and readability of information provided by an artificial intelligence (AI) interface ChatGPT (OpenAI, San Francisco). We searched ChatGPT for commonly asked questions by patients regarding anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA). MethodsChatGPT was used to answer 30 commonly asked questions by patients regarding aTSA and rTSA, inputted as “total shoulder replacement” and “reverse shoulder replacement”. These questions were categorized based on the Rothwell criteria into Fact, Policy, and Value. The answers generated were analyzed for quality, accuracy, and readability using the DISCERN scale, JAMA benchmark criteria, Flesch-Kincaid Reading Ease Score (FRES) and Grade Level (FKGL). ResultsFor both rTSA and aTSA the DISCERN score for Fact questions was 57, Policy questions was 61, and for Value questions was 58 (all were all considered “good”). The JAMA benchmark criteria was 0, representing the lowest score for Fact, Policy, and Value questions for both rTSA and aTSA questions. The FRES score for the aTSA answers for Fact was 15.15, for Policy was 11.14, and for Value questions was 10.95. The FRES score for rTSA questions for Fact is 48.02, Policy is 12.51, and Value is 17.22. The FKGL for aTSA answer for Fact was 17.48, Policy was 17.72 and Value was 17.96. The FKGL for rTSA questions for Fact are 8.10, Policy is 17.27, and Value is 16.56. ConclusionOverall, the quality of answers provided by AI open model, ChatGPT was considered “good.” The information provided had lower reliability, and lack of information regarding funding and disclosures. Most of the information generated by ChatGPT was also found to have the readability of “academic level text”, while Fact related information on reverse shoulder arthroplasty was found to have the readability of 9th grade level, which may be too complex for most patients. Overall, these results indicate that ChatGPT can provide correct answers to questions about aTSA and rTSA, although we would caution patients from utilizing this resource due to the lack of citations and complexity of the output that ChatGPT provides. Importantly, all answers provided by AI suggested reaching out to physicians to get more accurate and personalized advise, to factor into the shared decisions making model.

TrachGPT: Appraisal of tracheostomy care recommendations from an artificial intelligent Chatbot.

Safe home tracheostomy care requires engagement and troubleshooting by patients, who may turn to online, AI-generated information sources. This study assessed the quality of ChatGPT responses to such queries. In this cross-sectional study, ChatGPT was prompted with 10 hypothetical tracheostomy care questions in three domains (complication management, self-care advice, and lifestyle adjustment). Responses were graded by four otolaryngologists for appropriateness, accuracy, and overall score. The readability of responses was evaluated using the Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKRGL). Descriptive statistics and ANOVA testing were performed with statistical significance set to p < .05. On a scale of 1-5, with 5 representing the greatest appropriateness or overall score and a 4-point scale with 4 representing the highest accuracy, the responses exhibited moderately high appropriateness (mean = 4.10, SD = 0.90), high accuracy (mean = 3.55, SD = 0.50), and moderately high overall scores (mean = 4.02, SD = 0.86). Scoring between response categories (self-care recommendations, complication recommendations, lifestyle adjustments, and special device considerations) revealed no significant scoring differences. Suboptimal responses lacked nuance and contained incorrect information and recommendations. Readability indicated college and advanced levels for FRE (Mean = 39.5, SD = 7.17) and FKRGL (Mean = 13.1, SD = 1.47), higher than the sixth-grade level recommended for patient-targeted resources by the NIH. While ChatGPT-generated tracheostomy care responses may exhibit acceptable appropriateness, incomplete or misleading information may have dire clinical consequences. Further, inappropriately high reading levels may limit patient comprehension and accessibility. At this point in its technological infancy, AI-generated information should not be solely relied upon as a direct patient care resource.

Quality and readability of online information and materials on post-surgery breast seroma.

Aims/Background Seroma formation is the most common complication following breast surgery. However, there is little evidence on the readability of online patient education materials on this issue. This study aimed to assess the accessibility and readability of the relevant online information. Methods This systematic review of the literature identified 37 relevant websites for further analysis. The readability of each online article was assessed through using a range of readability formulae. Results The average Flesch-Reading Ease score for all patient education materials was 53.9 (± 21.9) and the average Flesch-Kincaid reading grade level was 7.32 (± 3.1), suggesting they were 'fairly difficult' to read and is higher than the recommended reading level. Conclusion Online patient education materials regarding post-surgery breast seroma are at a higher-than-recommended reading grade level for the public. Improvement would allow all patients, regardless of literacy level, to access such resources to aid decision-making around undergoing breast surgery.

A cross-sectional quantitative analysis of the readability and quality of online resources regarding thumb carpometacarpal joint replacement surgery

BackgroundThumb carpometacarpal (CMC) joint osteoarthritis is a common degenerative condition that affects up to 15 ​% of the population older than 30 years. Poor readability of online health resources has been associated with misinformation, inappropriate care, incorrect self-treatment, worse health outcomes, and increased healthcare resource waste. This study aims to assess the readability and quality of online information regarding thumb carpometacarpal (CMC) joint replacement surgery. MethodsThe terms “thumb joint replacement surgery”, “thumb carpometacarpal joint replacement surgery”, “thumb cmc joint replacement surgery”, “thumb arthroplasty”, “thumb carpometacarpal arthroplasty”, and “thumb cmc arthroplasty” were searched in Google and Bing. Readability was determined using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Reading Grade Level (FKGL). FRES >65 or a grade level score of sixth grade and under was considered acceptable. Quality was assessed using the Patient Education Materials Assessment Tool (PEMAT) and a modified DISCERN tool. PEMAT scores below 70 were considered poorly understandable and poorly actionable. ResultsA total of 34 websites underwent qualitative analysis. The average FRES was 54.60 ​± ​7.91 (range 30.30–67.80). Only 3 (8.82 ​%) websites had a FRES score >65. The average FKGL score was 8.19 ​± ​1.80 (range 5.60–12.90). Only 3 (8.82 ​%) websites were written at or below a sixth-grade level. The average PEMAT percentage score for understandability and actionability was 76.82 ​± ​9.43 (range 61.54–93.75) and 36.18 ​± ​24.12 (range 0.00–60.00) respectively. Although 22 (64.71 ​%) of websites met the acceptable standard of 70 ​% for understandability, none of the websites met the acceptable standard of 70 ​% for actionability. The average total DISCERN score was 32.00 ​± ​4.29 (range 24.00–42.00). ConclusionsMost websites reviewed were written above recommended reading levels. Most showed acceptable understandability but none showed acceptable actionability. To avoid the negative outcomes of poor patient understanding of online resources, providers of these resources should optimise accessibility to the average reader by using simple words, avoiding jargon, and analysing texts with readability software before publishing the materials online. Websites should also utilise visual aids and provide clearer pre-operative and post-operative instructions.

Assessing the missions and visions of NCI-designated cancer centers and their affiliated hospitals.

e23118 Background: Mission and vision statements (MVS) help convey an institution's identity and priorities. Missions describe objectives and responsibilities, while visions describe overarching directions for the future. Effectively communicating MVS is particularly important for cancer centers and hospitals seeking to inspire and reach patients and employees. We sought to assess the composition, readability, and topics addressed for MVS provided by NCI-designated cancer centers and their affiliated hospitals. Methods: We extracted MVS data from institutional websites for 65 NCI-designated cancer centers and 59 affiliated hospitals in 2023 (6 cancer centers did not have an affiliated hospital). We determined the composition of statements using word counts and time to read. We assessed the readability of statements using Flesch-Kincaid (FK) reading ease and grade level scores. We reviewed the MVS to determine the presence of four topics: equity, quality care, training, and research. We used descriptive statistics to compare these issues in MVS for cancer center versus the hospital. Results: Among the 65 cancer centers, we found mission statements for 93.9% (61/65) and vision statements for 63.1% (41/65). All affiliated hospitals provided a mission statement and 86.4% (51/59) had a vision statement. Data for composition, readability, and topics addressed for MVS are provided in the table. Overall, readability of MVS was difficult based on FK reading ease scores, with relatively high reading grade levels. Mission statements for cancer centers had a significantly lower reading ease and higher grade level compared with hospitals. In general, mission statements were slightly longer than vision statements, more frequently addressing the topics of quality care, training, and research. Compared with hospitals, cancer centers had a significantly lower frequency of mentioning the topic "training" in their MVS. Conclusions: We found that the majority of NCI-designated cancer centers report mission statements, but fewer provide vision statements. We demonstrated difficult readability for the MVS provided, highlighting a need to consider simpler diction and structure of language. Additionally, our work provides an overview of topics addressed for these statements including equity, quality care, training, and research. These topics underscore the values and priorities of an institution, and this review of MVS provides insights into the messaging that cancer centers communicate. [Table: see text]

Exploring AI-generated content and professional guidelines in cancer symptom management: A comparative analysis between ChatGPT and NCCN guidelines.

e13610 Background: Artificial intelligence (AI)-driven tools, like ChatGPT, have become widely-available sources for online health information. Limited research has explored the congruity between AI-generated content and professional treatment guidelines. This study seeks to compare recommendations for cancer-related symptoms generated from ChatGPT with guidelines from National Comprehensive Cancer Network (NCCN), a provider focused source requiring users to log in or register to access its recommendations. Utilizing a provider-focused source like NCCN serves as a benchmark to assess whether ChatGPT recommendations align with the standards typically endorsed by clinicians. Methods: We extracted treatment recommendations from four NCCN Supportive Care webpages (Cancer Pain, Antiemesis, Cancer-Related Fatigue, and Distress Management) and five subsections of the NCCN Palliative Care webpage (dyspnea, constipation, diarrhea, sleep disturbances, and anorexia/cachexia). We then entered "How can I reduce my cancer-related [symptom]" into ChatGPT 3.5 and extracted its recommendations. We calculated and compared word count and Flesch-Kincaid Grade Level readability for each NCCN and ChatGPT section. We completed a comparative content analysis focusing on recommendations for medications, consultations, and non-pharmacological strategies. Results: Across the nine NCCN Supportive Care and Palliative Care webpages, the mean word count was 2393.8 (SD=2601.4) and the mean Flesch-Kincaid Grade Level was 17.3 (SD=1.4) vs 382.4 (SD=29.6) and 11.6 (SD=0.8) for ChatGPT. The mean percent agreement between NCCN and ChatGPT recommendations was 44.6% (range 14.3%-81.8%). ChatGPT and NCCN guidelines shared fewer than half of their symptom-related recommendations in all but one section, fatigue. NCCN offered specific medication recommendations across all sections. ChatGPT's recommendations lacked the specificity observed in NCCN's guidelines including often not suggesting any medications. ChatGPT recommended specific medications in the shortness of breath and diarrhea sections that were not recommended by NCCN. NCCN's guidelines often did not include recommendations related to spirituality or palliative care consults, areas that ChatGPT addressed. Conclusions: While ChatGPT provides concise, accessible supportive care advice including many non-medical support recommendations, discrepancies with guidelines raise concerns for patient-facing symptom management recommendations. Overall, AI-generated content like ChatGPT can assist in providing preliminary information but should be researched in conjunction with comprehensive, evidence-based guidance provided by healthcare professionals. Healthcare providers should work with AI developers to ensure data sources are high-quality and accurate.

Reliability and readability analysis of ChatGPT-4 and Google Bard as a patient information source for the most commonly applied radionuclide treatments in cancer patients

PurposeSearching for online health information is a popular approach employed by patients to enhance their knowledge for their diseases. Recently developed AI chatbots are probably the easiest way in this regard. The purpose of the study is to analyze the reliability and readability of AI chatbot responses in terms of the most commonly applied radionuclide treatments in cancer patients. MethodsBasic patient questions, thirty about RAI, PRRT and TARE treatments and twenty-nine about PSMA-TRT, were asked one by one to GPT-4 and Bard on January 2024. The reliability and readability of the responses were assessed by using DISCERN scale, Flesch Reading Ease(FRE) and Flesch-Kincaid Reading Grade Level(FKRGL). ResultsThe mean (SD) FKRGL scores for the responses of GPT-4 and Google Bard about RAI, PSMA-TRT, PRRT and TARE treatmens were 14.57 (1.19), 14.65 (1.38), 14.25 (1.10), 14.38 (1.2) and 11.49 (1.59), 12.42 (1.71), 11.35 (1.80), 13.01 (1.97), respectively. In terms of readability the FRKGL scores of the responses of GPT-4 and Google Bard about RAI, PSMA-TRT, PRRT and TARE treatments were above the general public reading grade level. The mean (SD) DISCERN scores assesses by nuclear medicine phsician for the responses of GPT-4 and Bard about RAI, PSMA-TRT, PRRT and TARE treatments were 47.86 (5.09), 48.48 (4.22), 46.76 (4.09), 48.33 (5.15) and 51.50 (5.64), 53.44 (5.42), 53 (6.36), 49.43 (5.32), respectively. Based on mean DISCERN scores, the reliability of the responses of GPT-4 and Google Bard about RAI, PSMA-TRT, PRRT, and TARE treatments ranged from fair to good. The inter-rater reliability correlation coefficient of DISCERN scores assessed by GPT-4, Bard and nuclear medicine physician for the responses of GPT-4 about RAI, PSMA-TRT, PRRT and TARE treatments were 0.512(95% CI 0.296: 0.704), 0.695(95% CI 0.518: 0.829), 0.687(95% CI 0.511: 0.823) and 0.649 (95% CI 0.462: 0.798), respectively (p < 0.01). The inter-rater reliability correlation coefficient of DISCERN scores assessed by GPT-4, Bard and nuclear medicine physician for the responses of Bard about RAI, PSMA-TRT, PRRT and TARE treatments were 0.753(95% CI 0.602: 0.863), 0.812(95% CI 0.686: 0.899), 0.804(95% CI 0.677: 0.894) and 0.671 (95% CI 0.489: 0.812), respectively (p < 0.01). The inter-rater reliability for the responses of Bard and GPT-4 about RAİ, PSMA-TRT, PRRT and TARE treatments were moderate to good. Further, consulting to the nuclear medicine physician was rarely emphasized both in GPT-4 and Google Bard and references were included in some responses of Google Bard, but there were no references in GPT-4. ConclusionAlthough the information provided by AI chatbots may be acceptable in medical terms, it can not be easy to read for the general public, which may prevent it from being understandable. Effective prompts using 'prompt engineering' may refine the responses in a more comprehensible manner. Since radionuclide treatments are specific to nuclear medicine expertise, nuclear medicine physician need to be stated as a consultant in responses in order to guide patients and caregivers to obtain accurate medical advice. Referencing is significant in terms of confidence and satisfaction of patients and caregivers seeking information.

A Cross-Sectional Analysis of the Readability of Online Information Regarding Hip Osteoarthritis.

Introduction Osteoarthritis (OA) is an age-related degenerative joint disease. There is a 25% risk of symptomatic hip OA in patients who live up to 85 years of age. It can impair a person's daily activities and increase their reliance on healthcare services. It is primarily managed with education, weight loss and exercise, supplemented with pharmacological interventions. Poor health literacy is associated with negative treatment outcomes and patient dissatisfaction. A literature search found there are no previously published studies examining the readability of online information about hip OA. Objectives To assess the readability of healthcare websites regarding hip OA. Methods The terms "hip pain", "hip osteoarthritis", "hip arthritis", and "hip OA" were searched on Google and Bing. Of 240 websites initially considered, 74 unique websites underwent evaluation using the WebFX online readability software (WebFX®, Harrisburg, USA). Readability was determined using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Reading Grade Level (FKGL), Gunning Fog Index (GFI), Simple Measure of Gobbledygook (SMOG), Coleman-Liau Index (CLI), and Automated Readability Index (ARI). In line with recommended guidelines and previous studies, FRES >65 or a grade level score of sixth grade and under was considered acceptable. Results The average FRES was 56.74±8.18 (range 29.5-79.4). Only nine (12.16%) websites had a FRES score >65. The average FKGL score was 7.62±1.69 (range 4.2-12.9). Only seven (9.46%) websites were written at or below a sixth-grade level according to the FKGL score. The average GFI score was 9.20±2.09 (range 5.6-16.5). Only one (1.35%) website was written at or below a sixth-grade level according to the GFI score.The average SMOG score was 7.29±1.41 (range 5.4-12.0). Only eight (10.81%) websites were written at or below a sixth-grade level according to the SMOG score. The average CLI score was 13.86±1.75 (range 9.6-19.7). All 36 websites were written above a sixth-grade level according to the CLI score. The average ARI score was 6.91±2.06 (range 3.1-14.0). Twenty-eight(37.84%) websites were written at or below a sixth-grade level according to the ARI score. One-sample t-tests showed that FRES (p<0.001, CI -10.2 to -6.37), FKGL (p<0.001, CI 1.23 to 2.01), GFI (p<0.001, CI 2.72 to 3.69), SMOG (p<0.001, CI 0.97 to 1.62), CLI (p<0.001, CI 7.46 to 8.27), and ARI (p<0.001, CI 0.43 to 1.39) scores were significantly different from the accepted standard. One-way analysis of variance (ANOVA) testing of FRES scores (p=0.009) and CLI scores (p=0.009) showed a significant difference between categories. Post hoc testing showed a significant difference between academic and non-profit categories for FRES scores (p=0.010, CI -15.17 to -1.47) and CLI scores (p=0.008, CI 0.35 to 3.29). Conclusions Most websites regarding hip OA are written above recommended reading levels, hence exceeding the comprehension levels of the average patient. Readability of these resources must be improved to improve patient access to online healthcare information which can lead to improved patient understanding of their own condition and treatment outcomes.

Evaluation of the Current Status of Artificial Intelligence for Endourology Patient Education: A Blind Comparison of ChatGPT and Google Bard Against Traditional Information Resources.

Introduction: Artificial intelligence (AI) platforms such as ChatGPT and Bard are increasingly utilized to answer patient health care questions. We present the first study to blindly evaluate AI-generated responses to common endourology patient questions against official patient education materials. Methods: Thirty-two questions and answers spanning kidney stones, ureteral stents, benign prostatic hyperplasia (BPH), and upper tract urothelial carcinoma were extracted from official Urology Care Foundation (UCF) patient education documents. The same questions were input into ChatGPT 4.0 and Bard, limiting responses to within ±10% of the word count of the corresponding UCF response to ensure fair comparison. Six endourologists blindly evaluated responses from each platform using Likert scales for accuracy, clarity, comprehensiveness, and patient utility. Reviewers identified which response they believed was not AI generated. Finally, Flesch-Kincaid Reading Grade Level formulas assessed the readability of each platform response. Ratings were compared using analysis of variance (ANOVA) and chi-square tests. Results: ChatGPT responses were rated the highest across all categories, including accuracy, comprehensiveness, clarity, and patient utility, while UCF answers were consistently scored the lowest, all p < 0.01. A subanalysis revealed that this trend was consistent across question categories (i.e., kidney stones, BPH, etc.). However, AI-generated responses were more likely to be classified at an advanced reading level, while UCF responses showed improved readability (college or higher reading level: ChatGPT = 100%, Bard = 66%, and UCF = 19%), p < 0.001. When asked to identify which answer was not AI generated, 54.2% of responses indicated ChatGPT, 26.6% indicated Bard, and only 19.3% correctly identified it as the UCF response. Conclusions: In a blind evaluation, AI-generated responses from ChatGPT and Bard surpassed the quality of official patient education materials in endourology, suggesting that current AI platforms are already a reliable resource for basic urologic care information. AI-generated responses do, however, tend to require a higher reading level, which may limit their applicability to a broader audience.

Quality, Reliability, and Readability of Online Information on Idiopathic Intracranial Hypertension.

To evaluate the quality & reliability, technical quality, and readability of patient-targeted online information on idiopathic intracranial hypertension. In this cross-sectional study, we searched Google and Bing search engines for the terms "idiopathic intracranial hypertension" and "pseudotumor cerebri." We evaluated the first 50 search outcomes for each search term. Peer-reviewed articles, targeted advertisements, book chapters, videos, personal blogs, websites intended for healthcare professionals, and non-English websites were excluded. Websites were classified as institutional or private. Two independent reviewers assessed each website for 1) quality and reliability using DISCERN, Health on the Net Code of Conduct (HONcode), and JAMA criteria and 2) technical quality based on 11 criteria. Readability was evaluated using 6 measures (Flesch-Kincaid Reading Ease score and grade level, Gunning Fog, Simple Measure of Gobbledygook (Smog) index, Coleman-Liau index, and automated readability index). Sixty-three websites (37 institutional) were included. The mean scores for the DISCERN, HONcode, and JAMA criteria were 3.6 ± 1 (1-4.8; 1: worse, 5: best), 10.3 ± 2.8 (2-16; 0: worse, 16: best), and 2 ± 1.3 (0-4; 0: worse, 4: best), respectively. The mean technical quality score was 0.8 ± 0.1 (0.5-1). The mean Flesch-Kincaid grade level score was 8.9 ± 1.8 (3.3-13.3). For Flesch-Kincaid grade level, 47 (74.6%) websites scored a grade level of 10 or less. For Gunning Fog Score, 35 websites (55.6%) scored from 7.3 to 10.8. For the Smog Index, 46 (73%) websites scored 5.7-8. The mean Coleman Liau Index was 16 ± 2.1 (9.6-22.2). For the automated readability index, 30 (50.7%) websites scored less than the eighth grade. No statistically significant difference was present between institutional and private websites except for JAMA, which scored higher in private websites (2.4 vs 1.7, P = 0.03). Our findings suggest that online information on idiopathic intracranial hypertension/pseudotumor cerebri for patients generally demonstrates moderate to good quality and reliability. Although most websites maintain credible technical quality, the readability exceeds recommended levels for the average American reader. This may impede understanding, emphasizing the need for future improvements.

Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury.

The aim of our study was to evaluate the potential role of Artificial Intelligence tools like ChatGPT in patient education. To do this, we assessed both the quality and readability of information provided by ChatGPT 3.5 and 4 in relation to Anterior Cruciate Ligament (ACL) injury and treatment. ChatGPT 3.5 and 4 were used to answer common patient queries relating to ACL injuries and treatment. The quality of the information was assessed using the DISCERN criteria. Readability was assessed with the use of seven readability formulae: the Flesch-Kincaid Reading Grade Level, the Flesch Reading Ease Score, the Raygor Estimate, the SMOG, the Fry, the FORCAST, and the Gunning Fog. The mean reading grade level (RGL) was compared with the recommended 8th-grade reading level, the mean RGL among adults in America. The perceived quality and mean RGL of answers given by both ChatGPT 3.5 and 4 was also compared. Both ChatGPT 3.5 and 4 yielded DISCERN scores suggesting "good" quality of information, with ChatGPT 4 slightly outperforming 3.5. However, readability levels for both versions significantly exceeded the average 8th-grade reading level for American patients. ChatGPT 3.5 had a mean RGL of 18.08, while the mean RGL of ChatGPT 4 was 17.9, exceeding the average American reading grade level by 10.08 grade levels and 9.09 grade levels, respectively. While ChatGPT can provide both reliable and good quality information on ACL injuries and treatment options, the readability of the content may limit its utility. Additionally, the consistent lack of source citation represents a significant area of concern for patients and clinicians alike. If AI is to play a role in patient education, it must reliably produce information which is accurate, easily comprehensible, and clearly sourced.

Evaluation of Oropharyngeal Cancer Information from Revolutionary Artificial Intelligence Chatbot.

With burgeoning popularity of artificial intelligence-based chatbots, oropharyngeal cancer patients now have access to a novel source of medical information. Because chatbot information is not reviewed by experts, we sought to evaluate an artificial intelligence-based chatbot's oropharyngeal cancer-related information for accuracy. Fifteen oropharyngeal cancer-related questions were developed and input into ChatGPT version 3.5. Four physician-graders independently assessed accuracy, comprehensiveness, and similarity to a physician response using 5-point Likert scales. Responses graded lower than three were then critiqued by physician-graders. Critiques were analyzed using inductive thematic analysis. Readability of responses was assessed using Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKRGL) scales. Average accuracy, comprehensiveness, and similarity to a physician response scores were 3.88 (SD = 0.99), 3.80 (SD = 1.14), and 3.67 (SD = 1.08), respectively. Posttreatment-related questions were most accurate, comprehensive, and similar to a physician response, followed by treatment-related, then diagnosis-related questions. Posttreatment-related questions scored significantly higher than diagnosis-related questions in all three domains (p < 0.01). Two themes of the physician critiques were identified: suboptimal education value and potential to misinform patients. The mean FRE and FKRGL scores both indicated greater than an 11th grade readability level-higher than the 6th grade level recommended for patients. ChatGPT responses may not educate patients to an appropriate degree, could outright misinform them, and read at a more difficult grade level than is recommended for patient material. As oropharyngeal cancer patients represent a vulnerable population facing complex, life-altering diagnoses, and treatments, they should be cautious when consuming chatbot-generated medical information. NA Laryngoscope, 134:2252-2257, 2024.

Investigating the Readability and Linguistic, Psychological, and Emotional Characteristics of Digital Dementia Information Written in the English Language: Multitrait-Multimethod Text Analysis.

Past research in the Western context found that people with dementia search for digital dementia information in peer-reviewed medical research articles, dementia advocacy and medical organizations, and blogs written by other people with dementia. This past work also demonstrated that people with dementia do not perceive English digital dementia information as emotionally or cognitively accessible. In this study, we sought to investigate the readability; linguistic, psychological, and emotional characteristics; and target audiences of digital dementia information. We conducted a textual analysis of 3 different types of text-based digital dementia information written in English: 300 medical articles, 35 websites, and 50 blogs. We assessed the text's readability using the Flesch Reading Ease and Flesch-Kincaid Grade Level measurements, as well as tone, analytical thinking, clout, authenticity, and word frequencies using a natural language processing tool, Linguistic Inquiry and Word Count Generator. We also conducted a thematic analysis to categorize the target audiences for each information source and used these categorizations for further statistical analysis. The median Flesch-Kincaid Grade Level readability score and Flesch Reading Ease score for all types of information (N=1139) were 12.1 and 38.6, respectively, revealing that the readability scores of all 3 information types were higher than the minimum requirement. We found that medical articles had significantly (P=.05) higher word count and analytical thinking scores as well as significantly lower clout, authenticity, and emotional tone scores than websites and blogs. Further, blogs had significantly (P=.48) higher word count and authenticity scores but lower analytical scores than websites. Using thematic analysis, we found that most of the blogs (156/227, 68.7%) and web pages (399/612, 65.2%) were targeted at people with dementia. Website information targeted at a general audience had significantly lower readability scores. In addition, website information targeted at people with dementia had higher word count and lower emotional tone ratings. The information on websites targeted at caregivers had significantly higher clout and lower authenticity scores. Our findings indicate that there is an abundance of digital dementia information written in English that is targeted at people with dementia, but this information is not readable by a general audience. This is problematic considering that people with <12 years of education are at a higher risk of developing dementia. Further, our findings demonstrate that digital dementia information written in English has a negative tone, which may be a contributing factor to the mental health crisis many people with dementia face after receiving a diagnosis. Therefore, we call for content creators to lower readability scores to make the information more accessible to a general audience and to focus their efforts on providing information in a way that does not perpetuate overly negative narratives of dementia.

Online Patient Education Resources for Anterior Cruciate Ligament Reconstruction: An Assessment of the Accuracy and Reliability of Information on the Internet Over the Past Decade.

The purpose of this study is to evaluate the quality of patient education materials accessible through popular online search engines regarding anteriorcruciateligament (ACL) injuries and anteriorcruciateligament reconstruction (ACLR). Methods: Two search terms ("ACL surgery" and "ACL reconstruction") were entered into three search engines (Google, Yahoo, and Bing). The quality of information was scored using a novel scoring system developed and overseen by sports medicine orthopedic clinical research fellows and fellowship-trained orthopedic surgeons. Website quality, credibility, and readability were further assessed by the DISCERN score, Journal of the American Medical Association (JAMA) benchmark criteria, and Flesch-Kincaid Reading Grade Level (FKRGL), respectively. The Health On the Net Code of Conduct (HONcode) certification was also utilized to assess the transparency of health information for each website. We evaluated 39 websites. The average score for all websites was 11.2±5.6 out of 28 total points. Six out of the 39 websites (41%) were HONcodecertified. The websites that contained HONcode certification had a higher average JAMA benchmark score (3.5±0.7) and DISCERN score (44.6±14.7) when compared to the websites without the certification, 2.2±1.2and 37.6 ± 15.9for JAMA and DISCERN, respectively. The mean JAMA benchmark score was 2.7±1.2 (67.5%) for all websites out of a possible four points. The average FKRGL for all 39 websites was 10.0±2.0 (range: 5.4-13). The quality of patient education materials accessible on the internet regarding ACL injuries and ACLR can be misleading and directly impact the patient's decision-making process essential to the patient-physician relationship over the past decade. The internet can be a helpful online resource, however, surgeon clarification and consultation with qualified healthcare professionals are strongly recommended prior to clinical decision-making regarding potential treatment options.

Evaluating the readability of online patient-facing resources for alcohol use disorder

The goal of this study was to assess the readability of online resources pertaining to Alcohol Use Disorder (AUD) as perceived by patients seeking treatment. The National Institutes of Health (NIH) and American Medical Association (AMA) have recommended that medical resources should be written at a 6th-grade reading level. However, prior investigations in various medical fields have revealed that online materials often fail to adhere to these guidelines. An internet search was conducted to simulate the experience of patients seeking information on AUD treatment. The first 30 websites that did not require login credentials were examined using established readability tests. The main outcomes included: Flesch-Kincaid Reading Grade Level, Gunning Fog index, Simple Measure of Gobbledygook (SMOG) Readability Formula, and Coleman-Liau index. Thirty records were identified where the mean readability level was 12.37 (2.54). There were no significant differences in mean readability across readability indices or author type. None of the 30 records met the reading level recommendations as set by the NIH and AMA. In order to enhance accessibility and ultimately improve AUD health outcomes, it is recommended that patient-oriented resources be crafted with adherence to these guidelines. Consequently, future AUD resources ought to prioritize the enhancement of their readability.

Evaluation High-Quality of Information from ChatGPT (Artificial Intelligence—Large Language Model) Artificial Intelligence on Shoulder Stabilization Surgery

To analyze the quality and readability of information regarding shoulder stabilization surgery available using an online AI software (ChatGPT), using standardized scoring systems, as well as to report on the given answers by the AI. An open AI model (ChatGPT) was used to answer 23 commonly asked questions from patients on shoulder stabilization surgery. These answers were evaluated for medical accuracy, quality, and readability using The JAMA Benchmark criteria, DISCERN score, Flesch-Kincaid Reading Ease Score (FRES) & Grade Level (FKGL). The JAMA Benchmark criteria score was 0, which is the lowest score, indicating no reliable resources cited. The DISCERN score was 60, which is considered a good score. The areas that open AI model did not achieve full marks were also related to the lack of available source material used to compile the answers, and finally some shortcomings with information not fully supported by the literature. The FRES was 26.2, and the FKGL was considered to be that of a college graduate. There was generally high quality in the answers given on questions relating to shoulder stabilization surgery, but there was a high reading level required to comprehend the information presented. However, it is unclear where the answers came from with no source material cited. It is important to note that the ChatGPT software repeatedly references the need to discuss these questions with an orthopaedic surgeon and the importance of shared discussion making, as well as compliance with surgeon treatment recommendations. As shoulder instability is an injury that predominantly affects younger individuals who may use the Internet for information, this study shows what information patients may be getting online.

Quality of information and appropriateness of ChatGPT outputs for urology patients.

The proportion of health-related searches on the internet is continuously growing. ChatGPT, a natural language processing (NLP) tool created by OpenAI, has been gaining increasing user attention and can potentially be used as a source for obtaining information related to health concerns. This study aims to analyze the quality and appropriateness of ChatGPT's responses to Urology case studies compared to those of a urologist. Data from 100 patient case studies, comprising patient demographics, medical history, and urologic complaints, were sequentially inputted into ChatGPT, one by one. A question was posed to determine the most likely diagnosis, suggested examinations, and treatment options. The responses generated by ChatGPT were then compared to those provided by a board-certified urologist who was blinded to ChatGPT's responses and graded on a 5-point Likert scale based on accuracy, comprehensiveness, and clarity as criterias for appropriateness. The quality of information was graded based on the section 2 of the DISCERN tool and readability assessments were performed using the Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKGL) formulas. 52% of all responses were deemed appropriate. ChatGPT provided more appropriate responses for non-oncology conditions (58.5%) compared to oncology (52.6%) and emergency urology cases (11.1%) (p = 0.03). The median score of the DISCERN tool was 15 (IQR = 5.3) corresponding to a quality score of poor. The ChatGPT responses demonstrated a college graduate reading level, as indicated by the median FRE score of 18 (IQR = 21) and the median FKGL score of 15.8 (IQR = 3). ChatGPT serves as an interactive tool for providing medical information online, offering the possibility of enhancing health outcomes and patient satisfaction. Nevertheless, the insufficient appropriateness and poor quality of the responses on Urology cases emphasizes the importance of thorough evaluation and use of NLP-generated outputs when addressing health-related concerns.

Quality and Readability of Accessible Facial Feminization Literature: Where Can We Improve?

Facial feminization surgery (FFS) consists of multiple, complex procedures. Well-informed patients have been shown to have better outcomes and expectations. However, there is limited data evaluating FFS patient-oriented material online. This study aims to evaluate the quality and readability of FFS literature online. Facial feminization surgery-related terms were queried in Google with location, cookies, and user account information disabled. Websites were analyzed for readability using Flesch Reading Ease and Flesch-Kincaid Reading Grade Level, suitability using the suitability assessment of materials (SAM), and quality using the DISCERN scale. Unpaired t tests and χ2 tests were used to compare the websites of community-based and academic or public institutions. One hundred twenty websites met inclusion criteria (71 community-based and 49 academic). The average reading grade level was 11.68 ± 1.71. The average SAM score was 53.11 ± 11.75, denoting adequate readability. Only 16.67% of websites contained visual aids to assist in explaining procedures and benefits. Overall, 68.33% of websites' DISCERN score was rated very poor or poor, whereas only 8.33% were rated as good. The following DISCERN elements had the lowest scores across all graded websites: "clear source and date of information," "details of additional source of support," "refers to areas of uncertainty," and "describes risks of each treatment." Websites published by community-based institutions were significantly better in describing how FFS works and the benefits of each procedure. Academic sites overall were less biased (4.84 vs 4.62, P = 0.03) and provided additional sources of support (2.35 vs 1.32, P = <0.001). Online FFS patient resources should be written at a more inclusive reading level and should increase the use of pictorial aids to improve patient comprehension. Facial feminization surgery patient resources were significantly above the recommended sixth grade reading level for health literature. Academic and community-based institutions should include more information about procedural risks and limitations in surgical outcomes to ensure a broader scope of understanding.

The Quality and Readability of Online Patient Information on Positional Head Shape Conditions.

Families increasingly use online resources to acquire medical information about their child's condition with little understanding of the legitimacy of the source of information or of the information itself. We evaluate the quality and readability of online information related to positional head shape conditions and identify unmet needs for healthcare providers to improve online patient education. The search terms "flat head baby," "brachycephaly," and "plagiocephaly" were queried on the Google search engine and the first 20 websites for each were reviewed. Included websites were evaluated for quality using the DISCERN Instrument and readability using the Flesch-Kincaid Reading Grade Level (FKGL) and Flesch Reading Ease Score (FRES). Websites were categorized by upload source and results were compared using one-way ANOVA. 38 websites met inclusion criteria. There was no significant correlation between DISCERN score and Google search rank between the three search terms. Professional organizations provided websites with the highest mean DISCERN score (56.3) and commercial websites with the lowest score (36.6, P = .003), indicating "good" and "poor" quality content, respectively. Readability assessments showed an overall average FKGL of 9.9 and FRES of 54.4, suggesting "fairly difficult". Hospitals provided the most website results and tended to publish lower quality information, yet are the most readable. High quality websites written at an appropriate reading level for the general public are lacking. A review of online resources for positional head shape conditions can be used to derive recommendations to improve the content of online patient education for pediatric healthcare.