Flesch-Kincaid Grade Level Scores Research Articles

Evaluation of Online AI-Generated Foot and Ankle Surgery Information

As a natural progression from educational pamphlets to the worldwide web, and now artificial intelligence (AI), OpenAI chatbots provide a simple way of obtaining pathology-specific patient information, however, little is known concerning the readability and quality of foot and ankle surgery information. This investigation compares such information using the commercially available OpenAI ChatGPT Chatbot and FootCareMD®. A list of common foot and ankle pathologies from FootCareMD® were queried and compared with similar results using ChatGPT. From both resources, the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL) scores were calculated for each condition. Qualitative analysis of each query was performed using the JAMA Benchmark Criteria Score and the DISCERN Score.The overall ChatGPT and FootCareMD® FRES scores were 31.12±7.86 and 55.18±7.27, respectively (p<0.0001). The overall ChatGPT and FootCareMD® FKGL scores were 13.79±1.22 and 9.60±1.24 respectively (p<0.0001), except for the pilon fracture FKGL scores (p=0.09). The average JAMA Benchmark for all information obtained through ChatGPT and FootCareMD® were 0±0 and 1.95±0.15 (p < 0.001), respectively. The DISCERN Score for all information obtained through ChatGPT and FootCareMD® were 52.53±5.39 and 66.93±4.57 (p < 0.001), respectively. AI-assisted queries concerning common foot and ankle pathologies are written at a higher grade level and with less reliability and accuracy compared to similar information available on FootCareMD®. With the ease of use and increase in AI technology, consideration should be given to the nature and quality of information being shared with respect to the diagnosis and treatment of foot and ankle conditions. Level of EvidenceIV

Quality of Popular Online Resources About Vestibular Migraine.

To evaluate the readability, understandability, actionability, and accuracy of online resources covering vestibular migraine (VM). Cross-sectional descriptive study design. Digital collection of websites appearing on Google search. Google searches were conducted to identify common online resources for VM. We examined readability using the Flesch Reading Ease(FRE) and Flesch-Kincaid Grade Level scores, understandability and actionability using the Patient Education Materials Assessment Tool (PEMAT), and accuracy by comparing the website contents to the consensus definition of "probable vestibular migraine." Eleven of the most popular websites were analyzed. Flesch-Kincaid Grade Level averaged at a 13th-gradelevel (range: 9th-18th). FRE scores averaged 35.5 (range: 9.1-54.4). No website had a readability grade level at the USAgency for Healthcare Research and Quality recommended 5th-grade level or an equivalent FRE score of 90 or greater. Understandability scores varied ranging from 49% to 88% (mean 70%). Actionability scores varied more, ranging from 12% to 87% (mean 44%). There was substantial inter-rater agreement for both PEMAT understandability scoring (mean κ = 0.76, SD = 0.08) and actionability scoring (mean κ = 0.65, SD = 0.08). Three sites included all 3 "probable vestibular migraine" diagnostic criteria as worded in the consensus statement. The quality of online resources for VM is poor overall in terms of readability, actionability, and agreement with diagnostic criteria.

ChatGPT Responses to Frequently Asked Questions on Ménière's Disease: A Comparison to Clinical Practice Guideline Answers.

Evaluate the quality of responses from Chat Generative Pre-Trained Transformer (ChatGPT) models compared to the answers for "Frequently Asked Questions" (FAQs) from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS) Clinical Practice Guidelines (CPG) for Ménière's disease (MD). Comparative analysis. The AAO-HNS CPG for MD includes FAQs that clinicians can give to patients for MD-related questions. The ability of ChatGPT to properly educate patients regarding MD is unknown. ChatGPT-3.5 and 4.0 were each prompted with 16 questions from the MD FAQs. Each response was rated in terms of (1) comprehensiveness, (2) extensiveness, (3) presence of misleading information, and (4) quality of resources. Readability was assessed using Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease Score (FRES). ChatGPT-3.5 was comprehensive in 5 responses whereas ChatGPT-4.0 was comprehensive in 9 (31.3% vs 56.3%, P = .2852). ChatGPT-3.5 and 4.0 were extensive in all responses (P = 1.0000). ChatGPT-3.5 was misleading in 5 responses whereas ChatGPT-4.0 was misleading in 3 (31.3% vs 18.75%, P = .6851). ChatGPT-3.5 had quality resources in 10 responses whereas ChatGPT-4.0 had quality resources in 16 (62.5% vs 100%, P = .0177). AAO-HNS CPG FRES (62.4 ± 16.6) demonstrated an appropriate readability score of at least 60, while both ChatGPT-3.5 (39.1 ± 7.3) and 4.0 (42.8 ± 8.5) failed to meet this standard. All platforms had FKGL means that exceeded the recommended level of 6 or lower. While ChatGPT-4.0 had significantly better resource reporting, both models have room for improvement in being more comprehensive, more readable, and less misleading for patients.

What Questions Do Patients Ask About Biologic Therapies for Chronic Sinusitis With Nasal Polyps?

Objective: The objective of this study is to explore the questions commonly asked online about biologic therapies for use in treatment of chronic sinusitis with nasal polyps (CRSwNP) and the quality of the available content. Methods: Most common search terms were identified via Google Trends. People Also Ask (PAA) questions were identified and extracted with their associated website using an online data-scraping program [Search Engine Optimization (SEO) Minion, Keywords Everywhere]. Sources were evaluated using Flesch Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE, higher number = better) score for readability; Journal of American Medical Association (JAMA) Benchmark criteria for quality assessment (0-4, 4 = all criteria met). Results: A total of 143 unique PAA questions and unique websites were identified. Questions were organized into 3 categories: questions about biologic therapies overall (38.46%), about CRSwNP (37.76%), and about treatment options for CRSwNP (23.78%). Websites answering PAA questions were from commercial (60.14%), medical practice (13.99%), academic (13.29%), and government (12.59%) sources. FKGL scores found the average reading level to be at approximately a 12th grade level (SD = 3.297) alongside a low reading ease FRE score of 37.6 (SD = 16.77). Mean JAMA criteria scores were 0.9895 (SD = 0.848), indicating largely low-quality materials. Conclusion: Biologic therapies are a novel treatment option for CRSwNP, and participants are seeking more information about these treatments and disease state. Online resources regarding biologics should be presented at a lower reading level. Sources with evidence-based information are needed. Physicians should be aware of these limitations in online material and counsel accordingly by curating and directing patients to good sources.Level of Evidence: Step 4.

Evaluation of information accuracy and clarity: ChatGPT responses to the most frequently asked questions about premature ejaculation.

Premature ejaculation (PE) is the most prevalent sexual dysfunction in men, and like many diseases and conditions, patients use Internet sources like ChatGPT, which is a popular artificial intelligence-based language model, for queries about this andrological disorder. The objective of this research was to evaluate the quality, readability, and understanding of texts produced by ChatGPT in response to frequently requested inquiries on PE. In this study we used Google Trends to identify the most frequently searched phrases related to PE. Subsequently, the discovered keywords were methodically entered into ChatGPT, and the resulting replies were assessed for quality using the Ensuring Quality Information for Patients (EQIP) program. The produced texts were assessed for readability using the Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES), and DISCERN metrics. This investigation has identified substantial concerns about the quality of texts produced by ChatGPT, highlighting severe problems with reading and understanding. The mean EQIP score for the texts was determined to be 45.93 ± 4.34, while the FRES was 15.8 ± 8.73. Additionally, the FKGL score was computed to be 15.68 ± 1.67 and the DISCERN score was 38.1 ± 3.78. The comparatively low average EQIP and DISCERN scores suggest that improvements are required to increase the quality and dependability of the presented information. In addition, the FKGL scores indicate a significant degree of linguistic intricacy, requiring a level of knowledge comparable to about 14 to 15years of formal schooling in order to understand. The texts about treatment, which are the most frequently searched items, are more difficult to understand compared to other texts about other categories. The results of this research suggest that compared to texts on other topics the PE texts produced by ChatGPT exhibit a higher degree of complexity, which exceeds the recommended reading threshold for effective health communication. Currently, ChatGPT is cannot be considered a substitute for comprehensive medical consultations. This study is to our knowledge the first reported research investigating the quality and comprehensibility of information generated by ChatGPT in relation to frequently requested queries about PE. The main limitation is that the investigation included only the first 25 popular keywords in English. ChatGPT is incapable of replacing the need for thorough medical consultations.

Assessing the missions and visions of NCI-designated cancer centers and their affiliated hospitals.

e23118 Background: Mission and vision statements (MVS) help convey an institution's identity and priorities. Missions describe objectives and responsibilities, while visions describe overarching directions for the future. Effectively communicating MVS is particularly important for cancer centers and hospitals seeking to inspire and reach patients and employees. We sought to assess the composition, readability, and topics addressed for MVS provided by NCI-designated cancer centers and their affiliated hospitals. Methods: We extracted MVS data from institutional websites for 65 NCI-designated cancer centers and 59 affiliated hospitals in 2023 (6 cancer centers did not have an affiliated hospital). We determined the composition of statements using word counts and time to read. We assessed the readability of statements using Flesch-Kincaid (FK) reading ease and grade level scores. We reviewed the MVS to determine the presence of four topics: equity, quality care, training, and research. We used descriptive statistics to compare these issues in MVS for cancer center versus the hospital. Results: Among the 65 cancer centers, we found mission statements for 93.9% (61/65) and vision statements for 63.1% (41/65). All affiliated hospitals provided a mission statement and 86.4% (51/59) had a vision statement. Data for composition, readability, and topics addressed for MVS are provided in the table. Overall, readability of MVS was difficult based on FK reading ease scores, with relatively high reading grade levels. Mission statements for cancer centers had a significantly lower reading ease and higher grade level compared with hospitals. In general, mission statements were slightly longer than vision statements, more frequently addressing the topics of quality care, training, and research. Compared with hospitals, cancer centers had a significantly lower frequency of mentioning the topic "training" in their MVS. Conclusions: We found that the majority of NCI-designated cancer centers report mission statements, but fewer provide vision statements. We demonstrated difficult readability for the MVS provided, highlighting a need to consider simpler diction and structure of language. Additionally, our work provides an overview of topics addressed for these statements including equity, quality care, training, and research. These topics underscore the values and priorities of an institution, and this review of MVS provides insights into the messaging that cancer centers communicate. [Table: see text]

Evaluating the accuracy and readability of ChatGPT in providing parental guidance for adenoidectomy, tonsillectomy, and ventilation tube insertion surgery

ObjectivesThis study examined the potential of ChatGPT as an accurate and readable source of information for parents seeking guidance on adenoidectomy, tonsillectomy, and ventilation tube insertion surgeries (ATVtis). MethodsChatGPT was tasked with identifying the top 15 most frequently asked questions by parents on internet search engines for each of the three specific surgical procedures. We removed repeated questions from the initial set of 45. Subsequently, we asked ChatGPT to generate answers to the remaining 33 questions. Seven highly experienced otolaryngologists individually assessed the accuracy of the responses using a four-level grading scale, from completely incorrect to comprehensive. The readability of responses was determined using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) scores. The questions were categorized into four groups: Diagnosis and Preparation Process, Surgical Information, Risks and Complications, and Postoperative Process. Responses were then compared based on accuracy grade, FRE, and FKGL scores. ResultsSeven evaluators each assessed 33 AI-generated responses, providing a total of 231 evaluations. Among the evaluated responses, 167 (72.3 %) were classified as ‘comprehensive.’ Sixty-two responses (26.8 %) were categorized as ‘correct but inadequate,’ and two responses (0.9 %) were assessed as ‘some correct, some incorrect.’ None of the responses were adjudged ‘completely incorrect’ by any assessors. The average FRE and FGKL scores were 57.15(±10.73) and 9.95(±1.91), respectively. Upon analyzing the responses from ChatGPT, 3 (9.1 %) were at or below the sixth-grade reading level recommended by the American Medical Association (AMA). No significant differences were found between the groups regarding readability and accuracy scores (p > 0.05). ConclusionsChatGPT can provide accurate answers to questions on various topics related to ATVtis. However, ChatGPT's answers may be too complex for some readers, as they are generally written at a high school level. This is above the sixth-grade reading level recommended for patient information by the AMA. According to our study, more than three-quarters of the AI-generated responses were at or above the 10th-grade reading level, raising concerns about the ChatGPT text's readability.

Evaluation of Online Shoulder Instability-Related Patient Education Materials

Background: Younger patients are more likely than older patients to experience shoulder instability and to rely on online educational resources. Although the Internet has increased patient access to medical information, this may not translate to increased health literacy. Purpose: We sought to analyze the quality and readability of online information on shoulder instability. Methods: We conducted a Google search using 6 terms related to shoulder instability. We collected the first 20 non-sponsored results for each term. Readability was evaluated using the Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), and Gunning Fox Index (GFI) instruments. Quality was assessed using a Quality Grading Sheet (QGS) and the validated DISCERN instrument. Results: A total of 64 of 120 patient educational materials (PEMs) met the inclusion criteria. The mean FKGL, FRE, and GFI scores were 9.45 ± 0.552, 50.51 ± 3.4, and 11.5 ± 0.6, respectively. The mean DISCERN score and QGS rating were 33.09 ± 2.02 and 10.52 ± 1.28, respectively. While 49 (76.6%) articles discussed operative treatment for persistent shoulder instability, only 4 (6.3%) mentioned risks associated with surgery. Non-institutional sources had higher DISCERN scores than those from medical institutions. Conclusions: This review of online shoulder instability-related PEMs suggests that many do not meet current recommendations, with an average quality rating of “poor” and a mean ninth-grade reading level. Surgeons should be aware of the relative paucity of information on the risks and outcomes associated with operative treatment of shoulder instability contained in these PEMs.

Assessment of the Arabic patient-centered online information about orthodontic pain: A quality and readability assessment.

This study assesses the quality and readability of Arabic online information about orthodontic pain. With the increasing reliance on the internet for health information, especially among Arabic speakers, it's critical to ensure the accuracy and comprehensiveness of available content. Our methodology involved a systematic search using the Arabic term for (Orthodontic Pain) in Google, Bing, and Yahoo. This search yielded 193,856 results, from which 74 websites were selected based on predefined criteria, excluding duplicates, scientific papers, and non-Arabic content. For quality assessment, we used the DISCERN instrument, the Journal of the American Medical Association (JAMA) benchmarks, and the Health on the Net (HON) code. Readability was evaluated using the Simplified Measure of Gobbledygook (SMOG), Flesch Reading Ease Score (FRES), and Flesch-Kincaid Grade Level (FKGL) scores. Results indicated that none of the websites received the HONcode seal. The DISCERN assessment showed median total scores of 14.96 (± 5.65), with low overall quality ratings. In JAMA benchmarks, currency was the most achieved aspect, observed in 45 websites (60.81%), but none met all four criteria simultaneously. Readability scores suggested that the content was generally understandable, with a median FKGL score of 6.98 and a median SMOG score of 3.98, indicating middle school-level readability. This study reveals a significant gap in the quality of Arabic online resources on orthodontic pain, highlighting the need for improved standards and reliability. Most websites failed to meet established quality criteria, underscoring the necessity for more accurate and trustworthy health information for Arabic-speaking patients.

Evaluation of the quality of Internet information on masseter botox injection for bruxism.

Botulinum toxin, commonly referred to as botox, is frequently used in facial aesthetics, but also has therapeutic applications in various medical and surgical conditions such as bruxism treatment. The aim of this research is to evaluate the information quality of only English resources available on the Internet regarding the use of botulinum toxin in the treatment of bruxism. The terms "Botox + Tooth Grinding" selected for this study were searched on the Internet using the Google Search Engine, and the URLs of the top 150 websites containing only English pages were recorded and examined. The websites were assessed using DISCERN tools, the Journal of the American Medical Association (JAMA) benchmark, and HONcode tools. The website readability evaluation involved utilizing the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level (FKGL). A total of 130 websites met the inclusion criteria. The total mean DISCERN score (section 1 + section 2) for all websites was 39.75/75 (range: 15-67). The average FRES of all websites was 52.79, and the average FKGL score of all websites was 10.72. Internet blog websites had significantly higher DISCERN (Section1) scores than other websites (p= 0.007). Only one website had the HON code. Internet blog websites showed better compliance with JAMA (authorship and currency) benchmarks. No statistically significant difference was determined related to FRES and FKGL scores between groups (p> 0.05). There is moderate and difficult-to-read information on the Internet about masseter botox injection for bruxism. Relevant guidelines are needed to improve the quality of websites about botulinum toxin for the treatment of bruxism.

Readability of Neurosurgical Patient Education Resources by the American Association of Neurological Surgeons

The importance of patient education has become increasingly apparent in recent years. A prominent patient education tool in neurosurgery is the Neurosurgical Conditions and Treatments page provided by the American Association of Neurological Surgeons (AANS). This study aimed to investigate the readability of this resource page as many new articles have been incorporated in the past decade. Flesch-Kincaid Grade Level and Flesch-Kincaid Reading Ease scores were calculated for each article. One-way analysis of variance and Scheffé, Tukey-Kramer, Bonferroni, Fisher least significant difference, and Dunnett test post hoc analyses were conducted to determine differences between each category with respect to their mean grade level and reading ease scores. Overall mean (SD) Flesch-Kincaid Reading Ease score was 40.2 (12.24), and overall mean (SD) Flesch-Kincaid Grade Level score was 7.48 (1.26). Significant differences were found between mean reading ease scores between each categorization by the AANS (P= 0.014). No significant differences were found between mean grade level score for each categorization (analysis of variance, P= 0.154). As compared to a single previous investigation conducted 10 years ago, the readability of articles has changed modestly, and the reading grade level remains well above the recommendations by the American Medical Association and National Institutes of Health. The 6 new articles introduced in the past decade have demonstrated similar readability, presenting a persistent challenge in the realm of patient education in neurosurgery.

Analysis of the Most Popular Online Ankle Fracture-Related Patient Education Materials.

Given the increasing accessibility of Internet access, it is critical to ensure that the informational material available online for patient education is both accurate and readable to promote a greater degree of health literacy. This study sought to investigate the quality and readability of the most popular online resources for ankle fractures. After conducting a Google search using 6 terms related to ankle fractures, we collected the first 20 nonsponsored results for each term. Readability was evaluated using the Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), and Gunning Fog Index (GFI) instruments. Quality was evaluated using custom created Ankle Fracture Index (AFI). A total of 46 of 120 articles met the inclusion criteria. The mean FKGL, FRE, and GFI scores were 8.4 ± 0.5, 57.5 ± 3.2, and 10.5 ± 0.5, respectively. The average AFI score was 15.4 ± 1.4, corresponding to an "acceptable" quality rating. Almost 70% of articles (n = 32) were written at or below the recommended eighth-grade reading level. Most articles discussed the need for imaging in diagnosis and treatment planning while neglecting to discuss the risks of surgery or potential future operations. We found that online patient-facing materials on ankle fractures demonstrated an eighth-grade average reading grade level and an acceptable quality on content analysis. Further work should surround increasing information regarding risk factors, complications for surgery, and long-term recovery while ensuring that readability levels remain below at least the eighth-grade level.

Characteristics and Quality of Online Searches for Direct Anterior Versus Posterior Approach for Total Hip Arthroplasty

BackgroundOnline resources are important for patient self-education and reflect public interest. We described commonly asked questions regarding the direct anterior versus posterior approach (DAA, PA) to total hip arthroplasty (THA) and the quality of associated websites. MethodsWe extracted the top 200 questions and websites in Google’s “People Also Ask” section for 8 queries on January 8, 2023, and grouped websites and questions into DAA, PA, or comparison. Questions were categorized using Rothwell’s classification (fact, policy, value) and THA-relevant subtopics. Websites were evaluated by information source, Journal of the American Medical Association Benchmark Criteria (credibility), DISCERN survey (information quality), and readability. ResultsWe included 429 question/website combinations (questions: 52.2% DAA, 21.2% PA, 26.6% comparison; websites: 39.0% DAA, 11.0% PA, 9.6% comparison). Per Rothwell’s classification, 56.2% of questions were fact, 31.7% value, 10.0% policy, and 2.1% unrelated. The THA-specific question subtopics differed between DAA and PA (P < .001), specifically for recovery timeline (DAA 20.5%, PA 37.4%), indications/management (DAA 13.4%, PA 1.1%), and technical details (DAA 13.8%, PA 5.5%). Information sources differed between DAA (61.7% medical practice/surgeon) and PA websites (44.7% government; P < .001). The median Journal of the American Medical Association Benchmark score was 1 (limited credibility, interquartile range 1 to 2), with the lowest scores for DAA websites (P < .001). The median DISCERN score was 55 (“good” quality, interquartile range 43 to 65), with the highest scores for comparison websites (P < .001). Median Flesch-Kincaid Grade Level scores were 12th grade level for both DAA and PA (P = .94). ConclusionsPatients’ informational interests can guide counseling. Internet searches that explicitly compare THA approaches yielded websites that provide higher-quality information. Providers may also advise patients that physician websites and websites only describing the DAA may have less balanced perspectives, and limited information regarding surgical approaches is available from social media resources.

The Emerging Role of AI in Patient Education: A Comparative Analysis of the Accuracy of Large Language Models for Pelvic Organ Prolapse

Introduction: This study aimed to evaluate the accuracy, completeness, precision, and readability of outputs generated by three large language models (LLMs); these are GPT by OpenAI, BARD by Google, and Bing by Microsoft, in comparison to patient education material on pelvic organ prolapse (POP) provided by the Royal College of Obstetricians and Gynecologists (RCOG). Methods: A total of 15 questions were retrieved from the RCOG website and input into the three LLMs. Two independent reviewers evaluated the outputs for accuracy, completeness, and precision. Readability was assessed using the Simplified Measure of Gobbledygook (SMOG) score and the Flesch-Kincaid Grade Level (FKGL) score. Results: Significant differences were observed in completeness and precision metrics. ChatGPT ranked highest in completeness (66.7%), while Bing led in precision (100%). No significant differences were observed in accuracy across all models. In terms of readability, ChatGPT exhibited higher difficulty than BARD, Bing, and the original RCOG answers. Conclusion: While all models displayed a variable degree of correctness, ChatGPT excelled in completeness, significantly surpassing BARD and Bing. However, Bing led in precision, providing the most relevant and concise answers. Regarding readability, ChatGPT exhibited higher difficulty. We observed that while all LLMs showed varying degrees of correctness in answering RCOG questions on patient information for POP, ChatGPT was the most comprehensive, but its answers were harder to read. Bing, on the other hand, was the most precise. The findings highlight the potential of LLMs in health information dissemination and the need for careful interpretation of their outputs.

Quality, Reliability, and Readability of Online Information on Idiopathic Intracranial Hypertension.

To evaluate the quality & reliability, technical quality, and readability of patient-targeted online information on idiopathic intracranial hypertension. In this cross-sectional study, we searched Google and Bing search engines for the terms "idiopathic intracranial hypertension" and "pseudotumor cerebri." We evaluated the first 50 search outcomes for each search term. Peer-reviewed articles, targeted advertisements, book chapters, videos, personal blogs, websites intended for healthcare professionals, and non-English websites were excluded. Websites were classified as institutional or private. Two independent reviewers assessed each website for 1) quality and reliability using DISCERN, Health on the Net Code of Conduct (HONcode), and JAMA criteria and 2) technical quality based on 11 criteria. Readability was evaluated using 6 measures (Flesch-Kincaid Reading Ease score and grade level, Gunning Fog, Simple Measure of Gobbledygook (Smog) index, Coleman-Liau index, and automated readability index). Sixty-three websites (37 institutional) were included. The mean scores for the DISCERN, HONcode, and JAMA criteria were 3.6 ± 1 (1-4.8; 1: worse, 5: best), 10.3 ± 2.8 (2-16; 0: worse, 16: best), and 2 ± 1.3 (0-4; 0: worse, 4: best), respectively. The mean technical quality score was 0.8 ± 0.1 (0.5-1). The mean Flesch-Kincaid grade level score was 8.9 ± 1.8 (3.3-13.3). For Flesch-Kincaid grade level, 47 (74.6%) websites scored a grade level of 10 or less. For Gunning Fog Score, 35 websites (55.6%) scored from 7.3 to 10.8. For the Smog Index, 46 (73%) websites scored 5.7-8. The mean Coleman Liau Index was 16 ± 2.1 (9.6-22.2). For the automated readability index, 30 (50.7%) websites scored less than the eighth grade. No statistically significant difference was present between institutional and private websites except for JAMA, which scored higher in private websites (2.4 vs 1.7, P = 0.03). Our findings suggest that online information on idiopathic intracranial hypertension/pseudotumor cerebri for patients generally demonstrates moderate to good quality and reliability. Although most websites maintain credible technical quality, the readability exceeds recommended levels for the average American reader. This may impede understanding, emphasizing the need for future improvements.

Artificial intelligence insights into osteoporosis: assessing ChatGPT's information quality and readability.

The use of artificial intelligence(AI) applications as a source of information in the field of health is increasing. Readable and accurate information plays a critical role in empowering patients to make decisions about their disease. The aim was to examine the quality and readability of responses provided by ChatGPT, an AI chatbot, to commonly asked questions regarding osteoporosis, representing a major public health problem. "Osteoporosis," "female osteoporosis," and "male osteoporosis" were identified by using Google trends for the 25 most frequently searched keywords on Google. A selected set of 38 keywords was sequentially inputted into the chat interface of the ChatGPT. The responses were evaluated with tools of the Ensuring Quality Information for Patients (EQIP), the Flesch-Kincaid Grade Level (FKGL), and the Flesch-Kincaid Reading Ease (FKRE). The EQIP score of the texts ranged from a minimum of 36.36 to a maximum of 61.76 with a mean value of 48.71 as having "serious problems with quality." The FKRE scores spanned from 13.71 to 56.06 with a mean value of 28.71 and the FKGL varied between 8.48 and 17.63, with a mean value of 13.25. There were no statistically significant correlations between the EQIP score and the FKGL or FKRE scores. Although ChatGPT is easily accessible for patients to obtain information about osteoporosis, its current quality and readability fall short of meeting comprehensive healthcare standards.

Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis.

Generative artificial intelligence (AI) chatbots have recently been posited as potential sources of online medical information for patients making medical decisions. Existing online patient-oriented medical information has repeatedly been shown to be of variable quality and difficult readability. Therefore, we sought to evaluate the content and quality of AI-generated medical information on acute appendicitis. A modified DISCERN assessment tool, comprising 16 distinct criteria each scored on a 5-point Likert scale (score range 16-80), was used to assess AI-generated content. Readability was determined using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) scores. Four popular chatbots, ChatGPT-3.5 and ChatGPT-4, Bard, and Claude-2, were prompted to generate medical information about appendicitis. Three investigators independently scored the generated texts blinded to the identity of the AI platforms. ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 had overall mean (SD) quality scores of 60.7 (1.2), 62.0 (1.0), 62.3 (1.2), and 51.3 (2.3), respectively, on a scale of 16-80. Inter-rater reliability was 0.81, 0.75, 0.81, and 0.72, respectively, indicating substantial agreement. Claude-2 demonstrated a significantly lower mean quality score compared to ChatGPT-4 (p = 0.001), ChatGPT-3.5 (p = 0.005), and Bard (p = 0.001). Bard was the only AI platform that listed verifiable sources, while Claude-2 provided fabricated sources. All chatbots except for Claude-2 advised readers to consult a physician if experiencing symptoms. Regarding readability, FKGL and FRE scores of ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 were 14.6 and 23.8, 11.9 and 33.9, 8.6 and 52.8, 11.0 and 36.6, respectively, indicating difficulty readability at a college reading skill level. AI-generated medical information on appendicitis scored favorably upon quality assessment, but most either fabricated sources or did not provide any altogether. Additionally, overall readability far exceeded recommended levels for the public. Generative AI platforms demonstrate measured potential for patient education and engagement about appendicitis.

Thyroid Ethanol and Radiofrequency Ablation: A Call for Patient-Friendly Online Resources

Introduction: The multiple treatment options available to patients with thyroid nodules can generate uncertainty and confusion. Radiofrequency ablation (RFA) and ethanol ablation (EA) are two alternative modalities to manage thyroid nodules. As patients more frequently utilize online resources to guide their decision-making, the quality of such resources must be evaluated. The goal of this study was to assess the quality of online patient materials relating to RFA and EA compared to standard thyroidectomy. Methods: The terms “thyroidectomy,” “thyroid radiofrequency ablation,” and “thyroid ethanol ablation” were searched on Google. Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and Patient Education Materials Assessment Tool (PEMAT) understandability and actionability were calculated for each website. Statistical analysis was conducted on SPSS Statistics. Google trends were used to determine search interest for each term (May 2016 – May 2021). Results: Of the 77 websites that met our inclusion criteria (30 thyroidectomy sites, 30 RFA sites, and 17 EA sites), the average FRE, FKGL, and SMOG scores of the RFA websites were significantly worse than those of the thyroidectomy websites (p < 0.05). The FKGL and SMOG scores of the EA websites were significantly worse than those of the thyroidectomy websites (p < 0.05). The average understandability and actionability scores for thyroidectomy websites were significantly higher than those of RFA and EA websites (p < 0.05). Conclusion: Ablation websites have lower search interest, readability, validity, understandability, and actionability scores in comparison to traditional thyroidectomy websites. Our findings emphasize the need to consider readability and PEMAT scores when developing online educational resources for ablative alternatives to thyroidectomy to allow for greater patient accessibility.

Assessment of the Readability and Quality of Online Information for Patients and Their Families Regarding Schizophrenia.

To our knowledge, this is the first study that specifically aims to assess the readability and quality of online information about schizophrenia. The analysis is performed on 93 of 195 websites that appeared in an advanced Google search of the term "schizophrenia" performed on a single day. The websites were categorized as commercial, nonprofit, professional, and government. The websites were evaluated using the Health on the Net Foundation (HON) code certificate, DISCERN tool, and Journal of the American Medical Association ( JAMA ) benchmark criteria for quality and the Flesch Reading Ease Formula, Flesch-Kincaid Grade Level (FKGL) Formula, Simple Measure of Gobbledygook, and Gunning Fog indices for readability. A total of 21.5% of all websites had a HON code certificate, 50.5% were accepted as high quality ( JAMA score ≥3), and 25.8% reached the recommended readability level (FKGL ≤8). Only three websites scored at a fifth- to sixth-grade reading level. Commercial and government websites had significantly lower DISCERN scores. Commercial websites had significantly lower Flesch Reading Ease Score and FKGL score than nonprofit websites. In conclusion, the current findings indicate that the quality of online information on schizophrenia is generally acceptable, but the readability is insufficient. Website creators, physicians, and health authorities should be more sensitive to the readability of online information about schizophrenia, considering the poor cognitive capacity of the patients and the unique nature of the disease.

Evaluating insomnia queries from an artificial intelligence chatbot for patient education.

We evaluated the accuracy of ChatGPT in addressing insomnia-related queries for patient education and assessed ChatGPT's ability to provide varied responses based on differing prompting scenarios. Four identical sets of 20 insomnia-related queries were posed to ChatGPT. Each set differed by the context in which ChatGPT was prompted: no prompt, patient-centered, physician-centered, and with references and statistics. Responses were reviewed by 2 academic sleep surgeons, 1 academic sleep medicine physician, and 2 sleep medicine fellows across 4 domains: clinical accuracy, prompt adherence, referencing, and statistical precision, using a binary grading system. Flesch-Kincaid grade-level scores were calculated to estimate the grade level of the responses, with statistical differences between prompts analyzed via analysis of variance and Tukey's test. Interrater reliability was calculated using Fleiss's kappa. The study revealed significant variations in the Flesch-Kincaid grade-level scores across 4 prompts: unprompted (13.2 ± 2.2), patient-centered (8.1 ± 1.9), physician-centered (15.4 ± 2.8), and with references and statistics (17.3 ± 2.3, P < .001). Despite poor Fleiss kappa scores, indicating low interrater reliability for clinical accuracy and relevance, all evaluators agreed that the majority of ChatGPT's responses were clinically accurate, with the highest variability on Form 4. The responses were also uniformly relevant to the given prompts (100% agreement). Eighty percent of the references ChatGPT cited were verified as both real and relevant, and only 25% of cited statistics were corroborated within referenced articles. ChatGPT can be used to generate clinically accurate responses to insomnia-related inquiries. Alapati R, Campbell D, Molin N, etal. Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J Clin Sleep Med. 2024;20(4):583-594.