Flesch-Kincaid Research Articles

Pillar 3: The impact of language complexity on the preferences of commercial bank website users

This research explores the impact of document complexity and readability on user preferences for disclosure information on commercial bank web portals, with a focus on Pillar 3 disclosures. The study investigates the usage patterns of disclosed information by stakeholders. Through an analysis of web portal access variables and document complexity and readability measures, the study identifies correlations between text complexity/readability and user preferences. The experiment used traditional readability measures like the Gunning Fog Index and Flesch Reading Ease but also focused on less-used metrics such as LIX, various text characteristics, and part-of-speech metrics. The findings reveal that while stakeholders show interest in financial documents despite their complexity, preferences in using these documents vary based on document type and readability. Notably, documents with higher complexity tend to attract more attention, suggesting potential challenges in accessibility and comprehension. The paper highlights the importance of presenting disclosure information in a manner that enhances readability and accessibility to better engage stakeholders. Additionally, the research introduces user preference indicators aligned with web portal access metrics, providing insights into stakeholders’ behavior. The research followed up on previous findings of extremely low interest in Pillar 3 information from commercial bank stakeholders. The results showed that stakeholders are more interested in less demanding and more readable texts such as annual reports, and as Pillar 3 documents are more complex and harder to read, they are less interested in them. The research concludes that stakeholders’ increasing interest in this information requires finding approaches to presenting it in less complicated and more readable ways.

Still Using Only ChatGPT? The Comparison of Five Different Artificial Intelligence Chatbots' Answers to the Most Common Questions About Kidney Stones.

Objective: To evaluate and compare the quality and comprehensibility of answers produced by five distinct artificial intelligence (AI) chatbots-GPT-4, Claude, Mistral, Google PaLM, and Grok-in response to the most frequently searched questions about kidney stones (KS). Materials and Methods: Google Trends facilitated the identification of pertinent terms related to KS. Each AI chatbot was provided with a unique sequence of 25 commonly searched phrases as input. The responses were assessed using DISCERN, the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), the Flesch-Kincaid Grade Level (FKGL), and the Flesch-Kincaid Reading Ease (FKRE) criteria. Results: The three most frequently searched terms were "stone in kidney," "kidney stone pain," and "kidney pain." Nepal, India, and Trinidad and Tobago were the countries that performed the most searches in KS. None of the AI chatbots attained the requisite level of comprehensibility. Grok demonstrated the highest FKRE (55.6 ± 7.1) and lowest FKGL (10.0 ± 1.1) ratings (p = 0.001), whereas Claude outperformed the other chatbots in its DISCERN scores (47.6 ± 1.2) (p = 0.001). PEMAT-P understandability was the lowest in GPT-4 (53.2 ± 2.0), and actionability was the highest in Claude (61.8 ± 3.5) (p = 0.001). Conclusion: GPT-4 had the most complex language structure of the five chatbots, making it the most difficult to read and comprehend, whereas Grok was the simplest. Claude had the best KS text quality. Chatbot technology can improve healthcare material and make it easier to grasp.

Online Patient Education in Obstructive Sleep Apnea: ChatGPT versus Google Search.

The widespread implementation of artificial intelligence technologies provides an appealing alternative to traditional search engines for online patient healthcare education. This study assessed ChatGPT-3.5's capabilities as a source of obstructive sleep apnea (OSA) information, using Google Search as a comparison. Ten frequently searched questions related to OSA were entered into Google Search and ChatGPT-3.5. The responses were assessed by two independent researchers using the Global Quality Score (GQS), Patient Education Materials Assessment Tool (PEMAT), DISCERN instrument, CLEAR tool, and readability scores (Flesch Reading Ease and Flesch-Kincaid Grade Level). ChatGPT-3.5 significantly outperformed Google Search in terms of GQS (5.00 vs. 2.50, p < 0.0001), DISCERN reliability (35.00 vs. 29.50, p = 0.001), and quality (11.50 vs. 7.00, p = 0.02). The CLEAR tool scores indicated that ChatGPT-3.5 provided excellent content (25.00 vs. 15.50, p < 0.001). PEMAT scores showed higher understandability (60-91% vs. 44-80%) and actionability for ChatGPT-3.5 (0-40% vs. 0%). Readability analysis revealed that Google Search responses were easier to read (FRE: 56.05 vs. 22.00; FKGL: 9.00 vs. 14.00, p < 0.0001). ChatGPT-3.5 delivers higher quality and more comprehensive OSA information compared to Google Search, although its responses are less readable. This suggests that while ChatGPT-3.5 can be a valuable tool for patient education, efforts to improve readability are necessary to ensure accessibility and utility for all patients. Healthcare providers should be aware of the strengths and weaknesses of various healthcare information resources and emphasize the importance of critically evaluating online health information, advising patients on its reliability and relevance.

Evaluation of the quality of Internet information on masseter botox injection for bruxism.

Botulinum toxin, commonly referred to as botox, is frequently used in facial aesthetics, but also has therapeutic applications in various medical and surgical conditions such as bruxism treatment. The aim of this research is to evaluate the information quality of only English resources available on the Internet regarding the use of botulinum toxin in the treatment of bruxism. The terms "Botox + Tooth Grinding" selected for this study were searched on the Internet using the Google Search Engine, and the URLs of the top 150 websites containing only English pages were recorded and examined. The websites were assessed using DISCERN tools, the Journal of the American Medical Association (JAMA) benchmark, and HONcode tools. The website readability evaluation involved utilizing the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level (FKGL). A total of 130 websites met the inclusion criteria. The total mean DISCERN score (section 1 + section 2) for all websites was 39.75/75 (range: 15-67). The average FRES of all websites was 52.79, and the average FKGL score of all websites was 10.72. Internet blog websites had significantly higher DISCERN (Section1) scores than other websites (p= 0.007). Only one website had the HON code. Internet blog websites showed better compliance with JAMA (authorship and currency) benchmarks. No statistically significant difference was determined related to FRES and FKGL scores between groups (p> 0.05). There is moderate and difficult-to-read information on the Internet about masseter botox injection for bruxism. Relevant guidelines are needed to improve the quality of websites about botulinum toxin for the treatment of bruxism.

Quality and Readability Analysis of Turbinoplasty on the Web.

This study aimed to assess the quality, credibility, and readability of online health information concerning turbinoplasty, given the increasing reliance on internet resources for health education. Using four search terms related to turbinoplasty, we analyzed 71 text-based webpages from Google.com, Bing.com, and Yahoo.com. Readability was evaluated using the Flesch-Kincaid grade level, Gunning-Fog Index, SMOG Index, and Coleman-Liau Index were utilized. Web page quality was computed through the DISCERN Instrument (DISCERN), the Journal of the American Medical Association benchmark criteria (JAMA), Novel Turbinoplasty Index (NTI), and presence of code certification by Health on The Net (HON). Seventy-one text-based web pages were assessed. Information quality was measured by an average DISCERN score of 47.4±7.40, indicating "fair" quality. The average readability was a grade level of 9.7±1.57, notably higher than AMA and NIH recommendations. Of all web pages, only 11 (15.49%) proficiently met all 4 listed JAMA criteria. Significant correlations between web page classification and average DISCERN (P=0.0042), as well as JAMA score (P<0.001) were discovered. The web pages that had HON code certification revealed significantly higher quality metrics such as DISCERN scores (P<0.001), JAMA scores (P<0.001), and NTI scores (P=0.038). Online health information for turbinoplasty is of "fair" quality, and the average readability is several grade levels above current AMA and NIH recommendations. Health care providers should aim to guide their patients on finding appropriate educational resources and should improve the readability of their patient education materials.

Evaluation of the accuracy and readability of ChatGPT-4 and Google Gemini in providing information on retinal detachment: a multicenter expert comparative study

BackgroundLarge language models (LLMs) such as ChatGPT-4 and Google Gemini show potential for patient health education, but concerns about their accuracy require careful evaluation. This study evaluates the readability and accuracy of ChatGPT-4 and Google Gemini in answering questions about retinal detachment.MethodsComparative study analyzing responses from ChatGPT-4 and Google Gemini to 13 retinal detachment questions, categorized by difficulty levels (D1, D2, D3). Masked responses were reviewed by ten vitreoretinal specialists and rated on correctness, errors, thematic accuracy, coherence, and overall quality grading. Analysis included Flesch Readability Ease Score, word and sentence counts.ResultsBoth Artificial Intelligence tools required college-level understanding for all difficulty levels. Google Gemini was easier to understand (p = 0.03), while ChatGPT-4 provided more correct answers for the more difficult questions (p = 0.0005) with fewer serious errors. ChatGPT-4 scored highest on most challenging questions, showing superior thematic accuracy (p = 0.003). ChatGPT-4 outperformed Google Gemini in 8 of 13 questions, with higher overall quality grades in the easiest (p = 0.03) and hardest levels (p = 0.0002), showing a lower grade as question difficulty increased.ConclusionsChatGPT-4 and Google Gemini effectively address queries about retinal detachment, offering mostly accurate answers with few critical errors, though patients require higher education for comprehension. The implementation of AI tools may contribute to improving medical care by providing accurate and relevant healthcare information quickly.

Evaluación de la fiabilidad y legibilidad de las respuestas de los chatbots como recurso de información al paciente para las exploraciones PET-TC más communes

PurposeThis study aimed to evaluate the reliability and readability of responses generated by two popular AI-chatbots, ‘ChatGPT-4.0’ and ‘Google Gemini’, to potential patient questions about PET/CT scans. Materials and methodsThirty potential questions for each of [18F]FDG and [68Ga]Ga-DOTA-SSTR PET/CT, and twenty-nine potential questions for [68Ga]Ga-PSMA PET/CT were asked separately to ChatGPT-4 and Gemini in May 2024. The responses were evaluated for reliability and readability using the modified DISCERN (mDISCERN) scale, Flesch Reading Ease (FRE), Gunning Fog Index (GFI), and Flesch-Kincaid Reading Grade Level (FKRGL). The inter-rater reliability of mDISCERN scores provided by three raters (ChatGPT-4, Gemini, and a nuclear medicine physician) for the responses was assessed. ResultsThe median [min-max] mDISCERN scores reviewed by the physician for responses about FDG, PSMA and DOTA PET/CT scans were 3.5 [2–4], 3 [3–4], 3 [3–4] for ChatPT-4 and 4 [2–5], 4 [2–5], 3.5 [3–5] for Gemini, respectively. The mDISCERN scores assessed using ChatGPT-4 for answers about FDG, PSMA, and DOTA-SSTR PET/CT scans were 3.5 [3–5], 3 [3–4], 3 [2–3] for ChatGPT-4, and 4 [3–5], 4 [3–5], 4 [3–5] for Gemini, respectively. The mDISCERN scores evaluated using Gemini for responses FDG, PSMA, and DOTA-SSTR PET/CTs were 3 [2–4], 2 [2–4], 3 [2–4] for ChatGPT-4, and 3 [2–5], 3 [1–5], 3 [2–5] for Gemini, respectively. The inter-rater reliability correlation coefficient of mDISCERN scores for ChatGPT-4 responses about FDG, PSMA, and DOTA-SSTR PET/CT scans were 0.629 (95% CI = 0,32−0,812), 0.707 (95% CI = 0.458−0.853) and 0.738 (95% CI = 0.519−0.866), respectively (p < 0.001). The correlation coefficient of mDISCERN scores for Gemini responses about FDG, PSMA, and DOTA-SSTR PET/CT scans were 0.824 (95% CI = 0.677−0.910), 0.881 (95% CI = 0.78−0.94) and 0.847 (95% CI = 0.719−0.922), respectively (p < 0.001). The mDISCERN scores assessed by ChatGPT-4, Gemini, and the physician showed that the chatbots' responses about all PET/CT scans had moderate to good statistical agreement according to the inter-rater reliability correlation coefficient (p < 0,001). There was a statistically significant difference in all readability scores (FKRGL, GFI, and FRE) of ChatGPT-4 and Gemini responses about PET/CT scans (p < 0,001). Gemini responses were shorter and had better readability scores than ChatGPT-4 responses. ConclusionThere was an acceptable level of agreement between raters for the mDISCERN score, indicating agreement with the overall reliability of the responses. However, the information provided by AI-chatbots cannot be easily read by the public.

Internet Health Resources for Diabetic Foot Search: The Content, Quality, and Readability of Information on Web Sites

Background: As the incidence of diabetes mellitus increases, the incidence of diabetic foot also increases. This situation, which may lead to devastating complications and progress to limb loss for patients, exposes patients and their social environments to a big crisis. Thus, patients may seek secondary opinions from online sources about information they initially obtained from health institutions. We aimed to evaluate the information content related to diabetic foot on the Internet that is probably used by patients for Internet searching. Methods: After software optimization and reset, related queries with the keyword diabetic foot were determined on Google Trends. Selected keywords were searched in three search engines, and the results were examined. Web sites were classified into five subcategories (nongovernmental health institution, governmental institution, academic, blog, and university) and evaluated with an information content scale (ICS) based on the literature, Journal of the American Medical Association benchmark criteria, the Flesch-Kincaid readability test, and presence of the Health On the Net Foundation Code of Conduct certificate. The search engines, keywords, and Web site subcategories were investigated with the evaluation criteria. Results: In terms of finding Web sites eligible for assessment, the Google search engine listed more eligible Web sites than did Bing and Yahoo. Concerning the ICS, there was no significant difference between search engines for total scores (P &gt; .05). Concerning ICS diagnosis and evaluation and ICS total score, academic Web sites scored significantly higher than other subcategories. Conclusions: Results that can be obtained with an Internet search for diabetic foot depend on the proper keyword selection, Web site type, and search engine to help patients reach more appropriate content.

Accessing information provided via artificial intelligence regarding reverse and anatomic total shoulder arthroplasty

BackgroundThe purpose of this study is to analyze the quality, accuracy, and readability of information provided by an artificial intelligence (AI) interface ChatGPT (OpenAI, San Francisco). We searched ChatGPT for commonly asked questions by patients regarding anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA). MethodsChatGPT was used to answer 30 commonly asked questions by patients regarding aTSA and rTSA, inputted as “total shoulder replacement” and “reverse shoulder replacement”. These questions were categorized based on the Rothwell criteria into Fact, Policy, and Value. The answers generated were analyzed for quality, accuracy, and readability using the DISCERN scale, JAMA benchmark criteria, Flesch-Kincaid Reading Ease Score (FRES) and Grade Level (FKGL). ResultsFor both rTSA and aTSA the DISCERN score for Fact questions was 57, Policy questions was 61, and for Value questions was 58 (all were all considered “good”). The JAMA benchmark criteria was 0, representing the lowest score for Fact, Policy, and Value questions for both rTSA and aTSA questions. The FRES score for the aTSA answers for Fact was 15.15, for Policy was 11.14, and for Value questions was 10.95. The FRES score for rTSA questions for Fact is 48.02, Policy is 12.51, and Value is 17.22. The FKGL for aTSA answer for Fact was 17.48, Policy was 17.72 and Value was 17.96. The FKGL for rTSA questions for Fact are 8.10, Policy is 17.27, and Value is 16.56. ConclusionOverall, the quality of answers provided by AI open model, ChatGPT was considered “good.” The information provided had lower reliability, and lack of information regarding funding and disclosures. Most of the information generated by ChatGPT was also found to have the readability of “academic level text”, while Fact related information on reverse shoulder arthroplasty was found to have the readability of 9th grade level, which may be too complex for most patients. Overall, these results indicate that ChatGPT can provide correct answers to questions about aTSA and rTSA, although we would caution patients from utilizing this resource due to the lack of citations and complexity of the output that ChatGPT provides. Importantly, all answers provided by AI suggested reaching out to physicians to get more accurate and personalized advise, to factor into the shared decisions making model.

Resources on lymphedema surgery: How effective are they for patients?

Online patient education materials (PEMs) on lymphedema surgery were assessed for quality, readability, and content. A total of 37 PEMs were identified, primarily authored by academic/medical organizations. Readability scores indicated materials were difficult to read, with an average Flesch-Kincaid Grade Level of 10.4. PEM Assessment Tools showed acceptable understandability (72.3%) but poor actionability (28.5%). PEMs often lacked information on surgical risks, postoperative care, and long-term follow-up. Simplifying language and adding visual aids could improve PEM effectiveness.

Digesting Digital Health: A Study of Appropriateness and Readability of ChatGPT-Generated Gastroenterological Information.

The advent of artificial intelligence-powered large language models capable of generating interactive responses to intricate queries marks a groundbreaking development in how patients access medical information. Our aim was to evaluate the appropriateness and readability of gastroenterological information generated by ChatGPT. We analyzed responses generated by ChatGPT to 16 dialogue-based queries assessing symptoms and treatments for gastrointestinal conditions and 13 definition-based queries on prevalent topics in gastroenterology. Three board-certified gastroenterologists evaluated output appropriateness with a 5-point Likert-scale proxy measurement of currency, relevance, accuracy, comprehensiveness, clarity, and urgency/next steps. Outputs with a score of 4 or 5 in all 6 categories were designated as "appropriate." Output readability was assessed with Flesch Reading Ease score, Flesch-Kinkaid Reading Level, and Simple Measure of Gobbledygook scores. ChatGTP responses to 44% of the 16 dialogue-based and 69% of the 13 definition-based questions were deemed appropriate, and the proportion of appropriate responses within the 2 groups of questions was not significantly different (P = .17). Notably, none of ChatGTP's responses to questions related to gastrointestinal emergencies were designated appropriate. The mean readability scores showed that outputs were written at a college-level reading proficiency. ChatGPT can produce generally fitting responses to gastroenterological medical queries, but responses were constrained in appropriateness and readability, which limits the current utility of this large language model. Substantial development is essential before these models can be unequivocally endorsed as reliable sources of medical information.

Accuracy and Readability of Artificial Intelligence Chatbot Responses to Vasectomy-Related Questions: Public Beware.

Purpose Artificial intelligence (AI) has rapidly gained popularity with the growth of ChatGPT (OpenAI, San Francisco, USA) and other large-language model chatbots, and these programs have tremendous potential to impact medicine. One important area of consequence in medicine and public health is that patients may use these programs in search of answers to medical questions. Despite the increased utilization of AI chatbots by the public, there is little research to assess the reliability of ChatGPT and alternative programs when queried for medical information. This study seeks to elucidate the accuracy and readability of AI chatbots in answering patient questions regarding urology. As vasectomy is one of the most common urologic procedures, this study investigates AI-generated responses to frequently asked vasectomy-related questions. For this study, five popular and free-to-access AI platforms were utilized to undertake this investigation. Methods Fifteen vasectomy-related questions were individually queried to five AI chatbots from November-December 2023: ChatGPT (OpenAI, San Francisco, USA), Bard (Google Inc., Mountainview, USA) Bing (Microsoft, Redmond, USA) Perplexity (Perplexity AI Inc., San Francisco, USA), and Claude (Anthropic, San Francisco, USA). Responses from each platform were graded by two attending urologists, two urology research faculty, and one urological resident physician using a Likert (1-6) scale: (1-completely inaccurate, 6-completely accurate) based on comparison to existing American Urological Association guidelines. Flesch-Kincaid Grade levels (FKGL) and Flesch Reading Ease scores (FRES) (1-100) were calculated for each response. To assess differences in Likert, FRES, and FKGL, Kruskal-Wallis tests were performed using GraphPad Prism V10.1.0 (GraphPad,San Diego,USA) with Alpha set at 0.05. Results Analysis shows that ChatGPT provided the most accurate responses across the five AI chatbots with an average score of 5.04 on the Likert scale. Subsequently, Microsoft Bing (4.91), Anthropic Claude (4.65), Google Bard (4.43), and Perplexity (4.41) followed. All five chatbots were found to score, on average, higher than 4.41 corresponding to a score of at least "somewhat accurate." Google Bard received the highest Flesch Reading Ease score (49.67) and lowest Grade level (10.1) when compared to the other chatbots. Anthropic Claude scored 46.7 on the FRES and 10.55 on the FKGL. Microsoft Bing scored 45.57 on the FRES and 11.56 on the FKGL. Perplexity scored 36.4 on the FRES and 13.29 on the FKGL. ChatGPT had the lowest FRES of 30.4 and highest FKGL of 14.2. Conclusion This study investigates the use of AI in medicine, specifically urology, and it helps to determine whether large-language model chatbots can be reliable sources of freely available medical information. All five AI chatbots on average were able to achieve at least "somewhat accurate" on a 6-point Likert scale. In terms of readability, all five AI chatbots on average had Flesch Reading Ease scores of less than 50 and were higher than a 10th-grade level. In this small-scale study, there were several significant differences identified between the readability scores of each AI chatbot. However, there were no significant differences found among their accuracies. Thus, our study suggests that major AI chatbots may perform similarly in their ability to be correct but differ in their ease of being comprehended by the general public.

Assessment of the Quality and Readability of Web-Based Arabic Health Information on Halitosis: Infodemiological Study.

Halitosis, characterized by an undesirable mouth odor, represents a common concern. This study aims to assess the quality and readability of web-based Arabic health information on halitosis as the internet is becoming a prominent global source of medical information. A total of 300 Arabic websites were retrieved from Google using 3 commonly used phrases for halitosis in Arabic. The quality of the websites was assessed using benchmark criteria established by the Journal of the American Medical Association, the DISCERN tool, and the presence of the Health on the Net Foundation Code of Conduct (HONcode). The assessment of readability (Flesch Reading Ease [FRE], Simple Measure of Gobbledygook, and Flesch-Kincaid Grade Level [FKGL]) was conducted using web-based readability indexes. A total of 127 websites were examined. Regarding quality assessment, 87.4% (n=111) of websites failed to fulfill any Journal of the American Medical Association requirements, highlighting a lack of authorship (authors' contributions), attribution (references), disclosure (sponsorship), and currency (publication date). The DISCERN tool had a mean score of 34.55 (SD 7.46), with the majority (n=72, 56.6%) rated as moderate quality, 43.3% (n=55) as having a low score, and none receiving a high DISCERN score, indicating a general inadequacy in providing quality health information to make decisions and treatment choices. No website had HONcode certification, emphasizing the concern over the credibility and trustworthiness of these resources. Regarding readability assessment, Arabic halitosis websites had high readability scores, with 90.5% (n=115) receiving an FRE score ≥80, 98.4% (n=125) receiving a Simple Measure of Gobbledygook score <7, and 67.7% (n=86) receiving an FKGL score <7. There were significant correlations between the DISCERN scores and the quantity of words (P<.001) and sentences (P<.001) on the websites. Additionally, there was a significant relationship (P<.001) between the number of sentences and FKGL and FRE scores. While readability was found to be very good, indicating that the information is accessible to the public, the quality of Arabic halitosis websites was poor, reflecting a significant gap in providing reliable and comprehensive health information. This highlights the need for improving the availability of high-quality materials to ensure Arabic-speaking populations have access to reliable information about halitosis and its treatment options, tying quality and availability together as critical for effective health communication.

Patient education resources for oral mucositis: a google search and ChatGPT analysis.

Oral mucositis affects 90% of patients receiving chemotherapy or radiation for head and neck malignancies. Many patients use the internet to learn about their condition and treatments; however, the quality of online resources is not guaranteed. Our objective was to determine the most common Google searches related to "oral mucositis" and assess the quality and readability of available resources compared to ChatGPT-generated responses. Data related to Google searches for "oral mucositis" were analyzed. People Also Ask (PAA) questions (generated by Google) related to searches for "oral mucositis" were documented. Google resources were rated on quality, understandability, ease of reading, and reading grade level using the Journal of the American Medical Association benchmark criteria, Patient Education Materials Assessment Tool, Flesch Reading Ease Score, and Flesh-Kincaid Grade Level, respectively. ChatGPT-generated responses to the most popular PAA questions were rated using identical metrics. Google search popularity for "oral mucositis" has significantly increased since 2004. 78% of the Google resources answered the associated PAA question, and 6% met the criteria for universal readability. 100% of the ChatGPT-generated responses answered the prompt, and 20% met the criteria for universal readability when asked to write for the appropriate audience. Most resources provided by Google do not meet the criteria for universal readability. When prompted specifically, ChatGPT-generated responses were consistently more readable than Google resources. After verification of accuracy by healthcare professionals, ChatGPT could be a reasonable alternative to generate universally readable patient education resources.

Improving readability and comprehension levels of otolaryngology patient education materials using ChatGPT

ObjectiveA publicly available large language learning model platform may help determine current readability levels of otolaryngology patient education materials, as well as translate these materials to the recommended 6th-grade and 8th-grade reading levels. Study designCross-sectional analysis. SettingOnline using large language learning model, ChatGPT. MethodsThe Patient Education pages of the American Laryngological Association (ALA) and American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS) websites were accessed. Materials were input into ChatGPT (OpenAI, San Francisco, CA; version 3.5) and Microsoft Word (Microsoft, Redmond, WA; version 16.74). Programs calculated Flesch Reading Ease (FRE) scores, with higher scores indicating easier readability, and Flesch-Kincaid (FK) grade levels, estimating U.S. grade level required to understand text. ChatGPT was prompted to “translate to a 5th-grade reading level” and provide new scores. Scores were compared for statistical differences, as well as differences between ChatGPT and Word gradings. ResultsPatient education materials were reviewed and 37 ALA and 72 AAO-HNS topics were translated. Overall FRE scores and FK grades demonstrated significant improvements following translation of materials, as scored by ChatGPT (p < 0.001). Word also scored significant improvements in FRE and FK following translation by ChatGPT for AAO-HNS materials overall (p < 0.001) but not for individual topics or for subspecialty-specific categories. Compared with Word, ChatGPT significantly exaggerated the change in FRE grades and FK scores (p < 0.001). ConclusionOtolaryngology patient education materials were found to be written at higher reading levels than recommended. Artificial intelligence may prove to be a useful resource to simplify content to make it more accessible to patients.

Performance of large language models (LLMs) in providing prostate cancer information

PurposeThe diagnosis and management of prostate cancer (PCa), the second most common cancer in men worldwide, are highly complex. Hence, patients often seek knowledge through additional resources, including AI chatbots such as ChatGPT and Google Bard. This study aimed to evaluate the performance of LLMs in providing education on PCa.MethodsCommon patient questions about PCa were collected from reliable educational websites and evaluated for accuracy, comprehensiveness, readability, and stability by two independent board-certified urologists, with a third resolving discrepancy. Accuracy was measured on a 3-point scale, comprehensiveness was measured on a 5-point Likert scale, and readability was measured using the Flesch Reading Ease (FRE) score and Flesch–Kincaid FK Grade Level.ResultsA total of 52 questions on general knowledge, diagnosis, treatment, and prevention of PCa were provided to three LLMs. Although there was no significant difference in the overall accuracy of LLMs, ChatGPT-3.5 demonstrated superiority over the other LLMs in terms of general knowledge of PCa (p = 0.018). ChatGPT-4 achieved greater overall comprehensiveness than ChatGPT-3.5 and Bard (p = 0.028). For readability, Bard generated simpler sentences with the highest FRE score (54.7, p < 0.001) and lowest FK reading level (10.2, p < 0.001).ConclusionChatGPT-3.5, ChatGPT-4 and Bard generate accurate, comprehensive, and easily readable PCa material. These AI models might not replace healthcare professionals but can assist in patient education and guidance.

Assessing the Readability, Reliability, and Quality of AI-Modified and Generated Patient Education Materials for Endoscopic Skull Base Surgery.

Despite National Institutes of Health and American Medical Association recommendations to publish online patient education materials at or below sixth-grade literacy, those pertaining to endoscopic skull base surgery (ESBS) have lacked readability and quality. ChatGPT is an artificial intelligence (AI) system capable of synthesizing vast internet data to generate responses to user queries but its utility in improving patient education materials has not been explored. To examine the current state of readability and quality of online patient education materials and determined the utility of ChatGPT for improving articles and generating patient education materials. An article search was performed utilizing 10 different search terms related to ESBS. The ten least readable existing patient-facing articles were modified with ChatGPT and iterative queries were used to generate an article de novo. The Flesch Reading Ease (FRE) and related metrics measured overall readability and content literacy level, while DISCERN assessed article reliability and quality. Sixty-six articles were located. ChatGPT improved FRE readability of the 10 least readable online articles (19.7 ± 4.4 vs. 56.9 ± 5.9, p < 0.001), from university to 10th grade level. The generated article was more readable than 48.5% of articles (38.9 vs. 39.4 ± 12.4) and higher quality than 94% (51.0 vs. 37.6 ± 6.1). 56.7% of the online articles had "poor" quality. ChatGPT improves the readability of articles, though most still remain above the recommended literacy level for patient education materials. With iterative queries, ChatGPT can generate more reliable and higher quality patient education materials compared to most existing online articles and can be tailored to match readability of average online articles.

Development, validation, user testing of patient information leaflet to improve awareness of metabolic side effects in patients on atypical antipsychotics

BackgroundMetabolic adverse effects of second-generation antipsychotics (SGAs) have been well-documented in literature. This study aimed to develop, validate, and evaluate the usefulness of patient information leaflet (PIL) by user testing method to improve metabolic monitoring for psychiatric patients prescribed with SGAs. MethodsThe study included out-patients above 18 years of age diagnosed with schizophrenia, schizoaffective, bipolar disorders prescribed with SGAs. Database such as micromedex, guidelines on developing PIL and factsheets on antipsychotics were used to prepare the content of the leaflet and was given to experts for validation. The PIL covered information on definition of antipsychotic and its types, metabolic syndrome and its prevention such as lifestyle modifications and regular monitoring. Readability of the PIL was assessed using Flesch reading ease (FRE), Flesch Kincaid grade level (FK-GL) and user testing in 20 similar patient population. ResultsThe BALD scores on layout and design of the PILs was found to be 29. The FRE readability test score was found to be 63.5 and FKGL test score was found to be 7.6. Among the 20 patients, 5 of them favoured English PILs, while 15 chose Kannada PILs. The overall pre and post user-testing mean scores for knowledge assessment significantly increased from 60.62 ± 23.39 to 91.25 ± 11.54. Overall, about 85 % of patients gave the PIL leaflets a good rating. ConclusionPatient education materials of high quality that are created with pictures may aid in a better knowledge of the treatment for psychiatric patients who are on long term SGA therapy.

Evaluation of the Quality and Readability of Web-Based Information Regarding Foreign Bodies of the Ear, Nose, and Throat: Qualitative Content Analysis.

Foreign body (FB) inhalation, ingestion, and insertion account for 11% of emergency admissions for ear, nose, and throat conditions. Children are disproportionately affected, and urgent intervention may be needed to maintain airway patency and prevent blood vessel occlusion. High-quality, readable online information could help reduce poor outcomes from FBs. We aim to evaluate the quality and readability of available online health information relating to FBs. In total, 6 search phrases were queried using the Google search engine. For each search term, the first 30 results were captured. Websites in the English language and displaying health information were included. The provider and country of origin were recorded. The modified 36-item Ensuring Quality Information for Patients tool was used to assess information quality. Readability was assessed using a combination of tools: Flesch Reading Ease score, Flesch-Kincaid Grade Level, Gunning-Fog Index, and Simple Measure of Gobbledygook. After the removal of duplicates, 73 websites were assessed, with the majority originating from the United States (n=46, 63%). Overall, the quality of the content was of moderate quality, with a median Ensuring Quality Information for Patients score of 21 (IQR 18-25, maximum 29) out of a maximum possible score of 36. Precautionary measures were not mentioned on 41% (n=30) of websites and 30% (n=22) did not identify disk batteries as a risky FB. Red flags necessitating urgent care were identified on 95% (n=69) of websites, with 89% (n=65) advising patients to seek medical attention and 38% (n=28) advising on safe FB removal. Readability scores (Flesch Reading Ease score=12.4, Flesch-Kincaid Grade Level=6.2, Gunning-Fog Index=6.5, and Simple Measure of Gobbledygook=5.9 years) showed most websites (56%) were below the recommended sixth-grade level. The current quality and readability of information regarding FBs is inadequate. More than half of the websites were above the recommended sixth-grade reading level, and important information regarding high-risk FBs such as disk batteries and magnets was frequently excluded. Strategies should be developed to improve access to high-quality information that informs patients and parents about risks and when to seek medical help. Strategies to promote high-quality websites in search results also have the potential to improve outcomes.

Evaluation of Generative Language Models in Personalizing Medical Information: Instrument Validation Study.

Although uncertainties exist regarding implementation, artificial intelligence-driven generative language models (GLMs) have enormous potential in medicine. Deployment of GLMs could improve patient comprehension of clinical texts and improve low health literacy. The goal of this study is to evaluate the potential of ChatGPT-3.5 and GPT-4 to tailor the complexity of medical information to patient-specific input education level, which is crucial if it is to serve as a tool in addressing low health literacy. Input templates related to 2 prevalent chronic diseases-type II diabetes and hypertension-were designed. Each clinical vignette was adjusted for hypothetical patient education levels to evaluate output personalization. To assess the success of a GLM (GPT-3.5 and GPT-4) in tailoring output writing, the readability of pre- and posttransformation outputs were quantified using the Flesch reading ease score (FKRE) and the Flesch-Kincaid grade level (FKGL). Responses (n=80) were generated using GPT-3.5 and GPT-4 across 2 clinical vignettes. For GPT-3.5, FKRE means were 57.75 (SD 4.75), 51.28 (SD 5.14), 32.28 (SD 4.52), and 28.31 (SD 5.22) for 6th grade, 8th grade, high school, and bachelor's, respectively; FKGL mean scores were 9.08 (SD 0.90), 10.27 (SD 1.06), 13.4 (SD 0.80), and 13.74 (SD 1.18). GPT-3.5 only aligned with the prespecified education levels at the bachelor's degree. Conversely, GPT-4's FKRE mean scores were 74.54 (SD 2.6), 71.25 (SD 4.96), 47.61 (SD 6.13), and 13.71 (SD 5.77), with FKGL mean scores of 6.3 (SD 0.73), 6.7 (SD 1.11), 11.09 (SD 1.26), and 17.03 (SD 1.11) for the same respective education levels. GPT-4 met the target readability for all groups except the 6th-grade FKRE average. Both GLMs produced outputs with statistically significant differences (P<.001; 8th grade P<.001; high school P<.001; bachelors P=.003; FKGL: 6th grade P=.001; 8th grade P<.001; high school P<.001; bachelors P<.001) between mean FKRE and FKGL across input education levels. GLMs can change the structure and readability of medical text outputs according to input-specified education. However, GLMs categorize input education designation into 3 broad tiers of output readability: easy (6th and 8th grade), medium (high school), and difficult (bachelor's degree). This is the first result to suggest that there are broader boundaries in the success of GLMs in output text simplification. Future research must establish how GLMs can reliably personalize medical texts to prespecified education levels to enable a broader impact on health care literacy.