Patient Education Materials Assessment Tool Research Articles

Utilization of Artificial Intelligence in the Creation of Patient Information on Laryngology Topics.

To evaluate and compare the readability and quality of patient information generated by Chat-Generative Pre-Trained Transformer-3.5 (ChatGPT) and the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS) using validated instruments including Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease, DISCERN, and Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P). ENTHealth.org and ChatGPT-3.5 were queried for patient information on laryngology topics. ChatGPT-3.5 was queried twice for a given topic to evaluate for reliability. This generated three de-identified text documents for each topic: one from AAO-HNS and two from ChatGPT (ChatGPT Output 1, ChatGPT Output 2). Grade level and reading ease were compared between the three sources using a one-way analysis of variance and Tukey's post hoc test. Independent t-tests were used to compare DISCERN and PEMAT understandability and actionability scores between AAO-HNS and ChatGPT Output 1. Material generated from ChatGPT Output 1 and ChatGPT Output 2 were at least two reading grade levels higher than that of material from AAO-HNS (p < 0.001). Regarding reading ease, ChatGPT Output 1 and ChatGPT Output 2 documents had significantly lower mean scores compared to AAO-HNS (p < 0.001). Moreover, ChatGPT Output 1 material on vocal cord paralysis had a lower PEMAT-P understandability compared to that of AAO-HNS material (p > 0.05). Patient information on the ENTHealth.org website for select laryngology topics was, on average, of a lower grade level and higher reading ease compared to that produced by ChatGPT, but interestingly with largely no difference in the quality of information provided. NA Laryngoscope, 2024.

Comparative Analysis of Artificial Intelligence Platforms in Generating Post-Operative Instructions for Rhinologic Surgery

AbstractRecently, artificial intelligence (AI) platforms such as ChatGPT and Google Gemini have progressed at a rapid pace. To allow for optimal medical outcomes and patient safety, it is crucial that patients have clearly written post-operative instructions. Patients are increasingly turning to AI platforms for medical information. Our objective was to evaluate the ability of ChatGPT versus Gemini to develop accurate and understandable post-operative instructions for patients undergoing endoscopic sinus surgery (ESS). Postoperative instructions on ESS, balloon sinuplasty, Caldwell Luc surgery, and septoplasty were generated using ChatGPT and Gemini. Measures of readability were calculated. The Patient Education Materials Assessment Tool was utilized to determine Understandability and Actionability. ChatGPT and Gemini were capable of generating understandable post-operative instructions but were poor for actionability and readability. While AI has a promising ability to generate accessible medical information, patients should be aware of its drawbacks especially when seeking postoperative instructions.

Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma.

The potential impacts of artificial intelligence (AI) chatbots on care for patients with bone sarcoma is poorly understood. Elucidating potential risks and benefits would allow surgeons to define appropriate roles for these tools in clinical care. Eleven questions on bone sarcoma diagnosis, treatment, and recovery were inputted into three AI chatbots. Answers were assessed on a 5-point Likert scale for five clinical accuracy metrics: relevance to the question, balance and lack of bias, basis on established data, factual accuracy, and completeness in scope. Responses were quantitatively assessed for empathy and readability. The Patient Education Materials Assessment Tool (PEMAT) was assessed for understandability and actionability. Chatbots scored highly on relevance (4.24) and balance/lack of bias (4.09) but lower on basing responses on established data (3.77), completeness (3.68), and factual accuracy (3.66). Responses generally scored well on understandability (84.30%), while actionability scores were low for questions on treatment (64.58%) and recovery (60.64%). GPT-4 exhibited the highest empathy (4.12). Readability scores averaged between 10.28 for diagnosis questions to 11.65 for recovery questions. While AI chatbots are promising tools, current limitations in factual accuracy and completeness, as well as concerns of inaccessibility to populations with lower health literacy, may significantly limit their clinical utility.

Videos in Short-Video Sharing Platforms as Sources of Information on Colorectal Polyps: Cross-Sectional Content Analysis Study.

Short videos have demonstrated huge potential in disseminating health information in recent years. However, to our knowledge, no study has examined information about colorectal polyps on short-video sharing platforms. This study aimed to analyze the content and quality of colorectal polyps-related videos on short-video sharing platforms. The terms "" (intestinal polyps) or "" (colonic polyps) or "" (rectal polyps) or "" (colorectal polyps) or "" (polyps of large intestine) were used to search in TikTok (ByteDance), WeChat (Tencent Holdings Limited), and Xiaohongshu (Xingyin Information Technology Limited) between May 26 and June 8, 2024, and then the top 100 videos for each search term on different platforms were included and recorded. The Journal of American Medical Association (JAMA) score, the Global Quality Scale (GQS), the modified DISCERN, and the Patient Education Materials Assessment Tool (PEMAT) were used to evaluate the content and quality of selected videos by 2 independent researchers. SPSS (version 22.0; IBM Corp) and GraphPad Prism (version 9.0; Dotmatics) were used for analyzing the data. Descriptive statistics were generated, and the differences between groups were compared. Spearman correlation analysis was used to evaluate the relationship between quantitative variables. A total of 816 eligible videos were included for further analysis, which mainly conveyed disease-related knowledge (n=635, 77.8%). Most videos were uploaded by physicians (n=709, 86.9%). These videos had an average JAMA score of 2.0 (SD 0.6), GQS score of 2.5 (SD 0.8), modified DISCERN score of 2.5 (SD 0.8), understandability of 80.4% (SD 15.6%), and actionability of 42.2% (SD 36.1%). Videos uploaded by news agencies were of higher quality and received more likes and comments (all P<.05). The number of collections and shares of videos about posttreatment caveats were more than those for other content (P=.03 and P=.006). There was a positive correlation between the number of likes, comments, collections, and shares (all P<.001). The duration and the number of fans were positively correlated with the quality of videos (all P<.05). There are numerous videos about colorectal polyps on short-video sharing platforms, but the reliability and quality of these videos are not good enough and need to be improved.

Assessing the understandability and actionability of online resources for patients undergoing hemodialysis.

This study quantitatively evaluated the understandability and actionability of webpages for patients undergoing hemodialysis (HD). In June 2022, we searched on Google and Yahoo Japan using the top five keywords associated with end-stage kidney disease and HD to identify relevant materials. Subsequently, we used the Japanese version of the Patient Education Materials Assessment Tool to assess the understandability and actionability of these materials, rating them on a scale from 0% to 100%. The threshold was set at 70%. Among the included 194 materials, the overall understandability was 66.7%, and the actionability was 33.3%. Only 38.7% and 16.5% of the materials were deemed understandable and actionable, respectively. Challenges included the need for plain language and concise summaries to improve understandability, along with the use of effective visual aids for actionability. This study suggests that existing webpages on HD are not presented in a manner that patients can utilize.

Standard HIPAA Authorization Forms Decreased Response Rates for a Multi-site Pragmatic Trial.

The Health Insurance Portability and Accountability Act (HIPAA) aims to safeguard patient information; however, complex legal language may lead to confusion and mistrust, and hinder enrollment in clinical trials. To evaluate the effect of a standard HIPAA authorization included in mailed survey packets on study enrollment for a multi-site pragmatic trial. This study is nested within an advance care planning pragmatic trial at 50 primary care clinics across three University of California (UC) Health Systems. We included English and Spanish-speaking seriously ill patients. One third of eligible patients received and 2/3 did not receive the HIPAA authorization in their enrollment packet. We compared enrollment rates at 3months and assessed the readability, understandability, and actionability of the standard HIPAA form using the Federal Plain Language Guidelines Checklist for Plain Language, the Automatic Readability Checker consensus calculator (grade 8 is the average reading level for US adults), and the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P, 0-100%, 70% considered the minimum). Of 4632 eligible patients (mean age 71, 48% women, 11% Spanish-speaking, 40% racial/ethnic minority); 1543 received a mailed enrollment packet with a HIPAA form and 3089 did not. Patients mailed the HIPAA form were less likely to enroll (10.2% vs. 14.8%, p < 0.001). The standard HIPAA form scored at the 12th grade reading level, had a PEMAT-P Understandability score of 42%, had an Actionability score of 40%, and only met 50% of Federal Plain Language Guideline Checklist items. The inclusion of a standard HIPAA authorization in mailed enrollment packets for a large pragmatic trial led to lower rates of study enrollment. This study informs how HIPAA authorization forms should be redesigned to be more accessible to patients to prevent unnecessary barriers to research enrollment.

Development and Validation of an Artificial Intelligence-Assisted Patient Education Material for Ostomy Patients: A Methodological Study.

To develop and test the validity of an artificial intelligence-assisted patient education material for ostomy patients. A methodological study. The study was carried out in two main stages and five steps: (1) determining the information needs of ostomy patients, (2) creating educational content, (3) converting the educational content into patient education material, (4) validation of patient education material based on expert review and (5) measuring the readability of the patient education material. We used ChatGPT 4.0 to determine the information needs and create patient education material content, and Publuu Online Flipbook Maker was used to convert the educational content into patient education material. Understandability and applicability scores were assessed using the Patient Education Materials Assessment Tool submitted to 10 expert reviews. The tool inter-rater reliability was determined via the intraclass correlation coefficient. Readability was analysed using the Flesch-Kincaid Grade Level, Gunning Fog Index and Simple Measure of Gobbledygook formula. The mean Patient Education Materials Assessment Tool understandability score of the patient education material was 81.91%, and the mean Patient Education Materials Assessment Tool actionability score was 85.33%. The scores for the readability indicators were calculated to be Flesch-Kincaid Grade Level: 8.53, Gunning Fog: 10.9 and Simple Measure of Gobbledygook: 7.99. The AI-assisted patient education material for ostomy patients provided accurate information with understandable and actionable responses to patients, but is at a high reading level for patients. Artificial intelligence-assisted patient education materials can significantly increase patient information rates in the health system regarding ease of practice. Artificial intelligence is currently not an option for creating patient education material, and their impact on the patient is not fully known. The study followed the STROBE checklist guidelines. No patient or public contributions.

Oral Diabetes Medication Videos on Douyin: Analysis of Information Quality and User Comment Attitudes.

Oral diabetes medications are important for glucose management in people with diabetes. Although there are many health-related videos on Douyin (the Chinese version of TikTok), the quality of information and the effects on user comment attitudes are unclear. The purpose of this study was to analyze the quality of information and user comment attitudes related to oral diabetes medication videos on Douyin. The key phrase "oral diabetes medications" was used to search Douyin on July 24, 2023, and the final samples included 138 videos. The basic information in the videos and the content of user comments were captured using Python. Each video was assigned a sentiment category based on the predominant positive, neutral, or negative attitude, as analyzed using the Weiciyun website. Two independent raters assessed the video content and information quality using the DISCERN (a tool for assessing health information quality) and PEMAT-A/V (Patient Education Materials Assessment Tool for Audiovisual Materials) instruments. Doctors were the main source of the videos (136/138, 98.6%). The overall information quality of the videos was acceptable (median 3, IQR 1). Videos on Douyin showed relatively high understandability (median 75%, IQR 16.6%) but poor actionability (median 66.7%, IQR 48%). Most content on oral diabetes medications on Douyin related to the mechanism of action (75/138, 54.3%), precautions (70/138, 50.7%), and advantages (68/138, 49.3%), with limited content on indications (19/138, 13.8%) and contraindications (14/138, 10.1%). It was found that 10.1% (14/138) of the videos contained misinformation, of which 50% (7/14) were about the method of administration. Regarding user comment attitudes, the majority of videos garnered positive comments (81/138, 58.7%), followed by neutral comments (46/138, 33.3%) and negative comments (11/138, 8%). Multinomial logistic regression revealed 2 factors influencing a positive attitude: user comment count (adjusted odds ratio [OR] 1.00, 95% CI 1.00-1.00; P=.02) and information quality of treatment choices (adjusted OR 1.49, 95% CI 1.09-2.04; P=.01). Despite most videos on Douyin being posted by doctors, with generally acceptable information quality and positive user comment attitudes, some content inaccuracies and poor actionability remain. Users show more positive attitudes toward videos with high-quality information about treatment choices. This study suggests that health care providers should ensure the accuracy and actionability of video content, enhance the information quality of treatment choices of oral diabetes medications to foster positive user attitudes, help users access accurate health information, and promote medication adherence.

Evaluation of TikTok social media posts on side effect information for popular weight loss medications

AbstractIntroductionThe prevalence of overweight and obesity is increasing in America, contributing to various health risks. Glucagon‐like peptide‐1 (GLP‐1) and gastric inhibitory peptide (GIP) receptor agonists have bloomed in popularity due to their efficacy and United States Food and Drug Administration (FDA) approval for weight loss, igniting popularity on social media. This study focuses on the role of TikTok™ (ByteDance, Haidian, Beijing, China), a popular social media platform, in disseminating understandable and actionable information about the side effects of weight loss medications.MethodsThe study analyzed the top 50 TikTok videos for each of the following hashtags: #ozempicsideeffects, #semaglutidesideeffects, #mounjarosideeffects, and #wegovysideeffects. Videos were evaluated based on their understandability and actionability using the Patient Education Materials Assessment Tool (PEMAT) for audio‐visual (AV) products. PEMAT‐AV scores and popularity of videos were compared between hashtags utilizing one‐way analysis of variance (ANOVA). An alpha of 0.05 was used, with Bonferroni correction.ResultsA total of 165 videos were reviewed for PEMAT‐AV understandability and actionability scores. The total visibility of the videos reached over 19.4 million views. A majority of the videos were personal experiences (89%), with only 5% being health care education videos. The average understandability and actionability scores for all videos were 43% and 20%, respectfully. The data presented a difference between searched hashtags for both evaluation scores, p‐values of &lt;0.001 and 0.093, respectively.ConclusionTikTok videos covering weight loss medication side effects have poor understandability and actionability scores. This study highlights the opportunity for more educational videos produced by health care professionals on TikTok, and a potential need for health care organizations and/or colleges of pharmacy to provide education to health care providers on how to create quality content for medication education on social media‐based platforms. Further research is needed to evaluate the credibility of content.

Investigating the capabilities of advanced large language models in generating patient instructions and patient educational material

ObjectivesLarge language models (LLMs) with advanced language generation capabilities have the potential to enhance patient interactions. This study evaluates the effectiveness of ChatGPT 4.0 and Gemini 1.0 Pro in providing...

Suitability of ChatGPT as a Source of Patient Information for Screening Mammography.

ChatGPT3.5 and ChatGPT4 were released publicly in late November 2022 and March 2023, respectively, and have emerged as convenient sources of patient health education and information, including for screening mammography. ChatGPT4 offers enhanced capabilities; however, it is only available by paid subscription. The purported benefits of ChatGPT for health education need to be objectively evaluated. To assess performance differences, ChatGPT3.5 and GPT4 were used between 13 April and 29 May 2023 to generate breast screening patient information sheets, which were evaluated using the Patient Education Materials Assessment Tool for printed materials (PEMAT-P) and the CDC Clear Communication Index (CDC Index) Score Sheet; and benchmarked against gold standard content in BreastScreen NSW's patient information sheet. Mean scores were reported for comparison. GPT3.5 provided the appropriate tone and currency of information but lacked accuracy, omitting key insights: PEMAT-P understandability 68.0% (SD = 6.56) and actionability 36.7% (SD=20.4); CDC Index 58.8% (SD = 15.3). GPT4 was deemed superior to GPT3.5 but included several key omissions: PEMAT-P understandability 75.0% (SD = 17) and actionability 53.3% (SD = 11.54); CDC Index 66.0% (SD = 4.1). Both ChatGPT versions exhibited poor understandability and actionability and were unclear in their messaging. Those with poor health literacy will not benefit from accessing current versions of ChatGPT and may be further disadvantaged if they do not have access to a paid subscription. ChatGPT is evidenced to be an unreliable and inaccurate source of information concerning breast screening that may undermine participation and risk increased morbidity and mortality from breast cancer. ChatGPT may increase the demand on health care educators to rectify misinformation.

Online Plain Language Tool and Health Information Quality

Complex and ineffective health communication is a critical source of health inequity and occurs despite repeated policy directives to provide health information that is easy to understand and applies health literacy principles. To evaluate the effectiveness of the Sydney Health Literacy Lab Health Literacy Editor, an easy-to-use online plain language tool that supports health information providers to apply health literacy guidelines to written health information. This randomized clinical trial, conducted online in Australia from May 2023 to February 2024, included a convenience sample of health information providers with no previous experience using the Health Literacy Editor. Analysts were blinded to study group. Participants were randomized 1:1 to the intervention or control group. Participants in the intervention group were provided access to the Health Literacy Editor and a 30-minute online training program prior to editing 3 prespecified health texts. The Health Literacy Editor gives objective, real-time feedback on words and sentences. Control participants revised the texts using their own standard health information development processes. The preregistered primary outcome was the text school grade reading score (using a validated instrument, the Simple Measure of Gobbledygook). Secondary outcomes were text complexity (percentage of text using complex language), use of passive voice (number of instances), and subjective expert ratings (5-point Likert scale corresponding to items on the Patient Education Materials Assessment Tool). A total of 211 participants were randomized, with 105 in the intervention group and 106 in the control group. Of 181 participants in the intention-to-treat analysis (mean [SD] age, 41.0 [11.6] years; 154 women [85.1%]), 86 were in the intervention group and 95 in the control group. Texts revised in the intervention group had significantly improved grade reading scores (mean difference [MD], 2.48 grades; 95% CI, 1.84-3.12 grades; P < .001; Cohen d, 0.99), lower text complexity scores (MD, 6.86; 95% CI, 4.99-8.74; P < .001; Cohen d, 0.95), and less use of passive voice (MD, 0.95 instances; 95% CI, 0.44-1.47 instances; P < .001; Cohen d, 0.53) compared with texts revised in the control group in intention-to-treat analyses. Experts rated texts in the intervention group more favorably for word choice and style than those in the control group (MD, 0.44; 95% CI, 0.25-0.63; P < .001; Cohen d, 0.63), with no loss of meaning or content. In this randomized clinical trial, the Health Literacy Editor helped users simplify health information and apply health literacy guidelines to written text. The findings suggest the tool has high potential to improve development of health information for people who have low health literacy. As an online tool, the Health Literacy Editor is also easy to access and implement at scale. ANZCTR Identifier: ACTRN12623000386639.

Medical misinformation in social media: Representations of gastrointestinal disorders on a short video platform

Background and Objectives: Short video platforms have become one of the most common methods for disseminating medical information on social media. We analysed gastrointestinal (GI)-related content on TikTok, focusing on the creators’ background, patterns of content utilisation and overall content quality and understandability, using validated metrics to quantify the degree of misinformation. Methods: Twelve hashtags related to common GI conditions on TikTok and the 20 most engaging posts associated with each hashtag were selected for review. The ‘most engaging’ status was determined by the application’s internal algorithm. Six GI board-certified physicians evaluated the videos using two validated instruments: DISCERN quality criteria and Patient Education Materials Assessment Tool (PEMAT-A/V) survey. Results: In total, 253 videos were reviewed with 18.6 billion likes, 191,000 comments, 935,000 saves and 927,000 shares. IBS hashtag had the greatest number of views (2.2 billion), while videos on haemorrhoids had the most likes (4.3 million), comments (59,000) and shares (288,000). Most videos 177/253 (70%) were created by non-healthcare workers. The median video quality was 2/5 on DISCERN Likert-type scale with 23/75 DISCERN score. Of 186 videos, 114 (61.3%) were rated ‘very poor’; 58/186 (31.2%) ‘poor’, 13/186 (7%) ‘fair’ and only one video (0.5%) was rated ‘excellent’ on DISCERN grading system. The median PEMAT-A/V scores were 7/12 (73.9%) for understandability and 1/3 (33.3%) for actionability. Conclusion: Our study highlights significant public interest in common GI disorders. However, most posts lacked substantive information and did not have verifiable sources.

Using ChatGPT to Provide Patient-Specific Answers to Parental Questions in the PICU.

To determine if ChatGPT can incorporate patient-specific information to provide high-quality answers to parental questions in the PICU. We hypothesized that ChatGPT would generate high-quality, patient-specific responses. In this cross-sectional study, we generated assessments and plans for 3 PICU patients with respiratory failure, septic shock, and status epilepticus and paired them with 8 typical parental questions. We prompted ChatGPT with instructions, an assessment and plan, and 1 question. Six PICU physicians evaluated the responses for accuracy (1-6), completeness (yes/no), empathy (1-6), and understandability (Patient Education Materials Assessment Tool, PEMAT, 0% to 100%; Flesch-Kincaid grade level). We compared answer quality among scenarios and question types using the Kruskal-Wallis and Fischer's exact tests. We used percent agreement, Cohen's Kappa, and Gwet's agreement coefficient to estimate inter-rater reliability. All answers incorporated patient details, utilizing them for reasoning in 59% of sentences. Responses had high accuracy (median 5.0, [interquartile range (IQR), 4.0-6.0]), empathy (median 5.0, [IQR, 5.0-6.0]), completeness (97% of all questions), and understandability (PEMAT % median 100, [IQR, 87.5-100]; Flesch-Kincaid level 8.7). Only 4/144 reviewer scores were <4/6 in accuracy, and no response was deemed likely to cause harm. There was no difference in accuracy, completeness, empathy, or understandability among scenarios or question types. We found fair, substantial, and almost perfect agreement among reviewers for accuracy, empathy, and understandability, respectively. ChatGPT used patient-specific information to provide high-quality answers to parental questions in PICU clinical scenarios.

Assessing the Quality of Patient Education Materials on Cardiac Catheterization From Artificial Intelligence Chatbots: An Observational Cross-Sectional Study.

Background Health literacy empowers patients to participate in their own healthcare. Personal health literacy is one's ability to find, understand, and use information/resources to make well-informed health decisions. Artificial intelligence (AI) has become a source for the acquisition of health-related information through large language model (LLM)-driven chatbots. Assessment of the readability and quality of health information produced by these chatbots has been the subject of numerous studies to date. This study seeks to assess the quality of patient education materials on cardiac catheterization produced by AI chatbots. Methodology We asked a set of 10 questions about cardiac catheterization to four chatbots: ChatGPT (OpenAI, San Francisco, CA), Microsoft Copilot (Microsoft Corporation, Redmond, WA), Google Gemini (Google DeepMind, London, UK), and Meta AI (Meta, New York, NY). The questions and subsequent answers were utilized to make patient education materials on cardiac catheterization. The quality of these materials was assessed using two validated instruments for patient education materials: DISCERN and the Patient Education Materials Assessment Tool (PEMAT). Results The overall DISCERN scores were 4.5 for ChatGPT, 4.4 for Microsoft Copilot and Google Gemini, and 3.8 for Meta AI. ChatGPT, Microsoft Copilot, and Google Gemini tied for the highest reliability score at 4.6, while Meta AI had the lowest with 4.2. ChatGPT had the highest quality score at 4.4, while Meta AI had the lowest with 3.4. ChatGPT and Google Gemini had Understandability scores of 100%, while Meta AI had the lowest with 82%. ChatGPT, Microsoft Copilot, and Google Gemini all had Actionability scores of 75%, while Meta AI had one of 50%. Conclusions ChatGPT produced the most reliable and highest quality materials, followed closely by Google Gemini. Meta AI produced the lowest quality materials. Given the easy accessibility that chatbots provide patients and the high-quality responses that we obtained, they could be a reliable source for patients to obtain information about cardiac catheterization.

Abstract B025: Assessing the content and quality of online patient education material on clinical trials from NCI-designated comprehensive cancer centers

Abstract Background: Online patient educational material on cancer clinical trials is available on NCI- designated comprehensive cancer center websites and official YouTube channels. Online health information is an important resource for patients seeking to learn more about treatment options. Objective: Examine the quality and content themes in online patient educational material (PEM) related to cancer clinical trials published by comprehensive cancer centers. Methods: A random sample of comprehensive cancer centers (21) was chosen to represent the national network of cancer centers. Each website was searched for webpages containing patient-oriented clinical trial information. Official YouTube channels were also searched for relevant video material. Each webpage and video (N=196) were assessed for quality and content themes. Quality was measured using an abbreviated Patient Education Materials Assessment Tool which scores understandability and actionability of text and audio-visual materials, and content themes were assessed by the use of a content characteristic codebook. Results: Common content themes in the sample (N=196) of online PEM from comprehensive cancer centers were defines purpose of trials, physician speaker, animated or illustrated elements, and defines phases of trials. Online PEM had an average score of 83 for understandability and 26 for actionability, an area for improvement. Only 10% of materials reviewed scored high for understandability (&amp;gt;90) and for actionability (&amp;gt;70). Online webpages that mixed text and video scored highest on the quality assessment. Video-only content scored lowest on the quality assessment. Only a small percentage of PEM included FAQs (10%), instructions on how to find a trial (16%), mentioned consent (14%) or costs associated with trials (12%). Mentions or depictions of minority populations were not commonly observed in the material (11%). Conclusions: Online PEM published by comprehensive cancer centers is an important and easily accessible resource for patients looking to learn more about clinical trials. Cancer centers should add various audio-visual formats and interactive tools to their webpages to increase the quality of the clinical trial resources. Including representation of historically underserved populations and adding actionable advice for patients has the potential to engage patients’ interest and increase enrollment in cancer clinical trials. Citation Format: Trista Beard, Diana Morales, Fatimata Sanogo, Chanita Hughes Halbert. Assessing the content and quality of online patient education material on clinical trials from NCI-designated comprehensive cancer centers [abstract]. In: Proceedings of the 17th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2024 Sep 21-24; Los Angeles, CA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2024;33(9 Suppl):Abstract nr B025.

Abstract B105: Quality and Accessibility of Online Information Regarding CAR-T Therapy in Hematologic Malignancies

Abstract Background Chimeric antigen receptor T-cell (CAR-T) therapies have changed the treatment landscape for various hematologic malignancies and are currently under development and being tested in clinical trials for multiple solid tumors. Given that patients commonly use the internet as a source of health-related information, we sought to investigate the quality, understandability, and readability of online information regarding CAR-T therapy. Methods We conducted a search in a commonly used search engine (Google) of five pre-determined search terms to identify 100 unique web pages regarding CAR-T therapy. Scientific publications were excluded. Validated instruments were used to ascertain health information quality (DISCERN), understandability (Patient Education Materials Assessment Tool; PEMAT), and readability (Flesch-Kincaid, Gunning Fog, and SMOG indices). Results were summarized using descriptive statistics. Results Among the 100 web pages reviewed, 68% originated from academic medical centers, and 64% included links to clinical trials. Only 11% of the web pages included references to primary literature. 17% of web pages were available in a language other than English, with Spanish (15%) and Arabic (7%) being the most common. Moderate-to-high quality information (DISCERN ≥ 3) was presented by 47% of web pages, and only 31% of web pages were considered understandable (PEMAT ≥ 70%). Average readability scores were 10.6 (SD = 2.09), 12.6 (SD = 2.3), and 13.1 (SD = 1.7) using the Flesch-Kincaid, Gunning Fog, and SMOG indices, respectively. Conclusion Our study highlights that online information for CAR-T therapies commonly features low-quality health information that is typically not understandable to the general population. Further, contrary to recommendations from federal agencies and medical societies, available online information requires, on average, high school reading comprehension. Healthcare teams and stakeholders should strive to improve existing content and ensure patients have access to trusted, high-quality information given the unique complexities, processes, and toxicities associated with receiving CAR-T therapy. Citation Format: Luis C. Garcia, Ana I. Velázquez. Quality and Accessibility of Online Information Regarding CAR-T Therapy in Hematologic Malignancies [abstract]. In: Proceedings of the 17th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2024 Sep 21-24; Los Angeles, CA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2024;33(9 Suppl):Abstract nr B105.

Large language models: a new frontier in paediatric cataract patient education

Background/aimsThis was a cross-sectional comparative study. We evaluated the ability of three large language models (LLMs) (ChatGPT-3.5, ChatGPT-4, and Google Bard) to generate novel patient education materials (PEMs) and improve...

Artificial Intelligence-Prompted Explanations of Common Primary Care Diagnoses

Background: Artificial intelligence (AI)-generated explanations about medical topics may be clearer and more accessible than traditional evidence-based sources, enhancing patient understanding and autonomy. We evaluated different AI explanations for patients about common diagnoses to aid in patient care. Methods: We prompted ChatGPT 3.5 , Google Bard, HuggingChat, and Claude 2 separately to generate a short patient education paragraph about seven common diagnoses. We used the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) to evaluate the readability and grade level of the responses. We used the Agency for Healthcare Research and Quality’s Patient Education Materials Assessment Tool (PEMAT) grading rubric to evaluate the understandability and actionability of responses. Results: Claude 2 demonstrated scores of FRE (67.0), FKGL (7.4), and PEMAT, 69% for understandability, and 34% for actionability. ChatGPT scores were FRE (58.5), FKGL (9.3), PEMAT (69% and 31%, respectively). Google Bard scores were FRE (50.1), FKGL (9.9), PEMAT (52% and 23%). HuggingChat scores were FRE (48.7) and FKGL (11.6), PEMAT (57% and 29%). Conclusion: Claude 2 and ChatGPT demonstrated superior readability and understandability, but practical application and patient outcomes need further exploration. This study is limited by the rapid development of these tools with newer improved models replacing the older ones. Additionally, the accuracy and clarity of AI responses is based on that of the user-generated response. The PEMAT grading rubric is also mainly used for patient information leaflets that include visual aids and may contain subjective evaluations.

Expanding Accessibility in Cleft Care: The Role of Artificial Intelligence in Improving Literacy of Alveolar Bone Grafting Information.

The American Medical Association (AMA) recommends patient education materials (PEMs) be written at or below a sixth grade reading level. This study seeks to determine the quality, readability, and content of available alveolar bone grafting (ABG) PEMs and determine if artificial intelligence can improve PEM readability. Review of free online PEMs. Online ABG PEMs were retrieved from different authoring body types (hospital/academic center, medical society, or private practice). None. Content was assessed by screening PEMs for specific ABG-related topics. Quality was evaluated with the Patient Education Material Assessment Tool (PEMAT), which has measures of understandability and actionability. Open-access readability software (WebFX) determined readability with Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. PEMs were rewritten with ChatGPT, and readability metrics were reassessed. Quality, readability, and content of ABG PEMs. 34 PEMs were analyzed. Regarding quality, the average PEMAT-understandability score was 67.0 ± 16.2%, almost at the minimum acceptable score of 70.0% (p = 0.281). The average PEMAT-actionability score was low at 33.0 ± 24.1%. Regarding readability, the average Flesch Reading Ease score was 64.6 ± 12.8, categorized as "standard/plain English." The average Flesch-Kincaid Grade Level was 8.0 ± 2.3, significantly higher than AMA recommendations (p < 0.0001). PEM rewriting with ChatGPT improved Flesch-Kincaid Grade Level to 6.1 ± 1.3 (p < 0.0001). Available ABG PEMs are above the recommended reading level, yet ChatGPT can improve PEM readability. Future studies should improve areas of ABG PEMs that are most lacking, such as actionability.