Abstract 18734: Comparison of Hypertension Prevention Education Among Community Health Students, Online Chat-Based Artificial Intelligence Model, and Physician

Abstract

Introduction: Health literacy and preventive education remain the core of hypertension management. Emerging language processing, artificial intelligence (AI) tools have the potential to provide effective preventive information. We sought to compare the quality of responses to commonly asked questions about hypertension by community health students, a physician, and an AI tool. Methods: Through a structured, student service-learning program, 270 trained undergraduate students conducted 56 health fairs across 19 locations in NJ from Jan 2022 - March 2023. We identified the 10 most commonly asked questions related to blood pressure and asked a physician, a trained student, and ChatGPT to answer them in paragraph format within 200 words. Four independent, blinded expert reviewers graded each response from 1 to 3 (3=best, 1=worst response relative to others). We assessed the reliability across responses using the Fleiss' Kappa statistic. We used ANOVA with post-hoc pairwise comparisons to determine differences in average response score. Results: Among 30 responses, there was a significant difference in score by response type (physician, 2.0 [SD, 0.6]; student, 1.8 [0.5]; ChatGPT, 2.2 [0.6], p=0.001). Post-hoc, pairwise comparisons revealed student scores were significantly lower than ChatGPT scores (p=0.0038), but there was no difference between physician and ChatGPT responses (p=0.81). Scores were moderately reliable across expert reviewers (kappa = 0.56). Qualitatively, student and physician scores were rated to be more empathetic and personalized, but less comprehensive and actionable. Conclusions: A language-processing, AI-based tool performed as well, if not better, than a physician and trained community health students to provide preventive education related to blood pressure. Further quality improvement initiatives need to be done to better understand the utility of AI tools to augment preventive education and to more effectively train students/physicians to provide comprehensive and actionable strategies.