Assessing the accuracy of an online chat-based artificial intelligence model in providing recommendations on arterial hypertension in accordance with the 2018 ESC/ESH guidelines

Abstract

Abstract Background The rise of online chat-based artificial intelligence (AI) models has opened up new possibilities in the field of medicine. One of the promising contributions of AI chatbots could be their ability to provide accurate and accessible medical information to patients. This could help bridge the gap between patients and healthcare providers, especially in under-resourced areas where access to medical information and care is limited. Purpose We aimed to evaluate the ability of a commercially available AI-based online chat to provide answers to commonly asked questions related to arterial hypertension in accordance with the latest 2018 ESC/ESH guidelines (1). Methods We selected 30 commonly asked questions covering a range of arterial hypertension topics, including risk factors, symptoms, diagnosis, and treatment. These questions were submitted to the online AI interface, and the responses were recorded and evaluated by a team of four experienced cardiologists from our institution. To ensure consistency, each question was asked three times. All questions – except question 2 – were preceded by "according to the 2018 ESC/ESH Guidelines for the management of arterial hypertension". Each answer was rated as "accurate," "inaccurate," or "incomplete" based on the specialists’ clinical judgment, the 2018 ESC/ESH guidelines, and the content of the response. A response was considered "accurate" if it included all essential information, "inaccurate" if information provided was not in accordance with the guidelines, and "incomplete" if any essential information was missing. Results The AI model's responses to 24 out of the 30 questions (80%) were deemed accurate by cardiologists. Three responses were rated as incomplete (10%) and three responses were found to be inaccurate (10%). The AI responses were consistent in 28 out of the 30 questions (93%) (Tables 1-2). Conclusion We found that a popular online AI model provided largely accurate responses to commonly asked arterial hypertension questions. These answers were mostly in accordance with the 2018 ESC/ESH guidelines on the management of arterial hypertension. While the use of chat-based AI in medicine is still in its early stages and current models are not intended for medical use, the potential for such technology to improve the healthcare experience for patients is significant.Table 1:(1) Assessment of AnswersTable 2:(2) Assessment of Answers