Evaluation of the Readability, Understandability, and Accuracy of Artificial Intelligence Chatbots in Terms of Biostatistics Literacy

Abstract

Listen

Translate article

Objective: Chatbots have been frequently used in many different areas in recent years, such as diagnosis and imaging, treatment, patient follow-up and support, health promotion, customer service, sales, marketing, information and technical support. The aim of this study is to evaluate the readability, comprehensibility, and accuracy of queries made by researchers in the field of health through artificial intelligence chatbots in biostatistics. Methods: A total of 10 questions from the topics frequently asked by researchers in the field of health in basic biostatistics were determined by 4 experts. The determined questions were addressed to the artificial intelligence chatbots by one of the experts and the answers were recorded. In this study, free versions of most widely preferred ChatGPT4, Gemini and Copilot chatbots were used. The recorded answers were independently evaluated as “Correct”, “Partially correct” and “Wrong” by three experts who blinded to which chatbot the answers belonged to. Then, these experts came together and examined the answers together and made the final evaluation by reaching a consensus on the levels of accuracy. The readability and understandability of the answers were evaluated with the Ateşman readability formula, Sönmez formula, Çetinkaya-Uzun readability formula and Bezirci-Yılmaz readability formulas. Results: According to the answers given to the questions addressed to the artificial intelligence chatbots, it was determined that the answers were at the “difficult” level according to the Ateşman readability formula, “insufficient reading level” according to the Çetinkaya-Uzun readability formula, and “academic level” according to the Bezirci-Yılmaz readability formula. On the other hand, the Sönmez formula gave the result of “the text is understandable” for all chatbots. It was determined that there was no statistically significant difference (p=0.819) in terms of accuracy rates of the answers given by the artificial intelligence chatbots to the questions. Conclusion: It was determined that although the chatbots tended to provide accurate information, the answers given were not readable, understandable and their accuracy levels were not high.