Multi-criteria evaluation of health news stories

Abstract

The proliferation of digital and social media technologies has enabled quick and wide dissemination of news stories and press releases about new medical treatments. Evaluating these stories is difficult for two reasons. First, these stories are often not completely true or false. A nuanced approach that considers different aspects of these stories (e.g., the presence of inflated claims, suppression of risks associated with the treatment or withholding other essential information) is more appropriate for evaluation. Second, evaluating the quality and completeness of the arguments in the stories is costly and requires expertise in the relevant medical field, which laypeople do not have. To address this problem, in this study, we train different machine learning models on multi-criteria expert evaluations for health news stories about new medical treatments and compare their performance. We then compare the machine learning model evaluations to laypeople evaluations. We find that machine learning models overall outperform laypeople, who have a propensity to overestimate the comprehensiveness of the claims. Our machine learning models employ multi-criteria evaluation, which is different from most previous studies that evaluate news stories on whether they are true or false. We conclude by discussing the implications of this study for consumers of health news stories disseminated via social media.