Too many numbers and worse word choice: Why readers find data-driven news articles produced with automation harder to understand

Abstract

The production of data-driven journalism is becoming increasingly automated, impacting its composition and comprehensibility. Given the importance of data-driven reporting for democratic participation, this study investigates, firstly, how readers evaluate the composition of data-driven articles produced with and without the help of automation and, secondly, how these evaluations affect readers’ perceptions of the articles’ comprehensibility. In an online survey experiment, 3135 online news consumers evaluated 24 articles produced with or without automation using criteria developed in prior research. Our factor analysis reduced those criteria to five categories that matter in readers’ evaluations of the articles’ composition: numeric features, writing style, sentence and paragraph length, descriptive language, and word choice. Our results show, firstly, that although the perception of news stories produced with automation did not differ significantly from that of news stories produced without automation regarding sentence and paragraph length and writing style, the stories produced with automation were evaluated as significantly less comprehensible; and, secondly, that this can be explained partly by readers’ perceptions of some of the other article composition categories, which were rated significantly worse in automated articles. Our findings suggest that using automation to produce data-driven news articles changes their perceived composition in ways that negatively impact comprehensibility. However, this study also suggests how such articles could be made more comprehensible. Specifically, when ‘post-editing’ automated articles, journalists should aim to further reduce the quantity of numbers, better explain words that readers are unlikely to understand and change inappropriate wording.