Crowdsourcing Judgments of News Source Quality

Abstract

Reducing the spread of misinformation, especially on social media, is a major challenge. We investigate one potential approach: having social media platform algorithms preferentially display content from news sources which users rate as trustworthy. To do so, we ask whether crowdsourced trust ratings can effectively differentiate more versus less reliable sources. We ran two preregistered experiments (N=1,010 from Mechanical Turk and N=970 from Lucid) where individuals rated familiarity with, and trust in, 60 news sources from three categories: 1) mainstream media outlets, 2) hyper-partisan websites, and 3) websites that produce blatantly false content (“fake news”). Despite substantial partisan differences, we find that laypeople across the political spectrum rated mainstream sources as far more trustworthy than either hyper-partisan or fake news sources. Although this difference was larger for Democrats than Republicans – mostly due to distrust of mainstream sources by Republicans – every mainstream source (with one exception) was rated as more trustworthy than every hyper-partisan or fake news source across both studies when equally weighting ratings of Democrats and Republicans. Furthermore, politically balanced layperson ratings were strongly correlated (r=0.90) with ratings provided by professional fact-checkers. We also found that, particularly among liberals, individuals higher in cognitive reflection were better able to discern between low- and high-quality sources. Finally, we found that excluding ratings from participants who were not familiar with a given news source dramatically reduced the effectiveness of the crowd. Our findings indicate that having algorithms up-rank content from trusted media outlets may be a promising approach for fighting the spread of misinformation on social media.