Mechanical Turk and the “Don’t Know” Option

Abstract

ABSTRACTLuskin and Bullock’s (2011) randomized experiment on live-interview respondents found no evidence that American National Election Studies and Time-Sharing Experiments for the Social Sciences respondents hide knowledge behind the “don’t know” (DK) option. We successfully replicated their finding using two online platforms, the Cooperative Congressional Election Study and Google Surveys. However, we obtained different results on Amazon’s Mechanical Turk (MTurk). We attribute this difference to MTurkers’ experience with attention checks and other quality-control mechanisms, which condition them to avoid errors. This conditioning leads MTurkers to hide knowledge behind DK in ways not observed on other platforms. Researchers conducting political knowledge experiments or piloting surveys on MTurk should be aware of these differences.