A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Abstract

AbstractRecent scholarship indicates that explicitly listing eligibility requirements on Amazon’s Mechanical Turk can lead to eligibility falsification. Offering a conceptual replication of prior studies, we assessed the prevalence of eligibility falsification and its impact on data integrity. A screener survey collected the summer before the 2016 presidential election assessed political affiliation. Participants were then randomly assigned to be exposed to a second survey link for which they were eligible or ineligible. There was a significant interaction such that the differences between self‐reported Republicans and Democrats on outcome measures (e.g., attitudes toward Hillary Clinton), were smaller among participants that were falsifying eligibility (i.e., imposters) than those that were not (i.e., genuine participants). Moreover, for most outcomes, imposters put forth responses that were significantly different from the responses put forth by those in the political party with which imposters were pretending to be affiliated. Imposters’ responses were also significantly different from participants in the political party with which imposters initially claimed to genuinely belong. For example, those who self‐reported themselves as Democrats on the screener survey but responded to a survey for “only Republicans” (i.e., imposter Republicans), reported more favorable attitudes toward Donald Trump than genuine Democrats, but indicated less favorable attitudes toward Donald Trump than genuine Republicans. These results highlight the potential harms of explicitly listing eligibility requirements and emphasize the importance of minimizing imposter participation.