Too Anecdotal to Be True? Mechanical Turk Is Not All Bots and Bad Data: Response to Webb and Tangney (2022).

Abstract

In response to Webb and Tangney (2022) we call into question the conclusion that data collected on Amazon's Mechanical Turk (MTurk) was "at best-only 2.6% valid" (p. 1). We suggest that Webb and Tangney made certain choices during the study-design and data-collection process that adversely affected the quality of the data collected. As a result, the anecdotal experience of these authors provides weak evidence that MTurk provides low-quality data as implied. In our commentary we highlight best practice recommendations and make suggestions for more effectively collecting and screening online panel data.