Getting More Bang for Your Buck on MTurk

Abstract

Over the last several years, researchers have increasingly engaged online panels such as Amazon’s Mechanical Turk (MTurk) and Prolific to distribute surveys for academic research. This trend has sparked much debate over the quality of data generated from such panels. Thus far, researchers have overwhelmingly blamed panel participants for quality issues (e.g. worker distraction or deception); very little attention has been paid to the role that researchers may be playing in the matter. This paper considers the relation between the design and distribution of online surveys and data quality. In study 1, we review a sample of 5 Qualtrics surveys retrieved from the Open Science Foundation (OSF) platform, summarizing their strengths and weaknesses to identify common characteristics of survey quality—or lack thereof. In study 2, we empirically test whether three survey quality characteristics (title comprehensibility, device compatibility, and scale quality) affect sample composition (age, gender, ethnicity, education level, employment status, household income, or HITS completed) and data quality (attention check pass rate, item completion rate, and survey submission rate.) We also look more broadly for differences in means among responses between conditions. Our results indicate that title comprehensibility does not affect sample composition; device compatibility has a significant effect on attention check pass rates and means for multiple items; and scale quality has a significant effect on the reported level of psychological safety in participants’ place of employment. We conclude with a discussion of the results, post-hoc observations, and recommendations for researchers conducting online surveys.