An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk

Abstract

Data collection using Internet-based samples has become increasingly popular in many social science disciplines, including advertising. This research examines whether one popular Internet data source, Amazon's Mechanical Turk (MTurk), is an appropriate substitute for other popular samples utilized in advertising research. Specifically, a five-sample between-subjects experiment was conducted to help researchers who utilize MTurk in advertising experiments understand the strengths and weaknesses of MTurk relative to student samples and professional panels. In comparisons across five samples, results show that the MTurk data outperformed panel data procured from two separate professional marketing research companies across various measures of data quality. The MTurk data were also compared to two different student samples, and results show the data were at least comparable in quality. While researchers may consider MTurk samples as a viable alternative to student samples when testing theory-driven outcomes, precautions should be taken to ensure the quality of data regardless of the source. Best practices for ensuring data quality are offered for advertising researchers who utilize MTurk for data collection.