Are we measuring the same health constructs? Amazon’s Mechanical Turk versus a community sample

Abstract

Amazon’s Mechanical Turk (MTurk) platform has increasingly gained popularity because of its affordability and efficiency. The results of studies comparing MTurk respondents to community respondents have been mixed. The purpose of the present study was to compare an MTurk and a community sample to determine whether the psychometric properties of a measure completed in the two different formats were comparable. There were 957 MTurk participants and 837 from the community sample, with approximately equal numbers of males and females. Participants were asked to read a scenario depicting a family with a sick child, and then to complete a questionnaire that measured their perceived likelihood of hiring a Health Care Advocate (HCA). The results indicated some demographic differences between MTurk and community participants. There was an effect of medical condition in the MTurk sample, such that participants were more likely to perceive hiring an HCA for a child with leukemia than cystic fibrosis (p = .008). However, in the community sample, there was an effect of conception difficulty where participants were more likely to perceive hiring an HCA for a child who took 2 months to conceive than 5 years to conceive (p = .012). Despite some psychometric similarities between the two samples, there were some differences in the constructs measured in the two samples. Future researchers should continue to evaluate the reliability and validity of paper-and-pencil measurements for online administration.