Evaluating Amazon's Mechanical Turk for psychological research on the symbolic control of attention.

Abstract

The use of online crowdsourcing services like Amazon's Mechanical Turk (AMT) as a method of collecting behavioral data online has become increasingly popular in recent years. A growing body of contemporary research has empirically validated the use of AMT as a tool in psychological research by replicating a wide range of well-established effects that have been previously reported in controlled laboratory studies. However, the potential for AMT to be used to conduct spatialcuing experiments has yet to be investigated in depth. Spatialcuing tasks are typically very basic in terms of their stimulus complexity and experimental testing procedures, thus making them ideal for remote testing online that requires minimal task instruction. Studies employing the spatialcuing paradigm are typically aimed at unveiling novel facets of the symbolic control of attention, which occurs whenever observers orient their attention through space in accordance with the meaning of a spatial cue. Ultimately, the present study empirically validated the use of AMT to study the symbolic control of attention by successfully replicating four hallmark effects reported throughout the visual attention literature: the left/right advantage, cue type effect, cuedaxis effect, and cuedendpoint effect. Various recommendations for future endeavors using AMT as a means of remotely collecting behavioral data online are also provided. In sum, the present study provides a crucial first step toward establishing a novel tool for conducting psychological research that can be used to expedite not only our own scientific contributions, but also those of our colleagues.