Collecting experimental data online: How to maintain data quality when you can’t see your participants

Abstract

Researchers are often nervous about collecting experimental data online because unlike lab-based experiments they cannot directly control their participants’ hardware, software or their location. They also cannot directly observe participants’ behaviour during the experiment. I will review the key issues that arise from using remote participants, and suggest a systematic approach to (i) participant recruitment, (ii) data collection and (iii) data analysis that ensures high levels of data quality. I will argue that it is inevitable that online recruitment will lead to the collection of unsuitable data, and that while we should do all we can to improve the quality of our collected data, it is perhaps more important to focus on ensuring that poor quality data is not included in our final analyses. A cornerstone of this approach is the careful pre-registration of exclusion criteria based on task piloting. I will also suggest that this principled approach to data quality control is often missing in lab-based settings and that the lessons learned due to running experiments online can be broadly applied to a range of experimental paradigms.