Alternative methods for selecting web survey samples

Abstract

Probability-based sampling is the gold standard for general population surveys. However, when interested in more specific populations (e.g., consumers of a particular brand), a lot of research uses data from non-probability-based online panels. This article investigates different ways to select a sample in an opt-in panel: without previous information, using profiling information, or using passive data from a tracker installed on the panelists’ devices. Moreover, it investigates the effect of sending the survey closer to the “moment-of-truth,” which is expected to reduce memory limitations in recall questions. Using additional information (profiling or passive) to select the sample leads to clear improvements in terms of levels of participation and fieldwork efficiency, but not in terms of data quality (measured by the proportion of don’t know answers and the length of answers to open narrative questions) or accuracy (measured by comparing the answers to 14 questions to an external source of information). Doing the survey closer to the “moment-of-truth” further improves the fieldwork efficiency; however, there are still many challenges to implement true “in-the-moment” surveys. We also observed differences across the different samples in respondents’ socio-demographic characteristics and in the survey evaluation.