Social Media, Web, and Panel Surveys: Using Non‐Probability Samples in Social and Policy Research

Abstract

The use of online surveys has grown rapidly in social science and policy research, surpassing more established methods. We argue that a better understanding is needed, especially of the strengths and weaknesses of non‐probability online surveys, which can be conducted relatively quickly and cheaply. We describe two common approaches to non‐probability online surveys—river and panel sampling—and theorize their inherent selection biases: namely, topical self‐selection and economic self‐selection. We conduct an empirical comparison of two river samples (Facebook and web‐based sample) and one panel sample (from a major survey research company) with benchmark data grounded in a comprehensive population registry. The river samples diverge from the benchmark on demographic variables and yield much higher frequencies on non‐demographic variables, even after demographic adjustments; we attribute this to topical self‐selection. The panel sample is closer to the benchmark. When examining the characteristics of a non‐demographic subpopulation, we detect no differences between the river and panel samples. We conclude that non‐probability online surveys do not replace probability surveys, but augment the researcher's toolkit with new digital practices, such as exploratory studies of small and emerging non‐demographic subpopulations.