Estimating mode effects from a sequential mixed-mode experiment using structural moment models

Abstract

Until recently, the survey mode of the household panel study Understanding Society was mainly face-to-face interview, but it has now adopted a mixed-mode design where individuals can self-complete the questionnaire via the web. As mode is known to affect survey data, a randomized mixed-mode experiment was implemented during the first year of the two-year Wave 8 fieldwork period to assess the impact of this change. The experiment involved a sequential design that permits the identification of mode effects in the presence of nonignorable nonrandom mode selection. While previous studies have used instrumental variables regression to estimate the effects of mode on the means of the survey variables, we describe a more general methodology based on novel structural moment models that characterizes the overall effect of mode on a survey by its effects on the moments of the survey variables’ joint distribution. We adapt our estimation procedure to account for nonresponse and complex sampling designs and to include suitable auxiliary data to improve inference and relax key assumptions. Finally, we demonstrate how to estimate the effects of mode on the parameter estimates of generalized linear models and other exponential family models when both outcomes and predictors are subject to mode effects. This methodology is used to investigate the impact of the move to web mode on Wave 8 of Understanding Society.