Mode Effects in Mixed-Mode Economic Surveys: Insights from a Randomized Experiment

Abstract

Web-based surveys have become increasingly common in economic, marketing, and other social science research. However, questions exist about the comparability of data gathered using a web interview and data gathered using more traditional survey modes, particularly for surveys on household economic behavior. Differences between data from different survey modes may arise through two different mechanisms: sample selectivity due to (lack of) web access and mode effects. This study leverages the randomized experimental design of the mixed-mode Cognitive Economics Study to examine mode effects separately from sample selectivity issues. In particular, we examine differences in survey response rates, item nonresponse, and data quality due to mode effects. Our results indicate that, in contrast to mail mode, web mode surveys (1) attain higher response rates among web users, (2) display lower item nonresponse, and (3) elicit more precise values for financial measures. We conclude that, for web-using populations, web mode surveys appear to result in more usable data than mail mode surveys, and these data appear to be of high quality. However, we also find no systematic mode differences in the categorical distributions of responses to items, providing no evidence that pooling data from the two modes is inadvisable.