The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs

Abstract

Social scientists increasingly wish to use field experiments to test theories. However, common experimental designs for studying the effects of treatments delivered in the field on individuals’ attitudes are too expensive for most researchers and vulnerable to bias. We detail an alternative field experiment design exploiting multiple waves of panel surveys delivered online with multiple measures of outcomes and, if applicable, a placebo control. This design can make persuasion field experiments feasible by decreasing costs (often by nearly two orders of magnitude), allows experiments to test additional theories, and facilitates the evaluation of design assumptions. We then report an original application study, a field experiment implementing the design to evaluate a persuasive canvass targeting abortion attitudes. This study estimated a precise zero, suggesting the design can successfully evade demand effects. We conclude by discussing potential limitations and extensions.