Variation in Motivational Appeals to Survey Completion: Lessons from a Randomized Experiment with Teachers

Abstract

Sample attrition increases the risk of statistical bias and hinders the ability to plausibly estimate causal effects when patterns of nonresponse are correlated with key variables of interest. Drawing on leverage-salience theory and other work in the behavioral psychology field, we empirically capture the impact of distinct motivational appeals on the survey response rates of elementary education teachers in a large urban school district in the northeastern United States. During spring 2017, teachers were randomized to receive one of six motivational appeals and were rerandomized to receive a different appeal each subsequent week, conditional on not having completed the survey. We observe the results on four different margins that range in their time intensity (open email and click, start, and complete survey). We find that extrinsic rewards improved teacher response across all four margins, and the social norm of reciprocity substantially improved teacher response along margins of lower time intensity. As researchers continue to conduct multitreatment arm studies and large-scale evaluations that can suffer from serious issues of sample attrition, this work highlights the contribution of message framing in survey response.