The Progress and Pitfalls of Using Survey Experiments in Political Science

Abstract

Survey experiments are now quite common in political science. A recent analysis of the number of mentions of this term in political science journal articles demonstrates a dramatic increase from 2000 to 2013. In addition, the term survey experiment has been picked up by many other disciplines, by researchers in a variety of different countries. Given the large number of survey experiments already published, the goal here is not to review the numerous excellent studies using this methodology, because there are far too many, spanning too many different topics. Instead, this juncture—marked by both progress and the proliferation of this method—is used to highlight some of the issues that have arisen as this methodological approach has come of age. How might research using this methodology improve in political science? What are the greatest weaknesses of survey experimental studies in this discipline to date? The explosive growth of survey experiments in political science speaks to their popularity as a means of establishing causal inference. In his reflection on the origins of survey experiments, Paul Sniderman has suggested that their quick rise in popularity was due to two factors: a) their ability to meet expected standards of external validity within the discipline without sacrificing internal validity, and b) the lower marginal cost per study relative to studies that were representative national surveys. Collaborative data collection efforts such as the Multi-Investigator Project and Time-sharing Experiments for the Social Sciences (TESS) made it possible for more scholars to execute population-based survey experiments at a lower cost per study than traditional surveys. Using shared platforms, researchers can execute many experiments for the price of one representative survey. These explanations make perfect sense in the context of a field such as political science, where external validity traditionally has been valued more highly than internal validity. It may be surprising to younger colleagues to learn that, not all that long ago, experiments were deemed completely inappropriate within the discipline of political science, unless they were field experiments executed in the real world. Experiments involving interventions in naturally occurring political environments were deemed tolerable, but only political psychologists were likely to find experimentation more broadly acceptable due to their strong ties to psychology. In political science, survey experiments were a means of promoting experimental methods in an external-validity-oriented discipline. Survey experiments freed political scientists from college sophomores as subjects and promised that external validity need not be sacrificed for strong causal inference. Times have obviously changed, and political scientists now embrace a much broader array of methodologies including both observational and experimental methods. This occasion provides an opportunity to re-evaluate the strengths and weaknesses of this innovative method, in theory and in practice.