Building strong designs through multi-method research

Abstract

This book has sought to provide a comprehensive—though far from exhaustive—discussion of the discovery, analysis, and evaluation of natural experiments. I have emphasized that the strongest natural experiments contribute markedly to causal inference. In the best case, natural experiments allow us to learn about the effects of causes that are difficult to manipulate experimentally, while obviating the substantial problems of confounding associated with conventional observational studies. At the same time, I have underscored the potential limitations of this approach and identified three dimensions on which specific natural experiments may fall short: plausibility of as-if random, credibility of models, and relevance of interventions. Using natural experiments is not easy terrain. As with other methods, there is no single algorithm for success, even if some types of natural experiments are increasingly replicated across contexts. Natural experiments that are persuasive on one evaluative dimension might well fall short on another; sources of natural experiments that are compelling in one substantive setting or for one research question might elsewhere raise difficult issues of interpretation. In each application, researchers are therefore challenged to provide evidence that can bolster the credibility of the underlying assumptions and enhance the persuasiveness of their findings. This evidence comes in disparate forms, including both quantitative and qualitative data, and deep knowledge of substance and context is often essential. Data collection can also be costly and time-consuming.