The Paradox of Policy-Relevant Natural Experiments

Abstract

Conventional wisdom holds that natural experiments represent especially credible bases for econometric inference facilitating evidence-based policymaking. We illustrate the fragility of policy-relevant evidence generated by first-stage randomizations by way of parable economies. In the first (second) economy, the econometric analysis is relevant (irrelevant) as the government is able (powerless) to alter policy in the future. In the first economy, but not the second, econometric evidence is contaminated by endogeneity after-the-fact, with treatment responses contingent upon (unknown) parameters of the government objective function into which the empirical estimates will be fed. We have the following paradox: The irrelevant natural experiment is bias-free, while the relevant experiment is biased after-the-fact. Extending this argument, we show a perceived-credible natural experiment is biased by Hawthorne Effects while a perceived-non-credible experiment is not. Importantly, since we consider measure zero experimental subjects, the Hawthorne Effects we illustrate must be understood as arising from rational expectations as distinct from behavioral motives posited in the psychology literatures.