Statistics and Causality: Separated to Reunite-Commentary on Bryan Dowd's “Separated at Birth”

Abstract

Bryan Dowd (2010) should be commended for laying before us the historical roots of the tensions between statisticians and econometricians which, until today, perpetuate the myth that causal inference is somehow confusing, enigmatic, or controversial. While modern analysis has proven this myth baseless, it is often the historical accounts that put things in the proper perspective. I see the tension between statistics and economics or, more generally, between statistics and causality, to be rooted in a more fundamental schism than the one portrayed in Dowd's account. Moreover, and contrary to Dowd's narrative, I believe that the schism was justified, necessary, and not sufficiently emphasized. In fact, it was only after the distinction between statistical and causal concepts was made crisp and formal through new mathematical notation that a productive symbiosis has emerged which now benefits both paradigms. Dowd's account portrays the schism as a product of unfortunate circumstances that could have been avoided, if only the players were more aware of each other work. Economists, we are told, developed causal inference techniques that yield regressional estimates of causal effects (e.g., IV, confounding-control) and, since regression is a proud invention of statistics, there was no reason for statisticians to shun causal analysis as strongly as they did. If they did oppose structural equations, instrumental variables, and observational studies, it must have been due to an unfortunate rhetorical distinction or, perhaps, a fluke in the history of science. Upon reading Dowd's account on how close statisticians were to develop causal inference techniques by themselves, readers might be tempted to conclude that the distinction between causal and statistical inference is perhaps unwarranted, and that the former is but a nuance of the latter. This would be a mistake and would not cohere with the lesson I draw from the history of causal analysis. While there has been indeed a century of tension and misunderstanding between statisticians and econometricians regarding causation, the tension was justified: economists were prepared to posit untestable structural assumptions and derive their consequences, but statisticians were not. As a result, economists developed IV estimators and methods of controlling for confounders—all based on theoretical, untestable assumptions—while statisticians were alienated by those assumptions and found refuge in Fisher's controlled randomized trial (CRT), where the only assumptions needed were those concerning the nature of randomization. Dowd recognizes this basic difference but attempts to minimize its importance, arguing that statistical analysis too is laden with untested assumptions, for example, that randomization balances “unobserved confounders,” or that certain measurements were taken under identical conditions, or that a certain error is Gaussian, or that Bayesian priors have certain values, or that an experimental group is “representative” of a target population. There are, however, fundamental differences between the assumptions that underlie statistical studies and those needed for causal inference in observational studies. The first difference is that most of the assumptions in conventional statistical studies, while untested perhaps in a given study, are testable in principle, given sufficiently large sample and sufficiently refined measurements. Causal assumptions, in contrast, cannot be tested even in principle, unless one resorts to experimental control. This difference stands out in Bayesian analysis. Although the priors that Bayesians commonly assign to statistical parameters are untested quantities, the sensitivity to these priors tends to diminish with increasing sample size. In contrast, sensitivity to prior causal assumptions, say that treatment does not change gender, remains substantial regardless of sample size. The second, and perhaps deeper difference between statistical and causal information is that the latter cannot be expressed in probability calculus—the standard language of statistical analysis. Any mathematical approach to causal analysis must acquire new notation—probability calculus is insufficient. (Skeptics are invited to write down a mathematical expression for the English sentence: “The rooster crow does not cause the sun to rise.”) This notational requirement, which economists tried to circumvent using structural equations, was unacceptable to statisticians who insisted that all empirical information be expressed in contingency tables, probability functions, or covariance matrices. In particular, Fisher's generation of statisticians could not accept the ambiguities associated with structural equations. Indeed, if we examine Dowd's first equation (1) from a statistical perspective, the ambiguities are overwhelming. How is one to distinguish this equation from a regression equation, in which the error is automatically orthogonal to Ti? What empirical information is conveyed by this equation, if any? Does βT have a causal interpretation? Does this interpretation vary with the statistics of ui or with the existence of a confounder W? The ambiguities associated with interpreting this seemingly innocent equation became, to no exaggeration, the greatest confusion of the 20th century. Paul Holland (1995, p. 54), for example, writes: “I am speaking, of course, about the equation: {y=a+bx+ɛ}. What does it mean? The only meaning I have ever determined for such an equation is that it is a shorthand way of describing the conditional distribution of {y} given {x}.” Today we know, of course, that the structural interpretation of equation (1) has nothing to do with the conditional distribution of {y} given {x}; rather, it conveys causal and counterfactual information that is orthogonal to the statistical properties of {x} and {y} (see Pearl 2009, Chapter 7). But such an understanding was not to be expected from traditional statisticians who, even as late as the 1990s, considered structural equations to be “meaningless” (Wermuth 1992). Naturally, statisticians gravitated to Fisher's experiments for as long as they could and, when mathematical analysis of causal relations became necessary, they invented the Neyman–Rubin “potential outcome” notation (Rubin 1974) and continued to oppose structural equations as a threat to principled science (Rubin 2004, 2009, 2010; Sobel 2008), not recognizing that the two languages are in fact equivalent (Pearl 2009, pp. 98–102). David Freedman, another staunch critics of structural equations found them not only ambiguous but utterly “self-contradictory” (Freedman 1987, p. 114). Today we understand that Freedman's error was to interpret structural equations as ordinary equations when in fact they are nonalgebraic—they change meaning under legitimate algebraic operations such as moving terms from one side of the equation to the other. What Freedman and others failed to realize is that the equality sign in equation (1) stands not for algebraic equality but for an assignment operator (:=) which Nature invokes to assign values to Yi, based on the current values of Xi, Ti, and ui (Pearl 2009, p. 138). Remarkably, the recent flair-up of interest in instrumental variables and other structurally based identification methods (Angrist and Pischke 2010) does not reflect statisticians' acceptance of structural equations as legitimate carriers of scientific knowledge; rather, it reflects an uncritical reliance on the semblance between quasi-experiment (e.g., instrumental variables) and certain properties (e.g., balance) of Fisher's randomized experiment—the supreme compass of statistical right and wrong. Members of the so-called “experimentalist” camp in econometrics still refuse to recognize structural equation modeling for what it is—a transparent, formal language for causal and counterfactual information, logically equivalent to the opaque jargon of structureless models,1 within which powerful identification methods could be both justified and derived (Heckman 2010; Keane 2010; Leamer 2010; Nevo and Whinston 2010;).