Resurrecting the Third Variable: A Critique of Pearl's Causal Analysis of Simpson's Paradox

Abstract

Pearl argued that Simpson's Paradox would not be considered paradoxical but for statisticians’ unwillingness to acknowledge the role of causality in resolving an instance of it. He proposed using a causal calculus to determine which set of contradictory findings in an instance of the paradox should be accepted—the aggregated data or the data disaggregated by conditioning on the third variable. Pearl used the example of a hypothetical quasi-experiment to argue that when third variables are not causal, one should not condition on them, and—assuming no other sources of confounding—the aggregated data should be accepted. Pearl was precipitate in his argument that it would be inappropriate to condition on the noncausal third variables in the example. Whether causal or not, third variables can convey critical information about a first-order relationship, study design, and previously unobserved variables. Any conditioning on a nontrivial third variable that produces Simpson's Paradox should be carefully examined before either the aggregated or the disaggregated findings are accepted, regardless of whether the third variable is thought to be causal. In some cases, neither set of data is trustworthy; in others, both convey information of value. Pearl's hypothetical example is used to illustrate this argument.