Taking Causal Heterogeneity Seriously: Implications for Case Choice and Case Study-Based Generalizations

Abstract

In mixed methods approaches, statistical models are used to identify “nested” cases for intensive, small- n investigation for a range of purposes, including notably the examination of causal mechanisms. This article shows that under a commonsense interpretation of causal effects, large- n models allow no reliable conclusions about effect sizes in individual cases—even if we choose “onlier” cases as is usually suggested. Contrary to established practice, we show that choosing “reinforcing” outlier cases—where outcomes are stronger than predicted in the statistical model—is appropriate for testing preexisting hypotheses on causal mechanisms, as this reduces the risk of false negatives. When investigating mechanisms inductively, researchers face a choice between “onlier” and reinforcing outlier cases that represents a trade-off between false negatives and false positives. We demonstrate that the inferential power of nested research designs can be much increased through paired comparisons of cases. More generally, this article provides a new conceptual framework for understanding the limits to and conditions for causal generalization from case studies.