Using Case Studies to Test and Refine Regressions

Abstract

The last chapter talked about the contributions regression can make to causal inference, as well as about the fact that too much is sometimes expected from regressions. Regression analysis cannot by itself resolve key issues of causal inference. Instead, causal inferences that use regression always depend centrally on additional information that comes from some other source. Such information may involve knowledge that the treatment was experimentally designed, evidence that a credible natural experiment took place, insights that specify a complete and causally justified set of control variables, and so forth. Regardless of the specific form of additional information used to justify using a regression for causal inference, the central point remains: regression can never be a stand-alone basis for causal inference. Instead, it is an incredibly useful tool for summarizing data – and by extension a powerful way of simplifying and quantifying the results of causal inferences that derive their justifications from some other source. This reframing of the role of regression in justifying causal inference resets the parameters of the debate over the role that qualitative analysis can play in multi-method research. Because regression analysis cannot itself settle causal questions, there is no point in asking whether there is any meaningful role left for case-study work in the wake of a regression. Instead, the question we must face is this: can qualitative research bridge the gap between the descriptive statistical summaries provided by regression and the complex counterfactual knowledge necessary for causal inference? On rare occasions, it may be true that qualitative research can provide the bolstering information necessary to clinch a causal argument, in conjunction with regression analysis. Usually, however, case-study analysis will fall short; causal inference can simply require too much knowledge that is difficult to acquire in an observational context. Nonetheless, qualitative research when skillfully combined with regression analysis can identify the next step forward toward successful causal inference. In this chapter, I characterize four essential contributions that qualitative research can make to causal inferences that depend centrally on regression analysis. First, I discuss the value of case studies that are fortunate enough to turn up evidence, usually involving interrupted time series, that helps bound or identify the causal counterfactual for one or a few cases.