The qualitative method of impact analysis

Abstract

Consider the qualitative approach to evaluation design (as opposed to measurement) to be typified by a case study with a sample of just one. Although there have certainly been elaborate and emphatic defenses of the qualitative approach to program evaluation, such defenses rarely attempt to qualify the approach explicitly and rigorously as a method of impact analysis. The present paper makes that attempt. The problem with seeking to advance a qualitative method of impact analysis is that impact is a matter of causation and a non-quantitative approach to design is apparently not well suited to the task of establishing causal relations. The root of the difficulty is located in the counterfactual definition of causality, which is our only broadly accepted formal definition of causality for social science. It is not, however, the only definition we use informally. Another definition, labeled “physical causality,” is widely used in practice and has recently been formalized. Physical causality can be applied to the present problem. For example, it explains the persuasiveness of Scriven's “Modus Operandi” approach to causal inference in evaluation. Under this conceptualization, a tailored case study design with a sample size of one becomes in principle as strong a basis for making inferences about program impact as a randomized experiment. Crucial for the application of the method to program evaluation is the finding that people's “operative reasons” for doing what they do are the physical causes of their actions. Lastly, it is shown that external validity using this qualitative approach would have exceptional strengths.