Quantifying the share of total apparent effect that is genuinely due to a treatment: A comparison of methods

Abstract

Estimating true treatment effects in the presence of selection bias is a common problem in program evaluation, but it is uncommon to quantify the share of a total apparent effect that is “true”. We identify multiple methods for doing so, in the context of isolating the true causal influence of the built environment on travel behavior in the presence of residential self-selection (RSS). Among the different approaches for dealing with self-selection, we use three: statistical control (SC) modeling, propensity score-based techniques (PS), and sample selection (SS) modeling. Our objective is to identify multiple ways to estimate the proportion of the total apparent effect of the built environment on travel behavior that is due to the built environment itself, which we call the “built environment proportion”, or BEP. We present and evaluate three categories of methods for estimating this measure (each native to one of the approaches): variance explained, modular effects, and treatment effects. A BEP formula associated with a given method can be applied to its native approach or cross-applied to other approaches. We identify and enumerate 47 potential BEPs. The methods presented here can be applied in other contexts involving treatment evaluation in the presence of selection bias.