Evaluating the Effects of a Mandatory Government Program using Matched Groups within a Similar Geographic Location

Abstract

Program evaluators have recently turned their attention to matching methods in their search for a solution to the evaluation problem. Three nearest neighbour matching estimators are considered in this paper and compared with experimental benchmark results, following the literature initiated by Lalonde (1986). The first estimator is based on propensity score nearest neighbour matching and estimates the propensity score using a logit model. The second estimator estimates the propensity score using boosted regression, a non-parametric approach borrowed from the machine learning literature which can automatically model non-linearities between covariates and the propensity score. The third estimator is a covariate matching estimator using a distance metric suggested by Abadie and Imbens (2004, 2006). In addition, taking advantage of a statewide implementation of welfare reform, in-state comparison groups of varying sizes are created to determine the importance of the unweighted treatment to comparison group ratio in matching with replacement, and the importance of adjusting for aggregate geographic differences using county level data. Overall, no estimator was found to be dominant and regression adjustment using county level variables produced mixed results. However, one clear finding is that having a smaller treatment to comparison group ratio after matching and a larger overall sample size can help one's chances of obtaining unbiased matching estimates.