Abstract
Abstract In some applications, researchers using the synthetic control method (SCM) to evaluate the effect of a policy may struggle to determine whether they have identified a “good match” between the control group and treated group. In this paper, we demonstrate the utility of the mean and maximum Absolute Standardized Mean Difference (ASMD) as a test of balance between a synthetic control unit and treated unit, and provide guidance on what constitutes a poor fit when using a synthetic control. We explore and compare other potential metrics using a simulation study. We provide an application of our proposed balance metric to the 2013 Los Angeles (LA) Firearm Study [9]. Using Uniform Crime Report data, we apply the SCM to obtain a counterfactual for the LA firearm-related crime rate based on a weighted combination of control units in a donor pool of cities. We use this counterfactual to estimate the effect of the LA Firearm Study intervention and explore the impact of changing the donor pool and pre-intervention duration period on resulting matches and estimated effects. We demonstrate how decision-making about the quality of a synthetic control can be improved by using ASMD. The mean and max ASMD clearly differentiate between poor matches and good matches. Researchers need better guidance on what is a meaningful imbalance between synthetic control and treated groups. In addition to the use of gap plots, the proposed balance metric can provide an objective way of determining fit.
Highlights
Researchers using the synthetic control method (SCM) to evaluate the effect of a strategy may struggle to determine whether they have identified a “good match” between the control group and treated group
Given the construction of the metrics, the units of the metrics are not directly comparable – while the Absolute Standardized Mean Difference (ASMD) metrics are on an effect size scale, the Ben-Michael estimated bias and RMSE are relative to the scale of the outcome, and SRMSE is on the scale of a root mean squared effect size
We present metrics to help analysts determine whether they have a good enough match to conduct causal inference using SCM
Summary
Researchers using the synthetic control method (SCM) to evaluate the effect of a strategy may struggle to determine whether they have identified a “good match” between the control group and treated group. It is recommended that researchers not use the SCM for causal inference, but there is little guidance about how to determine whether the estimated synthetic control is satisfactory [1]. While the SCM has proven valuable in empirical crime research [2–6, e.g.,], we focus on an example in which it is unclear whether the SCM provides a useful counterfactual. In 2005, an interagency working group of California law enforcement officials and crime researchers entered a partnership to design interventions to reduce gun violence in Los Angeles (LA).
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
