When is a Match Sufficient? A Score-based Balance Metric for the Synthetic Control Method

Beth Ann Griffin,Priscillia Hunt,David Powell,Layla Parast

doi:10.1515/jci-2020-0013

Beth Ann Griffin, Priscillia Hunt

Open Access

https://doi.org/10.1515/jci-2020-0013

Copy DOI

Abstract

Abstract In some applications, researchers using the synthetic control method (SCM) to evaluate the effect of a policy may struggle to determine whether they have identified a “good match” between the control group and treated group. In this paper, we demonstrate the utility of the mean and maximum Absolute Standardized Mean Difference (ASMD) as a test of balance between a synthetic control unit and treated unit, and provide guidance on what constitutes a poor fit when using a synthetic control. We explore and compare other potential metrics using a simulation study. We provide an application of our proposed balance metric to the 2013 Los Angeles (LA) Firearm Study [9]. Using Uniform Crime Report data, we apply the SCM to obtain a counterfactual for the LA firearm-related crime rate based on a weighted combination of control units in a donor pool of cities. We use this counterfactual to estimate the effect of the LA Firearm Study intervention and explore the impact of changing the donor pool and pre-intervention duration period on resulting matches and estimated effects. We demonstrate how decision-making about the quality of a synthetic control can be improved by using ASMD. The mean and max ASMD clearly differentiate between poor matches and good matches. Researchers need better guidance on what is a meaningful imbalance between synthetic control and treated groups. In addition to the use of gap plots, the proposed balance metric can provide an objective way of determining fit.

Highlights

Researchers using the synthetic control method (SCM) to evaluate the effect of a strategy may struggle to determine whether they have identified a “good match” between the control group and treated group
Given the construction of the metrics, the units of the metrics are not directly comparable – while the Absolute Standardized Mean Difference (ASMD) metrics are on an effect size scale, the Ben-Michael estimated bias and RMSE are relative to the scale of the outcome, and SRMSE is on the scale of a root mean squared effect size
We present metrics to help analysts determine whether they have a good enough match to conduct causal inference using SCM

Summary

Introduction

Researchers using the synthetic control method (SCM) to evaluate the effect of a strategy may struggle to determine whether they have identified a “good match” between the control group and treated group. It is recommended that researchers not use the SCM for causal inference, but there is little guidance about how to determine whether the estimated synthetic control is satisfactory [1]. While the SCM has proven valuable in empirical crime research [2–6, e.g.,], we focus on an example in which it is unclear whether the SCM provides a useful counterfactual. In 2005, an interagency working group of California law enforcement officials and crime researchers entered a partnership to design interventions to reduce gun violence in Los Angeles (LA).

Methods

Results

Conclusion