Use of synthetic control methodology for evaluating public health interventions: a literature review

Abstract

BackgroundThe synthetic control method (SCM) improves causal inference in non-randomised studies by building a counterfactual using a weighted combination of potential control units. Widely used in other disciplines, it is not widely used in public health research. Our objectives were to identify the use of SCM studies in health and to summarise strengths and limitations identified in the literature. MethodsWe included studies that used a SCM design to investigate a public health outcome of any intervention in any population. We searched for the term “synthetic control method” in 26 health, social science, and grey literature databases, and looked for additional studies by key authors. No restrictions were placed on language or date. Searches were completed by Feb 16, 2016. We summarised key information about the studies including setting, number of treated and control units, intervention and outcome, number of pre-intervention and post-intervention data points available, and other methods used in the same study. FindingsSearches identified 35 health-related studies of which 23 were from US settings and investigated one treated unit. Most studies had at least ten control units. Interventions that were investigated included health finance and health systems reform, industry regulation, and taxation policies. Common outcomes were mortality rates, and insurance rates and health-care access. Most studies had more than four pre-implementation and post-implementation data points. SCM was most commonly used alongside difference-in-difference methods. Advantages were that it does not rely on parallel pre-implementation trends and that it allows for time-varying unmeasured confounders. Limitations included the need for suitable data on both the treated unit and a pool of potential controls, difficulties if the treated unit is an outlier, and the inapplicability of traditional statistical tests because of the small number of treated and control units and because they have not been randomly allocated. Falsification tests were generally used as an alternative. InterpretationThis comprehensive literature review suggests that SCM has been little used in public health despite some advantages over existing methods. Future research incorporating the method, ideally in combination with other methods, would be of value. FundingFP and PC are funded by the UK Medical Research Council (MC_UU_12017/13, MC_UU_12017/15) and the Scottish Government Chief Scientist Office (SPHSU13 and SPHSU15)