A comparison of synthetic control approaches for the evaluation of policy interventions using observational data: Evaluating the impact of redesigning urgent and emergency care in Northumberland.

Abstract

To compare the original synthetic control (OSC) method with alternative approaches (Generalized [GSC], Micro [MSC], and Bayesian [BSC] synthetic control methods) and re-evaluate the impact of a significant restructuring of urgent and emergency care in Northeast England, which included the opening of the UK's first purpose-built specialist emergency care hospital. Simulations and data from Secondary Uses Service data, a single comprehensive repository for patient-level health care data in England. Hospital use of individuals exposed and unexposed to the restructuring is compared. We estimate the impact using OSC, MSC, BSC, and GSC applied at the general practice level. We contrast the estimation methods' performance in a Monte Carlo simulation study. Hospital activity data from Secondary Uses Service for patients aged over 18 years registered at a general practice in England from April 2011 to March 2019. None of the methods dominated all simulation scenarios. GSC was generally preferred. In contrast to an earlier evaluation that used OSC, GSC reported a smaller impact of the opening of the hospital on Accident and Emergency (A&E) department (also known as emergency department or casualty) visits and no evidence for any impact on the proportion of A&E patients seen within 4 h. The simulation study highlights cases where the considered methods may lead to biased estimates in health policy evaluations. GSC was found to be the most reliable method of those considered. Considering more disaggregated data over a longer time span and applying GSC indicates that the specialist emergency care hospitals in Northumbria had less impact on A&E visits and waiting times than suggested by the original evaluation which applied OSC to more aggregated data.