Evaluation of statistical methods used in the analysis of interrupted time series studies: a simulation study

Simon L Turner,Amalia Karahalios,Andrew B Forbes,Joanne E Mckenzie,Monica Taljaard

doi:10.1186/s12874-021-01364-0

Simon L Turner, Amalia Karahalios + Show 3 more

Open Access

https://doi.org/10.1186/s12874-021-01364-0

Copy DOI

Abstract

BackgroundInterrupted time series (ITS) studies are frequently used to evaluate the effects of population-level interventions or exposures. However, examination of the performance of statistical methods for this design has received relatively little attention.MethodsWe simulated continuous data to compare the performance of a set of statistical methods under a range of scenarios which included different level and slope changes, varying lengths of series and magnitudes of lag-1 autocorrelation. We also examined the performance of the Durbin-Watson (DW) test for detecting autocorrelation.ResultsAll methods yielded unbiased estimates of the level and slope changes over all scenarios. The magnitude of autocorrelation was underestimated by all methods, however, restricted maximum likelihood (REML) yielded the least biased estimates. Underestimation of autocorrelation led to standard errors that were too small and coverage less than the nominal 95%. All methods performed better with longer time series, except for ordinary least squares (OLS) in the presence of autocorrelation and Newey-West for high values of autocorrelation. The DW test for the presence of autocorrelation performed poorly except for long series and large autocorrelation.ConclusionsFrom the methods evaluated, OLS was the preferred method in series with fewer than 12 points, while in longer series, REML was preferred. The DW test should not be relied upon to detect autocorrelation, except when the series is long. Care is needed when interpreting results from all methods, given confidence intervals will generally be too narrow. Further research is required to develop better performing methods for ITS, especially for short series.

Highlights

Interrupted time series (ITS) studies are frequently used to evaluate the effects of population-level interventions or exposures
Standard errors of level and slope change estimates Empirical standard errors Figure 3 and Supplementary Fig. S2 visually indicate the precision of the estimators in terms of the spread of the distributions therein
An exception to this occurred for the ordinary least squares (OLS) estimator which exhibited unusual behaviour for an autocorrelation of 0.8, with the Standard error (SE) initially increasing with an increasing number of points in the series, and decreasing

Summary

Introduction

Interrupted time series (ITS) studies are frequently used to evaluate the effects of population-level interventions or exposures. Interrupted time series (ITS) studies are frequently used to evaluate the impact of interventions or exposures that occur at a particular point in time [1,2,3,4]. A key feature of the design is that data from the pre-interruption interval can be used to estimate the underlying secular trend When this trend is modelled correctly, it can be projected into the postinterruption interval, providing a counterfactual for what would have occurred in the absence of the interruption. From this counterfactual, a range of effect measures can be constructed that characterise the impact of the interruption. Two commonly used measures include the ‘change in level’ – which represents the sustained change immediately after the interruption, and the ‘change in slope’ – which represents the difference in trends before and after the interruption

Methods

Results

Discussion

Conclusion