Comparison of six statistical methods for interrupted time series studies: empirical evaluation of 190 published series

Simon L Turner,Jeremy M Grimshaw,Joanne E Mckenzie,Andrew B Forbes,Monica Taljaard,Amalia Karahalios

doi:10.1186/s12874-021-01306-w

Simon L Turner, Jeremy M Grimshaw + Show 4 more

Open Access

https://doi.org/10.1186/s12874-021-01306-w

Copy DOI

Abstract

BackgroundThe Interrupted Time Series (ITS) is a quasi-experimental design commonly used in public health to evaluate the impact of interventions or exposures. Multiple statistical methods are available to analyse data from ITS studies, but no empirical investigation has examined how the different methods compare when applied to real-world datasets.MethodsA random sample of 200 ITS studies identified in a previous methods review were included. Time series data from each of these studies was sought. Each dataset was re-analysed using six statistical methods. Point and confidence interval estimates for level and slope changes, standard errors, p-values and estimates of autocorrelation were compared between methods.ResultsFrom the 200 ITS studies, including 230 time series, 190 datasets were obtained. We found that the choice of statistical method can importantly affect the level and slope change point estimates, their standard errors, width of confidence intervals and p-values. Statistical significance (categorised at the 5% level) often differed across the pairwise comparisons of methods, ranging from 4 to 25% disagreement. Estimates of autocorrelation differed depending on the method used and the length of the series.ConclusionsThe choice of statistical method in ITS studies can lead to substantially different conclusions about the impact of the interruption. Pre-specification of the statistical method is encouraged, and naive conclusions based on statistical significance should be avoided.

Highlights

The Interrupted Time Series (ITS) is a quasi-experimental design commonly used in public health to evaluate the impact of interventions or exposures
We collated datasets that were reported in the published paper or its supplement
Time series dataset acquisition Of the 230 ITS identified in the review [10] we obtained 10/230 (4%) datasets directly from the publication, 50/230 (22%) through email contact with the authors, and 184/230 (80%) through digital data extraction

Summary

Introduction

The Interrupted Time Series (ITS) is a quasi-experimental design commonly used in public health to evaluate the impact of interventions or exposures. Randomised trials are the gold standard design for investigating the impact of public health interventions, they cannot always be used. Differences between the counterfactual and observed data at various points post interruption can be estimated (e.g. immediate and longterm effects), having accounted for the underlying secular trend. A characteristic of data collected over time is that the data points tend to be correlated [5]. Autocorrelation may be observed between consecutive data points or over longer periods of time (e.g. seasonal effects). This characteristic of the data needs to be considered when designing and analysing ITS studies. If positive autocorrelation is present, larger sample sizes are required to provide power at the desired level [6] and if autocorrelation is not accounted for in the statistical analysis, standard errors may be underestimated [7]

Objectives

Methods

Results

Conclusion