Disentangling secular trends and policy impacts in health studies: use of interrupted time series analysis

Abstract

Although randomized control trials (RCTs) are the ‘gold standard’ to evaluate treatment effects in health care, they are frequently not practical, ethical or politically acceptable in the evaluation of many health system or public health interventions. A good examplewhere a health system intervention has undergone evaluation using an RCT design is the universal health insurance scheme, Seguro Popular, in Mexico. However, this is a rare exception mainly due to the academic background of the Mexican Health Minister, Julio Frenk, who introduced the scheme. More frequently, randomization is not feasible or practical, particularly when interventions target whole or large subgroups of populations. Because of political considerations, policy makers often want to implement changes quickly and refuse to wait several years to determine a new intervention’s effects. Further, they may be reluctant to be seen to withhold an intervention from a particular community, as was the case with the SureStart programme, which aimed to improve health and educational outcomes in young children in the UK. RCTs may also be unethical where clear evidence of benefit has been demonstrated from observational studies, as was the case with cervical cancer screening. Additionally, lack of funding often poses a hurdle to formal evaluation through an RCT, as RCTs can be very costly to carry out. In the absence of an RCT, evaluations often use quasi-experimental designs such as a pre-post study design with measurements before and after the intervention period. Figure 1, panel A shows an introductory example with an outcome measure subject to a secular time trend and an intervention without any impact. The standard approach to detect a significant impact would apply a t-test to compare the means of the preintervention phase with the post-intervention data. However, a t-test does not consider time but simply separates the data into two groups. In the example in panel A of Figure 1, the t-test would obtain a significant p-value although the difference is not due to the intervention but rather captures the secular time trend. In settings with a secular trend, a t-test (or another statistical test) is prone to false positives, as illustrated in the example, and false negatives in case of a negative secular trend. An interrupted time series (ITS) in contrast, adjusts for secular time trends and should be used instead. Panel B of Figure 1 contains a new data set with an intervention affecting the outcome. An ITS is a segmental linear regression model; preintervention and post-intervention are each modeled as a linear regression. Based on an ITS, a secular trend can therefore be captured in the regression line as shown in panel B of Figure 1. An ITS compares the intercept and slope of the regression line before the intervention with the intercept and slope after intervention. A one-time baseline effect of the intervention without influencing the secular trend can be detected as an intercept change. If the intervention changed the secular trend, there will also be a significant difference in the slope between the two periods. Later, we will see that an ITS also comprises more flexible models. Instead of using an ITS design, many studies use preand post-intervention groups without modeling the secular time trend. Examples include evaluations of major interventions DECLARATIONS