A Note on Testing for Intervention Effects on Binary Responses

Abstract

In some circumstances controlled trials are not feasible and treatments can only be evaluated using clinical databases. Here we consider the situation where treatment is introduced at a particular calendar time and can only be evaluated by comparison with historical controls. In these circumstances Heuer and Abel recommended using change-point methods to search for change in characteristics over the whole study period rather than simply comparing treated and untreated patients. Their recommendation is to only conclude that the intervention had an effect if a change-point could be demonstrated close in time to the introduction of the new treatment. This reduces the risk of false positives caused by confounding changes in population characteristics or changes in patient management. For binary data we develop a method that follows their philosophy and apply it to an observational study in the treatment of pin sites after orthopaedic surgery. Tests for change in binomial probabilities based on Brownian bridge and Hansen's approximation for maximally selected chi(2) statistics are compared to an exact test by Worsley. The approximate method is generalized to logistic regression models allowing for covariates. The agreement of the exact and approximate method is good for sample sizes of 100 or more. The actual test size of the Hansen approximate test allowing for covariates is close to the nominal level, whereas the Brownian bridge approximation is slightly conservative. The change in pin site treatment significantly reduces the risk of infection for both adults and children. We consider the Hansen approximation to provide a very good and very simple method for obtaining the p-value when testing for a change in binary data event probabilities, with or without covariates.