A Randomization-Based Theory for Preliminary Testing of Covariate Balance in Controlled Trials

Abstract

Randomized trials balance all covariates on average and are the gold standard for estimating treatment effects. Chance imbalances nevertheless exist more or less in realized treatment allocations and intrigue an important question: what should we do if the treatment groups differ with respect to some important baseline characteristics? A common strategy is to conduct a preliminary test of the balance of baseline covariates after randomization, and invoke covariate adjustment for subsequent inference if and only if the realized allocation fails some prespecified criterion. Although such practice is intuitive and popular among practitioners, the existing literature has so far only evaluated its properties under strong parametric model assumptions in theory and simulation, yielding results of limited generality. To fill this gap, we examine two strategies for conducting preliminary test-based covariate adjustment by regression, and evaluate the validity and efficiency of the resulting inferences from the randomization-based perspective. The main result is 2-fold. First, the preliminary-test estimator based on the analysis of covariance can be even less efficient than the unadjusted difference in means, and risks anticonservative confidence intervals based on normal approximation even with the robust standard error. Second, the preliminary-test estimator based on the fully interacted specification is less efficient than its counterpart under the always-adjust strategy, and yields overconservative confidence intervals based on normal approximation. In addition, although the Fisher randomization test is still finite-sample exact for testing the sharp null hypothesis of no treatment effect on any individual, it is no longer valid for testing the weak null hypothesis of zero average treatment effect in large samples even with properly studentized test statistics. These undesirable properties are due to the asymptotic non-normality of the preliminary-test estimators. Based on theory and simulation, we echo the existing literature and do not recommend the preliminary-test procedure for covariate adjustment in randomized trials.