The impact of treatment allocation procedures on nominal significance levels and bias

Abstract

Complete randomization, the simplest method for allocating treatments to patients in clinical trials, can serve as a basis for inferential procedures using standard permutation tests, because the method ensures that each sequence of allocations is equally likely. No other method of allocation possesses this property. However, many clinical trials employ allocation methods that force balance of covariates across treatment groups. With these methods, some allocation sequences are impossible or highly unlikely so that standard permutation tests are technically invalidated. In this article we investigate whether standard permutation tests for binary outcomes are likely to yield distorted nominal p values in practical applications of these alternative allocation methods. A sample of completed trials conducted by the Eastern Cooperative Oncology Group serves as a basis on which to construct simulations. Our results indicate that nominal p values can be conservative, but are not likely to be severely distorted if the analysis is stratified by important covariates used as allocation prompts. Moreover the inherent conservativeness of exact methods due to discreteness tends to dominate any additional conservativeness due to nonrandom designs. In addition, we investigate the relationship of treatment allocation methods with bias in estimates from a logistic model when important covariates are unknown. This bias is the same for all asymptotically balanced allocation methods and is significant but not disastrous.