Excessive false-positive errors in single-arm phase II trials: A simulation-based analysis

Abstract

6512 Background: The use of randomized phase II designs with an experimental arm and a standard-treatment control arm (R2PII) instead of a conventional single-arm design is clearly increasing in oncology. In practice, sample size, related cost issues, the belief that historical controls are adequate, and the use of a standard-treatment control arm in a phase II setting are frequently raised objections to R2PII trials. As the expense and complexity of definitive phase III trials increase, the ability of phase II trials to provide reliable and accurate results is critical. Methods: We investigated the ability of single arm vs R2PII trials to provide accurate conclusions by modeling variability in historical controls, patient outcome drifts independent of the tested therapy, and patient selection effects. Simulations compared R2PII and single-arm designs with binary endpoints under realistic parameters (e.g. alpha = beta = 0.10, historical control success rate = 20%, target success rate = 40%). Results: In the absence of variability in historical controls, estimated false positive and negative rates in both designs mirror the designated specifications. However, even in the presence of a modest drift effect in the population (mean 5% absolute shift in true control success rate), the false positive rate in single-arm designs is inflated two to three fold (to 20%-30%), while the R2PII retains the desired error rates. Greater confidence in historical controls in the single-arm design corrects only a small portion of the deviations. Increasing the sample size in each trial inflates the false positive error rate further to as much as 50%. Varying several sets of parameters gave similar results. Conclusions: In the presence of variability in historical controls, patient drift and/or selection effects, the false positive error rate of a single arm design is unacceptably high. In contrast, the R2PII design is reliable and robust despite the complexities in patient outcome drift and selection effects, and variability in historical control success rates. Given the rapid improvements in outcomes of many tumor types, the R2PII design should be the preferred method to evaluate novel agents in oncology in spite of the associated costs and the use of a reference control arm. No significant financial relationships to disclose.