Are Statistically Non-Significant Findings Necessarily Negative? A Review of All Phase III Randomized Controlled Trials in Hematology Conducted by NCI Sponsored Cooperative Groups.

Abstract

Background: Randomized clinical trials (RCT) remain the primary mean for advancement of new treatments for the prevention and cure of hematological malignancies. However, if a RCT fails to show a significant difference between the treatment and the control group, is it because treatment is not effective (evidence of absence of treatment effect) or the results were inconclusive (absence of evidence of treatment effect).Methods: We identified 143 studies related to hematological malignancies out of 566 trials conducted by 6 NCI sponsored cooperative groups (RTOG, COG, GOG, ECOG, NSABP and NCCTG). Data were extracted from the original research articles and protocols. Quality of trials was high. To determine whether non-significant trial findings were true negatives or inconclusive, we compared the a priori effect size used in the power calculation for the trial's pre-defined primary outcome to the confidence intervals (CIs) of the non-significant trial results. Trial results were construed as true negative if the effect size and the 95%CIs were entirely outside the limits of equivalence and inconclusive if the 95%CIs crossed the line of no effect and one or both limits of equivalence (BMJ 2004; 328:476–477).Results: 71% of studies resulted in non significant outcome (102/143, missing data for 11 studies). Comparing the calculated CI to the a priori specified effect size 48% (44/91) of studies resulted in true negative results (evidence of no effect) and 52% (47/91) were inconclusive (no evidence of effect). Most of the studies (73%, 66/91) had conducted a power analysis before the commencement of trials. The range of expected effect size for experimental treatments in this set of trials ranged from as low as 9% to high as 200% (improvement in survival).Conclusion: High-quality RCTs conducted by the most prestigious institutions often result in inconclusive or negative findings. Unrealistic expectation in treatment effect size appears to be a major culprit for continuing conduct of underpowered trials.