Commentary: The debate on non-inferiority trials: 'when meta-analysis alone is not helpful'

Abstract

In a typical clinical trial, two treatments are compared to determine which is better, or if both are the same. The design of the classic, parallel group randomized trial involves formulating a null hypothesis of no difference between interventions and identifying a clinically relevant difference ( ) that researchers do not wish to overlook on the primary end-point. These trials are refered as ‘superiority trials’ (STs) as investigators hope to reject the null hypothesis demonstrating a difference between interventions. In an ST, the type I error is falsely finding a treatment effect when there is none, and a type II error is failing to detect a treatment effect when truly one exists. In contrast, a non-inferiority trial (NIT) seeks to determine whether a new intervention is no worse than a reference intervention within a pre-specified non-inferiority margin (from – to 0)—that is, a clinically irrelevant difference—for the primary outcome. The null hypothesis under which an NIT is designed is that the experimental intervention is worse than the standard treatment and that the absence of a relevant difference can be demonstrated by rejecting it. In NITs, the null and alternative hypotheses are reversed compared with STs: a type I error is the erroneous acceptance of an inferior new treatment, whereas a type II error is the erroneous rejection of a truly non-inferior treatment. It is the very nature of the NIT design that makes it susceptible to bias and misuse unless (i) the research question has a strong rationale; (ii) the effectiveness of the standard treatment is solid and (iii) the end point(s) on which the has been chosen for assessing non-inferiority are appropriate. Recognizing that NIT may—under specified circumstances—be useful does not mean that difficulties in design, conduct, analysis and interpretation can be overlooked, especially as NITs can be (mis) used to study new marketable products with questionable, or no, innovation, producing results only to obtain regulatory authority approval. The a priori concern, together with empirical evidence of NITs’ inappropriate use, fuels the current debate between those supporting NIT and those detracting it on both pragmatic and ethical grounds. The paper by Soonowala et al. published in this issue of the Journal should be read against this background. The paper is stated to be an attempt to ‘to address the concerns by performing a meta-analysis of non inferiority trials to see whether the systematic use of too large non inferiority margins or systematic bias in designs, conduct or reporting skewed the overall results’. Authors searched relevant NITs across a variety of clinical questions and pooled data to see whether the suspicion of a systematic bias could, or could not, be confirmed. They conclude that ‘the experimental treatments that gain the verdict of non inferiority in published trials do not appear to be systematically less effective than the standard treatments’, and then go a step further stating that ‘the findings are reassuring considering the criticism that has been levelled at non-inferiority trials’. Do the data support these conclusions? Hardly so, and I will now briefly discuss why. The statistical methodologies used by the authors are appropriate and rigorous. However, the results of the paper, and even more its implications, are not easy to interpret and readers should consider whether (i) the study addressed the crucial questions about NTIs; (ii) the findings provide a better understanding of the issues and (iii) the results provide clear indications of where to go next. I believe that Soonowala et al.’s study has, in the above respects, important limitations that the authors largely acknowledge. Among the most important are (i) the search strategy was far from comprehensive; (ii) publication bias cannot be ruled out and may have led to the failure to identify some relevant NITs and (iii) the design of the NIT, in particular the clinical rationale for choice of the non-inferiority Published by Oxford University Press on behalf of the International Epidemiological Association