Improving the Health of Patients and Populations Requires Humility, Uncertainty, and Collaboration

Abstract

IN 2003 SMITH AND PELL 1 PUBLISHED A WITTY SYSTEMatic review of parachute use to prevent death and major trauma. They did not find any randomized comparisons of using vs not using a parachute when jumping from a height greater than 100 m and concluded that “everyone might benefit if the most radical protagonists of evidence-based medicine organized and participated in a double blind, randomized, placebo controlled, cross-over trial of the parachute.” In this issue of JAMA Pereira and colleagues report on an analysis of 3082 Cochrane reviews that included a total of 85 002 forest plots of meta-analyses. They found only 1 intervention with more than a 5-fold reduction in the odds of dying (extracorporeal membrane oxygenation [ECMO] compared with conventional ventilatory support for severe respiratory failure in newborns). Although this is, by their definition, a “very large effect,” the reduction in the risk (as opposed to the odds) of dying before discharge home was only a halving (risk ratio, 0.44; 95% CI, 0.31 to 0.61). This corresponds to a risk difference of −32% (95% CI, −44% to −20%), which means that 3 babies would need to be treated with ECMO rather than conventional ventilation to prevent 1 death. By comparison with other health care interventions, ECMO has a large effect. However, it is nowhere near as large as the effect of using a parachute compared with not using a parachute when jumping from an airplane. Parachute use is likely to reduce the risk of death or major trauma by more than a 1000-fold under normal conditions (assuming a risk of death or major trauma of less than 1 per 1000 with a parachute compared with a risk of greater than 999 per 1000 without a parachute). It is reassuring that Pereira and colleagues did not find randomized trials of interventions that are as effective as parachutes. Such trials would be unethical. However, very few health care interventions have such dramatic effects. Glasziou et al identified 16 examples of interventions with effects so dramatic that they concluded randomized trials are not needed. They suggested that 10-fold reductions in risk are highly likely to reflect real intervention effects, even if confounding factors associated with the intervention may have contributed to the size of the observed associations. Large effects, such as the 50% relative risk reduction with neonatal ECMO, are more common than really dramatic effects. Unlike 10-fold effects, 2-fold effects may be difficult to detect. For example, there was a “vociferous, if not vitriolic” debate about which anticonvulsant to use to treat women with eclampsia with no compelling evidence until the Collaborative Eclampsia Trial found that women given magnesium sulfate had a 2to 3-fold lower risk of a recurrent convulsion compared with diazepam or phenytoin (diazepam: risk ratio, 0.48; 95% CI, 0.36-0.63 vs phenytoin: risk ratio, 0.33; 95% CI, 0.21-0.53). Large effects may also be spurious, as documented by Pereira and colleagues because they found a large number of randomized trials of interventions that had apparently large but uncertain effects that were not confirmed. The main reason that most of the 20 573 examples of an odds ratio greater than 5 that the authors found were likely to be spurious is that the vast majority were from small trials with few events. In addition, many of the large odds ratios were for surrogate outcomes: the importance of those effects for patients is uncertain, even when the large odds ratios were confirmed. Less than 10% of index odds ratios greater than 5 were confirmed. Possible reasons that most large or very large effects were not confirmed by subsequent trials, in addition to insufficient sample sizes, include the risk of bias, a reluctance to undertake further trials, and true heterogeneity due, for example, to more intense interventions or more responsive patients in the index trials compared with subsequent trials. The findings of Pereira and colleagues may not be representative of the research literature. They are based on more than 225 000 measures of effect found in 3545 systematic reviews. However, that number of reviews is likely to be, at most, one-third of the number of reviews needed to reliably summarize what is known about the effects of health