The Need for Large-Scale Randomized Evidence Without Undue Emphasis on Small Trials, Meta-analyses, or Subgroup Analyses

Abstract

RANDOMIZED TRIALS OF ADEQUATE SIZE AND DURATION designed to test a priori hypotheses represent the most reliable design strategy to detect the most realistically small to moderate therapeutic effects of drugs. Such trials should achieve high adherence to an adequate dose of the drug and a sufficient number of clinical end points to distinguish reliably between the null hypothesis and the most plausible alternative hypothesis of small to moderate benefit or harm. With regard to the development of drugs to treat diabetes mellitus, the US Food and Drug Administration (FDA) has developed guidance for industry that somewhat overemphasizes results from meta-analyses of phase 2 trials that were not large enough to test realistic hypotheses about clinical cardiovascular (CV)events. Even inaggregate, suchresults should be considered more as hypothesis formulating than as hypothesis testing. The main need now is for trials that are large enough to have adequate statistical power. The quality and usefulness of any meta-analysis are dependent on the quality and comparability of data from the component trials. In particular, the trials combined should have high adherence and follow-up rates and should have reasonably comparable drugs, doses, and outcomes. The characteristics of the participants and the magnitude of effect from each trial must be sufficiently similar so that their combination will not produce a distorted estimate. Thus, metaanalyses can reduce the role of chance in the interpretation but may introduce bias and confounding. For example, a meta-analysis of rosiglitazone involved 42 randomized trials with a total of 26 011 patients who experienced 158 myocardial infarctions (MIs) and 61 deaths due to CV causes (CV death). The investigators concluded that rosiglitazone was associated with a significant increase in risk of MI (relative risk [RR],1.43; 95% confidence interval [CI], 1.03-1.98) as well as a nonsignificant increase in CV death (RR,1.64; 95% CI, 0.98-2.74). The chief value of this report should have been to formulate a hypothesis about one possible hazard of rosiglitazone that may offset any potential benefits. Furthermore, the widths of the CIs suggest that this meta-analysis was unable to distinguish reliably whether rosiglitazone conferred no hazard or a substantial hazard of CV events. Hypothesis formulation should lead to adequate hypothesis testing in another randomized trial of adequate size and duration designed a priori to address the question. The Rosiglitazone Evaluated for Cardiovascular Outcomes and Regulation of Glycaemia in Diabetes (RECORD) trial was designed a priori to assess the noninferiority of rosiglitazone added to metformin or sulfonylurea compared with dual therapy metformin and sulfonylurea on reducing CV events among 4447 patients with type 2 diabetes. The primary prespecified outcome was time to first CV hospitalization or CV death with a hazard ratio (HR) noninferiority margin of 1.20, which is the upper bound of the 95% CI as recommended by the FDA. During a mean 5.5-year follow up, there were 321 incident primary clinical events in the rosiglitazone group and 323 in the active comparator group (HR, 0.99; 95% CI, 0.85-1.16), meeting the criterion for noninferiority. Thus, the results of the large-scale trial did not support the hypothesis formulated from the meta-analysis of smaller trials. Likewise, in a meta-analysis of 7 small trials evaluating use of intravenous magnesium during suspected acute MI, there were 25 deaths among 657 patients in the magnesium group vs 53 deaths among 644 patients in the placebo group (HR, 0.45; 95% CI, 0.15-0.74). Although the existing totality of evidence was compatible with the possibility that intravenous magnesium was an effective, safe, and inexpensive intervention, a prudent approach was to await the results of the large fourth International Study of Infarct Survival (ISIS-4) trial before routinely using this therapy. In ISIS-4, 58 050 patients with suspected MI were randomized to receive either intravenous magnesium or usual care. Patients treated with magnesium had a nonsignificant 6% increase in mortality, as well as significant increases in heart failure, death attributable to cardiogenic shock, and bradycardia. In subgroup analyses (which are useful to formulate but not test hypotheses) no significant differences were found among patients treated less than 6 hours after the onset of symptoms, those who received intravenous magnesium within 2 hours after thrombolytic therapy, and those who received neither thrombolytic therapy