Meta-analysis of randomised controlled trials

Abstract

Your editorial,1Editorial. Meta-analysis under scrutiny.Lancet. 1997; 350: 675Summary Full Text Full Text PDF PubMed Scopus (38) Google Scholar inspired by LeLorier and colleagues'2LeLorier J Gregoire G Benhaddad A Lapierre J Derderian F Discrepancies between meta-analyses and subsequent large randomised controlled trials.N Engl J Med. 1997; 337: 536-542Crossref PubMed Scopus (990) Google Scholar report of serious discrepancies between meta-analyses of small trials and subsequent large trials, raised the question of whether meta-analyses can be trusted. Meta-analysis cannot be trusted when carried out mechanically and with no broader understanding of the issues under examination. For example, LeLorier and colleagues consider that meta-analyses of the influence of cholesterol lowering drugs on mortality were not supported by the outcome of later definitive studies. The drugs used in the earlier trials produced only small reductions in cholesterol compared with the substantial reductions produced by the statins. Meta-analyses had shown that the treatment effect was significantly related to the percentage reduction in cholesterol concentration3Davey G Smith Song F Sheldon T Cholesterol lowering and mortality:the importance of considering initial level of risk.BJM. 1993; 306: 1367-1373Crossref PubMed Scopus (361) Google Scholar and therefore one would not predict that the same effect would be seen in the statin trials as were seen in earlier trials. Do the earlier trials provide a basis for predicting the outcome of a current treatment regimen? There is an approach to the data included in a meta-analyses which can help here: inspection and statistical analysis of the funnel plot.4Egger M Davey Smith G Schneider M Minder C Bias in meta-analysis detected by a simple, graphical test.BMJ. 1997; 315: 629-634Crossref PubMed Scopus (32351) Google Scholar This plot allows examination of the association between the outcomes seen in trials (often odds ratios) and the statistical information (precision) contained within the trial, which is closely related to sample size. If an association is seen, with smaller trials producing larger beneficial effects, then the plot becomes asymmetrical and the meta-analyses may be seriously biased. The funnel plot of intravenous magnesium in the treatment of myocardial infarction is shown in the figure along with the meta-analysis and the latter large ISIS-4 trial, which failed to demonstrate the benefit seen in meta-analyses. The funnel plot is clearly asymmetrical, and a statistical test for asymmetry that we have developed4Egger M Davey Smith G Schneider M Minder C Bias in meta-analysis detected by a simple, graphical test.BMJ. 1997; 315: 629-634Crossref PubMed Scopus (32351) Google Scholar demonstrates significant (p=0·005) asymmetry. Conversely, in the case of streptokinase the funnel plot of trials published before the appearance of the large GISSI-1 and ISIS-2 trial was clearly symmetrical, with no statistical evidence of asymmetry in formal analysis. In this case the outcome of the large trials was almost identical to the result of the meta-analysis. There are several possible causes of asymmetry in funnel plots,4Egger M Davey Smith G Schneider M Minder C Bias in meta-analysis detected by a simple, graphical test.BMJ. 1997; 315: 629-634Crossref PubMed Scopus (32351) Google Scholar including publication bias, location bias due to negative findings being preferentially published in non-English language journals or receiving fewer citations than positive trials, and data irregularities. True heterogeneity may also lead to funnel plot asymmetry, if the size of the effect really differs according to sample size because, for example, of a greater intensity of intervention occurring in smaller trials or smaller trials being done in patients at higher initial level of risk, who receive greater benefit. In all these cases of asymmetry the pooled effect from a meta-analysis will be misleading, with the degree of asymmetry indicating the likelihood that bias is substantial. We suggest that funnel plots and formal statistical testing for asymmetry is routinely included in the performance and the reporting of meta-analyses.