The Meta-Analysis in Evidence-Based Medicine: High-Quality Research When Properly Performed

Abstract

The origins of evidence-based medicine (EBM) trace back to the mid-19th century1Freddi G Romàn-Pumar JL. Evidence-based medicine: What it can and cannot do.Ann Ist Super Sanita. 2011; 47: 22-25PubMed Google Scholar and the gradual evolution from personal journals to textbooks and eventually the advent of the peer-reviewed journal. Evidence-based medicine originally was defined by David Sackett as the “conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”2Sackett DL Rosenberg WM Gray JA et al.Evidence based medicine: What it is and what it isn't.BMJ. 1996; 312: 71-72Crossref PubMed Google Scholar Researchers from McMasters University began to introduce the term into the medical litrerature more systematically in the mid 1990s. Further analyzed, its aims include integrating individual clinical expertise (proficiency and judgment that individual clinicians acquire through clinical experience and practice) with the best available external clinical evidence from systematic research and patient's preferences to guide care.1Freddi G Romàn-Pumar JL. Evidence-based medicine: What it can and cannot do.Ann Ist Super Sanita. 2011; 47: 22-25PubMed Google Scholar Over the years, EBM has become the core guiding principle for those studying patient outcomes.3Masic I Miokovic M Muhamedagic B. Evidence based medicine – new approaches and challenges.Acta Inform Med. 2008; 16: 219-225Crossref PubMed Google Scholar The lower end of this paradigm includes unsystematic observations and narrative reviews, while at the opposite end lie methodologically sound, homogeneous, randomized clinical trials (RCTs). Systematic reviews and metaanalyses of homogeneous RCTS have been given the highest level of evidence. In its strictest definition, a metaanalysis (“analysis of analyses”) is a formal, epidemiologic, quantitative statistical technique that analyzes the results of different studies on the same topic and integrates the evidence into a single conclusion.4Lee YH. An overview of meta-analysis for clinicians.Korean J Intern Med. 2018; 33: 277-283Crossref PubMed Scopus (101) Google Scholar,5Moher D Cook DJ Eastwood S et al.Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement.Br J Surg. 2000; 87: 1448-1454Crossref PubMed Scopus (172) Google Scholar By combining individual studies, and thus using more data from a larger sample size, the precision and accuracy of the estimates in the individual studies can be improved upon—essentially improving the strength of the evidence. The key point is that a sound metaanalysis eliminates all bias by employing a strict methodologic/statistical approach and in the process provides evidence of the highest quality—with minimal heterogeneity. Additionally, if the individual studies were underpowered, combining them in a metaanalysis can increase the overall statistical power to detect an effect. When properly performed, meta-analyses can improve precision of effect estimates, generate clinically meaningful answers to questions that individual studies might not be able to accomplish, and even settle controversies from studies that may be conflicted.4Lee YH. An overview of meta-analysis for clinicians.Korean J Intern Med. 2018; 33: 277-283Crossref PubMed Scopus (101) Google Scholar The Cochrane Collaboration, an international organization, was developed to aid researchers generate systematic reviews of the highest quality and has become a key point of reference for researchers interested in producing and disseminating accurate data comparing treatment effectiveness.6Higgins JP Altman DG Gøtzsche PC et al.The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.BMJ. 2011; 343: d5928Crossref PubMed Scopus (18990) Google Scholar However, in the present day, the metaanalysis arguably has been abused, contributing to misleading conclusions rather than proper evidence-based information. The increasing popularity of metaanalyses can be traced to industrialized nations worldwide, most notably in China. According to PubMed data, 1.3% of publications by Chinese researchers in 2011 were metaanalyses, compared to 0.4% for authors from the rest of the world.7Ye XF Yu DH He J. The rise in meta-analyses from China.Epidemiology. 2013; 24: 335-336Crossref PubMed Scopus (10) Google Scholar This popularity is, in part, due to the pressure to publish on individual researchers for academic achievement and visibility7Ye XF Yu DH He J. The rise in meta-analyses from China.Epidemiology. 2013; 24: 335-336Crossref PubMed Scopus (10) Google Scholar instead of a meaningful contribution to the evidence, and has contributed to this pandemic with flawed and erroneous concepts in design, conduct, statistical analysis, and reporting that have plagued virtually every specialty of medicine. The incredible popularity of the metaanalysis has led to significant redundancy in data reporting, particularly in outcomes. The extent of redundancy, particularly in meta-analyses of randomized trials, has reached epidemic proportions.8Nunez Gil IJ Elizondo A Gradari S et al.Meta-analysis design and results in real life: Problem solvers or detour to maze. A critical review of meta-analysis of DAPT randomized controlled trials.Cardiovasc Revasc Med. 2019; 20: 897-906Crossref PubMed Scopus (1) Google Scholar Examples of this can be seen in multiple fields, with original meta-analyses showing statistically significant and clinical benefit, followed by subsequent similar meta-analyses, some with identical results and very few actually citing systematically among their references the prior metaanalyses on the same topic. This has led to the current perception of the metaanalysis as inferior research and a possibly diminished role in evidence-based clinical research. More so in cardiovascular medicine, journals are prone to have high levels of hidden biases in accepting manuscripts (articles from prominent researchers or major research institutes) despite significant redundancy of the topic in question in the medical literature. The sine qua non of the metaanalysis is the proper identification of existing literature and the systematic search of existing evidence. For this, a protocol that lists all potential sources and strategies should be established first. The authors believe that to help solidify the results and encourage transparency in the metaanalytic process, all metaanalyses should be registered in dedicated international databases of prospectively registered systematic reviews, such as PROSPERO.9Page MJ Shamseer L Tricco AC. Registration of systematic reviews in PROSPERO: 30,000 records and counting.Syst Rev. 2018; 7: 32Crossref PubMed Scopus (153) Google Scholar Registration allows those who are commissioning or planning studies to identify whether there are other metaanalyses already underway that are addressing their topic of interest. This helps provide transparency and avoid unintended and economically wasteful duplication of effort. It also serves as a safeguard against reporting biases by revealing differences between the outcomes reported and those planned in the registered protocol. This will improve the quality and credibility of metaanalyses endpoints and increase confidence behind policy or practice, informed by the findings of a systematic review drawing on the best-quality evidence. A significant factor instrumental in the final results of a metaanalysis lies in the search process; the most commonly used electronic databases may include MEDLINE and EMBASE.10Burns CS Shapiro 2nd, RM Nix T et al.Search results outliers among MEDLINE platforms.J Med Libr Assoc. 2019; 107: 364-373Crossref PubMed Scopus (4) Google Scholar,11Lam MT De Longhi C Turnbull J et al.Has Embase replaced MEDLINE since coverage expansion?.J Med Libr Assoc. 2018; 106: 227-234Crossref PubMed Google Scholar This can create an inherent bias from inception to find studies published only in English and excluding nonindexed studies in peer-reviewed journals, leading to only a sample of available evidence worldwide.12Montori VM Smieja M Guyatt GH. Publication bias: A brief review for clinicians.Mayo Clin Proc. 2000; 75: 1284-1288Abstract Full Text Full Text PDF PubMed Scopus (123) Google Scholar This is referred to as “publication bias”, and it is defined properly as the occurrence of studies showing positive effects being published and cited more frequently versus studies that show no significant results.12Montori VM Smieja M Guyatt GH. Publication bias: A brief review for clinicians.Mayo Clin Proc. 2000; 75: 1284-1288Abstract Full Text Full Text PDF PubMed Scopus (123) Google Scholar,13Dickersin K Min YI. Publication bias: The problem that won't go away.Ann N Y Acad Sci. 1993; 703: 135-146Crossref PubMed Scopus (305) Google Scholar This could lead to published data overestimating the actual degree of effect of the therapy or management studied.12Montori VM Smieja M Guyatt GH. Publication bias: A brief review for clinicians.Mayo Clin Proc. 2000; 75: 1284-1288Abstract Full Text Full Text PDF PubMed Scopus (123) Google Scholar,13Dickersin K Min YI. Publication bias: The problem that won't go away.Ann N Y Acad Sci. 1993; 703: 135-146Crossref PubMed Scopus (305) Google Scholar Trends show that larger studies with publication bias tend to get published regardless of their results. Studies that have small positive effects or reject the null hypothesis tend not to materialize in popular databases, whether it be due to publication delay, publications in languages other than English, publications in nonindexed journals, or studies not being published altogether. Therefore, reviewers may not be able to identify them. To minimize bias associated with narrow collection of sources and usage of only the most popular databases, one needs to broaden the scope of databases used. This includes electronic sources such as Cochrane Central,14Banno M Tsujimoto Y Kataoka Y. Using the Cochrane Central Register of Controlled Trials to identify clinical trial registration is insufficient: A cross-sectional study.BMC Med Res Methodol. 2020; 20: 200Crossref PubMed Scopus (5) Google Scholar the Federal Drug Administration, and internet websites such as Google Scholar. Examples of search strategies can be found in the Cochrane Collaboration's Reviewers’ handbook.15Zeng X Zhang Y Kwong JS et al.The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: A systematic review.J Evid Based Med. 2015; 8: 2-10Crossref PubMed Scopus (1093) Google Scholar The basic search strategy is built based on the research question formulation (ie, PICO or PICOS-Patient/ Problem, Intervention, Comparison and Outcome). Search strategies are constructed to include free-text terms (eg, in the title and abstract) and any appropriate subject indexing (eg, Medical Subject Headings) expected to retrieve eligible studies, with the help of an expert in the topic or an information specialist. Another critical component of the metaanalysis is defining heterogeneity and which cutoff value is defined as heterogeneous. Heterogeneity in metaanalysis refers to the variation in study outcomes between studies. There are three types of heterogeneity: clinical, statistical, and design-related. There is debate among researchers in how one would define heterogeneity—whether just based on a statistical variable using a quantitative test such as the Cochran's Q value or the I2 value.4Lee YH. An overview of meta-analysis for clinicians.Korean J Intern Med. 2018; 33: 277-283Crossref PubMed Scopus (101) Google Scholar It has become a common practice to establish an I2 of 50% (defining the study as heterogeneous) and to disregard the qualitative background of a study due to this cause. Clinical heterogeneity should be defined before quantitative analysis to overcome differences between sample characteristics and properly define the best method of analysis. Qualitative analysis of the heterogeneity due to the characteristics mentioned would provide value over analysis from a statistical test and could shed light over important determinants of treatment effect and increase the scientific and clinical value of the metaanalysis.16Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated.BMJ. 1994; 309: 1351-1355Crossref PubMed Scopus (749) Google Scholar Exploration of these differences often may yield new insights and hypotheses. Regarding statistical heterogeneity and mixing of studies, the authors’ approach to decrease this value includes lowering this threshold to a more rigorous accepted I2 of 25% or less to improve the inconsistency regarding selection of methods. The examination of all three types of heterogeneity is a crucial methodologic issue in systematic reviews and metaanalyses. Although a complete discussion of all models available is beyond the scope of this editorial, the authors briefly will discuss two statistical models for a metaanalysis, fixed-effect, and random-effect models. Fixed-effect models assume that the included studies have nonrandom variables and one true effect size, and that variation is caused by sample errors or chance (intrastudy errors).17Davey Smith G Egger M Meta-analyses of randomised controlled trials.Lancet. 1997; 350: 1182Abstract Full Text Full Text PDF PubMed Google Scholar Random-effect models assume the opposite, that the studies exhibit diversity and variability attributed not only to random chance (intrastudy and interstudy errors), with more than one effect size.18DerSimonian R Kacker R. Random-effects model for meta-analysis of clinical trials: An update.Contemp Clin Trials. 2007; 28: 105-114Abstract Full Text Full Text PDF PubMed Scopus (1547) Google Scholar The authors’ belief is that specific selection of a model should be justified based on types of studies, quantitative heterogeneity, and qualitative heterogeneity, and not solely on statistical heterogeneity. When there is discrepancy between models, both should be used as a sensitivity analysis, allowing the reader to interpret the data with more granularity. The flawed data from poor-quality metaanalyses can lead very easily to significant shifts in the delivery of healthcare from flawed guideline recommendations, which use metaanalytical data as the highest-quality evidence. Also, many clinicians, researchers, and editors may not be knowledgeable about how to differentiate between high- and low-quality studies, and some biases may evade the attention of even experienced methodologists and statistical editors To address publication bias, methodologists have proposed several statistical and graphic strategies to determine whether a metaanalysis is affected.12Montori VM Smieja M Guyatt GH. Publication bias: A brief review for clinicians.Mayo Clin Proc. 2000; 75: 1284-1288Abstract Full Text Full Text PDF PubMed Scopus (123) Google Scholar Although none of these strategies is accepted widely, the most commonly used is the funnel plot.19Pham B Platt R McAuley L et al.Is there a “best” way to detect and minimize publication bias? An empirical evaluation.Eval Health Prof. 2001; 24: 109-125PubMed Google Scholar However, a common problem with usage of the funnel plot to identify evidence of asymmetry and publication bias is that it is subjective and purely visual. It is not a formal statistical test. Presenting a funnel plot in a metaanalysis may be highly misleading, particularly when then are ten or fewer studies in the analysis.20Lau J Ioannidis JP Terrin N et al.The case of the misleading funnel plot.BMJ. 2006; 333: 597-600Crossref PubMed Scopus (1115) Google Scholar Studies using funnel-plot asymmetry to assess publication bias pose several problems, including subjectivity on whether a plot is asymmetric, the amount of asymmetry required to have evidence of bias, likelihood of asymmetry with publication bias, and whether asymmetric plots can occur by chance. A formal statistical test of asymmetry, such as Egger's test, generally should be preferred to funnel plots. Although Egger's test lacks statistical power, it can show appropriate type I error.21Simmonds M. Quantifying the risk of error when interpreting funnel plots.Syst Rev. 2015; 4: 24Crossref PubMed Scopus (100) Google Scholar To maintain the quality and standard of reporting metaanalyses, especially with randomized controlled trials (RCTs), there have been mandates over time to help classify important practices. One such mandate is the Quality of Reporting of Metaanalyses statement, which consists of a checklist and a flow diagram.5Moher D Cook DJ Eastwood S et al.Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement.Br J Surg. 2000; 87: 1448-1454Crossref PubMed Scopus (172) Google Scholar The checklist is organized into 21 headings and subheadings that encourage the authors to provide information on searches, selection, validity, assessment, data abstraction, study characteristics, quantitative data synthesis, and trial flow.5Moher D Cook DJ Eastwood S et al.Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement.Br J Surg. 2000; 87: 1448-1454Crossref PubMed Scopus (172) Google Scholar The flow diagram provides information about the number of RCTs identified, included, and excluded and the reasons for doing so.5Moher D Cook DJ Eastwood S et al.Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement.Br J Surg. 2000; 87: 1448-1454Crossref PubMed Scopus (172) Google Scholar This statement was revised and improved upon later with the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement. This template updates the format to consist of a 27-item checklist (inclusive of the Quality of Reporting of Metaanalyses criteria) and four-phase flow diagram, along with expanding its reach from metaanalyses to include other types of research that include RCTs.22Moher D Liberati A Tetzlaff J PRISMA GroupPreferred reporting items for systematic reviews and meta-analyses: The PRISMA statement.PLoS Med. 2009; 6Crossref Scopus (43490) Google Scholar The metaanalysis will remain a powerful tool in the research armamentarium, especially in cardiovascular medicine; there is, however, a pressing need for greater attention to the basic principles and crucial issues that the authors briefly have outlined. It will continue to provide high-level evidence for or against a therapy in question, but, as always, the devil is in the details. None.