UPDATE: SYSTEMATIC REVIEWS AND META‐ANALYSES

Abstract

Systematic reviews identify relevant studies, evaluate their quality, sum up the results using formal methods and draw conclusions based on the evidence. Meta-analysis is the term used when it is reasonable to combine the findings of a systematic review and perform a statistical analysis. The quality of articles in the surgical published work has been subjected to review. Chang et al. evaluated articles from the Archives of Surgery, Surgery, and Annals of Surgery for 3 months in 2005.1 They ‘followed commonly accepted study classifications that include: class I, prospective randomized controlled trials; class II, clinical studies in which the data were collected prospectively, or retrospective analyses that were based on prospectively maintained data (examples include observational studies, cohort studies, prevalence studies, and case control studies); and class III, studies based on retrospectively collected data (examples include clinical series, databases, and case reports)’. A total of 101 clinical articles were identified and classified as class I studies (8%), class II studies (34%) and class III studies (59%). There was no assessment of the quality of the studies. The authors comment that one limitation ‘is that this analysis would have missed many higher-level surgical studies that are often published in general medical journals, such as the New England Journal of Medicine or the Journal of the American Medical Association. Although there are many reasons to publish a study that has importance to all health-care providers in a more general journal, even if it was based on surgical patients or diseases, we believe this phenomenon results, in part, in the findings of this study, that our highest-quality journals dedicated to the care and treatment of surgical disease have a high prevalence of class III clinical evidence, perpetuating the practice of “eminence”-based medicine’. This study supports the call for a greater proportion of systematic reviews and meta-analyses within the surgical published work. Chang DC, Matsen SL, Simpkins CE. Why should surgeons care about clinical research methodology? J. Am. Coll. Surg. 2006; 203: 827–30. Most of the meta-analyses involving patients undergoing orthopaedic procedures meta-analyses contain methodological errors.1 So there a need for interested readers to be aware of the issues that arise when carrrying out and reporting a meta-analysis. Montori et al. have published a very sound introduction to meta-analyses.2 They explain that systematic reviews ‘are distinct from narrative reviews because they address a specific clinical question, require a comprehensive literature search, use explicit selection criteria to identify relevant studies, assess the methodologic quality of included studies, explore differences among study results, and either qualitatively or quantitatively synthesize study results’. By way of contrast, they point out that narrative reviews are ‘more likely to make recommendations weighted strongly by opinion, which may or may not reflect the evidence’. They quote the failure of narrative reviews to detect the benefits of thrombolytics after myocardial infarction or the futility of using lidocaine prophylaxis after myocardial infarction, although there was enough information available to come to these conclusions. They also describe how Cochrane, a British epidemiologist, stressed the need for summary evaluations of clinical trials for each specialty interest. The text also discusses the various steps that are needed to produce a sound meta-analysis. They explain that ‘Formulating the question typically is an iterative process of narrowing and expanding the focus to make the review feasible’. This is illustrated by using a very relevant example – thromboembolism prophylaxis for patients undergoing knee replacement surgery. They comment that ‘Although warfarin and various heparin agents have different mechanisms of action, the similar nature of their effect (to decrease the risk of thrombosis and increase the risk of bleeding by interfering with the coagulation cascade), and the relatively homogenous patient group appear similar enough to generate a pooled estimate of these studies to sensibly estimate the underlying truth about the efficacy of thromboembolism prevention with anticoagulants in patients having knee surgery’. They conclude by commenting that ‘Clinicians, teachers, and researchers in the discipline of orthopaedics should consider reading and doing not only clinical trials of orthopaedic interventions, but also systematic reviews of previous research’. Bhandari M, Morrow F, Kulkarni AV, Tornetta P III. Meta-analyses in orthopaedic surgery: a systematic review of their methodologies. J. Bone Joint Surg. 2001; 83A: 15–24. Montori VM, Swiontkowski MF, Cook DJ. Methodologic issues in systematic reviews and meta-analyses. Clin. Orthop. Relat. Res. 2003; 413: 43–54. Saha provides a useful account of ‘the functioning and the role of Cochrane collaboration and in particular, the systematic reviews undertaken by it, in the assimilation and dissemination of robust evidence for use in evidence-based surgery’.1 The gathering of useful information has been likened to ‘trying to get a sip of water from a fire-hose and you don’t even know the quality of the water’.2 The Cochrane centre was established in 1993 in Oxford, UK, and is an independent non-profit organization. It is named after Archie Cochrane, a British epidemiologist who introduced the concept of controlled clinical trials in the 1960s in the UK amidst stiff opposition from the clinicians. Not only did he revolutionize the way we now advocate treatments but also highlighted the economic benefits’ that can be derived from evidence-based clinical practice. There are 51 Cochrane review groups ‘dedicated to a particular topic, of which several are of relevance to surgeons; e.g., Breast cancer, upper GI and pancreatic diseases, colorectal cancer, prostatic diseases and urologic cancers, peripheral vascular disease, injuries, wounds etc. The Cochrane Collaboration aims to provide healthcare professionals with selected, critiqued reviews of the available evidence for specified clinical problems’. The entire Cochrane review index is available from http://www.cochrane.org/reviews/en/topics (accessed 30 November 2007). Saha S. The Cochrane database of systematic reviews: a treasure house for evidence-based surgery. Indian J. Surg. 2006; 68: 251–6. Butler D. The writing is on the web for science journals in print. Nature 1999; 397: 195–200. Sauerland and Seiler have outlined how systematic reviews and meta-analysis contribute towards evidence-based surgery.1 Their article contains well-informed comments on ‘potential weaknesses’ in meta-analyses ‘such as heterogeneity and publication bias, and highlights special situations when dealing with surgical trials’. The authors provide a comprehensive account of the seven-step approach to meta-analysis. Included is a clearly worded explanation of the underlying statistics including ‘forest plots’ and ‘funnel plots’. One of the weaknesses of meta-analyses is that they are trying to add ‘apples and oranges’. In other words, clinical trials evaluating a specific topic use different methods and study different types of patients – they can never be exactly the same. This phenomenon is called ‘heterogeneity’. Sauerland and Seiler comment that ‘Although heterogeneity may be due to chance alone, it can also be caused by clinical or methodologic differences among trials and might thereby result in systematic errors.1 Clinical considerations are also necessary when deciding whether a set of trials should be integrated into a common treatment estimate. Heterogeneity in surgical trials is likely to arise through diversity in technical expertise of trial surgeons’. The extent of heterogeneity can be evaluated by statistics and displayed as an illustration. If the results are highly heterogeneous, and the cause cannot be identified and allowed for, then the trial results should not be pooled for statistical comparison, that is, it is then called a systematic review rather than a meta-analysis. The authors conclude by stating: ‘The art of writing an overview has developed from the classic and unsystematic into a systematic, often quantitative review. For surgeons such articles provide the opportunity to obtain evidence-based summaries more quickly than from primary studies. Because heterogeneity, publication bias, and learning curve effects represent serious problems to systematic reviews, surgeons should have a basic understanding of methodology. Not every meta-analysis represents level I evidence’. This article is highly recommended as a comprehensive account of meta-analyses. Sauerland S, Seiler CM. Role of systematic reviews and meta-analysis in evidence-based medicine. World J. Surg. 2005; 29: 582–7. The American College of Surgeons has an Office of Continuous Quality Improvement to facilitate the performance of randomized controlled trials and systematic reviews. This led Panesar et al. to hypothesize ‘that there would exist greater momentum to undertake and publish studies employing these designs in the USA when compared to the UK’.1 They investigated this by comparing the ‘number and proportion of published randomized controlled trials and systematic reviews in the leading two US and UK general surgical journals’. The journals under evaluation were Annals of Surgery, Archives of Surgery, British Journal of Surgery, and Annals of the Royal College of Surgeons. Systematic searching uncovered 519 original reports in UK journals and 616 original reports in USA journals during 2004. The proportion of randomized controlled trials was 5.6%, which is similar to previous published reports. However, they found that systematic reviews ‘were significantly more commonly reported in the UK journals: UK 37/519 (7.1%) versus USA 22/616 (3.6%)’. It is clear that the American College of Surgeons initiative has yet to have an effect, which is understandable. It takes a lot of time to set up clinical trials and establish expertise in meta-analyses. The authors concluded that ‘There remains a need for funders, researchers and editors to prioritize the undertaking and reporting of studies employing robust designs in order to improve the body of evidence underpinning surgical practice’. Panesar SS, Thakrar R, Athanasiou T, Sheikh A. Comparison of reports of randomized controlled trials and systematic reviews in surgical journals: literature review. J. R. Soc Med. 2006; 99: 470–72. Attention has been directed towards minimum reporting standards for systematic reviews and meta-analysis. Mahid et al. have included this topic in their overview for ‘the surgeon scientist’.1 They also consider sources of bias, both quantitative and qualitative. They note that those ‘wishing to pursue clinical research will find that simple retrospective studies or narrative reviews no longer meet the standards of many journal editors’. Their aim is to provide ‘a clinical research primer for the surgeon by focusing on the methodology of conducting a systematic literature review and meta-analysis’. The body of the text starts with an account of the ‘preferred techniques’ for conducting a search of published work. The initial search usually starts ‘by using Medline, either through PubMed as the search engine or through Ovid’. Other databases include the Cochrane Database of Systematic Reviews, EMBASE, Cochrane Database of Abstracts of Reviews of Effectiveness (DARE), PsychINFO and the Cumulative Index of Nursing and Allied Health Literature (CINAHL). They also recommend the use of medical subject heading terms (MeSH) and Boolean operators (‘and’, ‘or’ and ‘not’) and ‘if the initial search yields only a handful of articles, the “explode” function or ‘related articles’ function’ of PubMed. They detail the selection of appropriate articles and the need for reporting standards – the Quality of Reporting of Meta-analyses (QUORUM) group has provided a 17-item checklist for reporting systematic reviews.2 Mahid et al. also provide a sound account of the relevant analyses and discuss sensitivity analyses that ‘probe’ the data to estimate the robustness of the results by varying the underlying assumptions.1 For instance, ‘if a study appears to be an outlier, then its influence on the results may be assessed by excluding it’ and ‘results may be calculated using all studies after using a quality scoring method, and then repeated after excluding poorer-quality studies’. They note that ‘Submitting data to sensitivity analysis permits the identification of possible sources of bias in the results. If such bias is not detected, this adds weight to the conclusions of the meta-analysis’. The authors conclude by stating that with ‘an ever increasing wealth of surgical literature and inevitable time constraints on the surgeon, the critical evaluation of clinical studies is of the utmost importance. The ability to recognize whether the results of a study are valid and whether or not they should impact on clinical practice must be accepted as a key part of every surgeon’s continuing medical education’. Mahid SS, Hornung CA, Minor KS, Turina M, Galandiuk S. Systematic reviews and meta-analysis for the surgeon scientist. Br. J. Surg. 2006; 93: 1315–24. Moher D, Cook DJ, Eastwood S et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement quality of reporting of meta-analyses. Lancet 1999; 354: 1896–1900. Reviews of the published work reviews help surgeons to keep up with the flow of information without the time-consuming and onerous chore of collecting, and then evaluating, each of the original articles.1 Meta-analyses are systematic reviews that use various statistical tests to evaluate the reliability of the conclusions. But, like any other type of study, there are both good and bad meta-analyses. Dixon et al. from Canada have assessed ‘the methodologic quality of meta-analyses of general surgery topics published in peer-reviewed journals’.2 They collected 51 admissible meta-analyses that were published between 1997 and 2002, which is important because others have shown a gradual increase in the quality of meta-analyses in the medical published reports. Approximately 80% of the articles were published in surgical journals. The investigators excluded from study Cochrane Reviews (because they have been shown to be of high quality) and those that did not address general surgery topics (including chemo/radiotherapy, anaesthesia/analgesia, vascular surgery and critical care). They assessed the quality of the meta-analyses using the 10-item Overview Quality Assessment Questionnaire (OQAQ), which has been validated as a useful tool for evaluating the quality of meta-analyses. It provides an overall rating between 1 (bad) and 7 (good). The median score of the studies under review was 3.3. Indeed, two-thirds of the meta-analyses failed to clearly report their search strategies: failure to declare a comprehensive search strategy defies one of the most basic concepts underlying meta-analyses, the universal collection of data to avoid selection bias. The article goes on to describe the various errors in detail. On average, each meta-analysis evaluated 27 studies and 11 853 patients, but there was great variation. The authors concluded that ‘This critical appraisal of meta-analyses published in the general surgery literature demonstrates frequent methodologic flaws. The quality of these reports limits the validity of the findings and the inferences that can be made about the primary studies reviewed. To improve the quality of future meta-analyses, we recommend following guidelines for the optimal conduct and reporting of meta-analyses in general surgery’. Ng TT, McGory ML, Ko CY, Maggard MA. Meta-analysis in surgery: methods and limitations. Arch. Surg. 2006; 141: 1125–30. Dixon E, Hameed M, Sutherland F, Cook DJ, Doig C. Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann. Surg. 2005; 241: 450–59. The high-quality evidence provided by competent systematic reviews is often incorporated into clinical guidelines. Haggard comments that the ‘relationship of evidence to guidelines, an important link between research and application, is not as simple and unidirectional as is often imagined.1 The recent emphasis on evidence of high quality and aggregation of evidence is appropriate but can be modified and extended. There are other important elements in guidelines besides the available evidence (which is centered on efficacy and effectiveness). Available evidence may be poor, or not be the most relevant to the decision at hand. To offset the limitations to which this predicament leads, effort is also appropriate to develop more rigorous principles for interpretation and evaluation of evidence, so as to make better use of what is available’. The text goes on to point out that an ‘evidence-based guideline’ can mislead readers when inappropriately used to promote guidelines based on low levels of evidence. Guidelines should cover ‘a small range of coherently related conditions and their treatments, or else a single condition of high volume or risk, where there are corresponding concerns about costs and effectiveness or about safety. Any wider and you have a textbook for a whole discipline. Any narrower and the needed guideline cannot be remembered or found in the large pile of guidelines on the shelf’. Haggard M. The relationship between evidence and guidelines. Otolaryngol. Head Neck Surg. 2007; 137 (4 Suppl.): S72–7.