Systematic Reviews Of Health Care Interventions Research Articles

Systematic reviews and meta-analyses in regional anesthesia and pain medicine (Part I): guidelines for preparing the review protocol

Comprehensive resources exist on how to plan a systematic review and meta-analysis. The objective of this article is to provide guidance to authors preparing their systematic review protocol in the...

User experience of applying AMSTAR 2 to appraise systematic reviews of healthcare interventions: a commentary

Background‘A Measurement Tool to Assess Systematic Reviews, version 2’ (AMSTAR 2) is a validated 16-item scale designed to appraise systematic reviews (SRs) of healthcare interventions and to rate the overall confidence in their results. This commentary aims to describe the challenges with rating of the individual items and the application of AMSTAR 2 from the user perspective.DiscussionA group of six experienced users (methodologists working in different clinical fields for at least 10 years) identified and discussed the challenges in rating of each item and the general use of AMSTAR 2 to appraise SRs. A group discussion was used to develop recommendations on how users could deal with the identified challenges. We identified various challenges with the content of items 2–16 and with the derivation of the overall confidence ratings on AMSTAR 2. These challenges include the need (1) to provide additional definitions (e.g., what constitutes major deviations from SR protocol on item 2), (2) to choose a rating strategy for multiple conditions on single items (e.g., how to rate item 5 if studies were selected in duplicate, but consensus between two authors was not reported), and (3) to determine rules for deriving the confidence ratings (e.g., what items are critical for such ratings). Based on these challenges we formulated specific recommendations for items 2–16 that AMSTAR 2 users could consider before applying the tool.SummaryOur commentary adds to the existing literature by providing the first in-depth examination of the AMSTAR 2 tool from the user perspective. The identified challenges could be addressed by additional decision rules including definitions for ambiguous items and guidance for rating of complex items and derivation of confidence ratings. We recommend that a team consensus regarding such decision rules is required before appraisal procedure begins.Trial registrationNot applicable.

Methodologic quality and risk-of-bias in systematic reviews of healthcare interventions: a review of methods

OBJECTIVE: To compare the characteristics of systematic reviews of healthcare interventions that assessed or did not methodologic quality/risk-of-bias of included studies. Additionally, to analyze methodologic features of those assessing the methodologic quality/risk-of-bias. METHODS: PubMed database was searched. From 25,571 systematic reviews retrieved, a random sample of 1,025 was screened. Frequencies were used to describe outcomes. Unadjusted and adjusted logistic regressions were performed to test the associations with methodologic quality/risk-of-bias results assessment. In a second analysis, systematic reviews that assessed methodologic quality/risk of bias were dichotomized according to the design of included studies (randomized clinical trials-only versus non-randomized studies of intervention or a combination of both). RESULTS: 303 systematic reviews were fully analyzed. Methodologic quality/risk of bias was assessed by 278 (92%). Methodologic quality/risk-of-bias assessment was associated with a higher number of databases searched (>4, P= 0.008), the presence of meta-analysis (P= 0.005), and the design of included studies (randomized clinical trials-only, P= 0.042). The chance of using a suitable tool and a tool designed for risk-of-bias assessment rather than methodologic quality was higher for randomized clinical trials-only systematic reviews (P< 0.05). The most used tool was Cochrane’s RoB Tool without a clear studies’ overall risk classification system. CONCLUSION: methodologic quality/risk-of-bias assessment was associated with included studies’ design (randomized clinical trials-only), a meta-analysis of data, and the number of databases searched (>4). The most used tool was Cochrane’s RoB Tool, with no clearly defined rating system. Methodologic quality/risk-of-bias assessment methods description, results, and impacts on meta-analysis, the certainty of evidence, and systematic reviews’ results are still to be consistently addressed.

The application of AMSTAR2 in 32 overviews of systematic reviews of interventions for mental and behavioural disorders: A cross-sectional study.

'A measurement tool to assess systematic reviews, version 2' (AMSTAR2) is a 16-item tool to critically appraise systematic reviews (SRs) of healthcare interventions. This study aimed to assess the methods and outcomes of AMSTAR2 appraisals in overviews of SRs of interventions for mental and behavioural disorders. The cross-sectional study was conducted using 32 overviews of SRs selected from three electronic databases in January 2021. Data items included overview and SR characteristics and AMSTAR2 appraisal methods and outcomes. Data were extracted by two authors independently and narratively synthesised using descriptive statistics (means ± SD and relative frequencies). SR characteristics were compared based on AMSTAR2 appraisal outcomes using chi-square tests. The 32 overviews appraised SRs of predominantly non-pharmacological interventions for mental disorders. AMSTAR2 appraisals were reported as confidence ratings in 25/32 overviews or individual item scores in 24/32 overviews. Most SRs/overview were non-Cochrane (mean=94%), included RCTs only (mean=77%) and were published before AMSTAR2 release (mean=79%). The confidence ratings derived in 25 overviews for 349 SRs were predominantly critically low (68%). Confidence ratings were similar for SRs with RCTs only versus RCTs+non-RCTs or SRs published before versus after AMSTAR2 release, while Cochrane SRs received more high+moderate than low+critically low confidence ratings (p < 0.01). Confidence ratings derived based on AMSTAR2 do not differentiate among SRs of healthcare interventions except for Cochrane SRs that fulfil the criteria for high confidence ratings. AMSTAR2 items should be consulted to avoid common weaknesses in future SRs.

Quality appraisal of systematic reviews of interventions for children with cerebral palsy reveals critically low confidence.

To evaluate the methodological quality of recent systematic reviews of interventions for children with cerebral palsy in order to determine the level of confidence in the reviews' conclusions. A comprehensive search of 22 databases identified eligible systematic reviews with and without meta-analysis published worldwide from 2015 to 2019. We independently extracted data and used A Measurement Tool to Assess Systematic Reviews-2 (AMSTAR-2) to appraise methodological quality. Eighty-three systematic reviews met strict eligibility criteria. Most were from Europe and Latin America and reported on rehabilitative interventions. AMSTAR-2 appraisal found critically low confidence in 88% (n=73) because of multiple and varied deficiencies. Only 7% (n=6) had no AMSTAR-2 critical domain deficiency. The number of systematic reviews increased fivefold from 2015 to 2019; however, quality did not improve over time. Most of these systematic reviews are considered unreliable according to AMSTAR-2. Current recommendations for treating children with CP based on these flawed systematic reviews need re-evaluation. Findings are comparable to reports from other areas of medicine, despite the general perception that systematic reviews are high-level evidence. The required use of current widely accepted guidance for conducting and reporting systematic reviews by authors, peer reviewers, and editors is critical to ensure reliable, unbiased, and transparent systematic reviews. What this paper adds Confidence was critically low in the conclusions of 88% of systematic reviews about interventions for children with cerebral palsy (CP). Quality issues in the sample were not limited to systematic reviews of non-randomized trials, or to those about certain populations of CP or interventions. The inclusion of meta-analysis did not improve the level of confidence in these systematic reviews. Numbers of systematic reviews on this topic increased over the 5 search years but their methodological quality did not improve.

Recommendations for assessing the risk of bias in systematic reviews of health-care interventions

ObjectivesRisk-of-bias assessment is a central component of systematic reviews, but little conclusive empirical evidence exists on the validity of such assessments. In the context of such uncertainty, we present pragmatic recommendations that promote transparency and reproducibility in processes, address methodological advances in the risk-of-bias assessment, and can be applied consistently across review topics. Study Design and SettingEpidemiological study design principles; available empirical evidence, risk-of-bias tools, and guidance; and workgroup consensus. ResultsWe developed recommendations for assessing the risk of bias of studies of health-care interventions specific to framing the focus and scope of risk-of-bias assessment; selecting the risk-of-bias categories; choosing assessment instruments; and conducting, analyzing, and presenting results of risk-of-bias assessments. Key recommendations include transparency and reproducibility of judgments, separating risk of bias from other constructs such as applicability and precision, and evaluating the risk of bias per outcome. We recommend against certain past practices, such as focusing on reporting quality, relying solely on study design or numerical quality scores, and automatically downgrading for industry sponsorship. ConclusionRisk-of-bias assessment remains a challenging but essential step in systematic reviews. We presented standards to promote transparency of judgments.

Could Embase Emtree Index Search Terms Be Focused To Reduce Numbers To Screen In Clinical Systematic Reviews?

Embase is one of the most commonly searched bibliographic databases when undertaking systematic reviews of healthcare interventions. Using a combination of text searching and index searching is accepted best practice when designing Embase search strategies. As the number of Emtree index terms assigned to each Embase record has increased over time, there has been a corresponding increase in overall search result numbers and, therefore, the size of the workload for reviewers. The objective of this research was to investigate whether focusing Emtree indexing terms using the ‘restrict to focus’ function, so that only those articles where the index term is key to that article are retrieved, could reduce the number of records for screening without loss of included studies. Embase searches conducted in three selected clinical reviews undertaken by the review team were retrospectively compared with searches in which the Emtree terms were focused. The records retrieved by the focused Emtree searches were checked to see what proportion of included studies identified by the original unfocused search strategy were retrieved. The subjects of reviews investigated were treatments for invasive fungal infections, type 2 diabetes mellitus, and acute lymphoblastic leukaemia. The data collected was analysed to identify total results with and without focusing Emtree terms, and to calculate the yield of included records. The focused searches retrieved 100% of the included papers in each review but results retrieved to screen were reduced by up to 30%. Focusing Emtree terms can potentially reduce screening burden in clinical systematic reviews without lowering retrieval of relevant records. Reducing the number of results in Embase searches without a loss of sensitivity could improve efficiency by reducing time spent and costs when undertaking systematic reviews.

Development of an algorithm to provide awareness in choosing study designs for inclusion in systematic reviews of healthcare interventions: a method study

ObjectivesTo develop an algorithm that aims to provide guidance and awareness for choosing multiple study designs in systematic reviews of healthcare interventions.DesignMethod study: (1) To summarise the literature base on...

Observational evidence and strength of evidence domains: case examples.

BackgroundSystematic reviews of healthcare interventions most often focus on randomized controlled trials (RCTs). However, certain circumstances warrant consideration of observational evidence, and such studies are increasingly being included as evidence in systematic reviews.MethodsTo illustrate the use of observational evidence, we present case examples of systematic reviews in which observational evidence was considered as well as case examples of individual observational studies, and how they demonstrate various strength of evidence domains in accordance with current Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) methods guidance.ResultsIn the presented examples, observational evidence is used when RCTs are infeasible or raise ethical concerns, lack generalizability, or provide insufficient data. Individual study case examples highlight how observational evidence may fulfill required strength of evidence domains, such as study limitations (reduced risk of selection, detection, performance, and attrition); directness; consistency; precision; and reporting bias (publication, selective outcome reporting, and selective analysis reporting), as well as additional domains of dose-response association, plausible confounding that would decrease the observed effect, and strength of association (magnitude of effect).ConclusionsThe cases highlighted in this paper demonstrate how observational studies may provide moderate to (rarely) high strength evidence in systematic reviews.

Efficacy of interventions to combat tobacco addiction: Cochrane update of 2012 reviews

The Cochrane Collaboration is an international not-for-profit organization which produces and disseminates systematic reviews of health-care interventions. This paper is the first in a series of annual updates of Cochrane reviews on tobacco addiction interventions. It also provides an up-to-date overview of review findings in this area to date and summary statistics for cessation reviews in which meta-analyses were conducted. In 2012, the Group published seven new reviews and updated 13 others. This update summarizes and comments on these reviews. It also summarizes key findings from all the other reviews in this area. New reviews in 2012 found that in smokers using pharmacotherapy, behavioural support improves success rates [risk ratio (RR) 1.16, 95% confidence interval (CI) = 1.09-1.24], and that combining behavioural support and pharmacotherapy aids cessation (RR 1.82, 95% CI = 1.66-2.00). Updated reviews established mobile phones as potentially helpful in aiding cessation (RR 1.71, 95% CI = 1.47-1.99), found that cytisine (RR 3.98, 95% CI = 2.01-7.87) and low-dose varenicline (RR 2.09, 95% CI = 1.56-2.78) aid smoking cessation, and found that training health professionals in smoking cessation improves patient cessation rates (RR 1.60, 95% CI = 1.26-2.03). The updated reviews confirmed the benefits of nicotine replacement therapy, standard dose varenicline and providing cessation treatment free of charge. Lack of demonstrated efficacy remained for partner support, expired-air carbon monoxide feedback and lung function feedback. Cochrane systematic review evidence for the first time establishes the efficacy of behavioural support over and above pharmacotherapy, as well as the efficacy of cytisine, mobile phone technology, low-dose varenicline and health professional training in promoting smoking cessation.

Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review.

BackgroundThe increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making.Methodology/Principal FindingsIn this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.ConclusionsThis update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

From the Trenches: A Cross-Sectional Study Applying the GRADE Tool in Systematic Reviews of Healthcare Interventions

BackgroundGRADE was developed to address shortcomings of tools to rate the quality of a body of evidence. While much has been published about GRADE, there are few empirical and systematic evaluations.ObjectiveTo assess GRADE for systematic reviews (SRs) in terms of inter-rater agreement and identify areas of uncertainty.DesignCross-sectional, descriptive study.MethodsWe applied GRADE to three SRs (n = 48, 66, and 75 studies, respectively) with 29 comparisons and 12 outcomes overall. Two reviewers graded evidence independently for outcomes deemed clinically important a priori. Inter-rater reliability was assessed using kappas for four main domains (risk of bias, consistency, directness, and precision) and overall quality of evidence.ResultsFor the first review, reliability was: κ = 0.41 for risk of bias; 0.84 consistency; 0.18 precision; and 0.44 overall quality. Kappa could not be calculated for directness as one rater assessed all items as direct; assessors agreed in 41% of cases. For the second review reliability was: 0.37 consistency and 0.19 precision. Kappa could not be assessed for other items; assessors agreed in 33% of cases for risk of bias; 100% directness; and 58% overall quality. For the third review, reliability was: 0.06 risk of bias; 0.79 consistency; 0.21 precision; and 0.18 overall quality. Assessors agreed in 100% of cases for directness. Precision created the most uncertainty due to difficulties in identifying “optimal” information size and “clinical decision threshold”, as well as making assessments when there was no meta-analysis. The risk of bias domain created uncertainty, particularly for nonrandomized studies.ConclusionsAs researchers with varied levels of training and experience use GRADE, there is risk for variability in interpretation and application. This study shows variable agreement across the GRADE domains, reflecting areas where further guidance is required.

Do Cochrane reviews provide a good model for social science? The role of observational studies in systematic reviews

Formalised research synthesis to underpin evidence-based policy and practice has become increasingly important in areas of public policy. In this paper we discuss whether the Cochrane standard for systematic reviews of healthcare interventions is appropriate for social research. We examine the formal criteria of the Cochrane Collaboration for including particular study designs and search the Cochrane Library to provide quantitative evidence on the de facto standard of actual Cochrane reviews. By identifying the sample of Cochrane reviews that consider observational designs, we are able to conclude that the majority of reviews appears limited to considering randomised controlled trials only. Because recent studies have delineated conditions for observational studies in social research to produce valid evidence, we argue that an inclusive approach is essential for truly evidence-based policy and practice in regard to social interventions.

Methodology in conducting a systematic review of systematic reviews of healthcare interventions.

BackgroundHundreds of studies of maternity care interventions have been published, too many for most people involved in providing maternity care to identify and consider when making decisions. It became apparent that systematic reviews of individual studies were required to appraise, summarise and bring together existing studies in a single place. However, decision makers are increasingly faced by a plethora of such reviews and these are likely to be of variable quality and scope, with more than one review of important topics. Systematic reviews (or overviews) of reviews are a logical and appropriate next step, allowing the findings of separate reviews to be compared and contrasted, providing clinical decision makers with the evidence they need.MethodsThe methods used to identify and appraise published and unpublished reviews systematically, drawing on our experiences and good practice in the conduct and reporting of systematic reviews are described. The process of identifying and appraising all published reviews allows researchers to describe the quality of this evidence base, summarise and compare the review's conclusions and discuss the strength of these conclusions.ResultsMethodological challenges and possible solutions are described within the context of (i) sources, (ii) study selection, (iii) quality assessment (i.e. the extent of searching undertaken for the reviews, description of study selection and inclusion criteria, comparability of included studies, assessment of publication bias and assessment of heterogeneity), (iv) presentation of results, and (v) implications for practice and research.ConclusionConducting a systematic review of reviews highlights the usefulness of bringing together a summary of reviews in one place, where there is more than one review on an important topic. The methods described here should help clinicians to review and appraise published reviews systematically, and aid evidence-based clinical decision-making.

Quality assessment of systematic reviews of health care interventions using AMSTAR

Appraisal of the methodological quality of systematic reviews would reflect on their utility for the clinicians and policymakers. This study was done to assess the quality of systematic reviews published in five leading Indian medical journals using AMSTAR. 22 systematic reviews of healthcare interventions were identified. The scores ranged 0 to 10 (mean 3.77 and median 2.5), 9 reviews scored > 4/11. Most frequent yes and no scores were: publication status as an inclusion criterion (12 /22), respectively and duplicate study selection and data extraction (17 /22). Several suboptimal aspects of methodological quality were identified in the reviews evaluated.

The Cochrane Collaboration and Meta-Analysis of Clinical Data

The Cochrane Collaboration is an international not-for-profit and independent organization, dedicated to making up-to-date, accurate information about the effects of healthcare readily available worldwide. It produces and disseminates systematic reviews of healthcare interventions and promotes the search for evidence in the form of clinical trials and other studies of interventions.

Nottingham systematic review course

The course is hosted by the Cochrane Schizophrenia Group for those interested in undertaking reviews of randomised trials. The Cochrane Schizophrenia Group is part of the Cochrane Collaboration, an international, non-profit organisation that publishes systematic reviews of healthcare interventions. New reviewers are encouraged to attend this course. It is not specialty specific and will attract anyone wishing to do a Cochrane style review. This was a four day course that involved small group work, interactive tutorials, workshops, and practical computer sessions. The first two days were a gentle introduction to the Cochrane Collaboration, focusing on developing and writing a …

Evidence-Based Podiatric Medicine

Due to the exponential increase in the quantity and quality of podiatric medicine-related research during the past decade, podiatric physicians are inundated with an insurmountable volume of research relevant to clinical practice. Systematic reviews can refine this literature by using explicit, rigorous, and reproducible methods to identify, critically appraise, and synthesize the best evidence from all clinical trials to answer clearly defined clinical questions. The Cochrane Collaboration is an international not-for-profit organization created to improve the user-friendliness and accessibility of medical literature mainly through preparing and maintaining systematic reviews of health-care interventions. The Cochrane Library currently contains more than 50 podiatric medicine-relevant systematic reviews summarizing and synthesizing evidence from many hundreds of randomized controlled trials evaluating interventions for foot problems. Although more than 60 countries worldwide have open online access to The Cochrane Library, in the United States, only the state of Wyoming has free access to full-text reviews. In an era demanding an evidence-based approach for every clinical intervention, high-quality systematic reviews streamline podiatric medical literature by reducing the time, cost, and training necessary to establish a solid evidence base for practice.

Systematic review of the empirical evidence of study publication bias and outcome reporting bias.

BackgroundThe increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.Methodology/Principal FindingsWe review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.ConclusionsRecent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Cochrane reviews with no trials: pointless or pragmatic?

The Cochrane Collaboration is internationally renowned as the leading organization for the production of systematic reviews of healthcare interventions. Since the mid-1990s, Cochrane reviews have become synonymous with randomized controlled trials (RCTs), contributing to the elevation of RCTs as the gold standard for research in healthcare. However, this article demonstrates how the inclusion of non-randomized data in Cochrane Reviews is now up for debate. The Cochrane Review of ‘Caesarean section for non-medical reasons at term’ is the first from the Cochrane Pregnancy and Childbirth Group (PCG) to strongly recommend research methods, other than an RCT, to gather future data. Here, two of the review authors assert that Cochrane reviews with no trials may have their place in the work of the PCG.