Systematic Review Effectiveness Research Articles

Seeking adverse effects in systematic reviews of orthodontic interventions: a cross-sectional study (part 1)

BackgroundSystematic reviews that assess the benefits of interventions often do not completely capture all dimensions of the adverse effects. This cross-sectional study (part 1 of 2 studies) assessed whether adverse effects were sought, whether the findings on these effects were reported, and what types of adverse effects were identified in systematic reviews of orthodontic interventions.MethodsSystematic reviews of orthodontic interventions on human patients of any health status, sex, age, and demographics, and socio-economic status, in any type of setting assessing any type of adverse effect scored at any endpoint or timing were eligible. The Cochrane Database of Systematic Reviews and 5 leading orthodontic journals were manually searched for eligible reviews between August 1 2009 and July 31 2021. Study selection and data extraction was conducted by two researchers independently. Prevalence proportions were calculated for four outcomes on seeking and reporting of adverse effects of orthodontic interventions. Univariable logistic regression models were used to determine the association between each one of these outcomes and the journal in which the systematic review was published using the eligible Cochrane reviews as reference.ResultsNinety-eight eligible systematic reviews were identified. 35.7% (35/98) of reviews defined seeking of adverse effects as a research objective, 85.7% (84/98) sought adverse effects, 84.7% (83/98) reported findings related to adverse effects, and 90.8% (89/98) considered or discussed potential adverse effects in the review. Reviews in the journal Orthodontics and Craniofacial Research compared with Cochrane reviews had approximately 7 times the odds (OR 7.20, 95% CI 1.08 to 47.96) to define seeking of adverse effects in the research objectives. Five of the 12 categories of adverse effects accounted for 83.1% (162/195) of all adverse effects sought and reported.ConclusionsAlthough the majority of included reviews sought and reported adverse effects of orthodontic interventions, end-users of these reviews should beware that these findings do not give the complete spectrum on these effects and that they could be jeopardized by the risk of non-systematically assessing and reporting of adverse effects in these reviews and in the primary studies that feed them. Much research is ahead such as developing core outcome sets on adverse effects of interventions for both primary studies and systematic reviews.

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

ObjectiveTo assess the risk of bias associated with missing outcome data in systematic reviews.DesignImputation study.SettingSystematic reviews.Population100 systematic reviews that included a group level meta-analysis with a statistically significant effect on...

Unwanted effects of treatments for depression in children and adolescents: a mapping of systematic reviews

ObjectivesTo describe the results of a mapping review exploring the coverage of unwanted treatment effects in systematic reviews of the effects of various treatments for moderate to severe depression in...

Seeking adverse effects in systematic reviews of orthodontic interventions: protocol for a cross-sectional study

BackgroundBefore implementing healthcare interventions, clinicians need to weigh the beneficial and adverse effects of interventions. However, a large body of evidence has demonstrated that seeking and reporting of adverse effects is suboptimal in clinical trials and in systematic reviews of interventions. This cross-sectional study will investigate the status of this problem in orthodontics. This study will assess whether adverse effects were sought and whether findings related to adverse effects were reported in systematic reviews of orthodontic interventions in the five leading orthodontic journals and in the Cochrane Database of Systematic Reviews.MethodsSystematic reviews of clinical orthodontic interventions published between 01 August 2009 and 31 July 2019 in the five leading orthodontic journals and in the Cochrane Database will be included. Empty reviews will be excluded. The reporting of outcomes on adverse effects will not determine eligibility, i.e., reviews will not be excluded, because they did not report usable data. Study selection and data extraction will be conducted independently by two authors. Our primary outcome will be the prevalence of systematic reviews of orthodontic interventions that sought any findings related to adverse effects in the included studies. Additional prevalence statistics will be calculated on a series of items related to seeking of adverse effects in the eligible reviews. All statistics will be calculated for (1) all journals together, (2) the group of five orthodontic journals and the Cochrane Database of Systematic Reviews separately, and (3) each individual journal separately. Chi-square tests of independence will be used to compare these groups.DiscussionThis study will assess whether adverse effects were sought in systematic reviews of orthodontic interventions. This knowledge is important, because reviews that present an incomplete picture on adverse effects can have unfavorable consequences for the end-users. Also not reporting that no adverse effects were assessed in eligible studies included in a systematic review can mislead pertinent stakeholders. Our findings could have policy implications for making judgments on accepting or rejecting an intervention systematic review for publication, for example, by directing editors and peer-reviewers to adopt the various items on adverse effects defined in the MECIR standards and in the PRISMA harm checklist.

Bayesian mixed treatment comparisons meta-analysis for correlated outcomes subject to reporting bias.

Many randomized controlled trials (RCTs) report more than one primary outcome. As a result, multivariate meta-analytic methods for the assimilation of treatment effects in systematic reviews of RCTs have received increasing attention in the literature. These methods show promise with respect to bias reduction and efficiency gain compared to univariate meta-analysis. However, most methods for multivariate meta-analysis have focused on pairwise treatment comparisons (i.e., when the number of treatments is two). Current methods for mixed treatment comparisons (MTC) meta-analysis (i.e., when the number of treatments is more than two) have focused on univariate or very recently, bivariate outcomes. To broaden their application, we propose a framework for MTC meta-analysis of multivariate (≥ 2) outcomes where the correlations among multivariate outcomes within- and between-studies are accounted for through copulas, and the joint modeling of multivariate random effects, respectively. We consider a Bayesian hierarchical model using Markov Chain Monte Carlo methods for estimation. An important feature of the proposed framework is that it allows for borrowing of information across correlated outcomes. We show via simulation that our approach reduces the impact of outcome reporting bias (ORB) in a variety of missing outcome scenarios. We apply the method to a systematic review of RCTs of pharmacological treatments for alcohol dependence, which tends to report multiple outcomes potentially subject to ORB.

Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures

A critical step in systematic reviews of potential health hazards is the structured evaluation of the strengths and weaknesses of the included studies; risk of bias is a term often used to represent this process, specifically with respect to the evaluation of systematic errors that can lead to inaccurate (biased) results (i.e. focusing on internal validity). Systematic review methods developed in the clinical medicine arena have been adapted for use in evaluating environmental health hazards; this expansion raises questions about the scope of risk of bias tools and the extent to which they capture the elements that can affect the interpretation of results from environmental and occupational epidemiology studies and in vivo animal toxicology studies, (the studies typically available for assessment of risk of chemicals). One such element, described here as “sensitivity”, is a measure of the ability of a study to detect a true effect or hazard. This concept is similar to the concept of the sensitivity of an assay; an insensitive study may fail to show a difference that truly exists, leading to a false conclusion of no effect. Factors relating to study sensitivity should be evaluated in a systematic manner with the same rigor as the evaluation of other elements within a risk of bias framework. We discuss the importance of this component for the interpretation of individual studies, examine approaches proposed or in use to address it, and describe how it relates to other evaluation components. The evaluation domains contained within a risk of bias tool can include, or can be modified to include, some features relating to study sensitivity; the explicit inclusion of these sensitivity criteria with the same rigor and at the same stage of study evaluation as other bias-related criteria can improve the evaluation process. In some cases, these and other features may be better addressed through a separate sensitivity domain. The combined evaluation of risk of bias and sensitivity can be used to identify the most informative studies, to evaluate the confidence of the findings from individual studies and to identify those study elements that may help to explain heterogeneity across the body of literature.

Minimally important difference estimates and methods: a protocol

IntroductionPatient-reported outcomes (PROs) are often the outcomes of greatest importance to patients. The minimally important difference (MID) provides a measure of the smallest change in the PRO that patients perceive...

Transporting Evidence-Based Parenting Programs for Child Problem Behavior (Age 3–10) Between Countries: Systematic Review and Meta-Analysis

There has been rapid global dissemination of parenting interventions, yet little is known about their effectiveness when transported to countries different from where they originated, or about factors influencing success. This is the first systematic attempt to address this issue, focusing on interventions for reducing child behavior problems. Stage 1 identified evidence-based parenting interventions showing robust effects in systematic reviews; Stage 2 identified trials of these interventions in a new country. Systematic review/meta-analysis of transported programs was followed by subgroup analyses by trial- and country-level cultural, resource, and policy factors. We found 17 transported trials of 4 interventions, originating in United States or Australia, tested in 10 countries in 5 regions, (n = 1,558 children). Effects on child behavior were substantial (SMD −.71) in the (14) randomized trials, but nonsignificant in the (3) nonrandomized trials. Subgroup analyses of randomized trials found no association between effect size and participant or intervention factors (e.g., program brand, staffing). Interventions transported to “western” countries showed comparable effects to trials in origin countries; however, effects were stronger when interventions were transported to culturally more distant regions. Effects were higher in countries with survival-focused family/childrearing values than those ranked more individualistic. There were no differences in effects by country-level policy or resource factors. Contrary to common belief, parenting interventions appear to be at least as effective when transported to countries that are more different culturally, and in service provision, than those in which they were developed. Extensive adaptation did not appear necessary for successful transportation.

Glutamine and antioxidants: status of their use in critical illness.

Many studies in critically ill patients have addressed enteral or parenteral supplementation of glutamine and antioxidants to counteract assumed deficiencies and induce immune-modulating effects to reduce infections and improve outcome. Older studies showed marked reductions in mortality, infectious morbidity and length of stay. Recent studies no longer show beneficial effects and in contrast even demonstrated increased mortality. This opiniating review focuses on the latest information and the consequences for the use of glutamine and antioxidants in critically ill patients. Positive effects in systematic reviews and meta-analyses are based on results from older, smaller and mainly single-centre studies. New information has challenged the conditional deficiency hypothesis concerning glutamine in critically ill patients. The recent REDOXS and MetaPlus trials studying the effects of glutamine, selenium and other antioxidants have shown no benefits and increased mortality. Given that the first dictum in medicine is to do no harm, we cannot be confident that immune-modulating nutrient supplementation with glutamine and antioxidants is effective and well tolerated for critically ill patients. Until more data are available, it is probably better not to routinely administer glutamine and antioxidants in nonphysiological doses to mechanically ventilated critically ill patients.

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods.

BackgroundThresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.MethodsMethodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.ResultsWe propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (≈ the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.ConclusionsIf followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.Electronic supplementary materialThe online version of this article (doi:10.1186/1471-2288-14-120) contains supplementary material, which is available to authorized users.

Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm

BackgroundSystematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed).MethodsA health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa.ResultsThe proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance.ConclusionsUsers of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.

Tips and tricks for understanding and using SR results—no 13: adverse effects in systematic reviews

AbstractThis is the 13th ‘Tips and tricks for understanding and using SR results’ in Evidence‐Based Child Health. Like the previous articles, it is aimed at helping readers to understand the results of systematic reviews and to use the results in clinical practice. This time, we focus on adverse effects in systematic reviews. The information in this article is based on earlier papers, the Cochrane Handbook, and the collective experience of the authors in teaching evidence‐based medicine. Copyright © 2009 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd. The Cochrane Collaboration

The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed

Background and Objective Publication bias and other sample size effects are issues for meta-analyses of test accuracy, as for randomized trials. We investigate limitations of standard funnel plots and tests when applied to meta-analyses of test accuracy and look for improved methods. Methods Type I and type II error rates for existing and alternative tests of sample size effects were estimated and compared in simulated meta-analyses of test accuracy. Results Type I error rates for the Begg, Egger, and Macaskill tests are inflated for typical diagnostic odds ratios (DOR), when disease prevalence differs from 50% and when thresholds favor sensitivity over specificity or vice versa. Regression and correlation tests based on functions of effective sample size are valid, if occasionally conservative, tests for sample size effects. Empirical evidence suggests that they have adequate power to be useful tests. When DORs are heterogeneous, however, all tests of funnel plot asymmetry have low power. Conclusion Existing tests that use standard errors of odds ratios are likely to be seriously misleading if applied to meta-analyses of test accuracy. The effective sample size funnel plot and associated regression test of asymmetry should be used to detect publication bias and other sample size related effects.

Assessing harmful effects in systematic reviews.

BackgroundBalanced decisions about health care interventions require reliable evidence on harms as well as benefits. Most systematic reviews focus on efficacy and randomised trials, for which the methodology is well established. Methods to systematically review harmful effects are less well developed and there are few sources of guidance for researchers. We present our own recent experience of conducting systematic reviews of harmful effects and make suggestions for future practice and further research.MethodsWe described and compared the methods used in three systematic reviews. Our evaluation focused on the review question, study designs and quality assessment.ResultsOne review question focused on providing information on specific harmful effects to furnish an economic model, the other two addressed much broader questions. All three reviews included randomised and observational data, although each defined the inclusion criteria differently. Standard methods were used to assess study quality. Various practical problems were encountered in applying the study design inclusion criteria and assessing quality, mainly because of poor study design, inadequate reporting and the limitations of existing tools. All three reviews generated a large volume of work that did not yield much useful information for health care decision makers. The key areas for improvement we identified were focusing the review question and developing methods for quality assessment of studies of harmful effects.ConclusionsSystematic reviews of harmful effects are more likely to yield information pertinent to clinical decision-making if they address a focused question. This will enable clear decisions to be made about the type of research to include in the review. The methodology for assessing the quality of harmful effects data in systematic reviews requires further development.