Bias In Meta-analysis Research Articles

Assessing small study effects and publication bias in orthodontic meta-analyses: a meta-epidemiological study

The aim of this study was to examine the presence and extent of small study effects and publication bias in meta-analyses (MAs) based on orthodontic studies. Following an extensive literature search, 25 MAs including 313 studies were identified and were possible to be re-analyzed. For the assessment of publication bias, contour-enhanced funnel plots were examined and their symmetry was tested using the Begg and Mazumdar rank correlation and Egger's linear regression tests. Robustness of MAs' results to publication bias was examined by Rosenthal's failsafe N, and adjusted effect sizes were calculated after consideration of publication bias using Duval and Tweedie's "trim and fill" procedure. Only few of the originally published MAs assessed the existence and effect of publication bias and some only partially. Inspection of the funnel plots indicated possible asymmetry, which was confirmed by Begg and Mazumdar's test in 12 % and by Egger's test in 28 % of the MAs. According to Rosenthal's criterion, 62 % of the MAs were robust, while adjusted effect estimates with unpublished studies differed from little to great from the unadjusted ones. Pooling of Egger's intercepts of included MAs indicated that evidence of asymmetry was found in the orthodontic literature, which was accentuated in medical journals and in diagnostic MAs. Small study effects and publication bias can often distort results of MAs. Since indications of publication bias in orthodontics were found, the influence of small trials on estimated treatment effects should be routinely and more carefully assessed by authors conducting MAs.

A model-based correction for outcome reporting bias in meta-analysis

It is often suspected (or known) that outcomes published in medical trials are selectively reported. A systematic review for a particular outcome of interest can only include studies where that outcome was reported and so may omit, for example, a study that has considered several outcome measures but only reports those giving significant results. Using the methodology of the Outcome Reporting Bias (ORB) in Trials study of (Kirkham and others, 2010. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. British Medical Journal 340, c365), we suggest a likelihood-based model for estimating the effect of ORB on confidence intervals and p-values in meta-analysis. Correcting for bias has the effect of moving estimated treatment effects toward the null and hence more cautious assessments of significance. The bias can be very substantial, sometimes sufficient to completely overturn previous claims of significance. We re-analyze two contrasting examples, and derive a simple fixed effects approximation that can be used to give an initial estimate of the effect of ORB in practice.

Development of a method for adjusting trial results for biases in meta-analysis

Randomised trials vary in methodological quality, and flaws in trial conduct can lead to biased estimation of the intervention effect. If a meta-analysis makes no allowance for methodological flaws, there's a danger that the results could be biased and over-precise, or that some lower-quality trials could be given too much influence in the meta-analysis. Cochrane review authors now use a specific ‘Risk of Bias' tool for assessing potential biases in the trials included in their review. This information is a valuable resource, but it is currently not clear how to use the ‘Risk of Bias' information to reduce the impact of suspected biases on the meta-analyses presented in the review. In this work, we are developing a method for adjusting trial results for biases in meta-analysis, making use of ‘Risk of Bias' information for each trial and also generic external evidence on biases. We aim to integrate two existing methods for adjusting for bias in meta-analysis, in order to gain the advantages of both. One of the existing methods uses generic evidence on bias from an external collection of meta-analyses, while the other method relies on detailed assessment of the methodological quality of each trial and elicitation of opinion about the degree of bias likely to result. We consider several different approaches for integrating generic and trial-specific evidence on bias and discuss the practicalities of these.

Assessing the presence of selection bias in meta-analyses of randomised trials using baseline heterogeneity

Results Five of the 12 meta-analyses exhibited heterogeneity (I >0.30) in age differences, when there should have been none, with two having significant or substantial heterogeneity (I > 0.50). In three meta-analyses we found that the age of intervention group was statistically significantly higher than in the control group. Two meta-analysis had a distribution of p-values that were inconsistent with chance. Meta-regression explained some of the observed heterogeneity in two meta-analyses as a consequence of poor allocation concealment. Conclusions The majority of a sample of recent meta-analyses showed that there were signs of imbalance and or heterogeneity in ages between treatment groups, when there should have been none. Systematic reviewers might consider using the techniques described here to assess the validity of their findings.

PC-FACS

PC-FACS

Comment lire une méta-analyse ?

Meta-analysis is aimed at assessing an exhaustive, unbiased, reproducible, quantified and accurate synthesis of a research problem. It is sustained by a systematic review of the literature and has statistical particularities. Sources of error and bias are numerous. In this paper, we describe them following the methodology steps of a well-conducted meta-analysis. Causes of divergent conclusions of meta-analyses are described and illustrated with the example of cancer risk assessment in TNF inhibitor-treated rheumatoid arthritis patients. Eventually, this article provides key-points to help readers to detect sources of error and bias in meta-analyses.

Funnel plots for population-based cancer survival: principles, methods and applications

Funnel plots are graphical tools designed to detect excessive variation in performance indicators by simple visual inspection of the data. Their main use in the biomedical domain so far has been to detect publication bias in meta-analyses, but they have also been recommended as the most appropriate way to display performance indicators for a vast range of health-related outcomes. Here, we extend the use of funnel plots to population-based cancer survival and several related measures. We present three applications to familiarise the reader with their interpretation. We propose funnel plots for various cancer survival measures, as well as age-standardised survival, trends in survival and excess hazard ratios. We describe the components of a funnel plot and the formulae for the construction of the control limits for each of these survival measures. We include three transformations to construct the control limits for the survival function: complementary log-log, logit and logarithmic transformations. We present applications of funnel plots to explore the following: (i) small-area and temporal variation in cancer survival; (ii) racial and geographical variation in cancer survival; and (iii) geographical variation in the excess hazard of death. Funnel plots provide a simple and informative graphical tool to display geographical variation and trend in a range of cancer survival measures. We recommend their use as a routine instrument for cancer survival comparisons, to inform health policy makers in planning and assessing cancer policies. We advocate the use of the complementary log-log or logit transformation to construct the control limits for the survival function.

Detecting, quantifying and adjusting for publication bias in meta-analyses: protocol of a systematic review on methods

BackgroundHealth professionals and policymakers aspire to make healthcare decisions based on the entire relevant research evidence. This, however, can rarely be achieved because a considerable amount of research findings are not published, especially in case of ‘negative’ results - a phenomenon widely recognized as publication bias. Different methods of detecting, quantifying and adjusting for publication bias in meta-analyses have been described in the literature, such as graphical approaches and formal statistical tests to detect publication bias, and statistical approaches to modify effect sizes to adjust a pooled estimate when the presence of publication bias is suspected. An up-to-date systematic review of the existing methods is lacking.Methods/designThe objectives of this systematic review are as follows:• To systematically review methodological articles which focus on non-publication of studies and to describe methods of detecting and/or quantifying and/or adjusting for publication bias in meta-analyses.• To appraise strengths and weaknesses of methods, the resources they require, and the conditions under which the method could be used, based on findings of included studies.We will systematically search Web of Science, Medline, and the Cochrane Library for methodological articles that describe at least one method of detecting and/or quantifying and/or adjusting for publication bias in meta-analyses. A dedicated data extraction form is developed and pilot-tested. Working in teams of two, we will independently extract relevant information from each eligible article. As this will be a qualitative systematic review, data reporting will involve a descriptive summary.DiscussionResults are expected to be publicly available in mid 2013. This systematic review together with the results of other systematic reviews of the OPEN project (To Overcome Failure to Publish Negative Findings) will serve as a basis for the development of future policies and guidelines regarding the assessment and handling of publication bias in meta-analyses.

Meta‐analysis of clinical trials with early stopping: an investigation of potential bias

Clinical trials that stop early for benefit have a treatment difference that overestimates the true effect. The consequences of this fact have been extensively debated in the literature. Some researchers argue that early stopping, or truncation, is an important source of bias in treatment effect estimates, particularly when truncated studies are incorporated into meta-analyses. Such claims are bound to lead some systematic reviewers to consider excluding truncated studies from evidence synthesis. We therefore investigated the implications of this strategy by examining the properties of sequentially monitored studies conditional on reaching the final analysis. As well as estimation bias, we studied information bias measured by the difference between standard measures of statistical information, such as sample size, and the actual information based on the conditional sampling distribution. We found that excluding truncated studies leads to underestimation of treatment effects and overestimation of information. Importantly, the information bias increases with the estimation bias, meaning that greater estimation bias is accompanied by greater overweighting in a meta-analysis. Simulations of meta-analyses confirmed that the bias from excluding truncated studies can be substantial. In contrast, when meta-analyses included truncated studies, treatment effect estimates were essentially unbiased. Previous analyses comparing treatment effects in truncated and non-truncated studies are shown not to be indicative of bias in truncated studies. We conclude that early stopping of clinical trials is not a substantive source of bias in meta-analyses and recommend that all studies, both truncated and non-truncated, be included in evidence synthesis.

Potential Bias in Meta-Analyses of Effect Sizes in Imaging Genetics

The penetrance of genetic variation has been assumed to be higher at the level of neural phenotypes than at the level of behavioral phenotypes. One of the few attempts to validate this assumption is the study of Rose and Donohoe published in this issue. In this article, we will address 2 methodological issues we believe have to be considered for a better understanding of the present results. We briefly discuss potential solutions that might also help improve future meta-analyses of effect sizes in neuroimaging data.

Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for depression.

The aim of this study was to assess whether systematic reviews investigating psychotherapeutic interventions for depression are affected by publication bias. Only homogeneous data sets were included, as heterogeneous data sets can distort statistical tests of publication bias. We applied Begg and Mazumdar's adjusted rank correlation test, Egger's regression analysis, and the trim and fill procedure to assess the presence and magnitude of publication bias in all homogeneous data sets of systematic reviews published up to September 2010. Thirty-one data sets reported in 19 meta-analyses fulfilled our inclusion criteria. Significant bias was detected in 5 (16.13%; rank correlation test) and 6 (19.35%; Egger's regression analysis) of these data sets. Applying the trim and fill procedure to amend presumably missing studies rarely changed the assessment of the efficacy of therapeutic interventions, with 2 exceptions. In 1 data set psychotherapy was no longer found to be significantly more efficacious than pharmacotherapy in reducing dropout at posttreatment when publication bias was taken into account. In the 2nd data set, after correcting for publication bias, there was no longer evidence that depressed patients without comorbid personality disorder profited more from psychotherapy and pharmacotherapy than patients with comorbid personality disorder. The results suggest that taken together, psychotherapy research for depression is only marginally affected by the selective reporting of positive outcomes. With 2 notable exceptions, correcting for publication bias did not change the evaluation of the efficacy of psychotherapeutic interventions.

The Rules of the Game Called Psychological Science

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

A Likelihood-Based Sensitivity Analysis for Publication Bias in Meta-Analysis

Summary A common conjecture in the study of publication bias is that studies reporting a significant result are more likely to be selected for review than studies whose results are inconclusive. We envisage a population of studies following the standard random-effects model of meta-analysis, and a selection probability given by a function of the study's ‘t-statistic’. In practice it is difficult to estimate this function, and hence difficult to estimate its associated bias correction. The paper suggests the more modest aim of a sensitivity analysis in which the treatment effect is estimated by maximum likelihood constrained by given values of the marginal probability of selection. This gives a graphical summary of how the inference from a meta-analysis changes as we allow for increasing selection (as the marginal selection probability decreases from 1), with an associated diagnostic plot comparing the observed treatment effects with their fitted values implied by the corresponding selection model. The approach is motivated by a medical example in which the highly significant result of a published meta-analysis was subsequently overturned by the results of a large-scale clinical trial.

Performance of the Trim and Fill Method in Adjusting for the Publication Bias in Meta-Analysis of Continuous Data

Publication bias in meta-analysis is a serious issue as it may lead to biased estimates which appear to be precise. A popular method for detecting and adjusting the publication bias is the trim and fill method. This study uses simulated meta-analysis to quantify the effects of publication bias on the overall meta-analysis estimates of continuous data where the absolute mean difference was utilized as the measure of effect. It additionally evaluates the performance of the trim and fills method for adjusting the publication bias in terms of statistical bias, the standard errors and the coverage probability. The results demonstrate that if the publication bias is not adjusted it could lead to up to 40% biased in treatment effect estimates. Utilization of the trim and fill method has reduced the bias in the overall effect estimate by more than half. It is optimum in presence of moderate underlying bias but has minimal effects in presence of low and severe bias. Additionally, the trim and fill method improves the coverage probability by more than half when subjected to the same level of publication bias as those of the unadjusted data. However, the method tends to produce false positive results. A sensitivity analysis suggests that the trim and fill method will incorrectly adjust the data for publication bias between 10-45% of the time (for the 5% nominal level). Although the data was incorrectly adjusted, it was found that the Percentage Relative Bias (PRB) introduced into the estimates due to this adjustment is minimal (min: 0.007%, max: 0.109%) and coverage probability for estimates based on this incorrectly adjusted data is not significantly different from those of which is correctly not adjusted. Therefore the trim and fill method is recommended be routinely used when conducting meta-analysis.

Publication Bias

This article offers three contributions for conducting meta-analytic reviews in education research. First, we review publication bias and the challenges it presents for meta-analytic researchers. Second, we review the most recent and optimal techniques for evaluating the presence and influence of publication bias in meta-analyses. We then re-analyze two sets of meta-analytic data from the literacy literature that have been published in different journals. The analyses serve as case examples of the techniques reviewed, and the results demonstrate a range of findings from noticeable instances of publication bias to minimal or no bias. The conclusions have important implications for research, policymaking, and practice. Finally, we discuss recommendations for future research.

A Comparison of Methods to Detect Publication Bias for Meta-analysis of Continuous Data

Publication bias is a serious problem in meta-analysis. Various methods have been developed to detect the presence of publication bias in meta-analysis. These methods have been assessed and compared in many dichotomous studies utilizing the log-odds ratio as the measure of effect. This paper evaluates and compares the performance of three popular methods, namely the Egger’s linear regression method, the Begg and Mazumdar’s rank correlation method and the Duvall and Tweedie’s trim and fill method, on meta-analysis of continuous data. The data comprised simulated meta-analyses with different levels of primary studies in the absence and presence of induced publication bias. The performance of these methods were measured through the power and type 1 error rate for the tests. The results suggest the trim and fill method to be superior in terms of its ability to detect publication bias when it exists, even in presence of only 5% unpublished studies. However this method is not recommended for large meta-analysis as it produces high rate of false-positive results. Both linear regression and rank correlation method performed relatively well in moderate bias but should be avoided in small meta-analysis as their power is very low in this data.

Adjusting for Partial Verification or Workup Bias in Meta-Analyses of Diagnostic Accuracy Studies

A key requirement in the design of diagnostic accuracy studies is that all study participants receive both the test under evaluation and the reference standard test. For a variety of practical and ethical reasons, sometimes only a proportion of patients receive the reference standard, which can bias the accuracy estimates. Numerous methods have been described for correcting this partial verification bias or workup bias in individual studies. In this article, the authors describe a Bayesian method for obtaining adjusted results from a diagnostic meta-analysis when partial verification or workup bias is present in a subset of the primary studies. The method corrects for verification bias without having to exclude primary studies with verification bias, thus preserving the main advantages of a meta-analysis: increased precision and better generalizability. The results of this method are compared with the existing methods for dealing with verification bias in diagnostic meta-analyses. For illustration, the authors use empirical data from a systematic review of studies of the accuracy of the immunohistochemistry test for diagnosis of human epidermal growth factor receptor 2 status in breast cancer patients.

Generalization of trim and fill for application in meta‐regression

Trim and fill is a popular method of accounting for publication bias in meta-analysis. However, the use of trim and fill is limited to the setting in which all meta-analyzed studies represent a true common effect. In many practical settings, within-study effect estimates are a function of some covariate. Because methods of accounting for publication bias in meta-regression have received little attention, we propose here a generalization of trim and fill for application in meta-regression. The proposed algorithm preserves the computational features of trim and fill and adds only an assumption of symmetry in the hypothesized distribution of the measured covariate. By simulation, we evaluate properties (mean bias, root mean squared error, and coverage probability) of meta-regression parameter estimates and corresponding confidence intervals with application of the proposed algorithm in a range of scenarios, including violation of the aforementioned assumption of symmetry. We also evaluate the performance of common estimators of the number of suppressed studies. In general, we show that the proposed algorithm is successful in identifying suppression of studies and reducing the bias in regression parameters derived from the analysis of the augmented set of studies. We apply the proposed algorithm to an analysis of the effect of cognitive-behavioral therapy on the risk of recidivism. Copyright © 2012 John Wiley & Sons, Ltd.

Publication bias in psychological science: Comment on Ferguson and Brannick (2012).

It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.

Biases in clinical trials with sequential monitoring

Objectives It is well known that a sequentially monitored clinical trial that stops early for benefit has a crude treatment difference that overestimates the true treatment effect. This has led to extended debate in the literature, with some researchers arguing that early stopping is an important source of bias in meta-analyses of clinical trials. We therefore investigated the implications of excluding studies that stopped early, so-called truncated studies, from estimation of treatment effects.