Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews

Abstract

Publication bias is an over-representation of statistically significant results in the published literature and may exaggerate summary effect estimates in oncology systematic reviews. Omitting non-significant results in systematic reviews may therefore affect clinical decision-making. We investigate ways that systematic reviewers attempted to limit publication bias during the search process as well as the statistical methods used to evaluate it. For a subset of reviews not reporting publication bias evaluations, we carried out our own assessments for publication bias to determine its likelihood among these reviews. We examined systematic reviews from the top five highest impact factor oncology journals published between 2007 and 2015. Systematic reviews were screened for eligibility and qualifying reviews (n = 182) were coded for relevant publication bias study characteristics by two authors. A re-analysis of reviews not initially evaluating for publication bias was carried out using Egger's regression, trim-and-fill, and selection models. Of the 182 systematic reviews, roughly half carried out a hand search to locate additional studies. Conference abstracts were the most commonly reported form of gray literature, followed by clinical trials registries. Fifty-one reviews reported publication bias evaluations. The most common method was the funnel plot (80%, 41/51) followed by Egger's regression (59%, 30/51) and Begg's test (43%, 22/51). Our publication bias evaluations on non-reporting reviews suggest that the degree of publication bias depends on the method employed. Our study shows publication bias assessments are not frequently used in oncology systematic reviews. Furthermore, evidence of publication bias was found in a subset of non-reporting reviews. Systematic reviewers in oncology are encouraged to conduct such analyses when appropriate and to employ more robust methods for both mitigating and evaluating publication bias.