On Knowing What We Do Not Know

Abstract

The performance of methods for detecting publication bias in meta-analysis was evaluated using Monte Carlo methods. Four methods of bias detection were investigated: Begg’s rank correlation, Egger’s regression, funnel plot regression, and trim and fill. Five factors were included in the simulation design: number of primary studies in each meta-analysis, sample sizes of primary studies, population variances in primary studies, magnitude of population effect size, and magnitude of selection bias. Results were evaluated in terms of Type I error control and statistical power. Results suggest poor Type I error control in many conditions for all of the methods examined. One exception was the Begg’s rank correlation method using sample size rather than the estimated variance. Statistical power was typically very low for conditions in which Type I error rates were adequately controlled, although power increased with larger sample sizes in the primary studies and larger numbers of studies in the meta-analysis.