Uncovering null effects in null fields: the case of homeopathy

Abstract

Sigurdson, Sainani, and Ioannidis (this journal) discussed homeopathy as a prototypical example of a "null field" where true effects are nonexistent and positive effect sizes reflect bias only. Based on a sample of published randomized placebo-controlled trials, they observed a surprisingly large effect in favor of homeopathy (Hedges' g=0.36 Z g=0.36). In this comment, we propose selective publication of significant results as a parsimonious explanation of the overall bias evident in this field. We re-analyzed the data of Sigurdson and collaborators using a meta-analytic mixture model that accounts for selective publishing with two parameters only, (1) the true homeopathy effect and (2) the proportion of results published only when statistically significant in the predicted direction. The mixture model fitted the data. As expected, the estimate of the true homeopathy effect reduces to almost zero (dˆ=0.05, 95% CI: [-0.05 - 0.16]) when taking selective publishing into account. Inclusion of effect size measures adjusting for selective publication practices should become routine practice in meta-analyses. Null fields not only provide useful benchmarks for the overall bias evident in a field. They are also important for testing explanations of this bias and validating adjusted effect size measures.