A Case Study of Discordant Overlapping Meta-Analyses: Vitamin D Supplements and Fracture

Mark J Bolland,Andrew Grey,Joseph Devaney

doi:10.1371/journal.pone.0115934

Abstract

BackgroundOverlapping meta-analyses on the same topic are now very common, and discordant results often occur. To explore why discordant results arise, we examined a common topic for overlapping meta-analyses- vitamin D supplements and fracture.Methods and FindingsWe identified 24 meta-analyses of vitamin D (with or without calcium) and fracture in a PubMed search in October 2013, and analysed a sample of 7 meta-analyses in the highest ranking general medicine journals. We used the AMSTAR tool to assess the quality of the meta-analyses, and compared their methodologies, analytic techniques and results. Applying the AMSTAR tool suggested the meta-analyses were generally of high quality. Despite this, there were important differences in trial selection, data extraction, and analytical methods that were only apparent after detailed assessment. 25 trials were included in at least one meta-analysis. Four meta-analyses included all eligible trials according to the stated inclusion and exclusion criteria, but the other 3 meta-analyses “missed” between 3 and 8 trials, and 2 meta-analyses included apparently ineligible trials. The relative risks used for individual trials differed between meta-analyses for total fracture in 10 of 15 trials, and for hip fracture in 6 of 12 trials, because of different outcome definitions and analytic approaches. The majority of differences (11/16) led to more favourable estimates of vitamin D efficacy compared to estimates derived from unadjusted intention-to-treat analyses using all randomised participants. The conclusions of the meta-analyses were discordant, ranging from strong statements that vitamin D prevents fractures to equally strong statements that vitamin D without calcium does not prevent fractures.ConclusionsSubstantial differences in trial selection, outcome definition and analytic methods between overlapping meta-analyses led to discordant estimates of the efficacy of vitamin D for fracture prevention. Strategies for conducting and reporting overlapping meta-analyses are required, to improve their accuracy and transparency.

Highlights

The number of meta-analyses published in recent years has dramatically increased [1, 2]
To explore why discordant results arise, we examined a common topic for overlapping meta-analyses- vitamin D supplements and fracture
Among overlapping meta-analyses of vitamin D and fracture, there were substantial differences in the trials included, the data used from each trial, the analytical approach adopted, and the conclusions drawn, despite the metaanalyses being of high quality and published in the highest ranking medical journals

Summary

Introduction

The number of meta-analyses published in recent years has dramatically increased [1, 2]. Overlapping meta-analyses may report discordant results and conclusions, as the number of such analyses increases. The consequences of this include contradictory recommendations for clinical practice, confusion amongst clinicians and their patients, and public disenchantment with clinical science. To explore why discordant results arise, we examined a common topic for overlapping meta-analyses- vitamin D supplements and fracture. Methods and Findings: We identified 24 meta-analyses of vitamin D (with or without calcium) and fracture in a PubMed search in October 2013, and analysed a sample of 7 meta-analyses in the highest ranking general medicine journals. Conclusions: Substantial differences in trial selection, outcome definition and analytic methods between overlapping meta-analyses led to discordant estimates of the efficacy of vitamin D for fracture prevention. Strategies for conducting and PLOS ONE | DOI:10.1371/journal.pone.0115934 December 31, 2014

Results

Discussion

Conclusion