A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects

Abstract

ObjectiveTo explore the impact of methodological choices on the results of meta-analyses (MAs), with acupuncture for smoking cessation as a case study. Study Design and SettingAfter performing an umbrella review (using MEDLINE, the COCHRANE Library, the Wan Fang database, and the Chinese Journal Full-text Database/March 2018) of MAs exploring the use of acupuncture for smoking cessation, we extracted all randomized controlled trials. Numerous MAs were performed as per every possible combination of various methodological choices (e.g., characteristics of the intervention and control procedures, outcome, publication status, language) to assess their vibration of effects or more precisely the existence of a Janus effect, that is, whether the 10th and 90th percentiles in the distribution of effect sizes were in opposite directions. ResultsAfter including 7 MAs and 39 randomized controlled trials, we performed 496,528 MAs. The effect size was negative at the 10th percentile (-0.1, favoring controls) and positive at the 90th percentile (1.17, favoring acupuncture). In all, 104,491 MAs showed a statistically significant difference in favor of acupuncture, whereas 392,037 failed to demonstrate the efficacy of acupuncture (including 96 that showed a statistically significant difference in favor of the control). ConclusionThe methodological choices made in performing pairwise MAs can result in substantial vibration of effects, occasionally leading to opposite results.