Do acupuncture trials have lower risk of bias over the last five decades? A methodological study of 4 715 randomized controlled trials.

Youlin Long,Liang Du,Jin Huang,Qiong Guo,Rui Chen,Shanxia Luo,Vance Berger

doi:10.1371/journal.pone.0234491

Abstract

ObjectiveTo evaluate the change of the risk of bias (RoB) of acupuncture randomized controlled trials (RCTs) in the past five decades.MethodsMultiple databases were searched. We included RCTs identified from systematic reviews (SRs) on acupuncture. General characteristics and RoB judgment for each domain were extracted. The proportions of RCTs at high and unclear RoB were calculated and the changes were examined by the Mann-Kendall test.ResultsWe included 368 SRs including 4 715 RCTs. The rates of RCTs at unclear RoB were the highest in allocation concealment (63%), and at the lowest in incomplete outcome data (35%); in the last five decades, statistically significant reductions were found for random sequence generation (P < 0.001) and selective reporting (P = 0.01), and increases for blinding of participants and personnel (P < 0.001), blinding of outcome assessment (P < 0.001) and incomplete outcome data (P = 0.04). For the proportions of RCTs at high RoB, blinding of participants and personnel (47%) and blinding of outcome assessment (35%) were the poorest domains; there were no significant differences in changes for all domains.ConclusionsAlthough improvements concerning unclear risk were observed for random sequence generation and selective reporting, major issues remain for allocation concealment and blinding. It is imperative to use valid randomization, specify how it is conducted, and try to test for selection bias and the success of masking by using the Berger Exner test.

Highlights

The Grading of Recommendations Assessment, Development and Evaluation (GRADE) system was often used to evaluate the quality of evidence [1]
The rates of randomized controlled trials (RCTs) at unclear risk of bias (RoB) were the highest in allocation concealment (63%), and at the lowest in incomplete outcome data (35%); in the last five decades, statistically significant reductions were found for random sequence generation (P < 0.001) and selective reporting (P = 0.01), and increases for blinding of participants and personnel (P < 0.001), blinding of outcome assessment (P < 0.001) and incomplete outcome data (P = 0.04)
Improvements concerning unclear risk were observed for random sequence generation and selective reporting, major issues remain for allocation concealment and blinding

Summary

Introduction

The Grading of Recommendations Assessment, Development and Evaluation (GRADE) system was often used to evaluate the quality of evidence [1]. Several studies have shown that a high risk of bias (RoB) was the main reason for rating down the quality of evidence [3,4,5], and the conclusion cannot be trusted if randomized controlled trials (RCTs) were rated as high RoB [6]. RoB of RCTs included selective reporting, failure to perform randomization, allocation concealment, blinding, and failure to consider the intention-to-treat principle for loss to follow-up [8]. Such limitations may lead to an underestimation or overestimation of the true effects of interventions [9]. The proportions of RCTs considered by the systematic reviewers to be at unclear and high RoB could be used as surrogates for poor reporting and inadequate methods, respectively [11]

Methods

Results

Conclusion