Risk of bias judgments for random sequence generation in Cochrane systematic reviews were\xa0frequently not in line with Cochrane Handbook

Ognjen Barcot,Ivana Vuka,Tina Poklepovic Pericic,Matija Boric,Marija Cavar,Svjetlana Dosenovic,Livia Puljak

doi:10.1186/s12874-019-0804-y

Ognjen Barcot, Ivana Vuka + Show 5 more

Open Access

https://doi.org/10.1186/s12874-019-0804-y

Copy DOI

Abstract

BackgroundAssessing the risk of bias (RoB) in included studies is one of the key methodological aspects of systematic reviews. Cochrane systematic reviews appraise RoB of randomised controlled trials (RCTs) with the Cochrane RoB tool. Detailed instructions for using the Cochrane RoB tool are provided in the Cochrane Handbook for Systematic Reviews of Interventions (The Cochrane Handbook). The purpose of this study was to analyse whether Cochrane authors use adequate judgments about the RoB for random sequence generation of RCTs included in Cochrane reviews.MethodsWe extracted authors’ judgments (high, low or unclear RoB) and supports for judgments (comments accompanying judgments which explain the rationale for a judgment) for random sequence generation of included RCTs from RoB tables of Cochrane reviews using automated data scraping. We categorised all supporting comments, analysed the number and type of various supporting comments and assessed adequacy of RoB judgment for randomisation in line with recommendations from the Cochrane Handbook.ResultsWe analysed 10,103 RCTs that were included in 704 Cochrane reviews. For 5,706 RCTs, randomisation was not described, but for the remaining RCTs, it was indicated that randomisation was performed using computer/software/internet (N = 2,850), random number table (N = 883), mechanical method (N = 359) or it was incomplete/inappropriate (N = 305).Overall, 1,220/10,103 trials (12%) did not have a RoB judgment in line with Cochrane Handbook guidance about randomisation. The highest proportion of misjudgements was found for trials with high RoB (28%), followed by those with low (20%) or unclear (3%). Therefore, one in eight judgments for the analysed domain in Cochrane reviews was not in line with Cochrane Handbook, and one in four if the judgment was "high risk".ConclusionAuthors of Cochrane reviews often make judgments about the RoB related to random sequence generation that are not in line with instructions given in the Cochrane Handbook, which compromises the reliability of the systematic reviews. Our results can help authors of both Cochrane and non-Cochrane reviews which use Cochrane RoB tool to avoid making common mistakes when assessing RoB in included trials.

Highlights

Assessing the risk of bias (RoB) in included studies is one of the key methodological aspects of systematic reviews
The objectives of this study were two-fold: to evaluate the rationales based on which judgments related to random sequence generation were made for trials in Cochrane reviews, and to investigate the proportion of inadequate judgments about randomisation based on independent reassessments using guidance from the Cochrane Handbook
There were 49 Randomized Controlled Trial (RCT) in which we found no specific reason for the missing RoB or domain, while 54 trials were not randomised, three webpage errors caused duplicate entries and six studies only had abstracts (Table 1)

Summary

Introduction

Assessing the risk of bias (RoB) in included studies is one of the key methodological aspects of systematic reviews. Randomised controlled trials (RCTs) are considered crucial for objectively testing the efficacy and safety of interventions [1]. They are often used to inform clinical practice and included in systematic reviews to obtain an even higher level of evidence through evidence synthesis. RCTs can be biased if flaws in their design and conduct lead to overestimation or underestimation of intervention effect estimates. Because of this potential for bias, assessment of the risk of bias (RoB) in trials is one of the usual methodological steps during the preparation of Cochrane systematic reviews. Bias has been defined as any systematic error that can negatively impact estimated effects of interventions and lead to incorrect conclusions about the efficacy and safety of analysed interventions [3]

Objectives

Methods

Results

Discussion

Conclusion