Abstract
BackgroundSystematic reviews of randomised trials guide policy and healthcare decisions. Yet, we observed that some reviews judge randomised trials as high or unclear risk of bias (ROB) for sequence generation, potentially introducing bias. However, to date, the extent of this issue has not been well examined. We evaluated the consistency in the ROB assessment for sequence generation of randomised trials in Cochrane and non-Cochrane reviews, and explored the reviewers’ judgement of the quality of evidence for the related outcomes.MethodsCochrane intervention reviews (01/01/2017–31/03/2017) were retrieved from the Cochrane Database of Systematic Reviews. We also searched for systematic reviews in ten general medical journals with highest impact factors (01/01/2016–31/03/2017). We examined the proportion of reviews that rated the sequence generation domain as high, low or unclear risk of selection bias. For reviews that had rated any randomised trials as high or unclear risk of bias, we examined the proportion that had assessed the quality of evidence.ResultsOverall, 100 systematic reviews were included in our analysis. We evaluated 64 Cochrane reviews which comprised of 984 randomised trials; 0.8% (n = 8) and 52.2% (n = 514) were rated as high and unclear ROB for sequence generation respectively. We further evaluated 36 non-Cochrane reviews which comprised of 1376 trials; 5.8% (n = 80) and 39.6% (n = 545) were rated as high and unclear ROB respectively. Ninety percent (n = 10) of non-Cochrane reviews which rated randomised trials as high ROB for sequence generation did not report an underlying reason. All Cochrane reviews assessed the quality of evidence (GRADE). For the non-Cochrane reviews, only just over half had assessed the quality of evidence.ConclusionSystematic reviews of interventions frequently rate randomised trials as high or unclear ROB for sequence generation. In general, Cochrane reviews were more transparent than non-Cochrane reviews in ROB and quality of evidence assessment. The scientific community should more strongly promote consistent ROB assessment for sequence generation to minimise selection bias and support transparent quality of evidence assessment. Consistency ensures that appropriate conclusions are drawn from the data.
Highlights
Systematic reviews of randomised trials guide policy and healthcare decisions
We examine the consistency in the risk of bias (ROB) assessment of the sequence generation domain of the Cochrane ROB tool for randomised trials in Cochrane and non-Cochrane reviews that use this tool, and, if conducted, we further explored the authors’ judgement of the quality of evidence for the related outcomes
For reviews that had rated any randomised trials as having high or unclear ROB for sequence generation, we examined the proportion that downgraded the quality of evidence (GRADE) for study limitations for all primary outcomes that included these high/unclear ROB studies
Summary
Systematic reviews of randomised trials guide policy and healthcare decisions. Yet, we observed that some reviews judge randomised trials as high or unclear risk of bias (ROB) for sequence generation, potentially introducing bias. We evaluated the consistency in the ROB assessment for sequence generation of randomised trials in Cochrane and non-Cochrane reviews, and explored the reviewers’ judgement of the quality of evidence for the related outcomes. Randomised trials are regarded as the optimal design to evaluate the effectiveness of healthcare interventions, and systematic reviews of intervention trials are an indisputable asset to clinical decision-making and evidence-based practice [2]. We have observed that some Cochrane and non-Cochrane reviews which a priori exclude non-randomised trials still report sequence generation as high or unclear ROB for some trials. Whereas a judgement of unclear ROB may be due to poor reporting in primary studies, it creates uncertainty whether non-randomised trials were truly excluded from the review. A judgement of high ROB for sequence generation suggests that non-randomised studies have likely been included
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
