Abstract
BackgroundSystematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of health interventions, specifically for reviews that planned to incorporate evidence from RCT and/or NRS.MethodsWe evaluated a random sample of non-Cochrane protocols for systematic reviews of interventions registered in PROSPERO between January 1 and October 12, 2018. For each protocol, we extracted data on the types of studies to be included (RCT and/or NRS) as well as the name and number of RoB tools planned to be used according to study design. We then conducted a longitudinal analysis of the most commonly reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records by year since the inception of the database (2011 to December 7, 2018), restricting the keyword search to the “Risk of bias (quality) assessment” field.ResultsIn total, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a single RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to use multiple RoB tools. Looking at trends over time, the planned use of the Cochrane RoB Tool and ROBINS-I seems to be increasing.ConclusionsWhile RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of study designs comprising NRS, multiple RoB tools tailored to specific designs may be required.
Highlights
Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT)
Included 2018 PROSPERO records In total, 471 of the 500 PROSPERO protocols from the 2018 random sample were included in the final analysis
We found that some protocol authors were not being specific in identifying RoB tools a priori or were inappropriately applying tools to assess risk of bias for non-randomized study (NRS)
Summary
Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The last two decades have seen a proliferation of tools developed to evaluate the risk of bias in NRS; a 2012 systematic review identified 74 tools developed for quality appraisal, of which risk of bias is a component, of non-experimental studies [7] None of these existing NRS quality appraisal tools are currently accepted as the gold standard [1, 8], and it is unclear which tools are the most rigorous and practical. If multiple checklists are used in systematic reviews that incorporate multiple study designs, review authors need to consider whether these tools are comparable, in terms of rating evidence within a grading system or when using a cut-off to determine which studies to include in a systematic review or meta-analysis
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
