26 Is the revised cochrane risk of bias tool research ready for the era of open science and preregistration?

Abstract

Objectives Risk of bias (RoB) is an important to assess scientific evidence. Lack of detailed preregistration obfuscates reporting transparency and cast doubts about the required bijective relation between study protocol and the final scientific report of outcomes. This contribution examines whether the lack of preregistration affects the judgments required for risk of bias assessment according to the RoB 2.0 tool and whether the tool can adequatly capture flaws in preregistration. Method We examined the literature on RoB 2.0 and thoroughly evaluated the definition and criteria for the three different categories ‘high risk of bias’, ‘low risk of bias’, and some concerns of risk of bias’. Moreover, we investigated the literature on meta-science and performed a conceptual analysis of the epistemic merits and methodological benefits arising from various forms of preregistration. Accordingly, selective endpoint reporting, or endpoint modification raise serious and severe doubts about the scientific validity of biomedical randomized controlled trials (RCT) and preregistration is obligatory according to Article 35 of the Declaration of Helsinki of 2013. Results The upcoming revised Cochrane handbook introduces RoB 2.0 as a new research tool for assessing the risk of bias of an RCT. RoB 2.0 requires a comparison between the pre-specified analysis intentions and the reported analyses in order to assess potential selection bias of multiple outcomes or endpoints. In case a preregistered analysis plan is met, ‘low risk bias’ is assigned. ‘High risk of bias’ is assigned only if it is likely that reported outcomes have been selected based on the results, i.e. a deviation from the preregistered protocol is detected. If no information is available, RoB 2.0 suggests ‘some concern’. Furthermore, in cases where preregistration is lacking, RoB 2.0 suggests the methods section of an article as a source of the analysis intentions. Therefore, the lack of preregistration does not by default lead to the evaluation of a ‘high risk of bias’. Conclusions Although lack of preregistration can lead to ‘some concerns of risk of bias’, there is by default no assignment of ‘high risk of bias’even if a preregistration protocol is completely lacking. In light of the epistemic arguments in favour of preregistration, RoB 2.0 presents an untoward loophole in the risk of bias assessment with regard to selective outcome reporting or post hoc endpoint modification. Because any RoB 2.0 assessment is very effort-intensive and time-consuming, it is of utmost importance that all sort of biases are adequate considered and, thus, that future systematic reviews and meta-analysis benefit from risk of bias assessment tools that account for lack of preregistration as a source of ‘high risk of bias’ by default.