The ROBINS-I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta-analysis.

Abstract

There is a lack of evidence on the usage of the quality assessment tool-the Risk Of Bias In Nonrandomized Studies-of Interventions (ROBINS-I). This article aimed to measure the reliability, criterion validity, and feasibility of the ROBINS-I and the Newcastle-Ottawa Scale (NOS). A sample of systematic reviews or meta-analyses of observational studies were selected from Medline (2013-2017) and assessed by two reviewers using ROBINS-I and the NOS. We reported on reliability in terms of the first-order agreement coefficient (AC1) statistic. Correlation coefficient statistic was used to explore the criterion validity of the ROBINS-I. We compared the feasibility of the ROBINS-I and NOS by recording the time to complete an assessment and the instances where assessing was difficult. Five systematic reviews containing 41 cohort studies were finally included. Interobserver agreement on the individual domain of the ROBINS-I as well as the NOS was substantial with a mean AC1 statistic of 0.67 (95% CI: 0.50-0.83) and 0.73 (95% CI: 0.65-0.81), respectively. The criterion validity of the ROBNS-I was moderate (K=0.52) against NOS. The time in assessing a single study by ROBINS-I varied from 7hours initially to 3hours compared with 30 minutes for the NOS. Both reviewers rated "bias due to departure from the intended interventions" the most time-consuming domain in the ROBINS-I, items in the NOS were equal. The ROBINS-I and the NOS seem to provide the same reliability but vary in applicability. The over-complicated feature of ROBINS-I may limit its usage and a simplified version is needed.