Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments

Carson Ka-Lok Lo,Dominik Mertz,Mark Loeb

doi:10.1186/1471-2288-14-45

Carson Ka-Lok Lo, Dominik Mertz + Show 1 more

Open Access

https://doi.org/10.1186/1471-2288-14-45

Copy DOI

Abstract

BackgroundLack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza.MethodsCohort studies included in the systematic review and published between 2008–2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics.ResultsAuthors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers’ assessment (median = 6; interquartile range [IQR] 6–6) compared with those by authors (median = 5, IQR 4–6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = −0.19, 0.48) to poor (K = −0.06, 95% CI = −0.22, 0.10). Reliability for the overall score was poor (K = −0.004, 95% CI = −0.11, 0.11).ConclusionsDifferences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews.

Highlights

Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials
We hypothesized that reviewers in the systematic review who had to rely on the published information would have assigned lower Newcastle-Ottawa Scale (NOS) scores compared with its authors, and that the agreement between authors and reviewers would be marginal
Six cohort studies were excluded, as there was no information on contacting authors (three articles published by the World Health Organization (WHO) and three articles by Centre for Disease Control and Prevention (CDC))

Summary

Introduction

Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza. The Newcastle-Ottawa Scale (NOS) is a risk of bias assessment tool for observational studies that is recommended by the Cochrane Collaboration [1,2]. In addition to the potentially limited inter-rater reliability, lack of reporting of methodological details in published articles may potentially distort the risk of bias assessment as it was shown for randomized controlled trials (RCT) [5,6]. We hypothesized that reviewers in the systematic review who had to rely on the published information would have assigned lower NOS scores (i.e., greater risk of bias) compared with its authors, and that the agreement between authors and reviewers would be marginal

Objectives

Methods

Results

Discussion

Conclusion