Abstract
BackgroundThe Risk of Bias (RoB) tool is used to assess internal validity of randomized controlled trials (RCTs). Our objectives were to: 1) evaluate inter-rater agreement of the RoB tool; 2) determine the time to access supplemental study information; 3) compare the RoB tool with the Jadad scale and Schulz allocation concealment (AC); and 4) examine the relationship between RoB and effect estimates.MethodsWe conducted a systematic review of long-acting beta agonists (LABA) combined with inhaled corticosteroids (ICS) for adults with persistent asthma. Two reviewers independently assessed 107 trials using RoB, Jadad, and AC. One reviewer searched for study protocols. We assessed inter-rater agreement using weighted Kappa (κ) and the correlation between tools using Kendall's Tau (τ). Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model.ResultsTrials had good Jadad scores (median 4, IQR 3-4); however, 85% had unclear AC and 87% high RoB. The factor that most influenced RoB was the potential inappropriate influence of study sponsors (95% industry funded). Agreement on RoB domains was fair (κ = 0.40) to almost perfect (κ = 0.86), and moderate for overall RoB (κ = 0.41). Median time to complete RoB assessments was 21 minutes (IQR 14-27) and 12 minutes (IQR 9-16) to search for protocols. Protocols were identified for 5/42 studies (12%); in 3 cases the assessment of selective outcome reporting changed. There was low correlation between overall RoB vs. Jadad (τ = 0.04, p = 0.3) and AC (τ = −0.02, p = 0.7). Analyses comparing effect estimates and risk showed no important patterns.ConclusionsInter-rater agreement on RoB assessments was better than previously reported suggesting that review-specific guidelines are important. The correlation between RoB and Jadad was low suggesting measurement of different constructs (risk of bias vs. quality of reporting). The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.
Highlights
Risk of bias, of studies included in a systematic review (SR) is a key methodological step and serves to identify the strengths and limitations of individual studies; investigate, and potentially explain, heterogeneity in findings across different studies included in a SR; and, contribute to grading the quality of a body of evidence for a given question
Overall methodological quality and risk of bias The median Jadad score across the 107 randomized controlled trials (RCTs) was 4 on a fivepoint scale (IQR 3, 4); scores greater or equal to 3 are considered good methodological quality [9;10]
Overall 95% of these long-acting beta agonists (LABA)/inhaled corticosteroids (ICS) trials were funded by the pharmaceutical industry
Summary
Risk of bias, of studies included in a systematic review (SR) is a key methodological step and serves to identify the strengths and limitations of individual studies; investigate, and potentially explain, heterogeneity in findings across different studies included in a SR; and, contribute to grading the quality of a body of evidence for a given question. There are numerous tools to assess methodological quality of primary studies; few have undergone extensive interrater reliability or validity testing. It is unknown whether, or to what extent, quality assessments based on existing tools differentiate studies that may yield biased results either by over or underestimating treatment effects. To what extent, quality assessments based on existing tools differentiate studies that may yield biased results either by over or underestimating treatment effects Such information is critical for decision-making in order to gain an accurate assessment of the potential benefits (or harms) of a given intervention. The Risk of Bias (RoB) tool is used to assess internal validity of randomized controlled trials (RCTs). Our objectives were to: 1) evaluate inter-rater agreement of the RoB tool; 2) determine the time to access supplemental study information; 3) compare the RoB tool with the Jadad scale and Schulz allocation concealment (AC); and 4) examine the relationship between RoB and effect estimates
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
