Abstract
BackgroundThe evaluation of imprecision is a key dimension of the grading of the confidence in the estimate. Grading of Recommendations Assessment, Development and Evaluation (GRADE) gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use. Trial Sequential Analysis (TSA) has been advocated for a more reliable assessment of imprecision. We aimed to evaluate reporting of and adherence to GRADE and to compare the assessment of imprecision of intervention effects assessed by GRADE and TSA in Cochrane systematic reviews.MethodsIn this cross-sectional study, we included 100 Cochrane reviews irrespective of type of intervention with a key dichotomous outcome meta-analyzed and assessed by GRADE. The methods and results sections of each review were assessed for adequacy of imprecision evaluation. We re-analyzed imprecision following the GRADE Handbook and the TSA Manual.ResultsOverall, only 13.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of the confidence intervals and the optimal information size. Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was re-analyzed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors’ assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% confidence interval [CI] 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors’ GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between our GRADE assessment following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79).ConclusionsIn a sample of Cochrane reviews, methods for assessing imprecision were rarely reported. GRADE according to Handbook guidelines and TSA led to more severe judgment of imprecision rather than GRADE adopted by reviews’ authors. Cochrane initiatives to improve adherence to GRADE Handbook are warranted. TSA may transparently assist in such development.
Highlights
The evaluation of imprecision is a key dimension of the grading of the confidence in the estimate
Cochrane systematic reviews were considered eligible for inclusion if (1) they were reviews of interventions, (2) they meta-analyzed at least two randomized controlled trials (RCTs) for a dichotomous outcome, and (3) the dichotomous outcome was listed in the summary of findings (SoF) table
Results of logistic regression analyses In the univariable logistic regression analysis, the type of the intervention (GRADE by review authors: p = 0.65; Trial Sequential Analysis (TSA) p = 0.78), the number of patients included in the meta-analysis (GRADE by the review authors: p = 0.08; TSA p = 0.07), the heterogeneity (GRADE by review authors: p = 0.12; TSA p = 0.38), and the meta-analysis technique (GRADE by review authors: p = 0.86; TSA p = 0.29) were not associated with downgrading due to imprecision
Summary
The evaluation of imprecision is a key dimension of the grading of the confidence in the estimate. Grading of Recommendations Assessment, Development and Evaluation (GRADE) gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use. We aimed to evaluate reporting of and adherence to GRADE and to compare the assessment of imprecision of intervention effects assessed by GRADE and TSA in Cochrane systematic reviews. The GRADE approach suggests evaluating five key dimensions that may affect our confidence in intervention effects: risks of bias (risks of systematic errors), imprecision (risks of random errors), inconsistency, indirectness, and publication bias [2, 3]. Systematic reviews employ multiple parameters to evaluate imprecision: accrued sample size, required or optimal information size (OIS) (meta-analytic “sample size”), alpha, beta, confidence intervals of the overall effect, and specified critical margins of “no effect,” “important benefit,” or “important harm” [2]. GRADE combines all components in a simple rule: “If the optimal information size criterion is not met, rate down for imprecision, unless the sample size is very large (at least 2000, and perhaps 4000 patients); if the optimal information size criterion is met and the 95% confidence interval (CI) excludes no effect, do not rate down for imprecision; if the optimal information size criterion is met, and the 95% CI overlaps no effect, rate down for imprecision if the CI fails to exclude important benefit or important harm” [5, 6]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
