Certainty ranges facilitated explicit and transparent judgments regarding evidence credibility

Abstract

ObjectivesThe Grading of Recommendations Assessment, Development and Evaluation approach to rating certainty of evidence includes five domains of reasons for rating down certainty. Only one of these, precision, is easily amenable—through the confidence interval—to quantitation. The other four (risk of bias, inconsistency, indirectness, and publication bias) are not. Nevertheless, conceptually, one could consider a quantified “certainty range” within which the true effect lies. The certainty range would be at least as wide as the confidence interval and would expand with each additional reason for uncertainty. Study Design and SettingWe have applied this concept to rating the certainty of evidence in the baseline risk of venous thromboembolism (VTE) and bleeding in patients undergoing urological surgery. We considered rating up moderate or low quality evidence when the net benefit of VTE prophylaxis was unequivocally positive, that is, when the smallest plausible value of VTE reduction was greater than the largest plausible value of increased bleeding. To establish whether the net benefit was unequivocally positive, we expanded the range of plausible values by 20% for each of the four nonquantitative domains in which there were serious limitations. ResultsWe present how we applied these methods to examples of open radical cystectomy and laparoscopic partial nephrectomy. In high-VTE risk laparoscopic partial nephrectomy patients and high- and medium-VTE risk open radical cystectomy patients, results proved robust to expanded certainty intervals, justifying rating up quality of evidence. In low-risk patients, the results were not robust, and rating up was therefore not appropriate. ConclusionThis work represents the first empirical application in a decision-making context of the previously suggested concept of certainty ranges and should stimulate further exploration of the associated theoretical and practical issues.