Sensitivity of Treatment Recommendations to Bias in Network Meta-Analysis

David M Phillippo,Vanessa Didelez,Nicky J Welton,Sofia Dias,A E Ades

doi:10.1111/rssa.12341

Abstract

SummaryNetwork meta‐analysis (NMA) pools evidence on multiple treatments to estimate relative treatment effects. Included studies are typically assessed for risk of bias; however, this provides no indication of the impact of potential bias on a decision based on the NMA. We propose methods to derive bias adjustment thresholds which measure the smallest changes to the data that result in a change of treatment decision. The methods use efficient matrix operations and can be applied to explore the consequences of bias in individual studies or aggregate treatment contrasts, in both fixed and random‐effects NMA models. Complex models with multiple types of data input are handled by using an approximation to the hypothetical aggregate likelihood. The methods are illustrated with a simple NMA of thrombolytic treatments and a more complex example comparing social anxiety interventions. An accompanying R package is provided.

Highlights

Network meta-analysis (NMA) compares the relative effectiveness of multiple treatments by combining the evidence from randomised controlled trials (RCTs), each of which only compares a subset of the treatments of interest (Lumley 2002, Caldwell et al 2005, Lu and Ades 2006)
Studies rated at high risk of bias due to issues with internal or external validity that have negligible influence on the treatment recommendation should be of little concern; whereas if they have a larger influence on the treatment recommendation they should be scrutinised carefully
Current approaches based around the GRADE framework (Puhan et al 2014, Salanti et al 2014) give a thorough qualitative evaluation of the quality of evidence behind such decisions, but fall short of describing the impact on treatment recommendations of any bias present in the evidence

Summary

INTRODUCTION

Network meta-analysis (NMA) compares the relative effectiveness of multiple treatments by combining the evidence from randomised controlled trials (RCTs), each of which only compares a subset of the treatments of interest (Lumley 2002, Caldwell et al 2005, Lu and Ades 2006). Two methods to extend the GRADE framework to NMA were proposed (Puhan et al 2014, Salanti et al 2014) Whilst such approaches can produce valuable and necessary qualitative assessments, they cannot tell how deficiencies in internal or external validity might affect the treatment recommendation. The two-stage NMA, where pairwise meta-analysis is performed in a first step and each of the pairwise estimates combined to give consistent NMA estimates, is only an approximation to the preferred one-stage NMA where all studies on all comparisons are synthesised at once (Lu et al 2011) Decision makers such as NICE recommend the one-stage method due to its accuracy and convenience when results are used in decision models (Dias et al 2011, NICE 2013, NICE 2014). Additional material, including detailed mathematical derivations and proofs, notes on computation, and an accompanying R package, is provided in a web appendix

Network Meta-Analysis

General form of bias adjustment thresholds

Bias adjustment thresholds for the fixed effect model

Example

DISCUSSION