Adjusting Trial Results for Biases in Meta-Analysis: Combining Data-Based Evidence on Bias With Detailed Trial Assessment

K M Rhodes,J P T Higgins,R Elbers,H E Jones,J Savović,J A C Sterne,N J Welton,R M Turner

doi:10.1111/rssa.12485

Abstract

SummaryFlaws in the conduct of randomized trials can lead to biased estimation of the intervention effect. Methods for adjustment of within‐trial biases in meta‐analysis include the use of empirical evidence from an external collection of meta‐analyses, and the use of expert opinion informed by the assessment of detailed trial information. Our aim is to present methods to combine these two approaches to gain the advantages of both. We make use of the risk of bias information that is routinely available in Cochrane reviews, by obtaining empirical distributions for the bias associated with particular bias profiles (combinations of risk of bias judgements). We propose three methods: a formal combination of empirical evidence and opinion in a Bayesian analysis; asking experts to give an opinion on bias informed by both summary trial information and a bias distribution from the empirical evidence, either numerically or by selecting areas of the empirical distribution. The methods are demonstrated through application to two example binary outcome meta‐analyses. Bias distributions based on opinion informed by trial information alone were most dispersed on average, and those based on opinions obtained by selecting areas of the empirical distribution were narrowest. Although the three methods for combining empirical evidence with opinion vary in ease and speed of implementation, they yielded similar results in the two examples.

Highlights

In a meta-analysis, results from a set of similar studies are combined to summarize evidence for a specific research question
The recent ‘Risk of bias in evidence synthesis’ (ROBES) study examined the association between the risk of bias judgements from Cochrane reviews with intervention effect estimates in 228 binary outcome meta-analyses with completed risk-of-bias tables (Savovic et al, 2017)
The objective of this research is to present methods in which the empirical data-based approach that was proposed by Welton et al (2009) and the opinion-based approach that was proposed by Turner et al (2009) can be combined to gain the advantages of both, while making use of the risk-of-bias information that is available in Cochrane reviews

Summary

Introduction

In a meta-analysis, results from a set of similar studies are combined to summarize evidence for a specific research question. Meta-epidemiological studies of large numbers of meta-analyses provide empirical evidence to suggest that aspects of trial design may lead to biased estimates of intervention effects. The recent ‘Risk of bias in evidence synthesis’ (ROBES) study examined the association between the risk of bias judgements from Cochrane reviews with intervention effect estimates in 228 binary outcome meta-analyses with completed risk-of-bias tables (Savovic et al, 2017). Cochrane riskof-bias tables include judgements of whether the risk of bias is low, high or unclear in relation to specific aspects of the trial methods. The results suggested that problems with randomization and lack of blinding are on average associated with a modest (approximately 10%) exaggeration of intervention effect estimates. There is evidence that trial results that are based on subjectively assessed outcomes are more susceptible to bias

Objectives

Findings

Discussion

Conclusion