Agreement was moderate between data-based and opinion-based assessments of biases affecting randomized trials within meta-analyses

Rebecca M Turner,Kirsty M Rhodes,Hayley E Jones,Julian P.T Higgins,Jessica A Haskins,Penny F Whiting,Asbjørn Hróbjartsson,Deborah M Caldwell,Richard W Morris,Barnaby C Reeves,Helen V Worthington,Isabelle Boutron,Jelena Savović

doi:10.1016/j.jclinepi.2020.05.009

Abstract

Background and ObjectiveRandomized trials included in meta-analyses are often affected by bias caused by methodological flaws or limitations, but the degree of bias is unknown. Two proposed methods adjust the trial results for bias using empirical evidence from published meta-epidemiological studies or expert opinion. MethodsWe investigated agreement between data-based and opinion-based approaches to assessing bias in each of four domains: sequence generation, allocation concealment, blinding, and incomplete outcome data. From each sampled meta-analysis, a pair of trials with the highest and lowest empirical model-based bias estimates was selected. Independent assessors were asked which trial within each pair was judged more biased on the basis of detailed trial design summaries. ResultsAssessors judged trials to be equally biased in 68% of pairs evaluated. When assessors judged one trial as more biased, the proportion of judgments agreeing with the model-based ranking was highest for allocation concealment (79%) and blinding (79%) and lower for sequence generation (59%) and incomplete outcome data (56%). ConclusionMost trial pairs found to be discrepant empirically were judged to be equally biased by assessors. We found moderate agreement between opinion and data-based evidence in pairs where assessors ranked one trial as more biased.

Highlights

Conflict of interests: I.B. is the co-convenor of the Cochrane Bias Methods Group.A meta-analysis of the results from relevant randomized trials is often regarded as the best evidence evaluating the effectiveness of a health care intervention [1]
Within each meta-analysis extracted from the Risk of Bias in Evidence Synthesis (ROBES) database, we selected the two trials with the highest and lowest model-based bias estimates, and elicited an opinion on which trial was judged to be more biased
We reported the proportion of trial pairs in which assessors chose one trial as more biased

Summary

Introduction

A meta-analysis of the results from relevant randomized trials is often regarded as the best evidence evaluating the effectiveness of a health care intervention [1]. Metaanalysis results summarize the findings from multiple studies and are more precise and usually more influential than the results from a single trial. Randomized trials included in meta-analyses are often affected by bias caused by methodological flaws or limitations, but the degree of bias is unknown. Two proposed methods adjust the trial results for bias using empirical evidence from published meta-epidemiological studies or expert opinion

Objectives

Methods

Results

Conclusion