Outcome Reporting Bias in Randomized Experiments on Substance Use Disorders

Abstract

This cohort study explores the prevalence and effect of suspected outcome reporting bias in clinical trials on substance use disorders. Protocols on the ClinicalTrials.gov registry are compared with the corresponding trial reports for 95 clinical trials across 3,162 outcomes. Variation in average effect size is examined by completeness and accuracy of reporting using ordinary least squares regression with robust standard errors (Eicker-Huber-White sandwich estimator). Trials reports are frequently incomplete and inconsistent with their protocol. The most commonly practiced biased reporting practices are added outcomes not prespecified on the protocol, insufficiently pre-specifying outcomes, and omitting outcomes that were pre-specified on the protocol. There is a linear trend between the number of different biased reporting practices the trialist(s) engaged in and mean study-level Cohen’s d (+ 0.214 with each additional type of biased reporting practice). Trials with omitted pre-specified outcomes have a significantly higher Cohen’s d on average when compared to trials that did not omit such outcomes (+ 0.315). Added outcomes have a Cohen’s d that is 0.385 higher in comparison to reported outcomes that were pre-specified on the protocol. The magnitude of outcome reporting bias raises considerable concern regarding inflated type I error rates. Implications for clinical trials on substance abuse, and randomized experiments in criminology, more generally, are discussed.