Are potentially clinically meaningful benefits misinterpreted in cardiovascular randomized trials? A systematic examination of statistical significance, clinical significance, and authors\u2019 conclusions

G Michael Allan,Simon Kwong,Oksana Babenko,Adriennne J Lindblad,James Mccormack,Michael R Kolber,Scott Garrison,Vivek Kumar,Caitlin R Finley,Emelie Braschi,Christina Korownyk

doi:10.1186/s12916-017-0821-9

G Michael Allan, Simon Kwong + Show 9 more

Open Access

https://doi.org/10.1186/s12916-017-0821-9

Copy DOI

Journal: BMC Medicine	Publication Date: Mar 20, 2017
Citations: 5	License type: open-access

Affiliation: University of Alberta, University of British Columbia

Abstract

BackgroundWhile journals and reporting guidelines recommend the presentation of confidence intervals, many authors adhere strictly to statistically significant testing. Our objective was to determine what proportions of not statistically significant (NSS) cardiovascular trials include potentially clinically meaningful effects in primary outcomes and if these are associated with authors’ conclusions.MethodsCardiovascular studies published in six high-impact journals between 1 January 2010 and 31 December 2014 were identified via PubMed. Two independent reviewers selected trials with major adverse cardiovascular events (stroke, myocardial infarction, or cardiovascular death) as primary outcomes and extracted data on trial characteristics, quality, and primary outcome. Potentially clinically meaningful effects were defined broadly as a relative risk point estimate ≤0.94 (based on the effects of ezetimibe) and/or a lower confidence interval ≤0.75 (based on the effects of statins).ResultsWe identified 127 randomized trial comparisons from 3200 articles. The primary outcomes were statistically significant (SS) favoring treatment in 21% (27/127), NSS in 72% (92/127), and SS favoring control in 6% (8/127). In 61% of NSS trials (56/92), the point estimate and/or lower confidence interval included potentially meaningful effects. Both point estimate and confidence interval included potentially meaningful effects in 67% of trials (12/18) in which authors’ concluded that treatment was superior, in 28% (16/58) with a neutral conclusion, and in 6% (1/16) in which authors’ concluded that control was superior. In a sensitivity analysis, 26% of NSS trials would include potential meaningful effects with relative risk thresholds of point estimate ≤0.85 and/or a lower confidence interval ≤0.65.ConclusionsPoint estimates and/or confidence intervals included potentially clinically meaningful effects in up to 61% of NSS cardiovascular trials. Authors’ conclusions often reflect potentially meaningful results of NSS cardiovascular trials. Given the frequency of potentially clinical meaningful effects in NSS trials, authors should be encouraged to continue to look beyond significance testing to a broader interpretation of trial results.

Highlights

While journals and reporting guidelines recommend the presentation of confidence intervals, many authors adhere strictly to statistically significant testing
We had two primary questions: (1) How often do the point estimates and confidence intervals of the primary outcome of not statistically significant (NSS) and statistically significant (SS) trials include potentially clinically meaningful effects? and (2) Are the authors’ conclusions in the abstract of NSS trials influenced by potentially clinically meaningful point estimates and confidence intervals? We focused on cardiovascular trials with major adverse cardiovascular events (MACE) because these are established, objective, patient-oriented outcomes that overlap between trials
Assessing potentially meaningful effects To assess if the primary outcome of an NSS trial included potentially meaningful effects, we focused on the point estimate and lower confidence interval

Summary

Introduction

While journals and reporting guidelines recommend the presentation of confidence intervals, many authors adhere strictly to statistically significant testing. Within CONSORT the use of confidence intervals is emphasized in preference to p-values. Confidence intervals describe the precision of the estimate and “are especially valuable in relation to differences that do not meet conventional statistical significance, for which they often indicate that the result does not rule out an important clinical difference” [1]. Editorials dating back almost 40 years have encouraged authors to use confidence intervals to describe the results of their studies rather than reporting the findings as statistically significant or not [2–4]. Confidence intervals are often used in a manner similar to p-values, to dichotomize outcomes as statistically significant (SS) or not. We have previously written about three important clinical controversies resulting from this dichotomous activity [6]

Objectives

Methods

Results

Discussion

Conclusion