Accuracy of event rate and effect size estimation in major cardiovascular trials

Abstract

Abstract Background During the design of a randomized controlled trial (RCT), the estimation of the expected event rate (ER) and effect size (ES) of an intervention is an essential step to calculate the sample size, which determines the scope of the trial. Overestimation of ER and ES might lead to an underpowered trial design. Objective To evaluate the accuracy of the estimation of ER and ES in contemporary cardiovascular RCTs. Methods We searched “Medline” for cardiovascular RCTs published in the New England Journal of Medicine, The Journal of the American Medical Association, or The Lancet between 2010 and 2019. Only multicenter RCTs with a dichotomous primary endpoint and with not more than 2 study arms were included. Data were extracted from the original publication or the associated study protocol. For trials that tested superiority of an intervention, the effect size measurements hazard ratio, odds ratio, or risk ratio were summarized as the effect size ratio (ESR). To determine the accuracy of estimation, we compared the observed to the hypothesized ER and ESR, respectively. The association of the accuracy of estimation with trial characteristics was determined by linear regression. Results Of 875 identified publications, 374 underwent full review. After exclusion of trials with insufficiently reported data for this analysis, data from 344 trials were analyzed. 145 trials (42.2%) were industry sponsored, 122 (35.5%) were investigator-initiated with industry-co funding, and 77 (22.1%) with government or public funding. The majority (186; 54.1%) tested a drug as the intervention. 263 (76.5%) were designed to test for superiority of the intervention. The primary endpoint consisted of at least 2 components in 247 (71.8%) trials. The median estimated sample size was 2386 (interquartile range [IQR] 800, 6000). The median observed event rate was 9.0% (IQR 4.3, 21.4%) and was significantly lower compared with the estimated ER (11.0% [IQR 6.0, 25.0%]; median deviation from estimation −6.9% [95% confidence interval, CI −10.0, −3.7%]; p<0.001). 196 trials (61%) overestimated the event rate (Figure 1A). The overestimation was consistent across several subgroups (Figure 2A). The median observed ESR was 0.910 (IQR 0.740, 0.990) and was significantly higher compared with the estimated ESR (0.724 [IQR 0.600, 0.795]; median deviation from estimated effect size 26.0% [95% CI, 19.0, 33.0%]). 216 (82.1%) trials overestimated the effect size (Figure 1B). This overestimation was consistent across all analyzed subgroups (Figure 2B) and was less pronounced in trials with use of a composite primary outcome. Conclusion 6 of 10 contemporary cardiovascular RCTs overestimated the event rate for the primary endpoint and 8 of 10 trials overestimated the effect sizes of an intervention. Overestimation may lead to inability to answer the trial hypothesis. Funding Acknowledgement Type of funding sources: None.