Assessing variability and uncertainty in orthopedic randomized controlled trials

Lauri Raittio,Aleksi Reito

doi:10.1080/17453674.2020.1755932

Lauri Raittio, Aleksi Reito

Open Access

https://doi.org/10.1080/17453674.2020.1755932

Copy DOI

Abstract

Background and purpose — Low statistical power remains endemic in clinical medicine including orthopedics and manifests as high uncertainty and wide confidence intervals (CI). We evaluated the reporting and correspondence between power calculation and observed data on key parameters of variability and uncertainty in orthopedic randomized controlled trials (RCTs).Material and methods — RCTs with 1:1 allocation published in 8 major orthopedic journals between 2016 and 2017 with one continuous primary outcome were included in the review. The components of power calculation and observed standard deviation (SD), mean difference (MD), and confidence interval (CI) of MD between groups were assessed for primary outcome.Results — 160 RCTs were included, of which 93 (58%) and 138 (86%) studies reported the estimated SD and MD in the power calculation, respectively. The median ratio of the estimated SD and SDs observed in the data was 1.0 (IQR –0.76 to 1.32) for 69 (43%) studies. Only 31 of 138 studies reported the CI of MD in primary outcome. In 42% of the negative studies, the estimated MD was included in the CI of the observed MD.Interpretation — The key parameters of data variability, both in power analyses and in final study results, were poorly reported. Low power in orthopedics may result from too high an estimated effect size due to an overoptimistic estimate of MD between study groups. In almost half of the studies, overlap of the CI of the observed MD and estimated MD suggested that the reported results of these studies were inconclusive.

Highlights

Low statistical power remains endemic in clinical medicine including orthopedics and manifests as high uncertainty and wide confidence intervals (CI)
The rationale for our study was to investigate the etiology of the suggested low power in orthopedic randomized controlled trials (RCTs) and the subsequent consequences, namely, are unreasonably small variability estimates to blame for low power and is the uncertainty of the primary outcome measured affected by small sample sizes? Assuming that average statistical power was low among orthopedic RCTs, we hypothesized that there would be poor correspondence in the estimated and observed variability of the primary outcome
Power calculations were used in most of the RCTs, but most of the studies lacked some of the essential components required by the CONSORT statement and the results required to replicate the analysis

Summary

Introduction

Low statistical power remains endemic in clinical medicine including orthopedics and manifests as high uncertainty and wide confidence intervals (CI). We evaluated the reporting and correspondence between power calculation and observed data on key parameters of variability and uncertainty in orthopedic randomized controlled trials (RCTs). The components of power calculation and observed standard deviation (SD), mean difference (MD), and confidence interval (CI) of MD between groups were assessed for primary outcome. 160 RCTs were included, of which 93 (58%) and 138 (86%) studies reported the estimated SD and MD in the power calculation, respectively. According to the CONSORT statement (Schulz et al 2010), power calculations are based on the estimated mean difference (MDest) between compared groups, the estimated standard deviation (SDest) or variability of the outcome at a particular point in time, and the chosen level of error, namely, type 1 and 2 errors.

Methods

Results

Discussion

Conclusion