Many High-Quality Randomized Controlled Trials in Sports Physical Therapy Are Making False-Positive Claims of Treatment Effect: A Systematic Survey.

Abstract

To examine the risk of false-positive reporting within high-quality randomized controlled trials (RCTs) in the sports physical therapy field. Cross-sectional. We searched the Physiotherapy Evidence Database for parallel-design, 2-arm RCTs reporting positive treatment effects, based on null-hypothesis significance testing, and scoring greater than 6/10 on the Physiotherapy Evidence Database scale. No restrictions were made on pathology, intervention, or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least 1 outcome variable. We estimated false-positive risk (FPR) with an online calculator, based on number of participants, P value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5, and 0.8 (optimistic hypothesis). We calculated the FPR associated with 189 statistically significant findings (P<.05) reported across 44 trials. The median FPR was 9% (25th-75th percentile, 2%-24%). Sixty-three percent of statistically significant results (119/189) had an FPR greater than 5%, and 18% (35/189) had an FPR greater than 50%. Changing the prior probability from skeptical to optimistic reduced the median FPR from 29% (25th-75th percentile, 9%-56%) to 2% (25th-75th percentile, 0.6%-7.0%). High-quality RCTs using null-hypothesis significance testing often overestimated treatment effects. The median FPR was 9%: in 1 in 10 trials, the researchers falsely concluded that there was a treatment effect. Future RCTs in sports physical therapy should be informed by prestudy odds and a minimum FPR estimation. J Orthop Sports Phys Ther 2020;50(2):104-109. doi:10.2519/jospt.2020.9264.