The Potential Effect of Lowering the Threshold of Statistical Significance From P < .05 to P < .005 in Orthopaedic Sports Medicine

Abstract

To determine how changing the P value threshold of statistical significance from .05 to .005 could affect the statistical significance of findings in previously published orthopaedic sports medicine randomized controlled trials (RCTs). The authors searched PubMed from January 1, 2016, to December 31, 2017, for RCTs published in the American Journal of Sports Medicine, Arthroscopy, and Knee Surgery, Sports Traumatology, Arthroscopy. Data were extracted blinded and in duplicate fashion by 2 of us. The authors then extracted P value data for primary end points, since RCTs are most often powered for these end points. Discrepancies were resolved by consensus. Google Forms were used for data extraction and STATA 15.1 for the data analysis. In total, 275 primary end points were identified from 132 trials. Analysis of primary end points found 45.8% (126/275) had a P value less than .05 and were classified as statistically significant under the current threshold, whereas 54.2% (149/275) had a P value greater than .05 and were not classified as suggestive. Of those end points that were previously considered statistically significant, 38.9% (49/126) were less than .005, whereas 61.1% (77/126) were between .005 and .05 and thereby would be reclassified as suggestive rather than statistically significant under the proposed threshold. Overall, when analyzing the 275 primary end points, we found only 49 (17.8%) of the end points were less than .005 and would hold statistical significance with the proposed threshold. The results suggest that if the threshold of statistical significance were to change to .005, the significance of orthopaedic sports medicine RCTs would be heavily altered. The authors also acknowledge the many issues research faces in regard to P value reliability and therefore interpretation of study results. Because P values from RCTs can often influence the ways physicians choose interventions, it is important to implement methodology that decreases incidence of bias and misrepresentation of these results. However, the authors also understand that lowering the P value could increase the needed sample size and by consequence increase study costs as well, while not directly correlating to clinical significance. Thus, the authors recommend that this proposed threshold should be further evaluated and cautiously interpreted. If the statistical significance threshold is changed, clinical practice guideline recommendations also may be affected.