Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study

Abstract

BackgroundReproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P-value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals. MethodsWe identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P-values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR50). The primary outcomes were the percentage of RCTs maintaining statistical significance at P<0.005 and minFPR50. ResultsWe included 318 RCTs. P-values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P<0.005 threshold. The mean (standard deviation) minFPR50 was 22% (20). At P=0.005, the minFPR50 was approximately 5%. ConclusionsThese proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P-value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.