AB1222 HOW DOES CHANGING THE TRADITIONAL P VALUE SIGNIFICANCE THRESHOLD TO .005 EFFECT OBSERVATIONAL STUDIES?

Abstract

Background:Changing the traditional p value significance level from 0.05 to 0.005 has been proposed as one remedy to the common ill understanding and use of the p values (1-2) especially leading to poor reproducibility. It was then reported that this change would decrease the number of statistically significant primary end points in randomized controlled trials (RCT) reported during 2017 in 3 main general medical journals by 1/3 (3).Objectives:We reasoned that the same change would affect in the right direction the significance conclusions more in observational studies (OS) in that OS generally have retrospective or cross-sectional designs, looser study inclusion criteria, many potential confounders and post – hoc analyses.Methods:We surveyed how the 0.05 to 0.005 change would affect the OS in the same 3 journals in reference 3 for the same year (2017) and in another year (2002), included to better assess the validity of our findings. RCT were also surveyed for both time periods. Two authors (MO, ZTD) surveyed all studies. Disagreements were settled by HY. Differing from the previous study (3) we analyzed the p values as related to the primary end points only in the abstracts. The RCT and OS between January 1, 2002 – December 31, 2002 and January 1, 2017 -December 31, 2017 in N Engl J Med, Lancet and JAMA were included.Results:Among the RCT in 2017 the decrease in the number of significant p values was quite similar between what was reported (3) and the current study (Table) (29.3 vs 30.0%) supporting the validity of the slightly different methodology we used. We were however surprised to see that changing the significance level to 0.005 did not decrease the number of significant p values in OS as compared to the RCT. In fact the percentage decrease in RCT was statistically significantly less in 2002 and numerically in 2017 (Table).Table.OS and RCT in 2002 and 2017OS (2002)RCT (2002)Number of articles305211Total number of p values related to primary outcome(s)323322p<0.05 n(%)260(80.4)240(74.5)p<0.005 n(%)190(58.8)121(37.5)Reductionin significant number of p values* (%)70(26.9)P=0.011119(49.5)OS (2017)RTC (2017)Number of articles167200Total number of p values related to primary outcome(s)187257p<0.05 n (%)162 (86.6)173(67.3)p<0.005 n (%)125(66.8%)121(47.0%)Reductionin significant number of p values* (%)37(22.8)P=0.5552(30.0)RCT:Randomized Controlled TrialsOS: Observational Studies*Number of p values <0.05 -Number of p values<0.005/(Number of p values<0.05Conclusion:Contrary to our initial reasoning we now realize that changing the traditional p threshold will not decrease the number of significant p values more in OS than in RCT. A limitation of our work was that we surveyed only 3, high impact factor, general medicine journals. We envisage that in studies with smaller sample sizes more commonly published in journals with lower impact factors, the proposed change in the significance threshold would be more useful. Smaller sample sizes lead to more unstable, fickle p (4) values and changing the threshold would conceivably more or less stabilize this fickleness both in OS and RCT.