P-values are misunderstood, but do not confound

Abstract

I agree with Professor Stang [1] that P-values cannot rule out the null hypothesis and do not substitute for measures of effect, and that these points bear repeating. But in cleaning up the language of published quantitative research,we should be careful not to throw P-values out with the bath water. The idea that the P-value ‘‘confounds’’ effect size and sample size is itself a fallacy ( pace Rothman) [2]. P is calculated using the effect size and sample size, but the two things are combined in such a way as to weigh the evidence against the null hypothesis, adjusting for sample size (large effects are less convincing in smaller studies) [3]. P-values and confidence intervals cannot substitute for each other because they answer different questions: for the confidence interval, this is ‘‘how big do we think the effect might be?’’; for the P-value, it is ‘‘do we think there is an effect at all?’’ In some situations, we may have moved beyond the latter, but in others this is the first question we ask, through an application of Occam’s razor. The real problem with the way people interpret P-values is calibration. Many find PZ 0.05 highly convincing, whereas realistically it does little more than tip the balance of probabilities in favor of the alternative hypothesis [3,4]. Smaller P-values are progressively more persuasive, which is why we need to see a P-value in addition to a 95% confidence interval. We should not abandon P-values, but instead try to understand them better.