Consequences of relying on statistical significance: Some illustrations.

Abstract

Despite regular criticisms of null hypothesis significance testing (NHST), a focus on testing persists, sometimes in the belief to get published and sometimes encouraged by journal reviewers. This paper aims to demonstrate known key limitations of NHST using simple nontechnical illustrations. The first illustration is based on simulated data of 20000 studies that compare two groups for an outcome event. The true effect size (difference in event rates) and sample size (20-100 per group) were varied. The second illustration used real data from a meta-analysis on alpha-blockers for the treatment of ureteric stones. The simulations demonstrated the large between-study variability in P-values (range between <.0001 and 1 for most simulation conditions). A focus on statistically significant effects (P<.05), notably in small to moderate samples, led to strongly overestimated effect sizes (up to 240%) and many false-positive conclusions, that is statistically significant effects that were, in fact, true null effects. Effect sizes also exerted strong between-study variability, but confidence intervals accounted for this: the interval width decreased with larger sample size, and the percentage of intervals that contained the true effect size was accurate across simulation conditions. Reducing alpha level, as recently suggested, reduced false-positive conclusions but strongly increased the overestimation of significant effects (up to 320%). Researchers and journals should abandon statistical significance as a pivotal element in most scientific publications. Confidence intervals around effect sizes are more informative, but should not merely be reported to comply with journal requirements.