Intersections of Statistical Significance and Substantive Significance: Pearson’s Correlation Coefficients Under a Known True Null Hypothesis

Abstract

The editors of a special issue of The American Statistician stated: “Regardless of whether it was ever useful, a declaration of “statistical significance” has today become meaningless.” This echoes the author's view, as “statistical significance” has been conflated with substantive significance. However, the author respectfully disagrees with the editors' call for “don’t use it.” With the help of relatively simple graphs and tables, this author demonstrates that small sample sizes (n < 1000) require Pearson’s correlation coefficients to be screened for statistical significance (p <.05) to reduce the number of effect size errors that would otherwise be considered substantively significant under a true null hypothesis. It's crucial to note here that the null hypothesis is not merely assumed true but is indeed known to be true.