Significance, truth and proof of p values: reminders about common misconceptions regarding null hypothesis significance testing

Abstract

Null hypothesis significance testing has successfully reduced the complexity of scientific inference to a dichotomous decision (i.e., 'reject' versus 'not reject'). As a consequence, p values and their associated statistical significance play an important role in the social and medical sciences. But do we truly understand what statistical significance testing and p values entail? Judging by the vast literature on controversies regarding their application and interpretation, this seems questionable. It has even been argued that significance testing should be abandoned all together [2]. We seek to extend Fayer's [3] paper on statistically significant correlations and to clarify some of the controversies regarding statistical significance testing by explaining that (1) the p value is not the probability of the null hypothesis; (2) rejecting the null hypothesis does not prove that the alternative hypothesis is true; (3) not rejecting the null hypothesis does not prove that the alternative hypothesis is false; (4) statistical significance testing is not necessarily an objective evaluation of results; and (5) the p value does not give an indication of the size of the effect. We note that this article does not raise new issues (see [4] for an extensive overview), but rather serves as a reminder of our responsibility as researchers to be knowledgeable about the methods we use in our scientific endeavors