The enormous difference between not rejecting a null hypothesis and stating that it is true

Abstract

Assuming that a hypothesis is true because insufficient evidence has been found to reject it is a very common error when interpreting the p-value of a test in biomedical research. For example, a value of p = 0.28 obviously does not mean the null hypothesis should be ruled out, but if we understand what it means (which is not a mathematical issue, but instead a purely logical one) that it is equally obvious that it cannot be stated that it is true. If the samples in a comparison of a new drug with an old one show that the new one has a higher healing percentage and the p-value of the test is 0.0004, for example, the scientific community concludes that the new one is better. However, if for example the p-value of the test is 0.14, the scientific community does not conclude that the new one is as good as the old one. It merely concludes that the new one has not been shown to outperform the other one. It is therefore possible that an extension of the study with more cases may demonstrate that the new one is better.