Answering Two Criticisms of Hypothesis Testing

Abstract

Two generations of methodologists have criticized hypothesis testing by claiming that most point null hypotheses are false and that hypothesis tests do not provide the probability that the null hypothesis is true. These criticisms are answered. (1) The point-null criticism, if correct, undermines only the traditional two-tailed test, not the one-tailed test or the little-known directional two-tailed test. The directional two-tailed test is the only hypothesis test that, properly used, provides for deciding the direction of a parameter, that is, deciding whether a parameter is positive or negative or whether it falls above or below some interesting nonzero value. The point-null criticism becomes unimportant if we replace traditional one- and two-tailed tests with the directional two-tailed test, a replacement already recommended for most purposes by previous writers. (2) If one interprets probability as a relative frequency, as most textbooks do, then the concept of probability cannot meaningfully be attached to the truth of an hypothesis; hence, it is meaningless to ask for the probability that the null is true. (3) Hypothesis tests provide the next best thing, namely, a relative frequency probability that the decision about the statistical hypotheses is correct. Two arguments are offered.