Adaptative significance levels using optimal decision rules: Balancing by weighting the error probabilities

Abstract

Our purpose is to recommend a change in the paradigm of testing by generalizing a very natural idea, originated perhaps in Jeffreys (1935, 1961) and clearly exposed by DeGroot (1975), with the aim of developing an approach that is attractive to all schools of statistics, re- sulting in a procedure better suited to the needs of science. The essential idea is to base testing statistical hypotheses on minimizing a weighted sum of type I and type II error probabilities instead of the prevailing paradigm, which is fixing type I error and minimizing type II error. For simple vs. simple hypotheses, the optimal criterion is to reject the null using the likelihood ratio as the evidence (ordering) statistic, with a fixed threshold value instead of a fixed tail probability. By defining expected type I and type II errors, we generalize the weighting approach and find that the optimal region is defined by the evidence ratio, that is, a ratio of averaged likelihoods (with respect to a prior measure) and a fixed threshold. This approach yields an optimal theory in complete general- ity, which the classical theory of testing does not. This can be seen as a Bayesian/Non-Bayesian compromise: using a weighted sum of type I and type II error probabilities is Frequentist, but basing the test criterion on a ratio of marginalized likelihoods is Bayesian. We give arguments to push the theory still further, so that the weighting measures (priors) of the likelihoods do not have to be proper and highly informative, but just well calibrated. That is, priors that give rise to the same evidence (marginal likelihoods) using minimal (smallest) training samples. The theory that emerges, similar to the theories based on objective Bayesian approaches, is a powerful response to criticisms of the pre- vailing approach of hypothesis testing. For criticisms see, for example, Ioannidis (2005 )a ndSiegfried (2010), among many others.