Use of the Binomial Theorem in Interpreting Results of Multiple Tests of Significance

Abstract

Often in the social and behavioral sciences, several individual tests of significance are used to determine whether some common or overall hypothesis should be rejected. Thus, it becomes necessary to interpret r significant results out of n tests. Many authors contend that one or more significant results should be interpreted as an overall significant result for the set of tests. The authors of this work suggest that a more appropriate approach would be to use the binomial theorem to compute the probability that r or more Type I errors would occur when all n of the null hypotheses are true, and use this result as the level of overall significance a*. It is shown that in the independent test situation, it is possible to set an action limit r for rejection of the overall hypothesis based on some required overall level of significance a*. In addition, an upper limit is obtained for a* when r significant test results are used to reject a set of n hypotheses when the tests are dependent to an unknown extent.