Explicit formulas for generalized family-wise error rates and unimprovable step-down multiple testing procedures

Abstract

Many multiple testing procedures (MTPs) are available today, and their number is growing. Also available are many type I error rates: the family-wise error rate (FWER), the false discovery rate, the proportion of false positives, and others. Most MTPs are designed to control a specific type I error rate, and it is hard to compare different procedures. We approach the problem by studying the exact level at which threshold step-down ( TSD) procedures (an important class of MTPs exemplified by the classic Holm procedure) control the generalized FWER defined as the probability of k or more false rejections. We find that level explicitly for any TSD procedure and any k . No assumptions are made about the dependency structure of the p -values of the individual tests. We derive from our formula a criterion for unimprovability of a procedure in the class of TSD procedures controlling the generalized FWER at a given level. In turn, this criterion implies that for each k the number of such unimprovable procedures is finite and is greater than one if k > 1 . Consequently, in this case the most rejective procedure in the above class does not exist.