Improving Holm's procedure using pairwise dependencies

Abstract

SUMMARY Seneta & Chen (2005) tightened the familywise error rate control of Holm’s procedure by sharpening its critical values using pairwise dependencies of the p-values. In this paper we further sharpen these critical values in the case where the distribution functions of the pairwise maxima of null p-values are convex, a property shown to hold in some applications of Holm’s procedure. The newer critical values are uniformly larger, providing tighter familywise error rate control than the approach of Seneta & Chen (2005), significantly so under high pairwise positive dependencies. The critical values can be further improved under exchangeable null p-values. Control of the familywise error rate, the probability of falsely rejecting at least one true null hypothesis, is commonly undertaken when testing multiple hypotheses. Among procedures for controlling this error rate, that of Holm (1979) is one of the most popular. Seneta & Chen (2005) attempted to improve upon Holm’s procedure in situations where pairwise dependencies among the p-values can be quantified. They applied an inequality of Kounias (1968) to obtain an upper bound for the distribution function of the minimum of a set of null p-values which is tighter than that provided by Bonferroni’s inequality, while modifying Holm’s critical values. The modification tightens the familywise error rate control of Holm’s procedure, and can be more powerful than the original step-up procedure of Hochberg (1988), as Seneta & Chen (2005) showed for some multiple testing problems associated with normally distributed test statistics with known correlations. We propose two improved versions of Holm’s step-down procedure for p-values such that the pairwise maxima of the null p-values have known convex distribution functions. The different versions depend on whether the null p-values are exchangeable, and each provides uniformly larger critical values than the method of Seneta & Chen (2005). The convexity of p-values is shown for some commonly used multivariate distributions. Numerical and simulation studies reveal that each of our proposed procedures is a better choice than the method of Seneta & Chen (2005), especially for small-scale multiple testing with high pairwise dependencies among the test statistics.