An Alpha-Exhaustive Multiple Testing Procedure

Mark Chang,Xuan Deng,John Balser

doi:10.3844/amjbsp.2016.30.41

Abstract

A multiple testing procedure can be a single-step procedure such as Bonferroni's method or a stepwise procedure such as Hochberg's stepup method and Hommel's method. It can be an α-exhaustive or α-conservative approach. We develop a single α-exhaustive procedure that can improve power 2-5% over Hochberg's and Hommel's methods in common situations when the test statistics are mutually independent. The method can also be generalized to dependent test statistics. The idea behind our method is to construct the rejection rules using the product of marginal p-values and by controlling the upper bounds of the kth order terms so that α is controlled for any configuration of k null hypotheses. Such upper bounds or critical values are determined progressively from k = 1 towards k = K, the number of null hypotheses in the problem.

Highlights

Multiple testing problems are common in pharmaceutical statistics and life-sciences in general
To achieve α-exhaustive, we use the marginal p-value product corresponding to each null hypothesis configuration and enforce it with an upper bound in the rejection rules
Such p-value product terms in the rejection rules ensure the synergy between the marginal p-values

Summary

Introduction

Multiple testing problems are common in pharmaceutical statistics and life-sciences in general. A MTP can be a single-step data-independent procedure such as Bonferroni’s method or a data-dependent stepwise procedure such as Hochberg’s stepup method and Hommel’s stepup method. It can be anα-exhaustive or α-conservative approach. Stepwise procedures are usually more powerful than single-step procedures and α-exhaustive procedures are usually more powerful than α-conservative approach Consider these two aspects together, comparisons of testing procedures are not that simple, often depending on the configuration of the alternative "hypotheses" or more precisely, the truths. The commonly used stepwise procedures include the Bonferroni-Holm stepdown method (Holm, 1979), the Holm stepdown method (Dmitrienko et al, 2009, p.65), Hommel’s stepup procedure (Hommel, 1988), Hochberg’s stepup method (Hochberg, 1988), the fallback procedure (Wiens, 2003) and the sequential test with fixed sequences (Westfall et al, 1999)

Stepdown Procedure

Stepup Procedure

Method

Summary and Discussion