Multiple comparison procedures for discrete test statistics

Abstract

Tarone (1990, Biometrics 46, 515–522) generalized the Bonferroni adjustment for multiple comparisons to obtain increased power by taking advantage of discreteness in the test statistics, but this paper shows that his procedure (denoted by T) does not have α-consistency (AC), i.e., a hypothesis that is accepted at a given α-level may be rejected at a lower α-level. Though lack of AC is theoretically very undesirable, practitioners might reasonably disagree about whether a procedure which lacks AC is preferable in a real application to one which has AC but is substantially less powerful. Several new procedures which are uniformly more powerful than T are proposed, with particular emphasis on two of them, one (denoted by T∗) which achieves AC and one (denoted by R) which fails to achieve AC but is usually much more powerful than T∗. The power difference is not surprising since T∗ is based on the Bonferroni inequality, while R is a discrete generalization of Hochberg's (1988, Biometrika 75, 800–802) improvement to Bonferroni. A variation of R (denoted by RMOD) generalizes Rom's (1990, Biometrika 77, 663–665) procedure instead of Hochberg's, and the arguments for preferring either of the two variations over the other are analogous to the arguments for preferring either Rom's or Hochberg's procedure in the continuous case. In the discrete case, unlike the continuous case, it matters whether strong control (SC) of the familywise error rate (FWE) is defined by requiring FWE<α or merely FWE⩽α. Versions of all of the above procedures (T,T∗, and both variations of R) are constructed to correspond to each of these two definitions of SC. In both cases, T∗ (like T) always achieves SC. A set of conditions closely related to SC of the FWE under Hochberg's [Rom's] original procedure, which is known to include the case of independence, is shown to be sufficient to guarantee SC under R [RMOD] as well. All of the proposed procedures are compared, and recommendations are made.