A Simple Method for Testing Global and Individual Hypotheses Involving a Limited Number of Possibly Correlated Outcomes

Abstract

Tests for the presence of a global treatment effect expressed by possibly correlated “primary” outcome variables taken together frequently use Bonferroni-type adjustments. These procedures accommodate an arbitrary number of comparisons, but can be conservative if the outcome variables are highly correlated. This conservatism can be ameliorated by a simple rule requiring essentially no calculation (and therefore convenient to apply when exact calculation is impractical) that is relatively robust to the correlation structure of the responses when the number of comparisons is not large (16 or less for 5 % level tests). The recommended global testing rule is: For a type 1 error rate of α and up to K(α) “primary” response variables, reject the global null hypothesis if (a) the smallest marginal p value is slightly less than α 1 = α/K, (b) the second smallest marginal p value is ≤ 2α 1, or (c) the third smallest marginal p value is ≤ α. Analytic expressions that do not assume independence or any particular distribution for the responses are provided for the probability of rejecting the global null hypothesis. The type 1 error rates and power generally are preserved regardless of the correlation structure. Individual comparisons can be tested if the global null hypothesis is rejected, with reasonable preservation of comparison-wise type 1 error rates and of the false discovery rates (FDRs).