A semi-Bayesian study of Duncan's Bayesian multiple comparison procedure

Abstract

Duncan's Bayesian decision-theoretic multiple comparison procedure requires a decision on the relative magnitudes of losses due to Type I and Type II errors. In this paper, the relative losses are chosen so that the procedure results in weak control of familywise error at the 0.05 level, i.e. the probability that all hypotheses are accepted is 0.95 when all hypotheses are true. Duncan's Bayesian formulation requires prior distributions and specification of associated hyperparameters for the variances of the population means and of the errors. With noninformative priors, the required ratio of these values can be estimated from the sample. From a frequentist point of view, this obviates the necessity for any prior specification for these distributions. However, Duncan's assumption of a prior normal distribution for the population means is required and is retained. A simulation study then compares the modified method, with respect to Bayes risk and average power, to several frequentist-based multiple comparison procedures for testing hypotheses concerning all pairwise comparisons among a set of means. Results indicate considerable similarity in both risk and average power between Duncan's modified procedure and the Benjamini and Hochberg (1995, J. Roy. Statist. Soc. 57, 289–300) FDR-controlling procedure, with the same weak familywise error control. Both risk and power of these procedures are close to the risk and power of individual t -tests of the mean comparisons, and considerably superior on both measures to the properties of the best symmetric simultaneous testing procedure, based on the range of normally distributed observations.