The Romano-Wolf Multiple Hypothesis Correction in Stata

Abstract

When considering multiple hypothesis tests simultaneously, standard statistical techniques will lead to over-rejection of null hypotheses unless the multiplicity of the testing framework is explicitly considered. In this paper we discuss the Romano-Wolf multiple hypothesis correction, and document its implementation in Stata. The Romano-Wolf correction (asymptotically) controls the familywise error rate (FWER), that is, the probability of rejecting at least one true null hypothesis in a family of hypotheses under test. This correction is considerably more powerful than earlier multiple testing procedures such as the Bonferroni and Holm corrections, given that it takes into account the dependence structure of the test statistics by resampling from the original data. We describe a Stata command rwolf that implements this correction, and provide a number of examples based on a wide range of models. We document and discuss the performance gains from using rwolf over other multiple correction procedures that control the FWER.