Reaching for the stars: attention to multiple testing problems and method recommendations using simulation for business research

Abstract

Multiple hypotheses tests decide on more than one hypothesis based upon the same data set. Despite the significant relevance for business research, we find that multiple testing methods are not systematically applied in our research area. As surprising this finding is, as crucial it is to the significance of research findings. False positive findings can inflate with the number of employed single tests. The focus on medicine and psychology related topics of current multiple testing literature might deter business researchers from employing the tests. Hence, we provide guidance to researchers by exposing the importance of multiple testing. Based on an application oriented systematization, Monte Carlo simulations, and a decision theoretic evaluation scheme, we highlight method recommendations for different conservatism definitions regarding errors. As implicit results we find, that the dependency structure in the data does not influence the method outcomes significantly. In addition, intuitive single step margin based approaches perform similar than sophisticated tests for typical multiple testing problems in a conservative setting. More liberal methods show a stable relation for all simulation designs.