Multiple Comparison Procedures

Abstract

This chapter first discusses the familywise error rate problem (Sect. 4.2), which may arise when a researcher applies statistical significance tests multiple times in an experiment. For example, if the researcher has four experimental systems and is interested in comparing every system pair, it is not advisable to conduct a regular t-test six times. This chapter then discusses two approaches to lower the familywise error rate, namely, the widely used but arguably obsolete Bonferroni correction (Sect. 4.3) and the more recommendable Tukey HSD (Honestly Significant Difference) test (Sect. 4.4). While many multiple comparison procedures for suppressing the familywise error rate have been proposed, the above two methods are parametric, single-step methods (Multiple comparison procedures in which the outcome of one hypothesis test determines what to do next are called stepwise methods. In contrast, multiple comparison procedures that can process all hypotheses at the same time are called single-step methods.) that are suitable for comparing every system pair (Nagata and Yoshida, Introduction to multiple comparison procedures (in Japanese). Scientist Press, 1997). However, the reader should be aware that the Bonferroni correction has low statistical power when handling many hypotheses. Finally, we discuss a distribution-free, computer-based version of the latter test, known as the randomised Tukey HSD test (Carterette, ACM TOIS 30(1):1–34, 2012; Sakai, Evaluation with informational and navigational intents. In: Proceedings of WWW 2012, pp 499–508, 2012), for situations where we have a matrix of scores such as a topic-by-run matrix of nDCG values (Sect. 4.5). The paired randomisation test is also discussed as a special case of this test.