Multiple comparisons: To compare or not to compare, that is the question.

Abstract

Researchers attempt to minimize Type-I errors (concluding there is a relationship between variables, when there in fact, isn't one) in their experiments by exerting control over the p-value thresholds or alpha level. If a statistical test is conducted only once in a study, it is indeed possible for the researcher to maintain control, so that the likelihood of a Type-I error is equal to or less than the significance (p-value) level. When making multiple comparisons in a study, however, the likelihood of making a Type-I error can dramatically increase. When conducting multiple comparisons, researchers frequently attempt to control for the increased risk of Type-I errors by making adjustments to their alpha level or significance threshold level. The Bonferroni adjustment is the most common of these types of adjustment. However, these, often rigid adjustments, are not without risk and are often applied arbitrarily. The objective of this review is to provide a balanced commentary on the advantages and disadvantages of making adjustments when undertaking multiple comparisons. A summary discussion of familiar- and experiment-wise error is also presented. Lastly, advice on when researchers should consider making adjustments in p-value thresholds and when they should be avoided, is provided.