Rational Approaches to Correcting for Multiple Tests

Abstract

The logic of the Bonferroni correction for multiple tests, or family-wise error, is to set the criterion to reduce the expected number of erroneous false positives, or Type I errors, below 1. This is a very stringent criterion for false positives in cases where the test may be applied millions of times, and will necessarily introduce a large proportion of false negatives (missed positives, or Type II errors). A proposed solution to this problem is to adjust the criterion for False Discovery Rate (Benjamini & Hochberg, 1995), which allows the number of false positives to increase proportionally to the number of true positives, though remaining at a small proportion, dramatically reducing the number of false negatives. This approach may be conceptualized as working with a relaxed confidence level that any one test is a true rather than a false positive, bringing the criterion more into line with our societal assessment of the validity of statements in general, and even in science, as having less than 100% certainty. The analytic strategy to the assessment of statistical significance provides a more intuitive approach to the identification of sparse signals in large datasets than the standard Bonferroni approach to correction for multiple tests.