Multiple testing: correcting for alpha error inflation with false discovery rate (FDR) or family-wise error rate?

Abstract

In 1995, Benjamini and Hochberg published an influential paper propagating the false discovery rate (FDR) as a means to correct for alpha error inflation due to multiple hypothesis testing (Benjamini & Hochberg 1995). The FDR was quickly taken up by the scientific research community, foremost within molecular biology. The popularity of this method is illustrated by the fact that by the time of writing of this commentary their paper has already been cited over 50 000 times. Over the last few years, the FDR has also gained popularity in behavioural biology, where, unfortunately, it is frequently applied incorrectly. This seems to be due to the misconception that the FDR is a less stringent alternative to the Bonferroni correction for alpha error inflation. This assumption is erroneous because FDR and Bonferroni corrections address different problems and are not interchangeable. The Bonferroni correction adjusts the rejection threshold, so that the overall likelihood of making at least one incorrect type 1 inference (i.e. reporting an effect when there is none and committing an alpha error) remains at the chosen significance threshold (usually 5%), even when multiple hypotheses are tested within one study. The FDR, on the other hand, resets the threshold level in such a way that only 5% of the positive inferences are false positive findings (type 1, or alpha errors). This is, clearly, a different and usually less ambitious criterion. But, what should we aim for? In the following sections, I first elaborate on both concepts and, then, summarize which one to use for which question and which kind of data within behavioural biology.