Comments on: Control of the false discovery rate under dependence using the bootstrap and subsampling

Abstract

I would like to commend the authors on a really nice piece of work. It is well written and gives a very general solution to the problem of bootstrap adjustment for multiplicity while controlling the false discovery rate (FDR). At the time that I was working on the normal-theory FDR controlling procedure (Troendle 2000), I had ideas about resampling-based FDR control. However, I have reservations about using FDRcontrolling procedures in applications, which led me to discontinue my research on them. The false discovery proportion (FDP) seems like the most natural thing to control when control of the familywise error rate is not needed. In applications there is only one FDP generated, and the bottom line question is “what can you claim about the likelihood of a large FDP with this set of rejected hypotheses?” Even with exact (as opposed to asymptotic) FDR control, the answer is “not much.” That is because the FDR is an expected value and says nothing about the tail behavior of the FDP. A simple realistic example give in Korn et al. (2004) showed that a procedure controlling the FDR at 0.1 has an actual FDP ≥ 0.29 with probability 0.1. One exciting possibility to take from this paper is that the subsampling ideas given in Sect. 6 might be extended to control of the FDP. The fact that the subsampling procedure did not behave well in the simulations for fairly small sample sizes is discouraging, but perhaps that can be overcome. It may take a lot of computation to get satisfactory results because the sample size should be large (for approximately asymptotic behavior to be expected), while the subsample size should also be large yet small relative to the sample size. There are a tremendous number of such subsets