A cross-validation based estimation of the proportion of true null hypotheses

Abstract

In the multiple testing context, a challenging problem is the estimation of the proportion π 0 of true null hypotheses. A large number of estimators of this quantity rely on identifiability assumptions that either appear to be violated on real data, or can be at least relaxed. The proposed estimator π ^ 0 results from density estimation by histograms, and cross-validation. Several consistency results are derived under independence. A new (plug-in) multiple testing procedure (MTP) is also described, based on the Benjamini and Hochberg procedure (BH-procedure) and the proposed estimator. This procedure is asymptotically optimal, provides the asymptotic desired false discovery rate (FDR) control, and is more powerful than the BH-procedure. The non-asymptotic behavior of π ^ 0 is finally assessed through several simulation experiments. It outperforms numerous existing estimators in usual settings, and remains accurate with “U-shape” densities where other estimators usually fail. It does not exhibit any strong sensitivity to dependence. With m block-structured dependent data, it stays reliable up to a within block correlation ρ = 0.5 , when m/50 blocks are used.