An adaptive null proportion estimator for false discovery rate control

Abstract

SUMMARY False discovery rate (FDR) is a commonly used criterion in multiple testing and the BenjaminiHochberg (BH) procedure is a standard approach for FDR control. To boost power, the adaptive BH procedure has been proposed by incorporating null proportion estimators, among which Storey’s estimator has gained substantial popularity. The performance of Storey’s estimator hinges on a critical hyper-parameter, where a pre-fixed configuration may lack power and existing data-driven hyper-parameters compromise the FDR control. In this work, we propose a novel class of adaptive hyper-parameters and establish the FDR control of the associated adaptive BH procedure using a martingale argument. Within this class of data-driven hyper-parameters, we further present a specific configuration designed to maximize the number of rejections and characterize its convergence to the optimal hyper-parameter under a mixture model. Our proposal exhibits significant power gains particularly in cases with a conservative null distribution common in composite null testing or a moderate proportion of weak non-nulls typically observed in biological experiments with enrichment processes.