Combining independent p-values in replicability analysis: a comparative study

Abstract

Given a family of null hypotheses , we are interested in the hypothesis that at most of these null hypotheses are false. Assuming that the corresponding p-values are independent, we are investigating combined p-values that are valid for testing . In various settings in which is false, we determine which combined p-value works well in which setting. Via simulations, we find that the Stouffer method works well if the null p-values are uniformly distributed and the signal strength is low, and the Fisher method works better if the null p-values are conservative, i.e. stochastically larger than the uniform distribution. The minimum method works well if the evidence for the rejection of is focused on only a few non-null p-values, especially if the null p-values are conservative. Methods that incorporate the combination of e-values work well if the null hypotheses are simple. Finally, we also consider nonparametric permutation-based combination methods. They are useful for addressing the conservativity of parametric p-values which are computed under least favourable parameter configurations.