Multiple Hypothesis Testing Procedures Research Articles

Suboptimal Diets Identified Among Adults in Two Rural States During the COVID-19 Pandemic

ABSTRACT Rural communities face unique food and nutrition challenges. Analyzing population-level survey data from 2021 to 2022 in two predominantly rural U.S. states using regression and multiple hypothesis testing procedures, we examined how well diets align with national recommendations and associations between diet, age, food security, and rurality. We found dietary intake of key foods and ingredients to differ significantly from recommendations and overall diet quality to increase with age. Food secure adults had higher diet quality than food insecure adults. Neither dietary intake nor dietary quality differed by degree of rurality. Continued action is needed to address nutrition challenges among underserved populations.

Genome-wide search algorithms for identifying dynamic gene co-expression via Bayesian variable selection.

A wealth of gene expression data generated by high-throughput techniques provides exciting opportunities for studying gene-gene interactions systematically. Gene-gene interactions in a biological system are tightly regulated and are often highly dynamic. The interactions can change flexibly under various internal cellular signals or external stimuli. Previous studies have developed statistical methods to examine these dynamic changes in gene-gene interactions. However, due to the massive number of possible gene combinations that need to be considered in a typical genomic dataset, intensive computation is a common challenge for exploring gene-gene interactions. On the other hand, oftentimes only a small proportion of gene combinations exhibit dynamic co-expression changes. To solve this problem, we propose Bayesian variable selection approaches based on spike-and-slab priors. The proposed algorithms reduce the computational intensity by focusing on identifying subsets of promising gene combinations in the search space. We also adopt a Bayesian multiple hypothesis testing procedure to identify strong dynamic gene co-expression changes. Simulation studies are performed to compare the proposed approaches with existing exhaustive search heuristics. We demonstrate the implementation of our proposed approach to study the association between gene co-expression patterns and overall survival using the RNA-sequencing dataset from The Cancer Genome Atlas breast cancer BRCA-US project.

Multiple hypothesis testing with persistent homology

In this paper we propose a computationally efficient multiple hypothesis testing procedure for persistent homology. The computational efficiency of our procedure is based on the observation that one can empirically simulate a null distribution that is universal across many hypothesis testing applications involving persistence homology. Our observation suggests that one can simulate the null distribution efficiently based on a small number of summaries of the collected data and use this null in the same way that p-value tables were used in classical statistics. To illustrate the efficiency and utility of the null distribution we provide procedures for rejecting acyclicity with both control of the Family-Wise Error Rate (FWER) and the False Discovery Rate (FDR). We will argue that the empirical null we propose is very general conditional on a few summaries of the data based on simulations and limit theorems for persistent homology for point processes.

Two-stage false discovery rate in microarray studies

In microarray and other genomic studies, in view of an abundance of genes, one statistical approach is to hold the family wise error rate to a prescribed limit while controlling the false discovery rate by suitable multiple hypothesis testing procedures, thus generally compromising the power properties to a certain extent. Since the genes are not generally independent or even marginally identically distributed, model flexibility is an essential task regardless of dependent structures among genes. In this respect, incorporating a version of the Chen-Stein theorem, two-stage procedure has been considered; it seems to have better average power without much elevation of false discovery rate compared to single-stage procedure. Simulation studies and applications in microarray data models are also stressed with the methodological developments.

Adaptive False Discovery Rate Control for Heterogeneous Data

Efforts to develop more efficient multiple hypothesis testing procedures for false discovery rate (FDR) control have focused on incorporating an estimate of the proportion of true null hypotheses (such procedures are called adaptive) or exploiting heterogeneity across tests via some optimal weighting scheme. This paper combines these approaches using a weighted adaptive multiple decision function (WAMDF) framework. Optimal weights for a flexible random effects model are derived and a WAMDF that controls the FDR for arbitrary weighting schemes when test statistics are independent under the null hypotheses is given. Asymptotic and numerical assessment reveals that, under weak dependence, the proposed WAMDFs provide more efficient FDR control even if optimal weights are misspecified. The robustness and flexibility of the proposed methodology facilitates the development of more efficient, yet practical, FDR procedures for heterogeneous data. To illustrate, two different weighted adaptive FDR methods for heterogeneous sample sizes are developed and applied to data.

Inferring monopartite projections of bipartite networks: an entropy-based approach

Bipartite networks are currently regarded as providing a major insight into the organization of many real-world systems, unveiling the mechanisms driving the interactions occurring between distinct groups of nodes. One of the most important issues encountered when modeling bipartite networks is devising a way to obtain a (monopartite) projection on the layer of interest, which preserves as much as possible the information encoded into the original bipartite structure. In the present paper we propose an algorithm to obtain statistically-validated projections of bipartite networks, according to which any two nodes sharing a statistically-significant number of neighbors are linked. Since assessing the statistical significance of nodes similarity requires a proper statistical benchmark, here we consider a set of four null models, defined within the exponential random graph framework. Our algorithm outputs a matrix of link-specific p-values, from which a validated projection is straightforwardly obtainable, upon running a multiple hypothesis testing procedure. Finally, we test our method on an economic network (i.e. the countries-products World Trade Web representation) and a social network (i.e. MovieLens, collecting the users’ ratings of a list of movies). In both cases non-trivial communities are detected: while projecting the World Trade Web on the countries layer reveals modules of similarly-industrialized nations, projecting it on the products layer allows communities characterized by an increasing level of complexity to be detected; in the second case, projecting MovieLens on the films layer allows clusters of movies whose affinity cannot be fully accounted for by genre similarity to be individuated.

Comparison of Methods for Estimating the Proportion of Null Hypotheses π0 in High Dimensional Data When the Test Statistics is Continuous

Advances in Genomics have re-energized interest in multiple hypothesis testing procedures but have simultaneously created new methodological and computational challenges. In Genomics for instance, it is now commonplace for experiments to measure expression levels in thousands of genes creating large multiplicity problems when thousands of hypotheses are to be tested simultaneously. Within this context we seek to identify differentially expressed genes, that is, genes whose expression levels are associated with a particular response or covariate of interest. The False Discovery Rate (FDR) is the preferred measure since the Family Wise Error Rates (FWERs) are usually overly restrictive. In the FDR methods, estimation of the proportion of null hypotheses (π0) is an important parameter that needs to be estimated. In this paper, we compare the effectiveness of 12 methods for estimating π0 when the test statistics are continuous using simulated data with independent, weak dependence, and moderate dependence structures.

Declining rainfall and regional variability changes in Jordan

AbstractJordan, with limited rainfall, has per capita water availability of 135 m3/yr making it one of the water‐poorest countries in the world. We analyzed the most comprehensive modern rainfall data set to date, consisting of 44 years of daily measurements from 58 stations primarily in the western, populated and agricultural portion of Jordan over the period 1970–2013 to assess temporal trends, variability, and spatial patterns. From 1995 to 2013, 13 of 19 years showed rainfall less than the mean, which has a probability <8.35% of chance occurrence. We used nonparametric statistical analysis and found 38 of 58 stations experienced an annual rainfall decrease at an average rate of 1.2 mm/yr. Over all 58 stations, the average decrease was 0.41 mm/yr. The annual coefficient of variation of daily rainfall showed a long‐term increase of ∼2–3% at 90% of stations. Analysis of annual variance of daily rainfall suggests decreasing variance in the low rainfall areas to the southwest and east and increasing variance in the high rainfall areas to the northwest, a pattern consistent with principal component analysis. Strict multiple hypothesis testing procedures using the k‐familywise error rate approach reinforced and confirmed the statistically significant regional rainfall decline as well as the spatial patterns of increasing and decreasing rainfall variability.

Optimal rejection curves for exact false discovery rate control

Finner et al. (2012) provided multiple hypothesis testing procedures based on a nonlinear rejection curve for exact false discovery rate control. This paper constructs classes of such procedures and compares the most powerful procedure in each class to competing procedures.

Computationally efficient banding of large covariance matrices for ordered data and connections to banding the inverse Cholesky factor.

In this article, we propose a computationally efficient approach to estimate (large) p-dimensional covariance matrices of ordered (or longitudinal) data based on an independent sample of size n. To do this, we construct the estimator based on a k-band partial autocorrelation matrix with the number of bands chosen using an exact multiple hypothesis testing procedure. This approach is considerably faster than many existing methods and only requires inversion of (k + 1)-dimensional covariance matrices. The resulting estimator is positive definite as long as k < n (where p can be larger than n). We make connections between this approach and banding the Cholesky factor of the modified Cholesky decomposition of the inverse covariance matrix (Wu and Pourahmadi, 2003) and show that the maximum likelihood estimator of the k-band partial autocorrelation matrix is the same as the k-band inverse Cholesky factor. We evaluate our estimator via extensive simulations and illustrate the approach using high-dimensional sonar data.

A Bayesian decision procedure for testing multiple hypotheses in DNA microarray experiments

DNA microarray experiments require the use of multiple hypothesis testing procedures because thousands of hypotheses are simultaneously tested. We deal with this problem from a Bayesian decision theory perspective. We propose a decision criterion based on an estimation of the number of false null hypotheses (FNH), taking as an error measure the proportion of the posterior expected number of false positives with respect to the estimated number of true null hypotheses. The methodology is applied to a Gaussian model when testing bilateral hypotheses. The procedure is illustrated with both simulated and real data examples and the results are compared to those obtained by the Bayes rule when an additive loss function is considered for each joint action and the generalized loss 0-1 function for each individual action. Our procedure significantly reduced the percentage of false negatives whereas the percentage of false positives remains at an acceptable level.

EU ETS market interactions: The case for multiple hypothesis testing approaches

As the European Union Emissions Trading Scheme (EU ETS) enters into the post-2012 Phase III period that runs to 2020, a timely assessment of European Union Allowance (EUA) and Certified Emissions Reduction (CER) price interactions over the Phase II period is presented. Vector autoregression and Granger causality testing is employed for this analysis. As a key contribution to the existing literature, the paper sets out the case for the application of sophisticated multiple hypothesis testing (MHT) procedures to control for the multiple comparisons problem, whereby under naive analysis the identification of significant results from performing multiple simultaneous hypothesis tests may occur by pure chance alone. Two classifications of procedures are employed to control for the multiple comparisons problem in the vector autoregression and Granger causality tests performed: (i) generalised familywise error rate procedures and (ii) false discovery proportion procedures. Over the latter part of Phase II, the generalised familywise error rate procedures show evidence of term structure interactions within both the EUA and CER forward curves. Interactions between the EUA and CER markets are also shown to exist. All of this evidence points to the improved structural reforms to Phase III of the EU ETS that include the extended greenhouse gas coverage, extended sectoral coverage (including aviation), single EU-wide emissions cap, introduction of auctioning and tighter offset limits. However, a word of caution is sounded as the false discovery proportion procedures fail to confirm any such interactions. This comparison of alternative MHT procedures in an empirical study provides important insights for researchers.

Discussion on “Change-Points: From Sequential Detection to Biology and Back” by David Siegmund

Inspired by Professor Siegmund's survey of change-point detection and related problems, I point out that some popular fixed-sample multiple hypothesis testing procedures also have a similar structure.

Multiple hypotheses testing procedures in clinical trials and genomic studies.

We review and compare multiple hypothesis testing procedures used in clinical trials and those in genomic studies. Clinical trials often employ global tests, which draw an overall conclusion for all the hypotheses, such as SUM test, Two-Step test, Approximate Likelihood Ratio test (ALRT), Intersection-Union Test (IUT), and MAX test. The SUM and Two-Step tests are most powerful under homogeneous treatment effects, while the ALRT and MAX test are robust in cases with non-homogeneous treatment effects. Furthermore, the ALRT is robust to unequal sample sizes in testing different hypotheses. In genomic studies, stepwise procedures are used to draw marker-specific conclusions and control family wise error rate (FWER) or false discovery rate (FDR). FDR refers to the percent of false positives among all significant results and is preferred over FWER in screening high-dimensional genomic markers due to its interpretability. In cases where correlations between test statistics cannot be ignored, Westfall-Young resampling method generates the joint distribution of P-values under the null and maintains their correlation structure. Finally, the GWAS data from a clinical trial searching for SNPs associated with nephropathy among Type 1 diabetic patients are used to illustrate various procedures.

EU ETS Market Interactions: A Multiple Hypothesis Testing Approach

As the European Union Emissions Trading Scheme (EU ETS) prepares for the post-2012 Phase III period that runs to 2020, a timely assessment of European Union Allowance (EUA) and Certified Emissions Reduction (CER) price interactions over the Phase II period is presented. Vector autoregression and Granger causality testing is employed for this analysis. The key contribution over existing literature is the application of sophisticated multiple hypothesis testing (MHT) procedures to control for the multiple comparisons problem, whereby under naive analysis the identification of significant results may occur by pure chance alone. Both generalisd familywise error rate and false discovery proportion procedures are used to control for this problem. Over the latter part of Phase II, the generalisd familywise error rate procedures show evidence of term structure interactions within the EUA and CER forward curves. Interactions between the EUA and CER markets are also shown to exist. However, a word of caution is sounded as the false discovery proportion procedures fail to confirm any such interactions. This comparison of alternative MHT procedures in the one empirical study provides important insights for researchers.

Randomised P-values and nonparametric procedures in multiple testing

The validity of many multiple hypothesis testing procedures for false discovery rate (FDR) control relies on the assumption that P-value statistics are uniformly distributed under the null hypotheses. However, this assumption fails if the test statistics have discrete distributions or if the distributional model for the observables is misspecified. A stochastic process framework is introduced that, with the aid of a uniform variate, admits P-value statistics to satisfy the uniformity condition even when test statistics have discrete distributions. This allows nonparametric tests to be used to generate P-value statistics satisfying the uniformity condition. The resulting multiple testing procedures are therefore endowed with robustness properties. Simulation studies suggest that nonparametric randomised test P-values allow for these FDR methods to perform better when the model for the observables is nonparametric or misspecified.

Flexible Regression Model Selection for Survival Probabilities: With Application to AIDS

Clinicians are often interested in the effect of covariates on survival probabilities at prespecified study times. Because different factors can be associated with the risk of short- and long-term failure, a flexible modeling strategy is pursued. Given a set of multiple candidate working models, an objective methodology is proposed that aims to construct consistent and asymptotically normal estimators of regression coefficients and average prediction error for each working model, that are free from the nuisance censoring variable. It requires the conditional distribution of censoring given covariates to be modeled. The model selection strategy uses stepup or stepdown multiple hypothesis testing procedures that control either the proportion of false positives or generalized familywise error rate when comparing models based on estimates of average prediction error. The context can actually be cast as a missing data problem, where augmented inverse probability weighted complete case estimators of regression coefficients and prediction error can be used (Tsiatis, 2006, Semiparametric Theory and Missing Data). A simulation study and an interesting analysis of a recent AIDS trial are provided.

Weighted multiple hypothesis testing procedures.

Multiple hypothesis testing is commonly used in genome research such as genome-wide studies and gene expression data analysis (Lin, 2005). The widely used Bonferroni procedure controls the family-wise error rate (FWER) for multiple hypothesis testing, but has limited statistical power as the number of hypotheses tested increases. The power of multiple testing procedures can be increased by using weighted p-values (Genovese et al., 2006). The weights for the p-values can be estimated by using certain prior information. Wasserman and Roeder (2006) described a weighted Bonferroni procedure, which incorporates weighted p-values into the Bonferroni procedure, and Rubin et al. (2006) and Wasserman and Roeder (2006) estimated the optimal weights that maximize the power of the weighted Bonferroni procedure under the assumption that the means of the test statistics in the multiple testing are known (these weights are called optimal Bonferroni weights). This weighted Bonferroni procedure controls FWER and can have higher power than the Bonferroni procedure, especially when the optimal Bonferroni weights are used. To further improve the power of the weighted Bonferroni procedure, first we propose a weighted Sidák procedure that incorporates weighted p-values into the Sidák procedure, and then we estimate the optimal weights that maximize the average power of the weighted Sidák procedure under the assumption that the means of the test statistics in the multiple testing are known (these weights are called optimal Sidák weights). This weighted Sidák procedure can have higher power than the weighted Bonferroni procedure. Second, we develop a generalized sequential (GS) Sidák procedure that incorporates weighted p-values into the sequential Sidák procedure (Scherrer, 1984). This GS idák procedure is an extension of and has higher power than the GS Bonferroni procedure of Holm (1979). Finally, under the assumption that the means of the test statistics in the multiple testing are known, we incorporate the optimal Sidák weights and the optimal Bonferroni weights into the GS Sidák procedure and the GS Bonferroni procedure, respectively. Theoretical proof and/or simulation studies show that the GS Sidák procedure can have higher power than the GS Bonferroni procedure when their corresponding optimal weights are used, and that both of these GS procedures can have much higher power than the weighted Sidák and the weighted Bonferroni procedures. All proposed procedures control the FWER well and are useful when prior information is available to estimate the weights.

The Impacts of International Migration on Remaining Household Members: Omnibus Results from a Migration Lottery Program

The impacts of international migration on development in the sending countries, and especially the effects on remaining household members, are increasingly studied. However, comparisons of households in developing countries with and without migrants are complicated by a double-selectivity problem: households self-select into migration, and among households involved in migration, some send a subset of members with the rest remaining whilst other households migrate en masse. We address these selectivity issues using the randomization provided by an immigration ballot under the Pacific Access Category (PAC) of New Zealand's immigration policy. We survey applicants to the 2002-05 PAC ballots in Tonga and compare outcomes for the remaining household members of emigrants with those for members of similar households who were unsuccessful in the ballots. The immigration laws determine which household members can accompany the principal migrant, providing an instrument to address the second selectivity issue. Using this natural experiment we examine the myriad impacts that migration has on remaining household members, focussing on labor supply, income, durable assets, financial service usage, diet and physical and mental health and use multiple hypothesis testing procedures to examine which impacts are robust. We find the overall impact on households left behind to be largely negative. We also find evidence that both sources of selectivity matter, leading studies which fail to adequately address them to misrepresent the impact of migration.

A clarifying comparison of methods for controlling the false discovery rate

Traditional multiple hypothesis testing procedures fix an error rate and determine the corresponding rejection region. In 2002 Storey proposed a fixed rejection region procedure and showed numerically that it can gain more power than the fixed error rate procedure of Benjamini and Hochberg while controlling the same false discovery rate (FDR). In this paper it is proved that when the number of alternatives is small compared to the total number of hypotheses, Storey's method can be less powerful than that of Benjamini and Hochberg. Moreover, the two procedures are compared by setting them to produce the same FDR. The difference in power between Storey's procedure and that of Benjamini and Hochberg is near zero when the distance between the null and alternative distributions is large, but Benjamini and Hochberg's procedure becomes more powerful as the distance decreases. It is shown that modifying the Benjamini and Hochberg procedure to incorporate an estimate of the proportion of true null hypotheses as proposed by Black gives a procedure with superior power.