Large-scale Multiple Testing Research Articles

Large-scale dependent multiple testing via higher-order hidden Markov models

ABSTRACT Taking into account the local dependence structure in large-scale multiple testing is expected to improve both the efficiency of the testing procedure and the interpretability of scientific findings. The hidden Markov model (HMM), as an effective model to describe the sequential dependence, has been successfully applied to large-scale multiple testing with local correlations. However, in many applications, the first-order Markov chain is not flexible enough to capture the complexity of local correlations. To address this issue, this paper proposes a novel multiple testing procedure that uses a higher-order Markov chain to better characterize local correlations among tests. The proposed procedure is validated by theoretical results and simulation studies, which show that it outperforms its competitors in terms of power. Finally, a real data analysis is presented to demonstrate the favorable performance of the proposed procedure.

ASTER: A Method to Predict Clinically Relevant Synthetic Lethal Genetic Interactions.

A Synthetic Lethal (SL) interaction is a functional relationship between two genes or functional entities where the loss of either entity is viable but the loss of both is lethal. Such pairs can be used to develop targeted anticancer therapies with fewer side effects and reduced overtreatment. However, finding clinically relevant SL interactions remains challenging. Leveraging unified gene expression data of both disease-free and cancerous samples, we design a new technique based on statistical hypothesis testing, called ASTER, to identify SL pairs. We empirically find that the patterns of mutually exclusivity ASTER finds using genomic and transcriptomic data provides a strong signal of synthetic lethality. For large-scale multiple hypothesis testing, we develop an extension called ASTER++ that can utilize additional input gene features within the hypothesis testing framework. Our computational and functional experiments demonstrate the efficacy of ASTER in identifying SL pairs with potential therapeutic benefits.

An Empirical Bayes Approach to Controlling the False Discovery Exceedance

In large-scale multiple hypothesis testing problems, the false discovery exceedance (FDX) provides a desirable alternative to the widely used false discovery rate (FDR) when the false discovery proportion (FDP) is highly variable. We develop an empirical Bayes approach to control the FDX. We show that, for independent hypotheses from a two-group model and dependent hypotheses from a Gaussian model fulfilling the exchangeability condition, an oracle decision rule based on ranking and thresholding the local false discovery rate (lfdr) is optimal in the sense that the power is maximized subject to the FDX constraint. We propose a data-driven FDX procedure that uses carefully designed computational shortcuts to emulate the oracle rule. We investigate the empirical performance of the proposed method using both simulated and real data and study the merits of FDX control through an application for identifying abnormal stock trading strategies.

Rejoinder on: statistical inference and large-scale multiple testing for high-dimensional regression models

Rejoinder on: statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Signal Classification in Large-Scale Multi-Sequence Integrative Analysis Under the HMM Dependence

The integrative analysis of multiple sequences of multiple tests has enjoyed increasing popularity in many applications, especially in large-scale genomics. In the context of large-scale multiple testing, the concept of signal classification has been developed recently for cases when the same features are involved in several independent studies, with the goal of classifying each feature into one of several classes. This article considers the problem of such signal classification in a generalized compound decision-making framework, where the observed data are assumed to be generated from an underlying four-state Cartesian hidden Markov model. Two oracle procedures are proposed for the total and set-specific control of misclassification rates, respectively, while the number of correct classifications is maximized. Optimal data-driven procedures are also proposed, with their asymptotic properties derived. It is shown that signal-classification could be improved significantly by taking into account the dependence structure among features, and the proposed procedures could have a better performance than their competitors that ignore the dependence structure. The proposed methods are applied to a psychiatric genetics study for detecting genetic variants that affect either or both of bipolar disorder and schizophrenia.

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Statistical inference and large-scale multiple testing for high-dimensional regression models

Statistical inference and large-scale multiple testing for high-dimensional regression models

Statistical Learning of Large-Scale Genetic Data: How to Run a Genome-Wide Association Study of Gene-Expression Data Using the 1000 Genomes Project Data

Teaching statistics through engaging applications to contemporary large-scale datasets is essential to attracting students to the field. To this end, we developed a hands-on, week-long workshop for senior high-school or junior undergraduate students, without prior knowledge in statistical genetics but with some basic knowledge in data science, to conduct their own genome-wide association study (GWAS). The GWAS was performed for open source gene expression data, using publicly available human genetics data. Assisted by a detailed instruction manual, students were able to obtain ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim$$\\end{document}1.4 million p-values from a real scientific study, within several days. This early motivation kept students engaged in learning the theories that support their results, including regression, data visualization, results interpretation, and large-scale multiple hypothesis testing. To further their learning motivation by emphasizing the personal connection to this type of data analysis, students were encouraged to make short presentations about how GWAS has provided insights into the genetic basis of diseases that are present in their friends or families. The appended open source, step-by-step instruction manual includes descriptions of the datasets used, the software needed, and results from the workshop. Additionally, scripts used in the workshop are archived on Github and Zenodo to further enhance reproducible research and training.

Multiple multi-sample testing under arbitrary covariance dependency.

Modern high-throughput biomedical devices routinely produce data on a large scale, and the analysis of high-dimensional datasets has become commonplace in biomedical studies. However, given thousands or tens of thousands of measured variables in these datasets, extracting meaningful features poses a challenge. In this article, we propose a procedure to evaluate the strength of the associations between a nominal (categorical) response variable and multiple features simultaneously. Specifically, we propose a framework of large-scale multiple testing under arbitrary correlation dependency among test statistics. First, marginal multinomial regressions are performed for each feature individually. Second, we use an approach of multiple marginal models for each baseline-category pair to establish asymptotic joint normality of the stacked vector of the marginal multinomial regression coefficients. Third, we estimate the (limiting) covariance matrix between the estimated coefficients from all marginal models. Finally, our approach approximates the realized false discovery proportion of a thresholding procedure for the marginal p-values for each baseline-category logit pair. The proposed approach offers a sensible trade-off between the expected numbers of true and false findings. Furthermore, we demonstrate a practical application of the method on hyperspectral imaging data. This dataset is obtained by a matrix-assisted laser desorption/ionization (MALDI) instrument. MALDI demonstrates tremendous potential for clinical diagnosis, particularly for cancer research. In our application, the nominal response categories represent cancer (sub-)types.

Change-detection-assisted multiple testing for spatiotemporal data

This paper considers a large-scale multiple testing problem for spatiotemporal data with multiple change points. A data-driven procedure that aims to fully utilize the clustering information is proposed. Specifically, we first develop a new change-point detection algorithm that integrates the kernel-based aggregation of spatial observations with a global loss function at the temporal level to group data into several sets, and then derive an FDR control scheme for set-wise multiple testing. Under some mild conditions on the spatiotemporal dependence structure, FDR is shown to be strongly controlled. Theoretical analysis and numerical studies demonstrate the advantages of the algorithm over competing methods.

Covariate-modulated large-scale multiple testing under dependence

Large-scale multiple testing, which calls for conducting tens of thousands of hypothesis testings simultaneously, has been applied in many scientific fields. Most conventional multiple testing procedures often focused on the control of false discovery rate (FDR) and largely ignored covariate information and the dependence structure among tests. A FDR control procedure, termed as Covariate-Modulated Local Index of Significance (cmLIS) procedure, which not only takes into account local correlations among tests but also accommodates the covariate information by leveraging a covariate-modulated hidden Markov model (HMM), has been proposed. In the oracle case where all parameters of the covariate-modulated HMM are known, the cmLIS procedure is shown to be valid and optimal in some sense. According to whether the number of mixed components in the non-null distribution is known, two Bayesian sampling algorithms are provided for parameter estimation. Extensive simulations are conducted to demonstrate the effectiveness of the cmLIS procedure over state-of-the-art multiple testing procedures. Finally, the cmLIS procedure is applied to an RNA sequencing data and a schizophrenia (SCZ) data.

Quantifying the dynamic transmission of COVID-19 asymptomatic and symptomatic infections: Evidence from four Chinese regions.

The dynamic transmission of asymptomatic and symptomatic COVID-19 infections is difficult to quantify because asymptomatic infections are not readily recognized or self-identified. To address this issue, we collected data on asymptomatic and symptomatic infections from four Chinese regions (Beijing, Dalian, Xinjiang, and Guangzhou). These data were considered reliable because the government had implemented large-scale multiple testing during the outbreak in the four regions. We modified the classical susceptible–exposure–infection–recovery model and combined it with mathematical tools to quantitatively analyze the number of infections caused by asymptomatic and symptomatic infections during dynamic transmission, respectively. The results indicated that the ratios of the total number of asymptomatic to symptomatic infections were 0.13:1, 0.48:1, 0.29:1, and 0.15:1, respectively, in the four regions. However, the ratio of the total number of infections caused by asymptomatic and symptomatic infections were 4.64:1, 6.21:1, 1.49:1, and 1.76:1, respectively. Furthermore, the present study describes the daily number of healthy people infected by symptomatic and asymptomatic transmission and the dynamic transmission process. Although there were fewer asymptomatic infections in the four aforementioned regions, their infectivity was found to be significantly higher, implying a greater need for timely screening and control of infections, particularly asymptomatic ones, to contain the spread of COVID-19.

Semi-parametric hidden Markov model for large-scale multiple testing under dependency

In this article, we propose a new semiparametric hidden Markov model (HMM) for use in the simultaneous hypothesis testing with dependency. The semi- or non-parametric HMM in the literature requires two conditions for its model identifiability, (a) the latent Markov chain (MC) is ergodic and its transition probability is full rank and (b) the observational distributions of different hidden states are disjoint or linearly independent. Unlike the existing models, our semiparametric HMM with two hidden states makes no assumption on the transition probability of the latent MC but assumes that observational distributions are extremal for the set of all stationary distributions of the model. To estimate the model, we propose a modified expectation-maximization algorithm, whose M-step has an additional purification step to make the observational distribution be extremal one. We numerically investigate the performance of the proposed procedure in the estimation of the model and compare it to two recent existing methods in various multiple testing error settings. In addition, we apply our procedure to analyzing two real data examples, the gas chromatography/mass spectrometry experiment to differentiate the origin of herbal medicine and the epidemiologic surveillance of an influenza-like illness.

Large-scale inference with block structure

The detection of weak and rare effects in large amounts of data arises in a number of modern data analysis problems. Known results show that in this situation the potential of statistical inference is severely limited by the large-scale multiple testing that is inherent in these problems. Here, we show that fundamentally more powerful statistical inference is possible when there is some structure in the signal that can be exploited, for example, if the signal is clustered in many small blocks, as is the case in some relevant applications. We derive the detection boundary in such a situation where we allow both the number of blocks and the block length to grow polynomially with sample size. We derive these results both for the univariate and the multivariate settings as well as for the problem of detecting clusters in a network. These results recover as special cases the sparse signal detection problem (Ann. Statist. 32 (2004) 962–994) where there is no structure in the signal, as well as the scan problem (Statist. Sinica 23 (2013) 409–428) where the signal comprises a single interval. We develop methodology that allows optimal adaptive detection in the general setting, thus exploiting the structure if it is present without incurring a relevant penalty in the case where there is no structure. The advantage of this methodology can be considerable, as in the case of no structure the means need to increase at the rate logn to ensure detection, while the presence of structure allows detection even if the means decrease at a polynomial rate.

Large-scale multiple testing via multivariate hidden Markov models

Large-scale multiple testing with correlated tests and auxiliary statistics arises in a wide range of scientific fields. Conventional multiple testing procedures largely ignored auxiliary information, such as sparsity information, and the dependence structure among tests. This may result in loss of testing efficiency. In this paper, we propose a procedure, called multivariate local index of significance (mvLIS) procedure, for large-scale multiple testing. The mvLIS procedure can not only characterize local correlations among tests via a Markov chain but also incorporates auxiliary information via multivariate statistics. We present that the oracle mvLIS procedure is valid, namely, it controls false discovery rate (FDR) at the pre-specified level, and show that it yields the smallest false non-discovery rate (FNR) at the same FDR level. Then a data-driven mvLIS procedure is developed to mimic the oracle procedure. Comprehensive simulation studies and a real data analysis of schizophrenia (SCZ) data are performed to illustrate the superior performance of the mvLIS procedure. Moreover, as a byproduct that is of independent interest, we generalize the single-index modulated (SIM) multiple testing procedure, which embeds prior information via 2-dimensional p-values, to allow for d-dimensional ( d ≥ 3 ) statistics in multiple testing. The detailed extension is deferred to Discussion.

OPTIMAL FALSE DISCOVERY RATE CONTROL FOR LARGE SCALE MULTIPLE TESTING WITH AUXILIARY INFORMATION.

Large-scale multiple testing is a fundamental problem in high dimensional statistical inference. It is increasingly common that various types of auxiliary information, reflecting the structural relationship among the hypotheses, are available. Exploiting such auxiliary information can boost statistical power. To this end, we propose a framework based on a two-group mixture model with varying probabilities of being null for different hypotheses a priori, where a shape-constrained relationship is imposed between the auxiliary information and the prior probabilities of being null. An optimal rejection rule is designed to maximize the expected number of true positives when average false discovery rate is controlled. Focusing on the ordered structure, we develop a robust EM algorithm to estimate the prior probabilities of being null and the distribution of p-values under the alternative hypothesis simultaneously. We show that the proposed method has better power than state-of-the-art competitors while controlling the false discovery rate, both empirically and theoretically. Extensive simulations demonstrate the advantage of the proposed method. Datasets from genome-wide association studies are used to illustrate the new methodology.

Testing and estimation for clustered signals

We propose a change-point detection method for large scale multiple testing problems with data having clustered signals. Unlike the classic change-point setup, the signals can vary in size within a cluster. The clustering structure on the signals enables us to effectively delineate the boundaries between signal and non-signal segments. New test statistics are proposed for observations from one and/or multiple realizations. Their asymptotic distributions are derived. We also study the associated variance estimation problem. We allow the variances to be heteroscedastic in the multiple realization case, which substantially expands the applicability of the proposed method. Simulation studies demonstrate that the proposed approach has a favorable performance. Our procedure is applied to an array based Comparative Genomic Hybridization (aCGH) dataset.

Reliable Post-Signal Fault Diagnosis for Correlated High-Dimensional Data Streams

Rapid advance of sensor technology is facilitating the collection of high-dimensional data streams (HDS). Apart from real-time detection of potential out-of-control (OC) patterns, post-signal fault diagnosis of HDS is becoming increasingly important in the filed of statistical process control to isolate abnormal data streams. The major limitations of the existing methods on that topic include (i) they cannot achieve reliable diagnostic results in the sense that their performance is highly variable, and (ii) the informative correlation among different streams is often neglected by them. This article elaborates the problem of reliable fault diagnosis for monitoring correlated HDS using the large-scale multiple testing. Under the framework of hidden Markov model dependence, new diagnostic procedures are proposed, which can control the missed discovery exceedance (MDX) at a desired level. Extensive numerical studies along with some theoretical results show that the proposed procedures can control MDX properly, leading to diagnostics with high reliability and efficiency. Also, their diagnostic performance can be improved significantly by exploiting the dependence among different data streams, which is especially appealing in practice for identifying clustered OC streams.