Ultra-high Dimensional Setting Research Articles

Nonparametric screening for additive quantile regression in ultra-high dimension

In practical applications, one often does not know the ‘true’ structure of the underlying conditional quantile function, especially in the ultra-high dimensional setting. To deal with ultra-high dimensionality, quantile-adaptive marginal nonparametric screening methods have been recently developed. However, these approaches may miss important covariates that are marginally independent of the response, or may select unimportant covariates due to their high correlations with important covariates. To mitigate such shortcomings, we develop a conditional nonparametric quantile screening procedure (complemented by subsequent selection) for nonparametric additive quantile regression models. Under some mild conditions, we show that the proposed screening method can identify all relevant covariates in a small number of steps with probability approaching one. The subsequent narrowed best subset (via a modified Bayesian information criterion) also contains all the relevant covariates with overwhelming probability. The advantages of our proposed procedure are demonstrated through simulation studies and a real data example.

Hypothesis testing on high dimensional quantile regression

Quantile regression has been an important analytical tool in econometrics since it was proposed in 1970s. Many advantages make it still popular in the era of big data. This paper focuses on the testing problems of high-dimensional quantile regression, supplementing to robust methods in the literature of high-dimensional hypothesis testing. We first construct a new test statistic based on the quantile regression score function. The new test statistic avoids the inverse operation of the covariance matrix, and hence becomes applicable to high-dimensional or even ultrahigh-dimensional settings. The proposed method retains robustness for non-Gaussian and heavy-tailed distributions. We then derive the limiting distributions of the proposed test statistic under both the null and the alternative hypotheses. We further investigate the case where the design matrix follows an elliptical distribution. We examined the finite sample performance of our proposed method through Monte Carlo simulations. The numerical comparisons exhibit that our proposed tests outperform some existing methods in terms of controlling Type I error rate and power, when the data deviate from the Gaussian assumptions or are heavy-tailed. We illustrate our proposed high-dimensional quantile testing in financial econometrics, through an empirical analysis of Chinese stock market data.

Prior Knowledge Guided Ultra-high Dimensional Variable Screening with Application to Neuroimaging Data.

Variable screening is a powerful and efficient tool for dimension reduction under ultrahigh dimensional settings. However, most existing methods overlook useful prior knowledge in specific applications. In this work, from a Bayesian modeling perspective, we develop a unified variable screening procedure for the linear regression model. We discuss different constructions of posterior mean screening (PMS) statistics to incorporate different types of prior knowledge according to specific applications. With non-informative prior specifications, PMS is equivalent to high-dimensional ordinary least-square projections (HOLP). We establish the screening consistency property for PMS with different types of prior knowledge. We show that PMS is robust to prior misspecifications; and when the prior knowledge provides correct information on summarizing the true parameter settings, PMS can substantially improve the selection accuracy compared to HOLP and other existing methods. We illustrate our method with extensive simulation studies and an analysis of neuroimaging data.

A Unified Framework for Correlation Mining in Ultra-High Dimension

Many applications benefit from theory relevant to the identification of variables having large correlations or partial correlations in high dimension. Recently there has been progress in the ultra-high dimensional setting when the sample size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> is fixed and the dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> tends to infinity. Despite these advances, the correlation screening framework suffers from practical, methodological and theoretical deficiencies. For instance, previous correlation screening theory requires that the population covariance matrix be sparse and block diagonal. This block sparsity assumption is however restrictive in practical applications. As a second example, correlation and partial correlation screening requires the estimation of dependence measures, which can be computationally prohibitive. In this paper, we propose a unifying approach to correlation and partial correlation mining that is not restricted to block diagonal correlation structure, thus yielding a methodology that is suitable for modern applications. By making connections to random geometric graphs, the number of highly correlated or partial correlated variables are shown to have compound Poisson finite-sample characterizations, which hold for both the finite <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> case and when <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> tends to infinity. The unifying framework also demonstrates a duality between correlation and partial correlation screening with theoretical and practical consequences.

Tests for high-dimensional single-index models

In this paper, we aim to test the overall significance of regression coefficients in high-dimensional single-index models. We first reformulate the hypothesis testing problem under elliptical distributions for predictors. Applying distribution-based transformation, we introduce a high-dimensional score-type test statistic. Notably, no moment condition for the error term is required. Our introduced procedures are thus robust with respect to outliers in response. Moreover our procedure is free of variance estimation of the error term. We establish the test statistic’s asymptotic normality under null hypothesis. Power analysis is also investigated. To further improve computational efficiency and enhance empirical powers, we also introduce a two-stage test procedure under ultrahigh-dimensional settings based on random data splitting. To eliminate the additional randomness induced by data splitting, we further develop a powerful ensemble algorithm based on multiple data splitting. We show that the ensemble algorithm can control the type I error rate at a given significance level. Extension to partial significance testing problem is also investigated. Lastly, numerical studies and real data analysis are conducted to compare with existing approaches and to illustrate the robustness and validity of our proposed test procedures.

An Additive Graphical Model for Discrete Data

We introduce a nonparametric graphical model for discrete node variables based on additive conditional independence. Additive conditional independence is a three-way statistical relation that shares similar properties with conditional independence by satisfying the semi-graphoid axioms. Based on this relation we build an additive graphical model for discrete variables that does not suffer from the restriction of a parametric model such as the Ising model. We develop an estimator of the new graphical model via the penalized estimation of the discrete version of the additive precision operator and establish the consistency of the estimator under the ultrahigh-dimensional setting. Along with these methodological developments, we also exploit the properties of discrete random variables to uncover a deeper relation between additive conditional independence and conditional independence than previously known. The new graphical model reduces to a conditional independence graphical model under certain sparsity conditions. We conduct simulation experiments and analysis of an HIV antiretroviral therapy dataset to compare the new method with existing ones. Supplementary materials for this article are available online.

Model Based Screening Embedded Bayesian Variable Selection for Ultra-high Dimensional Settings

We develop a Bayesian variable selection method, called SVEN, based on a hierarchical Gaussian linear model with priors placed on the regression coefficients as well as on the model space. Sparsity is achieved by using degenerate spike priors on inactive variables, whereas Gaussian slab priors are placed on the coefficients for the important predictors making the posterior probability of a model available in explicit form (up to a normalizing constant). Embedding a unique model based screening and using fast Cholesky updates, SVEN produces a highly scalable computational framework to explore gigantic model spaces, rapidly identify the regions of high posterior probabilities and make fast inference and prediction. A temperature schedule is used to further mitigate multimodal posterior distributions. The temperature value is guided by our model selection consistency results which hold even when the norm of mean effects solely due to the unimportant variables diverges. An appealing byproduct of SVEN is the construction of novel model weight adjusted prediction intervals. The performance of SVEN is demonstrated through a number of simulation experiments and a real data example from a genome wide association study with over half a million markers. Supplementary materials for this article are available online.

Communication-Efficient Distributed Learning for High-Dimensional Support Vector Machines

Distributed learning has received increasing attention in recent years and is a special need for the era of big data. For a support vector machine (SVM), a powerful binary classification tool, we proposed a novel efficient distributed sparse learning algorithm, the communication-efficient surrogate likelihood support vector machine (CSLSVM), in high-dimensions with convex or nonconvex penalties, based on a communication-efficient surrogate likelihood (CSL) framework. We extended the CSL for distributed SVMs without the need to smooth the hinge loss or the gradient of the loss. For a CSLSVM with lasso penalty, we proved that its estimator could achieve a near-oracle property for l1 penalized SVM estimators on whole datasets. For a CSLSVM with smoothly clipped absolute deviation penalty, we showed that its estimator enjoyed the oracle property, and that it used local linear approximation (LLA) to solve the optimization problem. Furthermore, we showed that the LLA was guaranteed to converge to the oracle estimator, even in our distributed framework and the ultrahigh-dimensional setting, if an appropriate initial estimator was available. The proposed approach is highly competitive with the centralized method within a few rounds of communications. Numerical experiments provided supportive evidence.

Constructing a polygenic risk score for childhood obesity using functional data analysis

Obesity is a highly heritable condition that affects increasing numbers of adults and, concerningly, of children. However, only a small fraction of its heritability has been attributed to specific genetic variants. These variants are traditionally ascertained from genome-wide association studies (GWAS), which utilize samples with tens or hundreds of thousands of individuals for whom a single summary measurement (e.g., BMI) is collected. An alternative approach is to focus on a smaller, more deeply characterized sample in conjunction with advanced statistical models that leverage longitudinal phenotypes. Novel functional data analysis (FDA) techniques are used to capitalize on longitudinal growth information from a cohort of children between birth and three years of age. In an ultra-high dimensional setting, hundreds of thousands of single nucleotide polymorphisms (SNPs) are screened, and selected SNPs are used to construct two polygenic risk scores (PRS) for childhood obesity using a weighting approach that incorporates the dynamic and joint nature of SNP effects. These scores are significantly higher in children with (vs. without) rapid infant weight gain—a predictor of obesity later in life. Using two independent cohorts, it is shown that the genetic variants identified in very young children are also informative in older children and in adults, consistent with early childhood obesity being predictive of obesity later in life. In contrast, PRSs based on SNPs identified by adult obesity GWAS are not predictive of weight gain in the cohort of young children. This provides an example of a successful application of FDA to GWAS. This application is complemented with simulations establishing that a deeply characterized sample can be just as, if not more, effective than a comparable study with a cross-sectional response. Overall, it is demonstrated that a deep, statistically sophisticated characterization of a longitudinal phenotype can provide increased statistical power to studies with relatively small sample sizes; and shows how FDA approaches can be used as an alternative to the traditional GWAS.

RaSE: A Variable Screening Framework via Random Subspace Ensembles

Variable screening methods have been shown to be effective in dimension reduction under the ultra-high dimensional setting. Most existing screening methods are designed to rank the predictors according to their individual contributions to the response. As a result, variables that are marginally independent but jointly dependent with the response could be missed. In this work, we propose a new framework for variable screening, random subspace ensemble (RaSE), which works by evaluating the quality of random subspaces that may cover multiple predictors. This new screening framework can be naturally combined with any subspace evaluation criterion, which leads to an array of screening methods. The framework is capable to identify signals with no marginal effect or with high-order interaction effects. It is shown to enjoy the sure screening property and rank consistency. We also develop an iterative version of RaSE screening with theoretical support. Extensive simulation studies and real-data analysis show the effectiveness of the new screening framework.

Individual Data Protected Integrative Regression Analysis of High-Dimensional Heterogeneous Data

Evidence-based decision making often relies on meta-analyzing multiple studies, which enables more precise estimation and investigation of generalizability. Integrative analysis of multiple heterogeneous studies is, however, highly challenging in the ultra high-dimensional setting. The challenge is even more pronounced when the individual-level data cannot be shared across studies, known as DataSHIELD contraint. Under sparse regression models that are assumed to be similar yet not identical across studies, we propose in this paper a novel integrative estimation procedure for data-Shielding High-dimensional Integrative Regression (SHIR). SHIR protects individual data through summary-statistics-based integrating procedure, accommodates between-study heterogeneity in both the covariate distribution and model parameters, and attains consistent variable selection. Theoretically, SHIR is statistically more efficient than the existing distributed approaches that integrate debiased LASSO estimators from the local sites. Furthermore, the estimation error incurred by aggregating derived data is negligible compared to the statistical minimax rate and SHIR is shown to be asymptotically equivalent in estimation to the ideal estimator obtained by sharing all data. The finite-sample performance of our method is studied and compared with existing approaches via extensive simulation settings. We further illustrate the utility of SHIR to derive phenotyping algorithms for coronary artery disease using electronic health records data from multiple chronic disease cohorts.

Graphical-model based high dimensional generalized linear models

We consider the problem of both prediction and model selection in high dimensional generalized linear models. Predictive performance can be improved by leveraging structure information among predictors. In this paper, a graphic model-based doubly sparse regularized estimator is discussed under the high dimensional generalized linear models, that utilizes the graph structure among the predictors. The graphic information among predictors is incorporated node-by-node using a decomposed representation and the sparsity is encouraged both within and between the decomposed components. We propose an efficient iterative proximal algorithm to solve the optimization problem. Statistical convergence rates and selection consistency for the doubly sparse regularized estimator are established in the ultra-high dimensional setting. Specifically, we allow the dimensionality grows exponentially with the sample size. We compare the estimator with existing methods through numerical analysis on both simulation study and a microbiome data analysis.

Ultrahigh-dimensional sufficient dimension reduction with measurement error in covariates

Analysis of sufficient dimension reduction (SDR) is an important topic and has attracts our attention in decades. Several methods have been proposed based on simple settings. In applications, however, the ultrahigh-dimensional setting with p≫n and covariate measurement error usually appear in the dataset, and it is not trivial to adopt the conventional methods to handle this problem. In this paper, we consider the SDR with both the ultrahigh-dimensional setting and covariate measurement error incorporated simultaneously. The theoretical results of the proposed method are established.

Supervised t-distributed stochastic neighbor embedding for data visualization and classification.

We propose a novel supervised dimension reduction method, called supervised t-distributed stochastic neighbor embedding (St-SNE), which achieves dimension reduction by preserving the similarities of data points in both feature and outcome spaces. The proposed method can be used for both prediction and visualization tasks, with the ability to handle high-dimensional data. We show through a variety of datasets that when compared with a comprehensive list of existing methods, St-SNE has superior prediction performance in the ultra-high dimensional setting where the number of features p exceeds the sample size n, and has competitive performance in the p ≤ n setting. We also show that St-SNE is a competitive visualization tool that is capable of capturing within cluster variations. In addition, we propose a penalized Kullback-Leibler divergence criterion to automatically select the reduced dimension size k for St-SNE.

Uniform joint screening for ultra-high dimensional graphical models

Identifying large-scale conditional dependence structures through graphical models is a challenging yet practical problem. Under ultra-high dimensional settings, a screening procedure is generally suggested before variable selection to reduce computational costs. However, most existing screening methods examine the marginal correlations, thus not suitable to discover the conditional dependence in graphical models. To overcome this issue, we propose a new procedure called graphical uniform joint screening (GUS) for edge identification in graphical models. Instead of screening out edges nodewisely, GUS utilizes a uniform threshold for all statistics indicating the significance of different edges to adapt to various kinds of graphical structures. We demonstrate that GUS enjoys the sure screening property and even the screening consistency by preserving the rankings of the significant edges. Furthermore, a scalable implementation of GUS is developed for big data applications. Simulation and real data studies are provided to illustrate the effectiveness of the proposed method.

A new joint screening method for right-censored time-to-event data with ultra-high dimensional covariates.

In an ultra-high dimensional setting with a huge number of covariates, variable screening is useful for dimension reduction before applying a more refined method for model selection and statistical analysis. This paper proposes a new sure joint screening procedure for right-censored time-to-event data based on a sparsity-restricted semiparametric accelerated failure time model. Our method, referred to as Buckley-James assisted sure screening (BJASS), consists of an initial screening step using a sparsity-restricted least-squares estimate based on a synthetic time variable and a refinement screening step using a sparsity-restricted least-squares estimate with the Buckley-James imputed event times. The refinement step may be repeated several times to obtain more stable results. We show that with any fixed number of refinement steps, the BJASS procedure retains all important variables with probability tending to 1. Simulation results are presented to illustrate its performance in comparison with some marginal screening methods. Real data examples are provided using a diffuse large-B-cell lymphoma (DLBCL) data and a breast cancer data. We have implemented the BJASS method using Matlab and made it available to readers through Github https://github.com/yiucla/BJASS .

Rates of convergence of the adaptive elastic net and the post-selection procedure in ultra-high dimensional sparse models

This article studies the convergence rates of the adaptive elastic net and proposes a post-selection method. In the low dimensional settings, Zou and Zhou proved the variable selection consistency of the adaptive elastic net. In this article, we show the probability of the adaptive elastic net selecting the wrong variables decays at an exponential rate in the ultra-high dimensional setting. The Irrepresentable condition isn’t necessary according to some appropriate initial estimator. Furthermore, we show that the MSE and bias of the adaptive elastic net may lead to an inferior rate depending on the choice of the tuning parameters. We also prove that there exists a bias of asymptotic normality in the adaptive elastic net. Hence we propose a post-selection procedure, called SSLS (Separate Selection from Least Squares), to improve the bias problem of the adaptive elastic net. It is shown that the SSLS’s bias decays at an exponential rate and it is MSE decays to zero. The variable selection consistency of SSLS implies its asymptotic normality. We show by simulations and financial modeling that the SSLS performs better than the other oracle-like methods.

Robust feature screening for elliptical copula regression model

In this paper, we propose a flexible semi-parametric regression model called Elliptical Copula Regression (ECR) model, which covers a large class of linear and nonlinear regression models such as the additive regression model and the linear transformation model. In addition, ECR model can capture the heavy-tail characteristic and tail dependence between variables, thus it can be widely applied in many areas such as econometrics and finance. We mainly focus on the feature screening problem for ECR model in an ultra-high dimensional setting here. We propose a robust feature screening procedure for ECR model, in which two types of correlation coefficients are involved: Kendall’s τ correlation and canonical correlation. Theoretical analysis shows that the procedure enjoys sure screening property, i.e., with probability tending to 1, the feature screening procedure selects out all important variables and substantially reduces the dimensionality to a moderate size against the sample size. Thorough numerical studies are conducted to illustrate its advantage over existing feature screening methods. At last, the proposed procedure is applied to a gene-expression real data set to show its empirical usefulness.

ROCKET: Robust confidence intervals via Kendall’s tau for transelliptical graphical models

Understanding complex relationships between random variables is of fundamental importance in high-dimensional statistics, with numerous applications in biological and social sciences. Undirected graphical models are often used to represent dependencies between random variables, where an edge between two random variables is drawn if they are conditionally dependent given all the other measured variables. A large body of literature exists on methods that estimate the structure of an undirected graphical model, however, little is known about the distributional properties of the estimators beyond the Gaussian setting. In this paper, we focus on inference for edge parameters in a high-dimensional transelliptical model, which generalizes Gaussian and nonparanormal graphical models. We propose ROCKET, a novel procedure for estimating parameters in the latent inverse covariance matrix. We establish asymptotic normality of ROCKET in an ultra high-dimensional setting under mild assumptions, without relying on oracle model selection results. ROCKET requires the same number of samples that are known to be necessary for obtaining a $\sqrt{n}$ consistent estimator of an element in the precision matrix under a Gaussian model. Hence, it is an optimal estimator under a much larger family of distributions. The result hinges on a tight control of the sparse spectral norm of the nonparametric Kendall’s tau estimator of the correlation matrix, which is of independent interest. Empirically, ROCKET outperforms the nonparanormal and Gaussian models in terms of achieving accurate inference on simulated data. We also compare the three methods on real data (daily stock returns), and find that the ROCKET estimator is the only method whose behavior across subsamples agrees with the distribution predicted by the theory.

Sparse Minimum Discrepancy Approach to Sufficient Dimension Reduction with Simultaneous Variable Selection in Ultrahigh Dimension

ABSTRACTSufficient dimension reduction (SDR) is known to be a powerful tool for achieving data reduction and data visualization in regression and classification problems. In this work, we study ultrahigh-dimensional SDR problems and propose solutions under a unified minimum discrepancy approach with regularization. When p grows exponentially with n, consistency results in both central subspace estimation and variable selection are established simultaneously for important SDR methods, including sliced inverse regression (SIR), principal fitted component (PFC), and sliced average variance estimation (SAVE). Special sparse structures of large predictor or error covariance are also considered for potentially better performance. In addition, the proposed approach is equipped with a new algorithm to efficiently solve the regularized objective functions and a new data-driven procedure to determine structural dimension and tuning parameters, without the need to invert a large covariance matrix. Simulations and a real data analysis are offered to demonstrate the promise of our proposal in ultrahigh-dimensional settings. Supplementary materials for this article are available online.