Ultrahigh-dimensional Data Analysis Research Articles

Model Selection for High Dimensional Nonparametric Additive Models via Ridge Estimation

In ultrahigh dimensional data analysis, to keep computational performance well and good statistical properties still working, nonparametric additive models face increasing challenges. To overcome them, we introduce a methodology of model selection for high dimensional nonparametric additive models. Our approach is to propose a novel group screening procedure via nonparametric smoothing ridge estimation (GRIE) to find the importance of each covariate. It is then combined with the sure screening property of GRIE and the model selection property of extended Bayesian information criteria (EBIC) to select the suitable sub-models in nonparametric additive models. Theoretically, we establish the strong consistency of model selection for the proposed method. Extensive simulations and two real datasets illustrate the outstanding performance of the GRIE-EBIC method.

High-dimensional robust approximated [formula omitted]-estimators for mean regression with asymmetric data

Asymmetry along with heteroscedasticity or contamination often occurs with the growth of data dimensionality. In ultra-high dimensional data analysis, such irregular settings are usually overlooked for both theoretical and computational convenience. In this paper, we establish a framework for estimation in high-dimensional regression models using Penalized Robust Approximated quadratic M-estimators (PRAM). This framework allows general settings such as random errors lack symmetry and homogeneity, or covariates are not sub-Gaussian. To reduce the possible bias caused by data’s irregularity in mean regression, PRAM adopts a loss function with an adaptive robustification parameter. Theoretically, we first show that, in the ultra-high dimension setting, PRAM estimators have local estimation consistency at the minimax rate enjoyed by the LS-Lasso. Then we show that PRAM with an appropriate non-convex penalty in fact agrees with the local oracle solution, and thus obtain its oracle property. Computationally, we compare the performances of six PRAM estimators (Huber, Tukey’s biweight or Cauchy loss function combined with Lasso or MCP penalty function). Our simulation studies and real data analysis demonstrate satisfactory finite sample performances of the PRAM estimator under different irregular settings.

Feature screening with latent responses.

A novel feature screening method is proposed to examine the correlation between latent responses and potential predictors in ultrahigh-dimensional data analysis. First, a confirmatory factor analysis (CFA) model is used to characterize latent responses through multiple observed variables. The expectation-maximization algorithm is employed to estimate the parameters in the CFA model. Second, R-Vector (RV) correlation is used to measure the dependence between the multivariate latent responses and covariates of interest. Third, a feature screening procedure is proposed on the basis of an unbiased estimator of the RV coefficient. The sure screening property of the proposed screening procedure is established under certain mild conditions. Monte Carlo simulations are conducted to assess the finite-sample performance of the feature screening procedure. The proposed method is applied to an investigation of the relationship between psychological well-being and the humangenome.

Censored mean variance sure independence screening for ultrahigh dimensional survival data

Feature screening has become an indispensable statistical modeling tool for ultrahigh dimensional data analysis. This article introduces a new model-free marginal feature screening approach for ultrahigh dimensional survival data with right censoring. The new procedure could be used for survival data with both ultrahigh dimensional categorical and continuous covariates. Motivated by Cui et al. (2015), a censored mean variance index (cMV) is proposed to measure the dependence between a survival outcome and a categorical covariate. Then a slice-and-fuse method is exploited to modify the cMV index adaptive to continuous covariates. The sure independence screening based on the censored mean variance index (cMV-SIS) is proposed to identify the important covariates for ultrahigh dimensional data with censored survival outcomes. It enjoys many appealing merits inherited in the mean variance index. It is model-free and thus robust to model misspecification. It is also robust to heavy tails and outliers in covariates. Moreover, the sure screening properties are theoretically investigated for both categorical and continuous covariates under some mild technical conditions. Extensive numerical simulations and a real data example have demonstrated the competitive performances of the proposed feature screening method.

Feature screening based on distance correlation for ultrahigh-dimensional censored data with covariate measurement error

Feature screening is an important method to reduce the dimension and capture informative variables in ultrahigh-dimensional data analysis. Its key idea is to select informative variables using correlations between the response and the covariates. Many methods have been developed for feature screening. These methods, however, are challenged by complex features pertinent to the data collection as well as the nature of the data themselves. Typically, incomplete response caused by right-censoring and covariate measurement error are often accompanying with survival analysis. Even though many methods have been proposed for censored data, little work has been available when both incomplete response and measurement error occur simultaneously. In addition, the conventional feature screening methods may fail to detect the truly important covariates that are marginally independent of the response variable due to correlations among covariates. In this paper, we explore this important problem and propose the model-free feature screening method in the presence of the censored response and error-prone covariates. In addition, we also develop the iteration method to improve the accuracy of selecting all important covariates. Numerical studies are reported to assess the performance of the proposed method. Finally, we implement the proposed method to a real dataset.

Nonparametric independence feature screening for ultrahigh-dimensional missing data

Missing data are common in medical and social science studies and often face a serious challenge in ultrahigh-dimensional data analysis. In this paper, a nonparametric feature screening approach based on the imputation technique is proposed for ultrahigh-dimensional data with responses missing at random, where the imputation technique is used to replacing each missing value with a set of plausible values. Our approach has many advantages. On one hand, the suggested method relies only on imputation, and its impact is considerably less than that of the feature screening procedure based on the inverse probability weighting approach in missing probabilities. On the other hand, our method does not rely on any model assumption and works generally for all kinds of situations. Simulation studies are conducted to examine the performance of our approach, and a real data example is also presented for illustration.

Feature screening under missing indicator imputation with non-ignorable missing response

This article develops a model-free variable screening technique with the non-ignorable missing response in ultrahigh-dimensional data analysis. Based on the common logistic model assumption of the propensity function, a novel screening procedure is proposed by borrowing hidden information of missingness indicator such that any variable screening method for ultrahigh-dimensional covariates with full data can be applied to the non-ignorable missing response case. And it is shown that the sure screening property can be kept as long as the corresponding screening method for full data is of sure screening property. The finite sample performances of the proposed method are demonstrated via some simulations and analysis of functional neuroimaging data.

Bayesian Penalized Method for Streaming Feature Selection

The online feature selection with streaming features has become more and more important in recent years. In contrast to standard feature selection method, streaming feature selection method can select feature dynamically without exploring full feature space. In some applications, when the information of all features is unknown in advance, standard feature selection methods cannot perform well in this setting. Moreover, in ultrahigh dimensional data analysis, especially considering interaction effects between features, the streaming feature selection method enjoy computational feasibility. In this paper, we proposed a Bayesian penalized method for streaming feature selection problem. The proposed approach adopts Bayesian regularization into penalized model which can adaptively estimates regularization parameter based on the coefficients of current model. Comparing with many existing streaming feature selection methods, our method can work for more general case of predictive model. The proposed method is evaluated extensively on various high-dimensional datasets. The experimental results show that the algorithm is competitive with many existing streaming and traditional feature selection algorithms.

Conditional quantile correlation screening procedure for ultrahigh-dimensional varying coefficient models

Ultrahigh-dimensional variable screening plays an increasingly important part in diverse scientific areas and statistical researches. This paper mainly proposes two new feature screening approaches for varying coefficient models in ultrahigh-dimensional data analysis. One of them use the conditional quantile correlation corresponding to CQSISτ as an utility measure of importance between the τth quantile of response and predictor conditioning on the index variable, and the other utilizes the conditional distribution correlation, which corresponds to CQSIS, as an utility measure of significance between the response and predictor conditioning on the index variable. Under some regularization conditions, we establish the theoretical properties, including ranking consistency property and sure screening property. Simulation studies are conducted to evaluate the performance of the proposed methodologies. The simulations results show that our proposed approaches CQSISτ and CQSIS significantly outperforms the existing methods in terms of varying coefficient models. We also illustrate the performance of CQSISτ and CQSIS through two real-data examples. Both theoretical and numerical studies demonstrate the effectiveness of the proposed methods.

Fused mean–variance filter for feature screening

A new model-free screening approach called as the slicing fused mean–variance filter is proposed for ultrahigh dimensional data analysis. The new method has the following merits: (i) its implementation does not require specifying a regression form of predictors and response variables; (ii) it can deal with various types of covariates and response variables including continuous, discrete and categorical variables; (iii) it works well even when the covariates/random errors are heavy-tailed, or the predictors are strongly correlated, or there are outliers; (iv) it is unsensitive to the slicing scheme. Under some regularity conditions, the sure screening and ranking consistency properties are established for the proposed procedure without assuming any moment conditions on the predictors. Simulation studies are conducted to investigate the finite sample performance of the proposed procedure. A real data example is illustrated to the proposed procedure.

A model-free feature screening approach based on kernel density estimation

ABSTRACTIn this article, a new model-free feature screening method named after probability density (mass) function distance (PDFD) correlation is presented for ultrahigh-dimensional data analysis. We improve the fused-Kolmogorov filter (F-KOL) screening procedure through probability density distribution. The proposed method is also fully nonparametric and can be applied to more general types of predictors and responses, including discrete and continuous random variables. Kernel density estimate method and numerical integration are applied to obtain the estimator we proposed. The results of simulation studies indicate that the fused-PDFD performs better than other existing screening methods, such as F-KOL filter, sure-independent screening (SIS), sure independent ranking and screening (SIRS), distance correlation sure-independent screening (DCSIS) and robust ranking correlation screening (RRCS). Finally, we demonstrate the validity of fused-PDFD by a real data example.

Model-free feature screening for ultrahigh-dimensional data conditional on some variables

In this paper, the conditional distance correlation (CDC) is used as a measure of correlation to develop a conditional feature screening procedure given some significant variables for ultrahigh-dimensional data. The proposed procedure is model free and is called conditional distance correlation-sure independence screening (CDC-SIS for short). That is, we do not specify any model structure between the response and the predictors, which is appealing in some practical problems of ultrahigh-dimensional data analysis. The sure screening property of the CDC-SIS is proved and a simulation study was conducted to evaluate the finite sample performances. Real data analysis is used to illustrate the proposed method. The results indicate that CDC-SIS performs well.

Model-Free Conditional Independence Feature Screening For Ultrahigh Dimensional Data.

Feature screening plays an important role in ultrahigh dimensional data analysis. This paper is concerned with conditional feature screening when one is interested in detecting the association between the response and ultrahigh dimensional predictors (e.g., genetic makers) given a low-dimensional exposure variable (such as clinical variables or environmental variables). To this end, we first propose a new index to measure conditional independence, and further develop a conditional screening procedure based on the newly proposed index. We systematically study the theoretical property of the proposed procedure and establish the sure screening and ranking consistency properties under some very mild conditions. The newly proposed screening procedure enjoys some appealing properties. (a) It is model-free in that its implementation does not require a specification on the model structure; (b) it is robust to heavy-tailed distributions or outliers in both directions of response and predictors; and (c) it can deal with both feature screening and the conditional screening in a unified way. We study the finite sample performance of the proposed procedure by Monte Carlo simulations and further illustrate the proposed method through two real data examples.

Interaction Screening for Ultrahigh-Dimensional Data

In ultrahigh-dimensional data analysis, it is extremely challenging to identify important interaction effects, and a top concern in practice is computational feasibility. For a dataset with n observations and p predictors, the augmented design matrix including all linear and order-2 terms is of size n × (p2 + 3p)/2. When p is large, say more than tens of hundreds, the number of interactions is enormous and beyond the capacity of standard machines and software tools for storage and analysis. In theory, the interaction-selection consistency is hard to achieve in high-dimensional settings. Interaction effects have heavier tails and more complex covariance structures than main effects in a random design, making theoretical analysis difficult. In this article, we propose to tackle these issues by forward-selection-based procedures called iFOR, which identify interaction effects in a greedy forward fashion while maintaining the natural hierarchical model structure. Two algorithms, iFORT and iFORM, are studied. Computationally, the iFOR procedures are designed to be simple and fast to implement. No complex optimization tools are needed, since only OLS-type calculations are involved; the iFOR algorithms avoid storing and manipulating the whole augmented matrix, so the memory and CPU requirement is minimal; the computational complexity is linear in p for sparse models, hence feasible for p ≫ n. Theoretically, we prove that they possess sure screening property for ultrahigh-dimensional settings. Numerical examples are used to demonstrate their finite sample performance. Supplementary materials for this article are available online.

Independent feature screening for ultrahigh-dimensional models with interactions

Feature selection is an important technique for ultrahigh-dimensional data analysis. Most feature selection methods such as SIS and its relevant versions heavily depend on the specified model structures. Furthermore, feature interactions are usually not taken into account in the existing literature. In this paper, we present a novel feature selection method for the model with variable interactions, without the use of structure assumption. Thus, the new ranking criterion is flexible and can deal with the models that contain interactions. Moreover, the new screening procedures are not complex, consequently, they are computationally efficient and the theoretical properties such as the ranking consistency and sure screening properties can be easily obtained. Several real and simulation examples are presented to illustrate the methodology.

Feature Screening via Distance Correlation Learning

This article is concerned with screening features in ultrahigh-dimensional data analysis, which has become increasingly important in diverse scientific fields. We develop a sure independence screening procedure based on the distance correlation (DC-SIS). The DC-SIS can be implemented as easily as the sure independence screening (SIS) procedure based on the Pearson correlation proposed by Fan and Lv. However, the DC-SIS can significantly improve the SIS. Fan and Lv established the sure screening property for the SIS based on linear models, but the sure screening property is valid for the DC-SIS under more general settings, including linear models. Furthermore, the implementation of the DC-SIS does not require model specification (e.g., linear model or generalized linear model) for responses or predictors. This is a very appealing property in ultrahigh-dimensional data analysis. Moreover, the DC-SIS can be used directly to screen grouped predictor variables and multivariate response variables. We establish the sure screening property for the DC-SIS, and conduct simulations to examine its finite sample performance. A numerical comparison indicates that the DC-SIS performs much better than the SIS in various models. We also illustrate the DC-SIS through a real-data example.

Testing the Statistical Significance of an Ultra-High-Dimensional Naive Bayes Classifier

The naive Bayes approach is one of the most popular methods used for classification. Nevertheless, how to test its statistical significance under an ultra-high-dimensional (UHD) setup is not well understood. To fill this important theoretical gap, we propose a novel testing statistic with a standard normal asymptotic null distribution, even if the predictor dimension is considerably larger than the sample size. This makes the proposed method useful for UHD data analysis. Simulation studies are presented to demonstrate its finite sample performance and a text classification example is described for illustration.

Robust Detection and Identification of Sparse Segments in Ultra-High Dimensional Data Analysis.

Copy number variants (CNVs) are alternations of DNA of a genome that results in the cell having a less or more than two copies of segments of the DNA. CNVs correspond to relatively large regions of the genome, ranging from about one kilobase to several megabases, that are deleted or duplicated. Motivated by CNV analysis based on next generation sequencing data, we consider the problem of detecting and identifying sparse short segments hidden in a long linear sequence of data with an unspecified noise distribution. We propose a computationally efficient method that provides a robust and near-optimal solution for segment identification over a wide range of noise distributions. We theoretically quantify the conditions for detecting the segment signals and show that the method near-optimally estimates the signal segments whenever it is possible to detect their existence. Simulation studies are carried out to demonstrate the efficiency of the method under different noise distributions. We present results from a CNV analysis of a HapMap Yoruban sample to further illustrate the theory and the methods.