Ultrahigh-dimensional Data Research Articles

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates

In this paper, a new robust and efficient estimation approach based on local modal regression is proposed for partially linear models with large-dimensional covariates. We show that the resulting estimators for both parametric and nonparametric components are more efficient in the presence of outliers or heavy-tail error distribution, and as asymptotically efficient as the corresponding least squares estimators when there are no outliers and the error distribution is normal. We also establish the asymptotic properties of proposed estimators when the covariate dimension diverges at the rate of $$o\left( {\sqrt{n} } \right) \mathrm{{ }}$$ . To achieve sparsity and enhance interpretability, we develop a variable selection procedure based on SCAD penalty to select significant parametric covariates and show that the method enjoys the oracle property under mild regularity conditions. Moreover, we propose a practical modified MEM algorithm for the proposed procedures. Some Monte Carlo simulations and a real data are conducted to illustrate the finite sample performance of the proposed estimators. Finally, based on the idea of sure independence screening procedure proposed by Fan and Lv (J R Stat Soc 70:849–911, 2008), a robust two-step approach is introduced to deal with ultra-high dimensional data.

Model-free conditional feature screening for ultra-high dimensional right censored data

ABSTRACTThis paper is concerned with the conditional feature screening for ultra-high dimensional right censored data with some previously identified important predictors. A new model-free conditional feature screening approach, conditional correlation rank sure independence screening, has been proposed and investigated theoretically. The suggested conditional screening procedure has several desirable merits. First, it is model free, and thus robust to model misspecification. Second, it has the advantage of robustness of heavy-tailed distributions of the response and the presence of potential outliers in response. Third, it is naturally applicable to complete data when there is no censoring. Through simulation studies, we demonstrate that the proposed approach outperforms the CoxCS of Hong et al. under some circumstances. A real dataset is used to illustrate the usefulness of the proposed conditional screening method.

Bayesian Multiresolution Variable Selection for Ultra-High Dimensional Neuroimaging Data.

Ultra-high dimensional variable selection has become increasingly important in analysis of neuroimaging data. For example, in the Autism Brain Imaging Data Exchange (ABIDE) study, neuroscientists are interested in identifying important biomarkers for early detection of the autism spectrum disorder (ASD) using high resolution brain images that include hundreds of thousands voxels. However, most existing methods are not feasible for solving this problem due to their extensive computational costs. In this work, we propose a novel multiresolution variable selection procedure under a Bayesian probit regression framework. It recursively uses posterior samples for coarser-scale variable selection to guide the posterior inference on finer-scale variable selection, leading to very efficient Markov chain Monte Carlo (MCMC) algorithms. The proposed algorithms are computationally feasible for ultra-high dimensional data. Also, our model incorporates two levels of structural information into variable selection using Ising priors: the spatial dependence between voxels and the functional connectivity between anatomical brain regions. Applied to the resting state functional magnetic resonance imaging (R-fMRI) data in the ABIDE study, our methods identify voxel-level imaging biomarkers highly predictive of the ASD, which are biologically meaningful and interpretable. Extensive simulations also show that our methods achieve better performance in variable selection compared to existing methods.

Expectile regression for analyzing heteroscedasticity in high dimension

High-dimensional data often display heteroscedasticity and this feature has attracted a lot of attention and discussion. In this paper, we propose regularized expectile regression with SCAD penalty for analyzing heteroscedasticity in high dimension when the error has finite moments. Since the corresponding optimization problem is nonconvex due to the SCAD penalty, we adopt the CCCP (coupling of the concave and convex procedure) algorithm to solve this problem. Under some regular conditions, we can prove that with probability tending to one, the proposed algorithm converges to the oracle estimator after several iterations. We should address that the higher order moment the error has, the higher dimension cardinality our procedure can handle with. If the error follows gaussian or sub-gaussian distribution, our method can be extended to deal with ultra high-dimensional data. Furthermore, by taking different expectile weight level α, we are able to detect heteroscedasticity and explore the entire conditional distribution of the response variable given all the covariates. We investigate the performances of our proposed method through Monte Carlo simulation study and real application and the numerical results show that the resulting estimator by our algorithm enjoys good performance and demonstrate the usefulness of our proposed method to analyze heteroscedasticity.

Conditional quantile correlation screening procedure for ultrahigh-dimensional varying coefficient models

Ultrahigh-dimensional variable screening plays an increasingly important part in diverse scientific areas and statistical researches. This paper mainly proposes two new feature screening approaches for varying coefficient models in ultrahigh-dimensional data analysis. One of them use the conditional quantile correlation corresponding to CQSISτ as an utility measure of importance between the τth quantile of response and predictor conditioning on the index variable, and the other utilizes the conditional distribution correlation, which corresponds to CQSIS, as an utility measure of significance between the response and predictor conditioning on the index variable. Under some regularization conditions, we establish the theoretical properties, including ranking consistency property and sure screening property. Simulation studies are conducted to evaluate the performance of the proposed methodologies. The simulations results show that our proposed approaches CQSISτ and CQSIS significantly outperforms the existing methods in terms of varying coefficient models. We also illustrate the performance of CQSISτ and CQSIS through two real-data examples. Both theoretical and numerical studies demonstrate the effectiveness of the proposed methods.

Model-Free Feature Screening for Ultrahigh Dimensional Data through a Modified Blum-Kiefer-Rosenblatt Correlation

Model-Free Feature Screening for Ultrahigh Dimensional Data through a Modified Blum-Kiefer-Rosenblatt Correlation

Fused mean–variance filter for feature screening

A new model-free screening approach called as the slicing fused mean–variance filter is proposed for ultrahigh dimensional data analysis. The new method has the following merits: (i) its implementation does not require specifying a regression form of predictors and response variables; (ii) it can deal with various types of covariates and response variables including continuous, discrete and categorical variables; (iii) it works well even when the covariates/random errors are heavy-tailed, or the predictors are strongly correlated, or there are outliers; (iv) it is unsensitive to the slicing scheme. Under some regularity conditions, the sure screening and ranking consistency properties are established for the proposed procedure without assuming any moment conditions on the predictors. Simulation studies are conducted to investigate the finite sample performance of the proposed procedure. A real data example is illustrated to the proposed procedure.

Nonparametric independence screening via favored smoothing bandwidth

We propose a flexible nonparametric regression method for ultrahigh-dimensional data. As a first step, we propose a fast screening method based on the favored smoothing bandwidth of the marginal local constant regression. Then, an iterative procedure is developed to recover both the important covariates and the regression function. Theoretically, we prove that the favored smoothing bandwidth based screening possesses the model selection consistency property. Simulation studies as well as real data analysis show the competitive performance of the new procedure.

Variable screening for ultrahigh dimensional heterogeneous data via conditional quantile correlations

In this article, we propose a new conditional quantile correlation and establish its connection with conditional quantile regression coefficient functions. We further introduce a conditional quantile screening method based on this metric for varying coefficient models with ultrahigh dimensional features. Under some technical conditions, the proposed approach is shown to enjoy desirable theoretical properties, including ranking consistency and sure screening properties. The extent of the new method’s dimensionality reduction is also qualified. To reduce the false selection rate, an iterative algorithm is proposed for improving the accuracy of variable screening. We conduct simulation studies to demonstrate that the proposed screening method can perform reasonably well, and we illustrate the proposed methodology through a real data analysis.

Sparse generalized linear model with L0 approximation for feature selection and prediction with big omics data

BackgroundFeature selection and prediction are the most important tasks for big data mining. The common strategies for feature selection in big data mining are L1, SCAD and MC+. However, none of the existing algorithms optimizes L0, which penalizes the number of nonzero features directly.ResultsIn this paper, we develop a novel sparse generalized linear model (GLM) with L0 approximation for feature selection and prediction with big omics data. The proposed approach approximate the L0 optimization directly. Even though the original L0 problem is non-convex, the problem is approximated by sequential convex optimizations with the proposed algorithm. The proposed method is easy to implement with only several lines of code. Novel adaptive ridge algorithms (L0ADRIDGE) for L0 penalized GLM with ultra high dimensional big data are developed. The proposed approach outperforms the other cutting edge regularization methods including SCAD and MC+ in simulations. When it is applied to integrated analysis of mRNA, microRNA, and methylation data from TCGA ovarian cancer, multilevel gene signatures associated with suboptimal debulking are identified simultaneously. The biological significance and potential clinical importance of those genes are further explored.ConclusionsThe developed Software L0ADRIDGE in MATLAB is available at https://github.com/liuzqx/L0adridge.

Robust feature screening for ultra-high dimensional right censored data via distance correlation

Ultra-high dimensional data with right censored survival times are frequently collected in large-scale biomedical studies, for which feature screening has become an indispensable statistical tool. In this paper, we propose two new feature screening procedures based on distance correlation. The first approach performs feature screening through replacing the response and covariate by their cumulative distribution functions’ Kaplan–Meier estimator and empirical distribution function respectively, while the second one modifies the distance correlation via an idea of composite quantile regression. The sure screening properties are established under some rather mild technical assumptions, which allow that the dimensionality increases at an exponential rate of the sample size. The proposed methods have three desirable characteristics. Firstly, they are model-free and thus robust to model misspecification. Secondly, they behave reliably when some features contain outliers or follow heavy-tailed distributions. Thirdly, our procedures have better convergence rate than that of distance correlation screening in Li et al. (2012b). Both simulated and real examples show that the proposed methods perform competitively.

A new nonparametric screening method for ultrahigh-dimensional survival data

For ultrahigh-dimensional data, sure independent screening methods can effectively reduce the dimensionality while ensuring that all the active variables can be retained with high probability. However, most existing screening procedures are developed for ultrahigh-dimensional complete data and cannot be applicable to censored survival data. To address the new challenges from censoring, a novel model-free screening method was proposed through the Kolmogorov–Smirnov test statistic that is specially tailored to the ultrahigh-dimensional survival data. The sure screening property was established under some mild regularity conditions, and its superior performance over existing screening methods is demonstrated by our extensive simulation studies. A real data example of gene expression is used to illustrate the application of the proposed fully nonparametric screening procedure.

Adjusted Pearson Chi-Square feature screening for multi-classification with ultrahigh dimensional data

Huang et al. (J Bus Econ Stat 32:237–244, 2014) first proposed a Pearson Chi-Square based feature screening procedure tailored to multi-classification problem with ultrahigh dimensional categorical covariates, which is a common problem in practice but has seldom been discussed in the literature. However, their work establishes the sure screening property only in a limited setting. Moreover, the p value based adjustments when the number of categories involved by each covariate is different do not work well in several practical situations. In this paper, we propose an adjusted Pearson Chi-Square feature screening procedure and a modified method for tuning parameter selection. Theoretically, we establish the sure screening property of the proposed method in general settings. Empirically, the proposed method is more successful than Pearson Chi-Square feature screening in handling non-equal numbers of covariate categories in finite samples. Results of three simulation studies and one real data analysis are presented. Our work together with Huang et al. (J Bus Econ Stat 32:237–244, 2014) establishes a solid theoretical foundation and empirical evidence for the family of Pearson Chi-Square based feature screening methods.

A Robust Model-Free Feature Screening Method for Ultrahigh-Dimensional Data

ABSTRACTFeature screening plays an important role in dimension reduction for ultrahigh-dimensional data. In this article, we introduce a new feature screening method and establish its sure independence screening property under the ultrahigh-dimensional setting. The proposed method works based on the nonparanormal transformation and Henze–Zirkler’s test, that is, it first transforms the response variable and features to Gaussian random variables using the nonparanormal transformation and then tests the dependence between the response variable and features using the Henze–Zirkler’s test. The proposed method enjoys at least two merits. First, it is model-free, which avoids the specification of a particular model structure. Second, it is condition-free, which does not require any extra conditions except for some regularity conditions for high-dimensional feature screening. The numerical results indicate that, compared to the existing methods, the proposed method is more robust to the data generated from heavy-tailed distributions and/or complex models with interaction variables. The proposed method is applied to screening of anticancer drug response genes. Supplementary material for this article is available online.

Sure explained variability and independence screening

ABSTRACTIn the era of Big Data, extracting the most important exploratory variables available in ultrahigh-dimensional data plays a key role in scientific researches. Existing researches have been mainly focusing on applying the extracted exploratory variables to describe the central tendency of their related response variables. For a response variable, its variability characteristic is as much important as the central tendency in statistical inference. This paper focuses on the variability and proposes a new model-free feature screening approach: sure explained variability and independence screening (SEVIS). The core of SEVIS is to take the advantage of recently proposed asymmetric and nonlinear generalised measures of correlation in the screening. Under some mild conditions, the paper shows that SEVIS not only possesses desired sure screening property and ranking consistency property, but also is a computational convenient variable selection method to deal with ultrahigh-dimensional data sets with more features than observations. The superior performance of SEVIS, compared with existing model-free methods, is illustrated in extensive simulations. A real example in ultrahigh-dimensional variable selection demonstrates that the variables selected by SEVIS better explain not only the response variables, but also the variables selected by other methods.

Large-Scale Online Feature Selection for Ultra-High Dimensional Sparse Data

Feature selection (FS) is an important technique in machine learning and data mining, especially for large-scale high-dimensional data. Most existing studies have been restricted to batch learning, which is often inefficient and poorly scalable when handling big data in real world. As real data may arrive sequentially and continuously, batch learning has to retrain the model for the new coming data, which is very computationally intensive. Online feature selection (OFS) is a promising new paradigm that is more efficient and scalable than batch learning algorithms. However, existing online algorithms usually fall short in their inferior efficacy. In this article, we present a novel second-order OFS algorithm that is simple yet effective, very fast and extremely scalable to deal with large-scale ultra-high dimensional sparse data streams. The basic idea is to exploit the second-order information to choose the subset of important features with high confidence weights. Unlike existing OFS methods that often suffer from extra high computational cost, we devise a novel algorithm with a MaxHeap-based approach, which is not only more effective than the existing first-order algorithms, but also significantly more efficient and scalable. Our extensive experiments validated that the proposed technique achieves highly competitive accuracy as compared with state-of-the-art batch FS methods, meanwhile it consumes significantly less computational cost that is orders of magnitude lower. Impressively, on a billion-scale synthetic dataset (1-billion dimensions, 1-billion non-zero features, and 1-million samples), the proposed algorithm takes less than 3 minutes to run on a single PC.

A model-free feature screening approach based on kernel density estimation

ABSTRACTIn this article, a new model-free feature screening method named after probability density (mass) function distance (PDFD) correlation is presented for ultrahigh-dimensional data analysis. We improve the fused-Kolmogorov filter (F-KOL) screening procedure through probability density distribution. The proposed method is also fully nonparametric and can be applied to more general types of predictors and responses, including discrete and continuous random variables. Kernel density estimate method and numerical integration are applied to obtain the estimator we proposed. The results of simulation studies indicate that the fused-PDFD performs better than other existing screening methods, such as F-KOL filter, sure-independent screening (SIS), sure independent ranking and screening (SIRS), distance correlation sure-independent screening (DCSIS) and robust ranking correlation screening (RRCS). Finally, we demonstrate the validity of fused-PDFD by a real data example.

Censored cumulative residual independent screening for ultrahigh-dimensional survival data.

For complete ultrahigh-dimensional data, sure independent screening methods can effectively reduce the dimensionality while retaining all the active variables with high probability. However, limited screening methods have been developed for ultrahigh-dimensional survival data subject to censoring. We propose a censored cumulative residual independent screening method that is model-free and enjoys the sure independent screening property. Active variables tend to be ranked above the inactive ones in terms of their association with the survival times. Compared with several existing methods, our model-free screening method works well with general survival models, and it is invariant to the monotone transformation of the responses, as well as requiring substantially weaker moment conditions. Numerical studies demonstrate the usefulness of the censored cumulative residual independent screening method, and the new approach is illustrated with a gene expression data set.

Variable Screening via Quantile Partial Correlation

ABSTRACT In quantile linear regression with ultrahigh-dimensional data, we propose an algorithm for screening all candidate variables and subsequently selecting relevant predictors. Specifically, we first employ quantile partial correlation for screening, and then we apply the extended Bayesian information criterion (EBIC) for best subset selection. Our proposed method can successfully select predictors when the variables are highly correlated, and it can also identify variables that make a contribution to the conditional quantiles but are marginally uncorrelated or weakly correlated with the response. Theoretical results show that the proposed algorithm can yield the sure screening set. By controlling the false selection rate, model selection consistency can be achieved theoretically. In practice, we proposed using EBIC for best subset selection so that the resulting model is screening consistent. Simulation studies demonstrate that the proposed algorithm performs well, and an empirical example is presented. Supplementary materials for this article are available online.

The cumulative Kolmogorov filter for model-free screening in ultrahigh dimensional data

We propose a cumulative Kolmogorov filter to improve the fused Kolmogorov filter proposed by Mai and Zou (2015) via cumulative slicing. We establish an improved asymptotic result under relaxed assumptions and numerically demonstrate its enhanced finite sample performance.