Model-free Variable Selection Research Articles

Gradient Learning With the Mode-Induced Loss: Consistency Analysis and Applications.

Variable selection methods aim to select the key covariates related to the response variable for learning problems with high-dimensional data. Typical methods of variable selection are formulated in terms of sparse mean regression with a parametric hypothesis class, such as linear functions or additive functions. Despite rapid progress, the existing methods depend heavily on the chosen parametric function class and are incapable of handling variable selection for problems where the data noise is heavy-tailed or skewed. To circumvent these drawbacks, we propose sparse gradient learning with the mode-induced loss (SGLML) for robust model-free (MF) variable selection. The theoretical analysis is established for SGLML on the upper bound of excess risk and the consistency of variable selection, which guarantees its ability for gradient estimation from the lens of gradient risk and informative variable identification under mild conditions. Experimental analysis on the simulated and real data demonstrates the competitive performance of our method over the previous gradient learning (GL) methods.

Robust Group Identification and Variable Selection in Sliced Inverse Regression Using Tukey's Biweight Criterion and Ball Covariance

The SSIR-PACS is a group identification and a model-free variable selection method under sufficient dimension reduction (SDR) settings. It combined the Pairwise Absolute Clustering and Sparsity (PACS) with sliced inverse regression (SIR) methods to produce solutions with sparsity and the ability of group identification. However, the SSIR-PACS depends on classical estimates for dispersion and location, squared loss function, and non-robust weights for outliers. In this paper, a robust version of SSIR-PACS (RSSIR-PACS) is proposed. We replaced the squared loss by the criterion of Tukey's biweight. Also, the non-robust weights to outliers, which depend on Pearson’s correlations, are substituted with robust weights based on recently developed ball correlation. Moreover, the estimates of the mean and covariance matrix are substituted by the median and ball covariance, respectively. The RSSIR-PACS is robust to outliers in both the response and covariates. According to the results of simulations, RSSIR-PACS produces very good results. If the outliers are existing, the efficacy of RSSIR-PACS is considerably better than the efficacy of the competitors. In addition, a robust criteria to estimate the structural dimension d is proposed. The RSSIR-PACS makes SSIR-PACS practically feasible. Also, we employed real data to demonstrate the utility of RSSIR-PACS.

Regularized MAVE through the elastic net with correlated predictors

In this article, we proposed a model-free variable selection method (SMAVE-EN). The concepts of sufficient dimension reduction (SDR) and regularization methods are combined to introduce SMAVE-EN. This method is proposed to produce a shrinkage estimation when the predictors are highly correlated under SDR settings. The advantage of SMAVE-EN is that SMAVE-EN extended Elastic net (EN) to nonlinear and multi-dimensional regression under SDR settings. From another side, the SMAVE-EN enables MAVE to work with problems were the predictors are highly correlated. In addition, SMAVE-EN can exhaustively estimate dimensions, while selecting informative covariates simultaneously under SDR framework. The effectiveness of SMAVE-EN is evaluated by both simulation and real data analysis.

Model-Free Variable Selection With Matrix-Valued Predictors

ABSTRACT We introduce a novel framework for model-free variable selection with matrix-valued predictors. To test the importance of rows, columns, and submatrices of the predictor matrix in terms of predicting the response, three types of hypotheses are formulated under a unified framework. The asymptotic properties of the test statistics under the null hypothesis are established and a permutation testing algorithm is also introduced to approximate the distribution of the test statistics. A maximum ratio criterion (MRC) is proposed to facilitate the model-free variable selection. Unlike the traditional stepwise regression procedures that require calculating p-values at each step, the MRC is a noniterative procedure that does not require p-value calculation and is guaranteed to achieve variable selection consistency under mild conditions. Performance of the proposed method is evaluated in extensive simulations and demonstrated through the analysis of an electroencephalography data. Supplementary materials for this article are available online.

Knockoff boosted tree for model-free variable selection.

The recently proposed knockoff filter is a general framework for controlling the false discovery rate (FDR) when performing variable selection. This powerful new approach generates a 'knockoff' of each variable tested for exact FDR control. Imitation variables that mimic the correlation structure found within the original variables serve as negative controls for statistical inference. Current applications of knockoff methods use linear regression models and conduct variable selection only for variables existing in model functions. Here, we extend the use of knockoffs for machine learning with boosted trees, which are successful and widely used in problems where no prior knowledge of model function is required. However, currently available importance scores in tree models are insufficient for variable selection with FDR control. We propose a novel strategy for conducting variable selection without prior model topology knowledge using the knockoff method with boosted tree models. We extend the current knockoff method to model-free variable selection through the use of tree-based models. Additionally, we propose and evaluate two new sampling methods for generating knockoffs, namely the sparse covariance and principal component knockoff methods. We test and compare these methods with the original knockoff method regarding their ability to control type I errors and power. In simulation tests, we compare the properties and performance of importance test statistics of tree models. The results include different combinations of knockoffs and importance test statistics. We consider scenarios that include main-effect, interaction, exponential and second-order models while assuming the true model structures are unknown. We apply our algorithm for tumor purity estimation and tumor classification using Cancer Genome Atlas (TCGA) gene expression data. Our results show improved discrimination between difficult-to-discriminate cancer types. The proposed algorithm is included in the KOBT package, which is available at https://cran.r-project.org/web/packages/KOBT/index.html. Supplementary data are available at Bioinformatics online.

Sparse minimum average variance estimation via the adaptive elastic net when the predictors correlated

A new model-free variable selection method was proposed in this article, which is called SMAVE-AdEN. We combined the effective sufficient dimension reduction method MAVE with the variable selection method which is called adaptive Elastic Net (AdEN) to introduce SMAVE-AdEN. The SMAVE-AdEN produces a sparse and accurate estimate when the predictors are highly correlated. The advantage of SMAVE-AdEN is that SMAVE-AdEN extended Adaptive Elastic net (AdEN) to nonlinear and multi-dimensional regression. Also, the SMAVE-AdEN enables MAVE to work with problems were the predictors are highly correlated. In addition, SMAVE-AdEN can exhaustively estimate dimensions, while selecting informative covariates simultaneously. The performance of SMAVE-AdEN is evaluated by both simulation and real data analysis.

Model-free variable selection for conditional mean in regression

A novel test statistic is proposed to identify important predictors for the conditional mean function in regression. The stepwise regression algorithm based on the proposed test statistic guarantees variable selection consistency without specifying the functional form of the conditional mean. When the predictors are ultrahigh dimensional, a model-free screening procedure is introduced to precede the stepwise regression algorithm. The screening procedure has the sure screening property when the number of predictors grows at an exponential rate of the available sample size. The finite-sample performances of our proposals are demonstrated via numerical studies.

Robust Variable Selection and Estimation Based on Kernel Modal Regression.

Model-free variable selection has attracted increasing interest recently due to its flexibility in algorithmic design and outstanding performance in real-world applications. However, most of the existing statistical methods are formulated under the mean square error (MSE) criterion, and susceptible to non-Gaussian noise and outliers. As the MSE criterion requires the data to satisfy Gaussian noise condition, it potentially hampers the effectiveness of model-free methods in complex circumstances. To circumvent this issue, we present a new model-free variable selection algorithm by integrating kernel modal regression and gradient-based variable identification together. The derived modal regression estimator is related closely to information theoretic learning under the maximum correntropy criterion, and assures algorithmic robustness to complex noise by replacing learning of the conditional mean with the conditional mode. The gradient information of estimator offers a model-free metric to screen the key variables. In theory, we investigate the theoretical foundations of our new model on generalization-bound and variable selection consistency. In applications, the effectiveness of the proposed method is verified by data experiments.

A model-free variable selection method for reducing the number of redundant variables

ABSTRACTUnder the sufficient dimension reduction () framework, we propose a model-free variable selection method for reducing the number of redundant predictors. The method adopts the distance correlation as a dependence measure to quantify the relevance and redundancy of a predictor, and searches for a set of the relevant but non-redundant predictors. Two forward screening algorithms are given to find an approximate solution to the set of the relevant but non-redundant predictors. The screening consistency of the proposed method and algorithms has been fully studied. The effectiveness of the proposed method and algorithms is illustrated by the simulation experiments and two real examples. The experimental results show that the proposed method can effectively exclude the redundant predictors and yield a more parsimonious subset of the relevant predictors.

On dual model-free variable selection with two groups of variables

In the presence of two groups of variables, existing model-free variable selection methods only reduce the dimensionality of the predictors. We extend the popular marginal coordinate hypotheses Cook (2004) in the sufficient dimension reduction literature and consider the dual marginal coordinate hypotheses, where the role of the predictor and the response is not important. Motivated by canonical correlation analysis (CCA), we propose a CCA-based test for the dual marginal coordinate hypotheses, and devise a joint backward selection algorithm for dual model-free variable selection. The performances of the proposed test and the variable selection procedure are evaluated through synthetic examples and a real data analysis.

Trace pursuit variable selection for multi-population data

ABSTRACTVariable selection is a very important tool when dealing with high dimensional data. However, most popular variable selection methods are model based, which might provide misleading results when the model assumption is not satisfied. Sufficient dimension reduction provides a general framework for model-free variable selection methods. In this paper, we propose a model-free variable selection method via sufficient dimension reduction, which incorporates the grouping information into the selection procedure for multi-population data. Theoretical properties of our selection methods are also discussed. Simulation studies suggest that our method greatly outperforms those ignoring the grouping information.

Gradient-induced Model-free Variable Selection with Composite Quantile Regression

Gradient-induced Model-free Variable Selection with Composite Quantile Regression

On marginal sliced inverse regression for ultrahigh dimensional model-free feature selection

Model-free variable selection has been implemented under the sufficient dimension reduction framework since the seminal paper of Cook [Ann. Statist. 32 (2004) 1062–1092]. In this paper, we extend the marginal coordinate test for sliced inverse regression (SIR) in Cook (2004) and propose a novel marginal SIR utility for the purpose of ultrahigh dimensional feature selection. Two distinct procedures, Dantzig selector and sparse precision matrix estimation, are incorporated to get two versions of sample level marginal SIR utilities. Both procedures lead to model-free variable selection consistency with predictor dimensionality $p$ diverging at an exponential rate of the sample size $n$. As a special case of marginal SIR, we ignore the correlation among the predictors and propose marginal independence SIR. Marginal independence SIR is closely related to many existing independence screening procedures in the literature, and achieves model-free screening consistency in the ultrahigh dimensional setting. The finite sample performances of the proposed procedures are studied through synthetic examples and an application to the small round blue cell tumors data.

Trace Pursuit: A General Framework for Model-Free Variable Selection

ABSTRACTWe propose trace pursuit for model-free variable selection under the sufficient dimension-reduction paradigm. Two distinct algorithms are proposed: stepwise trace pursuit and forward trace pursuit. Stepwise trace pursuit achieves selection consistency with fixed p. Forward trace pursuit can serve as an initial screening step to speed up the computation in the case of ultrahigh dimensionality. The screening consistency property of forward trace pursuit based on sliced inverse regression is established. Finite sample performances of trace pursuit and other model-free variable selection methods are compared through numerical studies. Supplementary materials for this article are available online.

Dimension reduction based linear surrogate variable approach for model free variable selection

Most of variable selection methods depend on the model assumptions, while sufficient dimension reduction is a nonparametric method to deal with high dimensional data. In this paper we aim at integrating sufficient dimension reduction into variable selection. A two stage procedure is proposed. First, we obtain dimension reduction directions and integrate them to construct a variable which is linearly dependent on predictors. Then by treating this constructed variable as a new response, we use the traditional variable selection methods such as adaptive LASSO to conduct variable selection. We call such a procedure as dimension reduction based linear surrogate variable (LSV) method. To illustrate that it has wide application, we also apply it to variable selection for the problem of missing responses. Extensive simulation studies show that it is more robust than the variable selection methods depending on model assumptions, and more efficient than the other model-free variable selection methods. Another advantage of the LSV is that it can be easily implemented. A real example is given to illustrate the proposed method.

Robust variable selection through MAVE

Dimension reduction and variable selection play important roles in high dimensional data analysis. The sparse MAVE, a model-free variable selection method, is a nice combination of shrinkage estimation, Lasso, and an effective dimension reduction method, MAVE (minimum average variance estimation). However, it is not robust to outliers in the dependent variable because of the use of least-squares criterion. A robust variable selection method based on sparse MAVE is developed, together with an efficient estimation algorithm to enhance its practical applicability. In addition, a robust cross-validation is also proposed to select the structural dimension. The effectiveness of the new approach is verified through simulation studies and a real data analysis.

Multiple Loci Mapping via Model-free Variable Selection

Multiple Loci Mapping via Model-free Variable Selection

Multiple loci mapping via model-free variable selection.

Despite recent flourish of proposals on variable selection, genome-wide multiple loci mapping remains to be challenging. The majority of existing variable selection methods impose a model, and often the homoscedastic linear model, prior to selection. However, the true association between the phenotypical trait and the genetic markers is rarely known a priori, and the presence of epistatic interactions makes the association more complex than a linear relation. Model-free variable selection offers a useful alternative in this context, but the fact that the number of markers p often far exceeds the number of experimental units n renders all the existing model-free solutions that require n > p inapplicable. In this article, we examine a number of model-free variable selection methods for small-n-large-p regressions in the context of genome-wide multiple loci mapping. We propose and advocate a multivariate group-wise adaptive penalization solution, which requires no model prespecification and thus works for complex trait-marker association, and handles one variable at a time so that works for n < p. Effectiveness of the new method is demonstrated through both intensive simulations and a comprehensive real data analysis across 6100 gene expression traits.

ASYMPTOTIC PROPERTIES OF SUFFICIENT DIMENSION REDUCTION WITH A DIVERGING NUMBER OF PREDICTORS.

We investigate asymptotic properties of a family of sufficient dimension reduction estimators when the number of predictors p diverges to infinity with the sample size. We adopt a general formulation of dimension reduction estimation through least squares regression of a set of transformations of the response. This formulation allows us to establish the consistency of reduction projection estimation. We then introduce the SCAD max penalty, along with a difference convex optimization algorithm, to achieve variable selection. We show that the penalized estimator selects all truly relevant predictors and excludes all irrelevant ones with probability approaching one, meanwhile it maintains consistent reduction basis estimation for relevant predictors. Our work differs from most model-based selection methods in that it does not require a traditional model, and it extends existing sufficient dimension reduction and model-free variable selection approaches from the fixed p scenario to a diverging p.

Bayesian ensemble methods for survival prediction in gene expression data.

We propose a Bayesian ensemble method for survival prediction in high-dimensional gene expression data. We specify a fully Bayesian hierarchical approach based on an ensemble 'sum-of-trees' model and illustrate our method using three popular survival models. Our non-parametric method incorporates both additive and interaction effects between genes, which results in high predictive accuracy compared with other methods. In addition, our method provides model-free variable selection of important prognostic markers based on controlling the false discovery rates; thus providing a unified procedure to select relevant genes and predict survivor functions. We assess the performance of our method several simulated and real microarray datasets. We show that our method selects genes potentially related to the development of the disease as well as yields predictive performance that is very competitive to many other existing methods. http://works.bepress.com/veera/1/.