Variable Selection In Linear Models Research Articles

An ensemble learning method for variable selection: application to high-dimensional data and missing values

Standard approaches for variable selection in linear models are not tailored to deal properly with high-dimensional and incomplete data. Currently, methods dedicated to high-dimensional data handle missing values by ad hoc strategies, like complete case analysis or single imputation, while methods dedicated to missing values, mainly based on multiple imputation, do not discuss the imputation method to use with high-dimensional data. Consequently, both approaches appear to be limited for many modern applications. With inspiration from ensemble methods, a new variable selection method is proposed. It extends classical variable selection methods in the case of high-dimensional data with or without missing data. Theoretical properties are studied and the practical interest is demonstrated through a simulation study, as well as through an application to models specification in sequential multiple imputation. In the low-dimensional case, the procedure improves the control of the error risks, especially type I error, even without missing values for stepwise, lasso or knockoff methods. With missing values, the method performs better than reference selection methods based on multiple imputation. Similar performances are obtained in the high-dimensional case with or without missing values.

Penalized joint generalized estimating equations for longitudinal binary data.

In statistical research, variable selection and feature extraction are a typical issue. Variable selection in linear models has been fully developed, while it has received relatively little attention for longitudinal data. Since a longitudinal study involves within-subject correlations, the likelihood function of discrete longitudinal responses generally cannot be expressed in analytically closed form, and standard variable selection methods cannot be directly applied. As an alternative, the penalized generalized estimating equation (PGEE) is helpful but very likely results in incorrect variable selection if the working correlation matrix is misspecified. In many circumstances, the within-subject correlations are of interest and need to be modeled together with the mean. For longitudinal binary data, it becomes more challenging because the within-subject correlation coefficients have the so-called Fréchet-Hoeffding upper bound. In this paper, we proposed smoothly clipped absolute deviation (SCAD)-based and least absolute shrinkage and selection operator (LASSO)-based penalized joint generalized estimating equation (PJGEE) methods to simultaneously model the mean and correlations for longitudinal binary data, together with variable selection in the mean model. The estimated correlation coefficients satisfy the upper bound constraints. Simulation studies under different scenarios are made to assess the performance of the proposed method. Compared to existing PGEE methods that specify a working correlation matrix for longitudinal binary data, the proposed PJGEE method works much better in terms of variable selection consistency and parameter estimation accuracy. A real data set on Clinical Global Impression is analyzed forillustration.

Variable selection in high-dimensional linear model with possibly asymmetric errors

In many application areas, the problem of the automatic variable selection in a linear model with asymmetric errors is encountered, when the number of explanatory variables diverges with the sample size. For this high-dimensional model, the penalized least squares method is not appropriate and the quantile framework makes the inference more difficult because of the non differentiability of the loss function. An estimation method by penalizing the expectile process with an adaptive LASSO penalty is proposed and studied. Two cases are considered: first with the number of model parameters is assumed to be much smaller than the sample size and afterwards it could be of the same order; the two cases being distinct by the adaptive penalties considered. For each case, the rate convergence is obtained and the oracle properties of the adaptive LASSO expectile estimator are established. The proposed estimators are evaluated through Monte Carlo simulations and compared with the adaptive LASSO quantile estimator. The proposed estimation method is also applied to real data in genetics.

Which bridge estimator is the best for variable selection?

We study the problem of variable selection for linear models under the high-dimensional asymptotic setting, where the number of observations $n$ grows at the same rate as the number of predictors $p$. We consider two-stage variable selection techniques (TVS) in which the first stage uses bridge estimators to obtain an estimate of the regression coefficients, and the second stage simply thresholds this estimate to select the “important” predictors. The asymptotic false discovery proportion ($\operatorname{AFDP}$) and true positive proportion (ATPP) of these TVS are evaluated. We prove that for a fixed ATPP, in order to obtain a smaller $\operatorname{AFDP}$, one should pick a bridge estimator with smaller asymptotic mean square error in the first stage of TVS. Based on such principled discovery, we present a sharp comparison of different TVS, via an in-depth investigation of the estimation properties of bridge estimators. Rather than “orderwise” error bounds with loose constants, our analysis focuses on precise error characterization. Various interesting signal-to-noise ratio and sparsity settings are studied. Our results offer new and thorough insights into high-dimensional variable selection. For instance, we prove that a TVS with Ridge in its first stage outperforms TVS with other bridge estimators in large noise settings; two-stage LASSO becomes inferior when the signal is rare and weak. As a by-product, we show that two-stage methods outperform some standard variable selection techniques, such as $\operatorname{LASSO}$ and Sure Independence Screening, under certain conditions.

Stochastic correlation coefficient ensembles for variable selection

ABSTRACTIn this paper, we propose a novel Max-Relevance and Min-Common-Redundancy criterion for variable selection in linear models. Considering that the ensemble approach for variable selection has been proven to be quite effective in linear regression models, we construct a variable selection ensemble (VSE) by combining the presented stochastic correlation coefficient algorithm with a stochastic stepwise algorithm. We conduct extensive experimental comparison of our algorithm and other methods using two simulation studies and four real-life data sets. The results confirm that the proposed VSE leads to promising improvement on variable selection and regression accuracy.

Rank-based group variable selection

A robust rank-based estimator for variable selection in linear models, with grouped predictors, is studied. The proposed estimation procedure extends the existing rank-based variable selection [Johnson, B.A., and Peng, L. (2008), ‘Rank-based Variable Selection’, Journal of Nonparametric Statistics, 20(3):241–252] and the ww-scad [Wang, L., and Li, R. (2009), ‘Weighted Wilcoxon-type Smoothly Clipped Absolute Deviation Method’, Biometrics, 65(2):564–571] to linear regression models with grouped variables. The resulting estimator is robust to contamination or deviations in both the response and the design space.The Oracle property and asymptotic normality of the estimator are established under some regularity conditions. Simulation studies reveal that the proposed method performs better than the existing rank-based methods [Johnson, B.A., and Peng, L. (2008), ‘Rank-based Variable Selection’, Journal of Nonparametric Statistics, 20(3):241–252; Wang, L., and Li, R. (2009), ‘Weighted Wilcoxon-type Smoothly Clipped Absolute Deviation Method’, Biometrics, 65(2):564–571] for grouped variables models. This estimation procedure also outperforms the adaptive hlasso [Zhou, N., and Zhu, J. (2010), ‘Group Variable Selection Via a Hierarchical Lasso and its Oracle Property’, Interface, 3(4):557–574] in the presence of local contamination in the design space or for heavy-tailed error distribution.

Power-Expected-Posterior Priors for Variable Selection in Gaussian Linear Models

In the context of the expected-posterior prior (EPP) approach to Bayesian variable selection in linear models, we combine ideas from power-prior and unit-information-prior methodologies to simultaneously (a) produce a minimally-informative prior and (b) diminish the effect of training samples. The result is that in practice our power-expected-posterior (PEP) methodology is sufficiently insensitive to the size n* of the training sample, due to PEP’s unit-information construction, that one may take n* equal to the full-data sample size n and dispense with training samples altogether. This promotes stability of the resulting Bayes factors, removes the arbitrariness arising from individual training-sample selections, and greatly increases computational speed, allowing many more models to be compared within a fixed CPU budget. We find that, under an independence Jeffreys (reference) baseline prior, the asymptotics of PEP Bayes factors are equivalent to those of Schwartz’s Bayesian Information Criterion (BIC), ensuring consistency of the PEP approach to model selection. Our PEP prior, due to its unit-information structure, leads to a variable-selection procedure that — in our empirical studies — (1) is systematically more parsimonious than the basic EPP with minimal training sample, while sacrificing no desirable performance characteristics to achieve this parsimony; (2) is robust to the size of the training sample, thus enjoying the advantages described above arising from the avoidance of training samples altogether; and (3) identifies maximum-a-posteriori models that achieve better out-of-sample predictive performance than that provided by standard EPPs, the g-prior, the hyper-g prior, non-local priors, the Least Absolute Shrinkage and Selection Operator (LASSO) and Smoothly-Clipped Absolute Deviation (SCAD) methods.

Robust Bayesian variable selection in linear models with spherically symmetric errors

This paper studies Bayesian variable selection in linear models with general spherically symmetric error distributions. We construct the posterior odds based on a separable prior, which arises as a class of mixtures of Gaussian densities. The posterior odds for comparing among nonnull models are shown to be independent of the error distribution, if this is spherically symmetric. Because of this invariance, we refer to our method as a robust Bayesian variable selection method. We demonstrate that our posterior odds have model selection consistency, and that our class of prior functions are the only ones within a large class which are robust in our sense.

Variable selection in linear models

Variable selection in linear models is essential for improved inference and interpretation, an activity which has become even more critical for high dimensional data. In this article, we provide a selective review of some classical methods including Akaike information criterion, Bayesian information criterion, Mallow's Cp and risk inflation criterion, as well as regularization methods including Lasso, bridge regression, smoothly clipped absolute deviation, minimax concave penalty, adaptive Lasso, elastic‐net, and group Lasso. We discuss how to select the penalty parameters. We also provide a review for some screening procedures for ultra high dimensions. WIREs Comput Stat 2014, 6:1–9. doi: 10.1002/wics.1284This article is categorized under: Statistical Models > Linear Models Statistical Learning and Exploratory Methods of the Data Sciences > Modeling Methods Statistical Models > Model Selection

LAD variable selection for linear models with randomly censored data

The least absolute deviations (LAD) variable selection for linear models with randomly censored data is studied through the Lasso. The proposed procedure can select significant variables in the parameters. With appropriate selection of the tuning parameters, we establish the consistency of this procedure and the oracle property of the resulting estimators. Simulation studies are conducted to compare the proposed procedure with an inverse-censoring-probability weighted LAD LASSO-estimator.

Applications of L1 regularisation

The lasso algorithm for variable selection in linear models, introduced by Tibshirani, works by imposing an $l_1$~norm bound constraint on the variables in a least squares model and then tuning the model estimation calculation using this bound. This introduction of the bound is interpreted as a form of regularisation step. It leads to a form of quadratic program which is solved by a straight-forward modification of a standard active set algorithm for each value of this bound. Considerable interest was generated by the discovery that the complete solution trajectory parametrised by this bound is piecewise linear and can be calculated very efficiently. Essentially it takes no more work than the solution of either the unconstrained least squares problem or the quadratic program at a single bound value. This has resulted in the study both of the selection problem for different objective and constraint choices and of applications to such areas as data compression and the generation of sparse solutions of very under-determined systems. One important class of generalisation is to quantile regression estimation problems. The original continuation idea extends to these polyhedral objectives in an interesting two phase procedure which involves both the constrained and Lagrangian forms of the problem at each step. However, it is significantly less computationally effective than is the original algorithm for least squares objectives. In contrast, the piecewise linear estimation problem can be solved for each value of the $l_1$~bound by a relatively efficient simplicial descent algorithm, and that this can be used to explore trajectory information in a manner which is at least competitive with the homotopy algorithm in this context. The form of line search used in the descent steps has an important bearing on the effectiveness of the algorithm. A comparison is given between the relative performance of the simplicial descent algorithm used and an interior point method on the piecewise linear estimation problem. References I. Barrodale and F. D. K. Roberts. An improved algorithm for $l_1$ linear approximation. SIAM J. Numer. Anal., 10:839--848, 1973. P. Bloomfield and W. L. Steiger. Least Absolute Deviations. Birkhauser, Boston, 1983. H. D. Bondell and B. J. Reich. Simultaneous regression shrinkage, variable selection, and supervising clustering of predictors with {OSCAR}. Biometrics, 64:115--123, 2008. doi:10.1111/j.1541-0420.2007.00843.x E. Candes and T. Tao. The Dantzig selector: statistical estimation when $p$ is much larger than $n$. Ann. Statist., 35(6):2313--2351, 2007. doi:10.1214/009053606000001523 D. L. Donoho and Y. Tsaig. Fast solution of $l_1$-norm minimization problems when the solution may be sparse. IEEE Trans. Inf. Theory, 54:4789--4812, 2008. doi:10.1109/TIT.2008.929958 R. Koenker. Quantile Regression. Cambridge University Press, 2005. Y. Li and J. Zhu. {$L_1$}-norm quantile regression. J. Comp. Graph. Stat., 17(1):163--185, 2008. doi:10.1198/106186008X289155 M. R. Osborne. Simplicial algorithms for minimizing polyhedral functions. Cambridge University Press, 2001. M. R. Osborne, Brett Presnell, and B. A. Turlach. On the Lasso and its dual. J. Comp. Graph. Stat., 9(2):319--337, 2000. http://www.jstor.org/stable/1390657 M. R. Osborne and B. A. Turlach. A homotopy algorithm for the quantile regression lasso and related piecewise linear problems. J. Comp. Graph. Stat., 2010. Accepted for publication. doi:10.1198/jcgs.2011.09184 S. Portnoy and R. Koenker. The Gaussian hare and the Laplacian tortoise: computability of squared error vs absolute error estimates. Stat. Sci., 12:279--300, 1997. http://www.jstor.org/stable/2246216 S. Rosset and Ji Zhu. Piecewise linear regularised solution paths. Ann. Stat., 35(3):1012--1030, 2007. doi:10.1214/009053606000001370 R. Tibshirani. Regression shrinkage and selection via the lasso. J. R. Stat. Soc., Ser. B, 58(1):267--288, 1996. http://www.jstor.org/stable/2346178 B. A. Turlach, W. N. Venables, and S. J. Wright. Simultaneous variable selection. Technometrics, 47(3):349--363, 2005. doi:10.1198/004017005000000139 Y. Yao and Y. Lee. Another look at linear programming for feature selection via methods of regularization. Technical Report 800, Department of Statistics, Ohio State University, 2007. M. Yuan and H. Zou. Efficient global approximation of generalised nonlinear {$\ell _1$} regularised solution paths and its applications. J. Am. Stat. Assoc., 104:1562--1573, 2009. doi:10.1198/jasa.2009.tm08287 J. Zhu, T. Hastie, S. Rosset, and R. Tibshirani. $l_1$-norm support vector machines. Adv. Neural Inf. Process. Syst., 16:49--56, 2004. http://books.nips.cc/papers/files/nips16/NIPS2003_AA07.pdf H. Zou and T. Hastie. Regularization and variable selection via the elastic net. J. R. Stat. Soc., Ser. B, 67:301--320, 2005. http://www.jstor.org/stable/3647580 H. Zou and M. Yuan. Regularised simultaneous model selection in multiple quantiles regression. Comp. Stat. Data Anal., 52:5296--5304, 2008. doi:10.1016/j.csda.2008.05.013

Random lasso

We propose a computationally intensive method, the random lasso method, for variable selection in linear models. The method consists of two major steps. In step 1, the lasso method is applied to many bootstrap samples, each using a set of randomly selected covariates. A measure of importance is yielded from this step for each covariate. In step 2, a similar procedure to the first step is implemented with the exception that for each bootstrap sample, a subset of covariates is randomly selected with unequal selection probabilities determined by the covariates' importance. Adaptive lasso may be used in the second step with weights determined by the importance measures. The final set of covariates and their coefficients are determined by averaging bootstrap results obtained from step 2. The proposed method alleviates some of the limitations of lasso, elastic-net and related methods noted especially in the context of microarray data analysis: it tends to remove highly correlated variables altogether or select them all, and maintains maximal flexibility in estimating their coefficients, particularly with different signs; the number of selected variables is no longer limited by the sample size; and the resulting prediction accuracy is competitive or superior compared to the alternatives. We illustrate the proposed method by extensive simulation studies. The proposed method is also applied to a Glioblastoma microarray data analysis.

Compatibility of Prior Specifications Across Linear Models

Bayesian model comparison requires the specification of a prior distribution on the parameter space of each candidate model. In this connection two concerns arise: on the one hand the elicitation task rapidly becomes prohibitive as the number of models increases; on the other hand numerous prior specifications can only exacerbate the well-known sensitivity to prior assignments, thus producing less dependable conclusions. Within the subjective framework, both difficulties can be counteracted by linking priors across models in order to achieve simplification and compatibility; we discuss links with related objective approaches. Given an encompassing, or full, model together with a prior on its parameter space, we review and summarize a few procedures for deriving priors under a submodel, namely marginalization, conditioning, and Kullback–Leibler projection. These techniques are illustrated and discussed with reference to variable selection in linear models adopting a conventional g-prior; comparisons with existing standard approaches are provided. Finally, the relative merits of each procedure are evaluated through simulated and real data sets.

On Measuring and Correcting the Effects of Data Mining and Model Selection

In the theory of linear models, the concept of degrees of freedom plays an important role. This concept is often used for measurement of model complexity, for obtaining an unbiased estimate of the error variance, and for comparison of different models. I have developed a concept of generalized degrees of freedom (GDF) that is applicable to complex modeling procedures. The definition is based on the sum of the sensitivity of each fitted value to perturbation in the corresponding observed value. The concept is nonasymptotic in nature and does not require analytic knowledge of the modeling procedures. The concept of GDF offers a unified framework under which complex and highly irregular modeling procedures can be analyzed in the same way as classical linear models. By using this framework, many difficult problems can be solved easily. For example, one can now measure the number of observations used in a variable selection process. Different modeling procedures, such as a tree-based regression and a projection pursuit regression, can be compared on the basis of their residual sums of squares and the GDF that they cost. I apply the proposed framework to measure the effect of variable selection in linear models, leading to corrections of selection bias in various goodness-of-fit statistics. The theory also has interesting implications for the effect of general model searching by a human modeler.

On Measuring and Correcting the Effects of Data Mining and Model Selection

Abstract In the theory of linear models, the concept of degrees of freedom plays an important role. This concept is often used for measurement of model complexity, for obtaining an unbiased estimate of the error variance, and for comparison of different models. I have developed a concept of generalized degrees of freedom (GDF) that is applicable to complex modeling procedures. The definition is based on the sum of the sensitivity of each fitted value to perturbation in the corresponding observed value. The concept is nonasymptotic in nature and does not require analytic knowledge of the modeling procedures. The concept of GDF offers a unified framework under which complex and highly irregular modeling procedures can be analyzed in the same way as classical linear models. By using this framework, many difficult problems can be solved easily. For example, one can now measure the number of observations used in a variable selection process. Different modeling procedures, such as a tree-based regression and a projection pursuit regression, can be compared on the basis of their residual sums of squares and the GDF that they cost. I apply the proposed framework to measure the effect of variable selection in linear models, leading to corrections of selection bias in various goodness-of-fit statistics. The theory also has interesting implications for the effect of general model searching by a human modeler.

Consistent Variable Selection in Linear Models

Abstract A method of estimating linear model dimension and variable selection is proposed. This new criterion, which generalizes the Cp criterion, the Akaike information criterion (AIC), the Bayes information criterion, and the phiv criterion and is consistent under certain conditions, is based on a new class of penalty functions and a procedure of sorting covariates based on t-statistics. In the course of introducing this method, we discuss the important role of the penalty function in the consistency of model dimension estimation and in variable selection. The proposed method requires less computation than resampling-based methods that search over all subsets of covariates for the true model. Simulation results show that the new method is superior to the Cp criterion and AIC in finite-sample situations as well.

Consistent Variable Selection in Linear Models

Abstract A method of estimating linear model dimension and variable selection is proposed. This new criterion, which generalizes the Cp criterion, the Akaike information criterion (AIC), the Bayes information criterion, and the phiv criterion and is consistent under certain conditions, is based on a new class of penalty functions and a procedure of sorting covariates based on t-statistics. In the course of introducing this method, we discuss the important role of the penalty function in the consistency of model dimension estimation and in variable selection. The proposed method requires less computation than resampling-based methods that search over all subsets of covariates for the true model. Simulation results show that the new method is superior to the Cp criterion and AIC in finite-sample situations as well.