Variable Selection Problem Research Articles

Variable selection, monotone likelihood ratio and group sparsity

In the pivotal variable selection problem, we derive the exact nonasymptotic minimax selector over the class of all s-sparse vectors, which is also the Bayes selector with respect to the uniform prior. While this optimal selector is, in general, not realizable in polynomial time, we show that its tractable counterpart (the scan selector) attains the minimax expected Hamming risk to within factor 2, and is also exact minimax with respect to the probability of wrong recovery. As a consequence, we establish explicit lower bounds under the monotone likelihood ratio property and we obtain a tight characterization of the minimax risk in terms of the best separable selector risk. We apply these general results to derive necessary and sufficient conditions of exact and almost full recovery in the location model with light tail distributions and in the problem of group variable selection under Gaussian noise and under more general anisotropic sub-Gaussian noise. Numerical results illustrate our theoretical findings.

Adaptive signal recovery with Subbotin noise

In this article, we study the problem of high-dimensional variable selection. Specifically, we consider a single vector X from a sparse sequence model with Subbotin noise and derive the regions of variable selection with respect to a Hamming loss function.

Variable selection in multivariate linear regression with random predictors

We propose a method for variable selection in multivariate regression with random predictors. This method is based on a criterion that permits to reduce the variable selection problem to a problem of estimating a suitable set. Then, an estimator for this set is proposed and the resulting method for selecting variables is shown to be consistent. A simulation study that permits to study several properties of the proposed approach and to compare it with existing methods is given.

Variable Selection for Nonlinear Cox Regression Model via Deep Learning

Variable selection problem for the nonlinear Cox regression model is considered. In survival analysis, one main objective is to identify the covariates that are associated with the risk of experiencing the event of interest. The Cox proportional hazard model is being used extensively in survival analysis in studying the relationship between survival times and covariates, where the model assumes that the covariate has a log-linear effect on the hazard function. However, this linearity assumption may not be satisfied in practice. In order to extract a representative subset of features, various variable selection approaches have been proposed for survival data under the linear Cox model. However, there exists little literature on variable selection for the nonlinear Cox model. To break this gap, we extend the recently developed deep learning-based variable selection model LassoNet to survival data. Simulations are provided to demonstrate the validity and effectiveness of the proposed method. Finally, we apply the proposed methodology to analyze a real data set on diffuse large B-cell lymphoma.

Improving Prediction Accuracy of Lasso and Ridge Regression as an Alternative to LS Regression to Identify Variable Selection Problems

This paper introduces the Lasso and Ridge Regression methods, which are two popular regularization approaches. The method they give a penalty to the coefficients differs in both of them. L1 Regularization refers to Lasso linear regression, while L2 Regularization refers to Ridge regression. As we all know, regression models serve two main purposes: explanation and prediction of scientific phenomena. Where prediction accuracy will be optimized by balancing each of the bias and variance of predictions, while explanation will be gained by constructing interpretable regression models by variable selection. The penalized regression method, also known as Lasso regression, adds bias to the model's estimates and reduces variance to enhance prediction. Ridge regression, on the other hand, introduces a minor amount of bias in the data to get long-term predictions. In the presence of multicollinearity, both regression methods have been offered as an alternative to the least square approach (LS). Because they deal with multicollinearity, they have the appropriate properties to reduce numerical instability caused by overfitting. As a result, prediction accuracy can be improved. For this study, the Corona virus disease (Covid-19) dataset was used, which has had a significant impact on global life. Particularly in our region (Kurdistan), where life has altered dramatically and many people have succumbed to this deadly sickness. Our data is utilized to analyze the benefits of each of the two regression methods. The results show that the Lasso approach produces more accurate and dependable or reliable results in the presence of multicollinearity than Ridge and LS methods when compared in terms of accuracy of predictions by using NCSS10, EViews 12 and SPSS 25.

Regularized Buckley-James method for right-censored outcomes with block-missing multimodal covariates.

High-dimensional data with censored outcomes of interest are prevalent in medical research. To analyze such data, the regularized Buckley-James estimator has been successfully applied to build accurate predictive models and conduct variable selection. In this paper, we consider the problem of parameter estimation and variable selection for the semiparametric accelerated failure time model for high-dimensional block-missing multimodal neuroimaging data with censored outcomes. We propose a penalized Buckley-James method that can simultaneously handle block-wise missing covariates and censored outcomes. This method can also perform variable selection. The proposed method is evaluated by simulations and applied to a multimodal neuroimaging dataset and obtains meaningful results.

Gibbs Priors for Bayesian Nonparametric Variable Selection with Weak Learners

We consider the problem of high-dimensional Bayesian nonparametric variable selection using an aggregation of so-called “weak learners.” The most popular variant of this is the Bayesian additive regression trees (BART) model, which is the natural Bayesian analog to boosting decision trees. In this article, we use Gibbs distributions on random partitions to induce sparsity in ensembles of weak learners. Looking at BART as a special case, we show that the class of Gibbs priors includes two recently proposed models—the Dirichlet additive regression trees (DART) model and the spike-and-forest model—as extremal cases, and we show that certain Gibbs priors are capable of achieving the benefits of both the DART and spike-and-forest models while avoiding some of their key drawbacks. We then show the promising performance of Gibbs priors for other classes of weak learners, such as tensor products of spline basis functions. A Pólya Urn scheme is developed for efficient computations. Supplementary materials for this article are available online.

Multi-threshold proportional hazards model and subgroup identification.

We propose a novel two-stage procedure for change point detection and parameter estimation in a multi-threshold proportional hazards model. In the first stage, we estimate the number of thresholds by formulating the threshold detection problem as a variable selection problem and applying the penalized partial likelihood approach. In the second stage, the change point locations are refined by a grid search and the standard inference for segment regression can then follow. The proposed model and estimation procedure could lend support to subgroup identification and personalized treatment recommendation in medical research. We establish the consistency of the threshold estimators and regression coefficient estimators under technical conditions. The finite sample performance of the method is demonstrated via simulation studies and two cancer data examples.

Adaptive random neighbourhood informed Markov chain Monte Carlo for high-dimensional Bayesian variable selection

We introduce a framework for efficient Markov chain Monte Carlo algorithms targeting discrete-valued high-dimensional distributions, such as posterior distributions in Bayesian variable selection problems. We show that many recently introduced algorithms, such as the locally informed sampler of Zanella (J Am Stat Assoc 115(530):852–865, 2020), the locally informed with thresholded proposal of Zhou et al. (Dimension-free mixing for high-dimensional Bayesian variable selection, 2021) and the adaptively scaled individual adaptation sampler of Griffin et al. (Biometrika 108(1):53–69, 2021), can be viewed as particular cases within the framework. We then describe a novel algorithm, the adaptive random neighbourhood informed sampler, which combines ideas from these existing approaches. We show using several examples of both real and simulated data-sets that a computationally efficient point-wise implementation (PARNI) provides more reliable inferences on a range of variable selection problems, particularly in the very large p setting.

Reversible Jump PDMP Samplers for Variable Selection

A new class of Markov chain Monte Carlo (MCMC) algorithms, based on simulating piecewise deterministic Markov processes (PDMPs), has recently shown great promise: they are nonreversible, can mix better than standard MCMC algorithms, and can use subsampling ideas to speed up computation in big data scenarios. However, current PDMP samplers can only sample from posterior densities that are differentiable almost everywhere, which precludes their use for model choice. Motivated by variable selection problems, we show how to develop reversible jump PDMP samplers that can jointly explore the discrete space of models and the continuous space of parameters. Our framework is general: it takes any existing PDMP sampler, and adds two types of trans-dimensional moves that allow for the addition or removal of a variable from the model. We show how the rates of these trans-dimensional moves can be calculated so that the sampler has the correct invariant distribution. We remove a variable from a model when the associated parameter is zero, and this means that the rates of the trans-dimensional moves do not depend on the likelihood. It is, thus, easy to implement a reversible jump version of any PDMP sampler that can explore a fixed model. Supplementary materials for this article are available online.

Bayesian sparse modeling to identify high-risk subgroups in meta-analysis of safety data.

Meta-analysis allows researchers to combine evidence from multiple studies, making it a powerful tool for synthesizing information on the safety profiles of new medical interventions. There is a critical need to identify subgroups at high risk of experiencing treatment-related toxicities. However, this remains quite challenging from a statistical perspective as there are a variety of clinical risk factors that may be relevant for different types of adverse events, and adverse events of interest may be rare or incompletely reported. We frame this challenge as a variable selection problem and propose a Bayesian hierarchical model which incorporates a horseshoe prior on the interaction terms to identify high-risk groups. Our proposed model is motivated by a meta-analysis of adverse events in cancer immunotherapy, and our results uncover key factors driving the risk of specific types of treatment-related adverse events.

Variable selection in linear-circular regression models

ABSTRACT Applications of circular regression models are ubiquitous in many disciplines, particularly in meteorology, biology and geology. In circular regression models, variable selection problem continues to be a remarkable open question. In this paper, we address variable selection in linear-circular regression models where uni-variate linear dependent and a mixed set of circular and linear independent variables constitute the data set. We consider Bayesian lasso which is a popular choice for variable selection in classical linear regression models. We show that Bayesian lasso in linear-circular regression models is not able to produce robust inference as the coefficient estimates are sensitive to the choice of hyper-prior setting for the tuning parameter. To eradicate the problem, we propose a robustified Bayesian lasso that is based on an empirical Bayes (EB) type methodology to construct a hyper-prior for the tuning parameter while using Gibbs Sampling. This hyper-prior construction is computationally more feasible than the hyper-priors that are based on correlation measures. We show in a comprehensive simulation study that Bayesian lasso with EB-GS hyper-prior leads to a more robust inference. Overall, the method offers an efficient Bayesian lasso for variable selection in linear-circular regression while reducing model complexity.

High-dimensional feature selection in competing risks modeling: A stable approach using a split-and-merge ensemble algorithm.

Variable selection is critical in competing risks regression with high-dimensional data. Although penalized variable selection methods and other machine learning-based approaches have been developed, many of these methods often suffer from instability in practice. This paper proposes a novel method named Random Approximate Elastic Net (RAEN). Under the proportional subdistribution hazards model, RAEN provides a stable and generalizable solution to the large-p-small-n variable selection problem for competing risks data. Our general framework allows the proposed algorithm to be applicable to other time-to-event regression models, including competing risks quantile regression and accelerated failure time models. We show that variable selection and parameter estimation improved markedly using the new computationally intensive algorithm through extensive simulations. A user-friendly R package RAEN is developed for public use. We also apply our method to a cancer study to identify influential genes associated with the death or progression from bladdercancer.

A Unifying Framework of High-Dimensional Sparse Estimation with Difference-of-Convex (DC) Regularizations

Under the linear regression framework, we study the variable selection problem when the underlying model is assumed to have a small number of nonzero coefficients. Nonconvex penalties in specific forms are well studied in the literature for sparse estimation. Recent work pointed out that nearly all existing nonconvex penalties can be represented as difference-of-convex (DC) functions, which are the difference of two convex functions, while itself may not be convex. There is a large existing literature on optimization problems when their objectives and/or constraints involve DC functions. Efficient numerical solutions have been proposed. Under the DC framework, directional-stationary (d-stationary) solutions are considered, and they are usually not unique. In this paper, we show that under some mild conditions, a certain subset of d-stationary solutions in an optimization problem (with a DC objective) has some ideal statistical properties: namely, asymptotic estimation consistency, asymptotic model selection consistency, asymptotic efficiency. Our assumptions are either weaker than or comparable with those conditions that have been adopted in other existing works. This work shows that DC is a nice framework to offer a unified approach to these existing works where nonconvex penalties are involved. Our work bridges the communities of optimization and statistics.

Sparse Index Tracking Portfolio with Sector Neutrality

As a popular passive investment strategy, a sparse index tracking strategy has advantages over a full index replication strategy because of higher liquidity and lower transaction costs. Sparsity and nonnegativity constraints are usually assumed in the construction of portfolios in sparse index tracking. However, none of the existing studies considered sector risk exposure of the portfolios that prices of stocks in one sector may fall at the same time due to sudden changes in policy or unexpected events that may affect the whole sector. Therefore, sector neutrality appeals to be critical when building a sparse index tracking portfolio if not using full replication. The statistical approach to sparse index tracking is a constrained variable selection problem. However, the constrained variable selection procedure using Lasso fails to produce a sparse portfolio under sector neutrality constraints. In this paper, we propose a high-dimensional constrained variable selection method using TLP for building index tracking portfolios under sparsity, sector neutrality and nonnegativity constraints. Selection consistency is established for the proposed method, and the asymptotic distribution is obtained for the sparse portfolio weights estimator. We also develop an efficient iteration algorithm for the weight estimation. We examine the performance of the proposed methodology through simulations and an application to the CSI 300 index of China. The results demonstrate the validity and advantages of our methodology.

Regularization in dynamic random‐intercepts models for analysis of longitudinal data

AbstractThis paper addresses the problem of simultaneous variable selection and estimation in the random‐intercepts model with the first‐order lag response. This type of model is commonly used for analyzing longitudinal data obtained through repeated measurements on individuals over time. This model uses random effects to cover the intra‐class correlation, and the first lagged response to address the serial correlation, which are two common sources of dependency in longitudinal data. We demonstrate that the conditional likelihood approach by ignoring correlation among random effects and initial responses can lead to biased regularized estimates. Furthermore, we demonstrate that joint modeling of initial responses and subsequent observations in the structure of dynamic random‐intercepts models leads to both consistency and Oracle properties of regularized estimators. We present theoretical results in both low‐ and high‐dimensional settings and evaluate regularized estimators' performances by conducting simulation studies and analyzing a real dataset. Supporting information is available online.

Gradient Learning under Tilted Empirical Risk Minimization.

Gradient Learning (GL), aiming to estimate the gradient of target function, has attracted much attention in variable selection problems due to its mild structure requirements and wide applicability. Despite rapid progress, the majority of the existing GL works are based on the empirical risk minimization (ERM) principle, which may face the degraded performance under complex data environment, e.g., non-Gaussian noise. To alleviate this sensitiveness, we propose a new GL model with the help of the tilted ERM criterion, and establish its theoretical support from the function approximation viewpoint. Specifically, the operator approximation technique plays the crucial role in our analysis. To solve the proposed learning objective, a gradient descent method is proposed, and the convergence analysis is provided. Finally, simulated experimental results validate the effectiveness of our approach when the input variables are correlated.

Group Logistic Regression Models with lp,q Regularization

In this paper, we proposed a logistic regression model with lp,q regularization that could give a group sparse solution. The model could be applied to variable-selection problems with sparse group structures. In the context of big data, the solutions for practical problems are often group sparse, so it is necessary to study this kind of model. We defined the model from three perspectives: theoretical, algorithmic and numeric. From the theoretical perspective, by introducing the notion of the group restricted eigenvalue condition, we gave the oracle inequality, which was an important property for the variable-selection problems. The global recovery bound was also established for the logistic regression model with lp,q regularization. From the algorithmic perspective, we applied the well-known alternating direction method of multipliers (ADMM) algorithm to solve the model. The subproblems for the ADMM algorithm were solved effectively. From the numerical perspective, we performed experiments for simulated data and real data in the factor stock selection. We employed the ADMM algorithm that we presented in the paper to solve the model. The numerical results were also presented. We found that the model was effective in terms of variable selection and prediction.

Variable selection for bivariate interval-censored failure time data under linear transformation models.

Variable selection is needed and performed in almost every field and a large literature on it has been established, especially under the context of linear models or for complete data. Many authors have also investigated the variable selection problem for incomplete data such as right-censored failure time data. In this paper, we discuss variable selection when one faces bivariate interval-censored failure time data arising from a linear transformation model, for which it does not seem to exist an established procedure. For the problem, a penalized maximum likelihood approach is proposed and in particular, a novel Poisson-based EM algorithm is developed for the implementation. The oracle property of the proposed method is established, and the numerical studies suggest that the method works well for practical situations.

Time-varying Group Lasso Granger Causality Graph for High Dimensional Dynamic system

Feature selection is a crucial preprocessing step in data analysis and machine learning. Since causal relationships imply the underlying mechanism of a system, causality-based feature selection methods have gradually attracted great attentions. For a high dimensional system undergoing dynamic transformation, because of the non-stationarity and sample scarcity, modeling the causal structure among these features is difficult. In this paper, we propose a time-varying Granger causal networks to capture the causal relations underlying high dimensional time-varying vector autoregressive models with high order lagged dependence. A kernel reweighted group lasso method is proposed, which overcomes the limitations of sample scarcity and transforms the problem of Granger causal structural learning into a group variable selection problem. The asymptotic consistency of the proposed algorithm is proved. We apply the time-varying Granger causal networks to simulation experiments and real data in the financial market. The study demonstrates that the method provides an efficient tool to detect changes and analysis characters of causal dependency structure in network evolution.