Variable Selection Problem Research Articles

Variable selection for bivariate interval-censored failure time data under linear transformation models.

Variable selection is needed and performed in almost every field and a large literature on it has been established, especially under the context of linear models or for complete data. Many authors have also investigated the variable selection problem for incomplete data such as right-censored failure time data. In this paper, we discuss variable selection when one faces bivariate interval-censored failure time data arising from a linear transformation model, for which it does not seem to exist an established procedure. For the problem, a penalized maximum likelihood approach is proposed and in particular, a novel Poisson-based EM algorithm is developed for the implementation. The oracle property of the proposed method is established, and the numerical studies suggest that the method works well for practical situations.

Time-varying Group Lasso Granger Causality Graph for High Dimensional Dynamic system

Feature selection is a crucial preprocessing step in data analysis and machine learning. Since causal relationships imply the underlying mechanism of a system, causality-based feature selection methods have gradually attracted great attentions. For a high dimensional system undergoing dynamic transformation, because of the non-stationarity and sample scarcity, modeling the causal structure among these features is difficult. In this paper, we propose a time-varying Granger causal networks to capture the causal relations underlying high dimensional time-varying vector autoregressive models with high order lagged dependence. A kernel reweighted group lasso method is proposed, which overcomes the limitations of sample scarcity and transforms the problem of Granger causal structural learning into a group variable selection problem. The asymptotic consistency of the proposed algorithm is proved. We apply the time-varying Granger causal networks to simulation experiments and real data in the financial market. The study demonstrates that the method provides an efficient tool to detect changes and analysis characters of causal dependency structure in network evolution.

Application of Metaheuristic Approaches for the Variable Selection Problem

Variable selection is an old topic from regression models. Besides many conventional approaches, some metaheuristic approaches from the realm of optimization such as GA (Genetic Algorithm) or simulated annealing have been suggested to date. These methods have a considerable advantage to deal with many problems over the classical methods, but they must control relevant fine-tuning parameters associated with cross-over or mutation, which can be difficult and time-consuming. In this paper, Jaya, one of several parameter-free approaches will be suggested and explored. Several metaheuristic methods will be compared using results from a real-world dataset and a simulated dataset. The impact of using local search will be analyzed.

Bayesian penalized Buckley-James method for high dimensional bivariate censored regression models.

For high dimensional gene expression data, one important goal is to identify a small number of genes that are associated with progression of the disease or survival of the patients. In this paper, we consider the problem of variable selection for multivariate survival data. We propose an estimation procedure for high dimensional accelerated failure time (AFT) models with bivariate censored data. The method extends the Buckley-James method by minimizing a penalized [Formula: see text] loss function with a penalty function induced from a bivariate spike-and-slab prior specification. In the proposed algorithm, censored observations are imputed using the Kaplan-Meier estimator, which avoids a parametric assumption on the error terms. Our empirical studies demonstrate that the proposed method provides better performance compared to the alternative procedures designed for univariate survival data regardless of whether the true events are correlated or not, and conceptualizes a formal way of handling bivariate survival data for AFT models. Findings from the analysis of a myeloma clinical trial using the proposed method are also presented.

The linear Lasso: A location model approach

AbstractWe use location model methodology to guide the least squares analysis in the Lasso problem of variable selection and inference. The nuisance parameter is taken to be an indicator for the selection of explanatory variables, and the interest parameter is the response variable itself. Recent theory eliminates the nuisance parameter by marginalization on the data space and then uses the resulting distribution for inference concerning the interest parameter. We develop this approach and find that primary inference is essentially one‐dimensional rather than n‐dimensional, inference focuses on the response variable itself rather than the least squares estimate (as variables are removed), computation is relatively easy, a scalar marginal model is available, and ineffective variables can be removed by distributional tilt or shift.

Bayesian model averaging: improved variable selection for matched case-control studies.

The problem of variable selection for risk factor modeling is an ongoing challenge in statistical practice. Classical methods that select one subset of exploratory risk factors dominate the medical research field. However, this approach has been criticized for not taking into account the uncertainty of the model selection process itself. This limitation can be addressed by a Bayesian model averaging approach: instead of focusing on a single model and a few factors, Bayesian model averaging considers all the models with non-negligible probabilities to make inference. This paper reports on a simulation study designed to emulate a matched case-control study and compares classical versus Bayesian model averaging selection methods. We used Matthews's correlation coefficient to measure the quality of binary classifications. Both classical and Bayesian model averaging were also applied and compared for the analysis of a matched case-control study of patients with methicillin-resistant Staphylococcus aureus infections after hospital discharge 2011-2013. Bayesian model averaging outperformed the classical approach with much lower false positive rates and higher Matthew's correlation scores. Bayesian model averaging also produced more reliable and robust effect estimates. Bayesian model averaging is a conceptually simple, unified approach that produces robust results. It can be used to replace controversial P-values for case-control study in medical research.

A general adaptive ridge regression method for generalized linear models: an iterative re-weighting approach

This article is concerned with the problem of variable selection and estimation for high dimensional generalized linear models. In this article, we introduce a general iteratively reweighted adaptive ridge regression method (GAR). We show that the GAR estimator possesses oracle property and grouping effect. A data-driven parameter γ is introduced in the GAR method to adapt the different cases of the true model. Then, such an adaptive parameter γ is adequately taken into consideration to establish a γ-dependent sufficient condition to guarantee the oracle property and the grouping effect. Furthermore, to apply the GAR method more efficiently, a coordinate-wise Newton algorithm is employed to successfully avoid the inverse matrix operation and the numerical instability caused by iteration. Extensive numerical simulation results show that the GAR method outperforms the commonly used methods, and the GAR method is tested on the gastric cancer dataset for further illustration.

The variable selection methods and algorithms in the multiple linear model

The Lasso and Ridge estimators are effective variable selection methods for a simple linear model. While the multiple linear model is often used in practice, only few studies have examined its problem of variable selection. To address this gap, we study such problem by taking advantage of the Lasso and Ridge estimate. We propose two variable selection methods in the multiple linear model and introduce the multiple random simulation (MRS) algorithm, whose efficiency is similar to that of the least angle regression (LARS) algorithm in a simple linear model. We verify the performance of these two algorithms by applying them to real diabetes data and find that the LARS algorithm cannot be easily applied in a multiple linear model. We also illustrate the excellent performance of MRS by applying this algorithm on a simulated dataset.

Simultaneous variable selection and estimation for longitudinal ordinal data with a diverging number of covariates

<abstract><p>In this paper, we study the problem of simultaneous variable selection and estimation for longitudinal ordinal data with high-dimensional covariates. Using the penalized generalized estimation equation (GEE) method, we obtain some asymptotic properties for these types of data in the case that the dimension of the covariates $ p_n $ tends to infinity as the number of cluster $ n $ approaches to infinity. More precisely, under appropriate regular conditions, all the covariates with zero coefficients can be examined simultaneously with probability tending to 1, and the estimator of the non-zero coefficients exhibits the asymptotic Oracle properties. Finally, we also perform some Monte Carlo studies to illustrate the theoretical analysis. The main result in this paper extends the elegant work of Wang et al. <sup>[<xref ref-type="bibr" rid="b1">1</xref>]</sup> to the multinomial response variable case.</p></abstract>

Variable Selection Based on Profile Forward Selection of Partial Linear Models with Interactive Terms

Due to the rapid development of information technology and data acquisition technology, the model which only considers the linear main effect can not provide accurate prediction results, and the interaction between the predictor and response variables can not be ignored, so the variable selection problem of the model with interaction terms has become an important research topic in the statistical analysis today. In this paper, we discuss the problem of variable selection for a partially linear model with interaction terms using the profile forward selection method under high dimensional data. We propose the two-stage interactive selection algorithm (iPFST) under strong genetic condition and the profile forward selection algorithm (iPFSM) under marginality principle respectively. Theoretically, we use the consistency of profile estimators to prove that profile estimators have uniform convergence rate, and use the screening consistency to prove that iPFST algorithm and iPFSM algorithm can uniformly identify all important linear main effect terms and important interaction effect terms with probability 1. Seven regularization conditions for the theorem are given. Numerical simulation shows the superiority of iPFST and iPFSM in variable selection, and the two algorithms are compared, then iPFST algorithm is better than iPFSM algorithm. Finally, we give detailed technical proof.

Variable Selection for Robust Mixture Regression Model with Skew Scale Mixtures of Normal Distributions

In this paper, we propose a robust mixture regression model based on the skew scale mixtures of normal distributions (RMR-SSMN) which can accommodate asymmetric, heavy-tailed and contaminated data better. For the variable selection problem, the penalized likelihood approach with a new combined penalty function which balances the SCAD and l2 penalty is proposed. The adjusted EM algorithm is presented to get parameter estimates of RMR-SSMN models at a faster convergence rate. As simulations show, our mixture models are more robust than general FMR models and the new combined penalty function outperforms SCAD for variable selection. Finally, the proposed methodology and algorithm are applied to a real data set and achieve reasonable results.

Wavelet-based robust estimation and variable selection in nonparametric additive models

This article studies M-type estimators for fitting robust additive models in the presence of anomalous data. The components in the additive model are allowed to have different degrees of smoothness. We introduce a new class of wavelet-based robust M-type estimators for performing simultaneous additive component estimation and variable selection in such inhomogeneous additive models. Each additive component is approximated by a truncated series expansion of wavelet bases, making it feasible to apply the method to nonequispaced data and sample sizes that are not necessarily a power of 2. Sparsity of the additive components together with sparsity of the wavelet coefficients within each component (group), results into a bi-level group variable selection problem. In this framework, we discuss robust estimation and variable selection. A two-stage computational algorithm, consisting of a fast accelerated proximal gradient algorithm of coordinate descend type, and thresholding, is proposed. When using nonconvex redescending loss functions, and appropriate nonconvex penalty functions at the group level, we establish optimal convergence rates of the estimates. We prove variable selection consistency under a weak compatibility condition for sparse additive models. The theoretical results are complemented with some simulations and real data analysis, as well as a comparison to other existing methods.

Economic variable selection

Regression plays a central role in the discipline of statistics and is the primary analytic technique in many research areas. Variable selection is a classical and major problem for regression. This article emphasizes the economic aspect of variable selection. The problem is formulated in terms of the cost of predictors to be purchased for future use: only the subset of covariates used in the model will need to be purchased. This leads to a decision‐theoretic formulation of the variable selection problems, which includes the cost of predictors as well as their effect. We adopt a Bayesian perspective and propose two approaches to address uncertainty about the model and model parameters. These approaches, termed the restricted and extended approaches, lead us to rethink model averaging. From an objective or robust Bayes point of view, the former is preferred. The proposed method is applied to three popular datasets for illustration.

Estimation in multivariate linear mixed models for longitudinal data with multiple outputs: Application to PBCseq data analysis

In many biomedical studies or clinical trials, we have data with more than one response variable on the same subject repeatedly measured over time. In analyzing such data, we adopt a multivariate linear mixed-effects longitudinal model. On the other hand, in longitudinal data, we often find features that do not impact modeling the response variable and are eliminated from the study. In this paper, we consider the problem of simultaneous variable selection and estimation in a multivariate t linear mixed-effects model (MtLMM) for analyzing longitudinally measured multioutcome data. This work's motivation comes from a cohort study of patients with primary biliary cirrhosis. The interest is eliminating insignificant variables using the smoothly clipped and absolute deviation penalty function in the MtLMM. The proposed penalized model offers robustness and flexibility to accommodate fat tails. An expectation conditional maximization algorithm is employed for the computation of maximum likelihood estimates of parameters. The calculation of standard errors is affected by an information-based method. The methodology is illustrated by analyzing Mayo Clinic Primary Biliary Cirrhosis sequential (PBCseq) data and a simulation study. We found drugs and sex can be eliminated from the PBCseq analysis, and over time the diseaseprogresses.

Global optimization for sparse solution of least squares problems

Finding solutions to least-squares problems with low cardinality has found many applications, including portfolio optimization, subset selection in statistics, and inverse problems in signal processing. Although most works consider local approaches that scale with high-dimensional problems, some others address its global optimization via mixed integer programming (MIP) reformulations. We propose dedicated branch-and-bound methods for the exact resolution of moderate-size, yet difficult, sparse optimization problems, through three possible formulations: cardinality-constrained and cardinality-penalized least-squares, and cardinality minimization under quadratic constraints. A specific tree exploration strategy is built. Continuous relaxation problems involved at each node are reformulated as -norm-based optimization problems, for which a dedicated algorithm is designed. The obtained certified solutions are shown to better estimate sparsity patterns than standard methods on simulated variable selection problems involving highly correlated variables. Problem instances selecting up to 24 components among 100 variables, and up to 15 components among 1000 variables, can be solved in less than 1000 s. Unguaranteed solutions obtained by limiting the computing time to 1s are also shown to provide competitive estimates. Our algorithms strongly outperform the CPLEX MIP solver as the dimension increases, especially for quadratically-constrained problems. The source codes are made freely available online.

Objective Bayesian variable selection in linear regression model

Variable selection in a regression model with k potential explanatory variables requires the choosing of a model among the possible submodels, which is a difficult task when the number of explanatory variables is moderately large. In this study, we propose the objective Bayesian variable selection procedures where the encompassing of the underlying nonnested linear models is crucial. Based on the encompassed models, objective priors for the multiple testing problem involved in the variable selection problem can be defined. The proposed approach provides a considerable reduction in the size of the compared models by restricting the posterior search for the right models, from to only k + 1, given k explanatory variables. Furthermore, the consistency of the proposed variable selection procedures was checked and their performance was examined using real examples and simulation analyzes by comparing the classical and Bayesian procedures of search in all possible submodels.

Variable selection for skew-normal mixture of joint location and scale models

Although there are many papers on variable selection methods based on mean model in the finite mixture of regression models, little work has been done on how to select significant explanatory variables in the modeling of the variance parameter. In this paper, we propose and study a novel class of models: a skew-normal mixture of joint location and scale models to analyze the heteroscedastic skew-normal data coming from a heterogeneous population. The problem of variable selection for the proposed models is considered. In particular, a modified Expectation-Maximization(EM) algorithm for estimating the model parameters is developed. The consistency and the oracle property of the penalized estimators is established. Simulation studies are conducted to investigate the finite sample performance of the proposed methodologies. An example is illustrated by the proposed methodologies.

A sparse approach for high-dimensional data with heavy-tailed noise

High-dimensional data have commonly emerged in diverse fields, such as economics, finance, genetics, medicine, machine learning, and so on. In this paper, we consider the sparse quantile regression problem of high-dimensional data with heavy-tailed noise, especially when the number of regressors is much larger than the sample size. We bring the spirit of L p -norm support vector regression into quantile regression and propose a robust L p -norm support vector quantile regression for high-dimensional data with heavy-tailed noise. The proposed method achieves robustness against heavy-tailed noise due to its use of the pinball loss function. Furthermore, L p -norm support vector quantile regression ensures that the most representative variables are selected automatically by using a sparse parameter. We use a simulation study to test the variable selection performance of L p -norm support vector quantile regression, where the number of explanatory variables greatly exceeds the sample size. The simulation study confirms that L p -norm support vector quantile regression is not only robust against heavy-tailed noise but also selects representative variables. We further apply the proposed method to solve the variable selection problem of index construction, which also confirms the robustness and sparseness of L p -norm support vector quantile regression.

Transferability of correlative and process-based species distribution models revisited: A response to Booth.

Here, we respond to Booth's criticism of our paper, “Predictive ability of a process‐based versus a correlative species distribution model.” Booth argues that our usage of the MaxEnt model was flawed and that the conclusions of our paper are by implication flawed. We respond by clarifying that the error Booth implies we made was not made in our analysis, and we repeat statements from the original manuscript which anticipated such criticisms. In addition, we illustrate that using BIOCLIM variables in a MaxEnt analysis as recommended by Booth does not change the conclusions of the original analysis. That is, high performance in the training data domain did not equate to reliable predictions in novel data domains, and the process model transferred into novel data domains better than the correlative model did. We conclude by discussing a hidden implication of our study, namely, that process‐based SDMs negate the need for BIOCLIM‐type variables and therefore reframe the variable selection problem in species distribution modeling.

Dynamic logistic regression and variable selection: Forecasting and contextualizing civil unrest

Civil unrest can range from peaceful protest to violent furor, and researchers are working to monitor, forecast, and assess such events to allocate resources better. Twitter has become a real-time data source for forecasting civil unrest because millions of people use the platform as a social outlet. Daily word counts are used as model features, and predictive terms contextualize the reasons for the protest. To forecast civil unrest and infer the reasons for the protest, we consider the problem of Bayesian variable selection for the dynamic logistic regression model and propose using penalized credible regions to select parameters of the updated state vector. This method avoids the need for shrinkage priors, is scalable to high-dimensional dynamic data, and allows the importance of variables to vary in time as new information becomes available. A substantial improvement in both precision and F1-score using this approach is demonstrated through simulation. Finally, we apply the proposed model fitting and variable selection methodology to the problem of forecasting civil unrest in Latin America. Our dynamic logistic regression approach shows improved accuracy compared to the static approach currently used in event prediction and feature selection.