Variable Selection Problem Research Articles

REMI: REGRESSION WITH MARGINAL INFORMATION AND ITS APPLICATION IN GENOME-WIDE ASSOCIATION STUDIES

In this study, we consider the problem of variable selection and estimation in high-dimensional linear regression models when the complete data are not accessible, but only certain marginal information or summary statistics are available. This problem is motivated from the Genome-wide association studies (GWAS) that have been widely used to identify risk variants underlying complex human traits/diseases. With a large number of completed GWAS, statistical methods using summary statistics become more and more important because of restricted accessibility to individual-level data sets. Theoretically guaranteed methods are highly demanding to advance the statistical inference with a large amount of available marginal information. Here we propose an $\ell_1$ penalized approach, REMI, to estimate high dimensional regression coefficients with marginal information and external reference samples. We establish an upper bound on the error of the REMI estimator, which has the same order as that of the minimax error bound of Lasso with complete individual-level data. In particular, when marginal information is obtained from a large number of samples together with a small number of reference samples, REMI yields good estimation and prediction results, and outperforms the Lasso because the sample size of accessible individual-level data can be limited. Through simulation studies and real data analysis of the NFBC1966 GWAS data set, we demonstrate that REMI can be widely applicable. The developed R package and the codes to reproduce all the results are available at this https URL

Multivariate variable selection by means of null-beamforming

This article aims to use beamforming, a covariate-assisted data projection method to solve the problem of variable selection for multivariate random-effects regression models. The new approach attempts to explore the covariance structure in the data with a small number of random-effects covariates. The basic premise behind the proposal is to scan through a covariate space with a series of forward filters named null-beamformers; each is tailored to a particular covariate in the space and resistant to interference effects originating from other covariates. Applying the proposed method to simulated and real multivariate regression data, we show that it can substantially outperform the existing methods of multivariate variable selection in terms of sensitivity and specificity. A theory on selection consistency is established under certain regularity conditions.

Sparse Learning with Non-convex Penalty in Multi-classification

Multi-classification is commonly encountered in data science practice, and it has broad applications in many areas such as biology, medicine, and engineering. Variable selection in multiclass problems is much more challenging than in binary classification or regression problems. In addition to estimating multiple discriminant functions for separating different classes, we need to decide which variables are important for each individual discriminant function as well as for the whole set of functions. In this paper, we address the multi-classification variable selection problem by proposing a new form of penalty, supSCAD, which first groups all the coefficients of the same variable associated with all the discriminant functions altogether and then imposes the SCAD penalty on the supnorm of each group. We apply the new penalty to both soft and hard classification and develop two new procedures: the supSCAD multinomial logistic regression and the supSCAD multi-category support vector machine. Our theoretical results show that, with a proper choice of the tuning parameter, the supSCAD multinomial logistic regression can identify the underlying sparse model consistently and enjoys oracle properties even when the dimension of predictors goes to infinity. Based on the local linear and quadratic approximation to the non-concave SCAD and nonlinear multinomial log-likelihood function, we show that the new procedures can be implemented efficiently by solving a series of linear or quadratic programming problems. Performance of the new methods is illustrated by simulation studies and real data analysis of the Small Round Blue Cell Tumors and the Semeion Handwritten Digit data sets.

Variable selection for linear regression in large databases: exact methods

This paper analyzes the variable selection problem in the context of Linear Regression for large databases. The problem consists of selecting a small subset of independent variables that can perform the prediction task optimally. This problem has a wide range of applications. One important type of application is the design of composite indicators in various areas (sociology and economics, for example). Other important applications of variable selection in linear regression can be found in fields such as chemometrics, genetics, and climate prediction, among many others. For this problem, we propose a Branch & Bound method. This is an exact method and therefore guarantees optimal solutions. We also provide strategies that enable this method to be applied in very large databases (with hundreds of thousands of cases) in a moderate computation time. A series of computational experiments shows that our method performs well compared to well-known methods in the literature and with commercial software.

Asymptotic theory of dependent Bayesian multiple testing procedures under possible model misspecification

We study asymptotic properties of Bayesian multiple testing procedures and provide sufficient conditions for strong consistency under general dependence structure. We also consider a novel Bayesian multiple testing procedure and associated error measures that coherently accounts for the dependence structure present in the model. We advocate posterior versions of FDR and FNR as appropriate error rates and show that their asymptotic convergence rates are directly associated with the Kullback–Leibler divergence from the true model. The theories hold regardless of the class of postulated models being misspecified. We illustrate our results in a variable selection problem with autoregressive response variables and compare our procedure with some existing methods through simulation studies. Superior performance of the new procedure compared to the others indicates that proper exploitation of the dependence structure by multiple testing methods is indeed important. Moreover, we obtain encouraging results in a maize dataset, where we select influential marker variables.

Spatially varying sparsity in dynamic regression models

Motivated by the problem of variable selection in spatially varying coefficients models for spatial econometrics data, a Bayesian spatially dynamic selection model based on spatial normal-gamma process (SNGP) is proposed, which pursues spatial varying sparsity in dynamic regression models. Theoretical properties of SNGP are discussed. Posterior samples are obtained by nimble, a powerful R package for Bayesian inference. A new tuning-free variable selection based on K-groups clustering is proposed for discriminating the signal and the noise. Simulation studies show that the proposed method has both good estimation performance and selection performance. Finally, the new method is applied to analyzing a county level income data of Louisiana.

Linearized symmetric multi-block ADMM with indefinite proximal regularization and optimal proximal parameter

The proximal term plays a significant role in the literature of proximal Alternating Direction Method of Multipliers (ADMM), since (positive-definite or indefinite) proximal terms can promote convergence of ADMM and further simplify the involved subproblems. However, an overlarge proximal parameter decelerates the convergence numerically, though the convergence can be established with it. In this paper, we thus focus on a Linearized Symmetric ADMM (LSADMM) with proper proximal terms for solving a family of multi-block separable convex minimization models, and we determine an optimal (smallest) value of the proximal parameter while convergence of this LSADMM can be still ensured. Our LSADMM partitions the data into two group variables and updates the Lagrange multiplier twice in different forms with suitable step sizes. The region of the proximal parameter, involved in the second group subproblems, is partitioned into the union of three different sets. We show the global convergence and sublinear ergodic convergence rate of LSADMM for the two cases, while a counter-example is given to illustrate that convergence of LSADMM can not be guaranteed for the remaining case. Theoretically, we obtain the optimal lower bound of the proximal parameter. Numerical experiments on the so-called latent variable Gaussian graphical model selection problems are presented to demonstrate performance of the proposed algorithm and the significant advantage of the optimal lower bound of the proximal parameter.

A Look at Robustness and Stability of $\ell_{1}$-versus $\ell_{0}$-Regularization: Discussion of Papers by Bertsimas et al. and Hastie et al.

We congratulate the authors Bertsimas, Pauphilet and van Parys (hereafter BPvP) and Hastie, Tibshirani and Tibshirani (hereafter HTT) for providing fresh and insightful views on the problem of variable selection and prediction in linear models. Their contributions at the fundamental level provide guidance for more complex models and procedures.

Learning Clique Subgraphs in Structural Brain Network Classification with Application to Crystallized Cognition

Structural brain networks constructed from diffusion MRI are important biomarkers for understanding human brain structure and its relation to cognitive functioning. There is increasing interest in learning differences in structural brain networks between groups of subjects in neuroimaging studies, leading to a variable selection problem in network classification. Traditional methods often use independent edgewise tests or unstructured generalized linear model (GLM) with regularization on vectorized networks to select edges distinguishing the groups, which ignore the network structure and make the results hard to interpret. In this paper, we develop a symmetric bilinear logistic regression (SBLR) with elastic-net penalty to identify a set of small clique subgraphs in network classification. Clique subgraphs, consisting of all the interconnections among a subset of brain regions, have appealing neurological interpretations as they may correspond to some anatomical circuits in the brain related to the outcome. We apply this method to study differences in the structural connectome between adolescents with high and low crystallized cognitive ability, using the crystallized cognition composite score, picture vocabulary and oral reading recognition tests from NIH Toolbox. A few clique subgraphs containing several small sets of brain regions are identified between different levels of functioning, indicating their importance in crystallized cognition.

Variable selection in high-dimensional linear model with possibly asymmetric errors

In many application areas, the problem of the automatic variable selection in a linear model with asymmetric errors is encountered, when the number of explanatory variables diverges with the sample size. For this high-dimensional model, the penalized least squares method is not appropriate and the quantile framework makes the inference more difficult because of the non differentiability of the loss function. An estimation method by penalizing the expectile process with an adaptive LASSO penalty is proposed and studied. Two cases are considered: first with the number of model parameters is assumed to be much smaller than the sample size and afterwards it could be of the same order; the two cases being distinct by the adaptive penalties considered. For each case, the rate convergence is obtained and the oracle properties of the adaptive LASSO expectile estimator are established. The proposed estimators are evaluated through Monte Carlo simulations and compared with the adaptive LASSO quantile estimator. The proposed estimation method is also applied to real data in genetics.

A fully Bayesian approach to sparse reduced-rank multivariate regression

In the context of high-dimensional multivariate linear regression, sparse reduced-rank regression (SRRR) provides a way to handle both variable selection and low-rank estimation problems. Although there has been extensive research on SRRR, statistical inference procedures that deal with the uncertainty due to variable selection and rank reduction are still limited. To fill this research gap, we develop a fully Bayesian approach to SRRR. A major difficulty that occurs in a fully Bayesian framework is that the dimension of parameter space varies with the selected variables and the reduced-rank. Due to the varying-dimensional problems, traditional Markov chain Monte Carlo (MCMC) methods such as Gibbs sampler and Metropolis-Hastings algorithm are inapplicable in our Bayesian framework. To address this issue, we propose a new posterior computation procedure based on the Laplace approximation within the collapsed Gibbs sampler. A key feature of our fully Bayesian method is that the model uncertainty is automatically integrated out by the proposed MCMC computation. The proposed method is examined via simulation study and real data analysis.

Which bridge estimator is the best for variable selection?

We study the problem of variable selection for linear models under the high-dimensional asymptotic setting, where the number of observations $n$ grows at the same rate as the number of predictors $p$. We consider two-stage variable selection techniques (TVS) in which the first stage uses bridge estimators to obtain an estimate of the regression coefficients, and the second stage simply thresholds this estimate to select the “important” predictors. The asymptotic false discovery proportion ($\operatorname{AFDP}$) and true positive proportion (ATPP) of these TVS are evaluated. We prove that for a fixed ATPP, in order to obtain a smaller $\operatorname{AFDP}$, one should pick a bridge estimator with smaller asymptotic mean square error in the first stage of TVS. Based on such principled discovery, we present a sharp comparison of different TVS, via an in-depth investigation of the estimation properties of bridge estimators. Rather than “orderwise” error bounds with loose constants, our analysis focuses on precise error characterization. Various interesting signal-to-noise ratio and sparsity settings are studied. Our results offer new and thorough insights into high-dimensional variable selection. For instance, we prove that a TVS with Ridge in its first stage outperforms TVS with other bridge estimators in large noise settings; two-stage LASSO becomes inferior when the signal is rare and weak. As a by-product, we show that two-stage methods outperform some standard variable selection techniques, such as $\operatorname{LASSO}$ and Sure Independence Screening, under certain conditions.

Understanding Marginal Structural Models for Time-Varying Exposures: Pitfalls and Tips.

Epidemiologists are increasingly encountering complex longitudinal data, in which exposures and their confounders vary during follow-up. When a prior exposure affects the confounders of the subsequent exposures, estimating the effects of the time-varying exposures requires special statistical techniques, possibly with structural (ie, counterfactual) models for targeted effects, even if all confounders are accurately measured. Among the methods used to estimate such effects, which can be cast as a marginal structural model in a straightforward way, one popular approach is inverse probability weighting. Despite the seemingly intuitive theory and easy-to-implement software, misunderstandings (or “pitfalls”) remain. For example, one may mistakenly equate marginal structural models with inverse probability weighting, failing to distinguish a marginal structural model encoding the causal parameters of interest from a nuisance model for exposure probability, and thereby failing to separate the problems of variable selection and model specification for these distinct models. Assuming the causal parameters of interest are identified given the study design and measurements, we provide a step-by-step illustration of generalized computation of standardization (called the g-formula) and inverse probability weighting, as well as the specification of marginal structural models, particularly for time-varying exposures. We use a novel hypothetical example, which allows us access to typically hidden potential outcomes. This illustration provides steppingstones (or “tips”) to understand more concretely the estimation of the effects of complex time-varying exposures.

Semi-analytic approximate stability selection for correlated data in generalized linear models

We consider the variable selection problem of generalized linear models (GLMs). Stability selection (SS) is a promising method proposed for solving this problem. Although SS provides practical variable selection criteria, it is computationally demanding because it needs to fit GLMs to many re-sampled datasets. We propose a novel approximate inference algorithm that can conduct SS without the repeated fitting. The algorithm is based on the replica method of statistical mechanics and vector approximate message passing of information theory. For datasets characterized by rotation-invariant matrix ensembles, we derive state evolution equations that macroscopically describe the dynamics of the proposed algorithm. We also show that their fixed points are consistent with the replica symmetric solution obtained by the replica method. Numerical experiments indicate that the algorithm exhibits fast convergence and high approximation accuracy for both synthetic and real-world data.

Application of integrated recurrent neural network with multivariate adaptive regression splines on SPC-EPC process

The integration of statistical process control and engineering process control has been reported as an effective way to monitor and control the autocorrelated process. However, because engineering process control compensates for the effects of underlying disturbances, the disturbance patterns become very hard to recognize, especially when various abnormal control chart patterns are mixed and co-existed in the engineering process. In this study, a new control chart pattern recognition model which integrates multivariate adaptive regression splines and recurrent neural network is proposed to not only address the problem of feature selection (i.e., lagged process measurements) but also improve the pattern recognition accuracy. The performance of the proposed method is evaluated by comparing the recognition results of multivariate adaptive regression splines and recurrent neural network with the results of four competing approaches (multivariate adaptive regression splines-extreme learning machine, multivariate adaptive regression splines-random forest, single recurrent neural network, and single random forest) on the simulated individual process data. The experimental study shows that the proposed multivariate adaptive regression splines and recurrent neural network approach can not only solve the problem of variable selection but also outperform other competing models. Moreover, according to the lagged process measurements selected by the proposed approach, lagged observations that exerted significant impact on the construction of the control chart pattern recognition model can be identified successfully. This study has significant implications for research and practice in production management and provides a valuable reference for manufacturing process managers to better understand and develop strategies for control chart pattern recognition.

A Dirichlet Process Prior Approach for Covariate Selection.

The variable selection problem in general, and specifically for the ordinary linear regression model, is considered in the setup in which the number of covariates is large enough to prevent the exploration of all possible models. In this context, Gibbs-sampling is needed to perform stochastic model exploration to estimate, for instance, the model inclusion probability. We show that under a Bayesian non-parametric prior model for analyzing Gibbs-sampling output, the usual empirical estimator is just the asymptotic version of the expected posterior inclusion probability given the simulation output from Gibbs-sampling. Other posterior conditional estimators of inclusion probabilities can also be considered as related to the latent probabilities distributions on the model space which can be sampled given the observed Gibbs-sampling output. This paper will also compare, in this large model space setup the conventional prior approach against the non-local prior approach used to define the Bayes Factors for model selection. The approach is exposed along with simulation samples and also an application of modeling the Travel and Tourism factors all over the world.

Bayesian One-Sided Variable Selection

This paper presents a novel Bayesian variable selection approach that accounts for the sign of the regression coefficients based on multivariate one-sided tests. We propose a truncated g prior to specify a prior distribution of coefficients with anticipated signs in a given model. Informative priors for the direction of the effects can be incorporated into prior model probabilities. The best subset of variables is selected by comparing the posterior probabilities of the possible models. The new Bayesian one-sided variable selection procedure has higher chance to include relevant variables and therefore select the best model, if the anticipated direction is accurate. For a large number of candidate variables, we present an adaptation of a Bayesian model search method for the one-sided variable selection problem to ensure fast computation. In addition, a fully Bayesian approach is used to adjust the prior inclusion probability of each one-sided model to correct for multiplicity. The performance of the proposed method is investigated using several simulation studies and two real data examples.

Covariate-driven factorization by thresholding for multiblock data.

Multiblock data, where multiple groups of variables from different sources are observed for a common set of subjects, are routinely collected in many areas of science. Methods for joint factorization of such multiblock data are being developed to explore the potentially joint variation structure of the data. While most of the existing work focuses on delineating joint components, shared across all data blocks, from individual components, which is only relevant to a single data block, we propose to model and estimate partially joint components across some, but not all, data blocks. If covariates, with potential multiblock structures, are available, then the components are further modeled to be driven by the covariate information. To estimate such a covariate-driven, block-structured factor model, we propose an iterative algorithm based on thresholding, by transforming the problem of signal segmentation into a grouped variable selection problem. The proposed factorization provides accurate estimation of individual and (partially) joint structures in multiblock data, as confirmed by simulation studies. In the analysis of a real multiblock genomic dataset from the Cancer Genome Atlas project, we demonstrate that the estimated block structures provide straightforward interpretation and facilitate subsequentanalyses.

New Approaches for Variable Selection in Longitudinal Studies: An Application to Genetic Association Studies and Bone Loss

We study the problem of variable selection in the context of longitudinal studies by linear mixed effect models when the dimensionality of predictors subject to selection is high. We consider a fully Bayesian formulation of the problem with spike-and-slab a prior distributions which is an appealing solution, however, we find it computationally demanding. As an alternative, we propose a general penalized log-likelihood framework based on marginalized observation equation and a heterogeneous error term. The parameters of the model are estimated by a block wise coordinate gradient descent algorithm. Given the formulation of the considered linear mixed effect models as random slope and intercept models, we consider yet another variable selection procedure which is more ad hoc but has an advantage of computational speed. The procedure consists of two steps and is based on assessing the significance of co-variates from biased predictors of subject-specific random effects. We study the impact of this mis-specification on the accuracy of variable selection. We demonstrate the applicability of the proposed frameworks by studying the association between single nucleotide polymorphisms and bone loss that contributes to the development of osteopenia and osteoporosis in post-menopausal women and aged men and is a strong risk factor for bones fractures and its related mortality. The data is obtained from the longitudinal Dubbo Osteoporosis Epidemiology Study.

Variable Selection and Forecasting in High Dimensional Linear Regressions with Structural Breaks

This paper is concerned with the problem of variable selection and forecasting in the presence of parameter instability. There are a number of approaches proposed for forecasting in the presence of breaks, including the use of rolling windows and exponential down-weighting. However, these studies start with a given model specification and do not consider the problem of variable selection, which is complicated by time variations in the effects of signal variables. In this study we investigate whether or not we should use weighted observations at the variable selection stage in the presence of structural breaks, particularly when the number of potential covariates is large. Amongst the extant variable selection approaches we focus on the recently developed One Covariate at a time Multiple Testing (OCMT) method. This procedure allows a natural distinction between the selection and forecasting stages. We establish three main theorems on selection, estimation post selection and in-sample .t. These theorems provide justification for using the full (not down-weighted) sample at the selection stage of OCMT and down-weighting of observations only at the forecasting stage (if needed). The benefits of the proposed method are illustrated by empirical applications to forecasting output growths and stock market returns.