Two-stage Variable Selection Research Articles

Relaxed Adaptive Lasso for Classification on High-Dimensional Sparse Data with Multicollinearity

High-dimensional sparse data with multicollinearity is frequently found in medical data. This problem can lead to poor predictive accuracy when applied to a new data set. The Least Absolute Shrinkage and Selection Operator (Lasso) is a popular machine-learning algorithm for variable selection and parameter estimation. Additionally, the adaptive Lasso method was developed using the adaptive weight on the l1-norm penalty. This adaptive weight is related to the power order of the estimators. Thus, we focus on 1) the power of adaptive weight on the penalty function, and 2) the two-stage variable selection method. This study aimed to propose the relaxed adaptive Lasso sparse logistic regression. Moreover, we compared the performances of the different penalty functions by using the mean of the predicted mean squared error (MPMSE) for the simulation study and the accuracy of classification for a real-data application. The results showed that the proposed method performed best on high-dimensional sparse data with multicollinearity. Along with, for classifier with the support vector machine, this proposed method was also the best option for the variable selection process.

Robust Variable Selection Based on Relaxed Lad Lasso

Least absolute deviation is proposed as a robust estimator to solve the problem when the error has an asymmetric heavy-tailed distribution or outliers. In order to be insensitive to the above situation and select the truly important variables from a large number of predictors in the linear regression, this paper introduces a two-stage variable selection method named relaxed lad lasso, which enables the model to obtain robust sparse solutions in the presence of outliers or heavy-tailed errors by combining least absolute deviation with relaxed lasso. Compared with lasso, this method is not only immune to the rapid growth of noise variables but also maintains a better convergence rate, which is Opn−1/2. In addition, we prove that the relaxed lad lasso estimator has the property of consistency at large samples; that is, the model selects the number of important variables with a high probability of convergence to one. Through the simulation and empirical results, we further verify the outstanding performance of relaxed lad lasso in terms of prediction accuracy and the correct selection of informative variables under the heavy-tailed distribution.

Relaxed Adaptive Lasso and Its Asymptotic Results

This article introduces a novel two-stage variable selection method to solve the common asymmetry problem between the response variable and its influencing factors. In practical applications, we cannot correctly extract important factors from a large amount of complex and redundant data. However, the proposed method based on the relaxed lasso and the adaptive lasso, namely, the relaxed adaptive lasso, can achieve information symmetry because the variables it selects contain all the important information about the response variables. The goal of this paper is to preserve the relaxed lasso’s superior variable selection speed while imposing varying penalties on different coefficients. Additionally, the proposed method enjoys favorable asymptotic properties, that is, consistency with a fast rate of convergence with Opn−1. The simulation demonstrates that the proper variable recovery, i.e., the number of significant variables selected, and prediction accuracy of the relaxed adaptive lasso in a limited sample is superior to the regular lasso, relaxed lasso and adaptive lasso estimators.

Variable selection in sparse GLARMA models

In this paper, we propose a novel and efficient two-stage variable selection approach for sparse GLARMA models, which are pervasive for modelling discrete-valued time series. Our approach consists in iteratively combining the estimation of the autoregressive moving average (ARMA) coefficients of GLARMA models with regularized methods designed for performing variable selection in regression coefficients of Generalized Linear Models (GLM). We first establish the consistency of the ARMA part coefficient estimators in a specific case. Then, we explain how to efficiently implement our approach. Finally, we assess the performance of our methodology using synthetic data, compare it with alternative methods and illustrate it on an example of real-world application. Our approach, which is implemented in the GlarmaVarSel R package, is very attractive since it benefits from a low computational load and is able to outperform the other methods in terms of recovering the non-null regression coefficients.

Detecting survival-associated biomarkers from heterogeneous populations

Detection of prognostic factors associated with patients’ survival outcome helps gain insights into a disease and guide treatment decisions. The rapid advancement of high-throughput technologies has yielded plentiful genomic biomarkers as candidate prognostic factors, but most are of limited use in clinical application. As the price of the technology drops over time, many genomic studies are conducted to explore a common scientific question in different cohorts to identify more reproducible and credible biomarkers. However, new challenges arise from heterogeneity in study populations and designs when jointly analyzing the multiple studies. For example, patients from different cohorts show different demographic characteristics and risk profiles. Existing high-dimensional variable selection methods for survival analysis, however, are restricted to single study analysis. We propose a novel Cox model based two-stage variable selection method called “Cox-TOTEM” to detect survival-associated biomarkers common in multiple genomic studies. Simulations showed our method greatly improved the sensitivity of variable selection as compared to the separate applications of existing methods to each study, especially when the signals are weak or when the studies are heterogeneous. An application of our method to TCGA transcriptomic data identified essential survival associated genes related to the common disease mechanism of five Pan-Gynecologic cancers.

A regularized functional regression model enabling transcriptome-wide dosage-dependent association study of cancer drug response.

Cancer treatments can be highly toxic and frequently only a subset of the patient population will benefit from a given treatment. Tumour genetic makeup plays an important role in cancer drug sensitivity. We suspect that gene expression markers could be used as a decision aid for treatment selection or dosage tuning. Using in vitro cancer cell line dose-response and gene expression data from the Genomics of Drug Sensitivity in Cancer (GDSC) project, we build a dose-varying regression model. Unlike existing approaches, this allows us to estimate dosage-dependent associations with gene expression. We include the transcriptomic profiles as dose-invariant covariates into the regression model and assume that their effect varies smoothly over the dosage levels. A two-stage variable selection algorithm (variable screening followed by penalized regression) is used to identify genetic factors that are associated with drug response over the varying dosages. We evaluate the effectiveness of our method using simulation studies focusing on the choice of tuning parameters and cross-validation for predictive accuracy assessment. We further apply the model to data from five BRAF targeted compounds applied to different cancer cell lines under different dosage levels. We highlight the dosage-dependent dynamics of the associations between the selected genes and drug response, and we perform pathway enrichment analysis to show that the selected genes play an important role in pathways related to tumorigenesis and DNA damage response.

Which bridge estimator is the best for variable selection?

We study the problem of variable selection for linear models under the high-dimensional asymptotic setting, where the number of observations $n$ grows at the same rate as the number of predictors $p$. We consider two-stage variable selection techniques (TVS) in which the first stage uses bridge estimators to obtain an estimate of the regression coefficients, and the second stage simply thresholds this estimate to select the “important” predictors. The asymptotic false discovery proportion ($\operatorname{AFDP}$) and true positive proportion (ATPP) of these TVS are evaluated. We prove that for a fixed ATPP, in order to obtain a smaller $\operatorname{AFDP}$, one should pick a bridge estimator with smaller asymptotic mean square error in the first stage of TVS. Based on such principled discovery, we present a sharp comparison of different TVS, via an in-depth investigation of the estimation properties of bridge estimators. Rather than “orderwise” error bounds with loose constants, our analysis focuses on precise error characterization. Various interesting signal-to-noise ratio and sparsity settings are studied. Our results offer new and thorough insights into high-dimensional variable selection. For instance, we prove that a TVS with Ridge in its first stage outperforms TVS with other bridge estimators in large noise settings; two-stage LASSO becomes inferior when the signal is rare and weak. As a by-product, we show that two-stage methods outperform some standard variable selection techniques, such as $\operatorname{LASSO}$ and Sure Independence Screening, under certain conditions.

Bayesian Variable Selection Using Spike-and-Slab Priors with Application to High Dimensional Electroencephalography Data by Local Modelling

SummaryBecause of the immense technological advances, very often we encounter data in high dimensions. Any set of measurements taken at multiple time points for multiple subjects leads to data of more than two dimensions (a matrix of covariates for each subject). We present a Bayesian variable-selection method to identify the active regions in the brain as a response to a certain stimulus. We build binary classification models of subject level responses by using binary regression with Gaussian models on the latent variables. We also study the scaled normal priors on the latent variables, as they cover a large family of distributions. We use continuous spike-and-slab priors to incorporate variable selection within the modelling. Because of the computational complexity, we build many local (at different time points) models and make predictions by utilizing the temporal structure between the local models. We perform two-stage variable selection for each of these local models. We demonstrate the effectiveness of such modelling through the results of a simulation study. We then present the performance of these models on multisubject neuroimaging (electroencephalography) data to study the effects on the functional states of the frontal cortex and parietal lobe for chronic exposure of alcohol.

Feature screening based on ultrahigh dimensional competing risks models

In survival analysis, variable selection methods have been proposed in the literature for ultrahigh dimensional Cox model which deals with standard time-to-event data. However, in the cases where subjects may suffer from multiple mutually exclusive events, or competing risks, these methods cannot be applied directly. A popular model for data with competing risks turns out to be proportional subdistribution hazard (PSH) model. In this paper, we develop a two-stage variable selection strategy based on sure joint screening (SJS) and penalized approximized log partial likelihood for ultrahigh PSH model, and establish the sure screening property for the proposed method; in other words, with probability tending to 1, the set of selected variable includes all actual significant covariates.The performance of our proposed method is demonstrated by Monte Carlo simulation and compared with sure independence screening for competing data. We also conduct an analysis of a public dataset about bladder cancer to illustrate our method.

A two-stage variable selection and classification approach for Parkinson’s disease detection by using voice recording replications

Background and ObjectiveIn the scientific literature, there is a lack of variable selection and classification methods considering replicated data. The problem motivating this work consists in the discrimination of people suffering Parkinson’s disease from healthy subjects based on acoustic features automatically extracted from replicated voice recordings. MethodsA two-stage variable selection and classification approach has been developed to properly match the replication-based experimental design. The way the statistical approach has been specified allows that the computational problems are solved by using an easy-to-implement Gibbs sampling algorithm. ResultsThe proposed approach produces an acceptable predictive capacity for PD discrimination with the considered database, despite the fact that the sample size is relatively small. Specifically, the accuracy rate, sensitivity and specificity are 86.2%, 82.5%, and 90.0%, respectively. However, the most important fact is that there is an improvement in the interpretability of the results at the same time that it is shown a better chain mixing and a lower computation time with respect to the only-classification approaches presented in the scientific literature. ConclusionsTo the best of the authors’ knowledge, this is the first approach developed to properly consider intra-subject variability for variable selection and classification. Although the proposed approach has been applied for PD discrimination, it can be applied in other contexts with similar replication-based experimental designs.

Feature screening and variable selection for partially linear models with ultrahigh-dimensional longitudinal data

This paper is concerned with longitudinal partially linear models (LPLM) with ultrahigh-dimensional covariates and predictors. As flexible extension of linear regression models by allowing nonparametric intercept function to capture the overall trend over time, the LPLM are expected to be highly potential statistical models for analyzing high-dimensional longitudinal data such as longitudinal genetic data and functional magnetic resonance image data. Feature screening and variable selection are indispensable for LPLM in the presence of ultrahigh-dimensional covariates such as genetic markers and all pixels in image data. This paper proposes a two-stage variable selection procedure that consists of a quick screening stage and a post-screening refining stage, for the ultrahigh dimensional longitudinal partially linear models. The proposed approach is based on the partial residual method for dealing with the nonparametric baseline function. We establish the sure screening property of the proposed screening procedure in the first stage. Simulation results demonstrate the validity of this two-stage method. We further demonstrate the proposed methodology by an empirical analysis of a real data set collected in a soybean plant longitudinal genetic study.

Beyond support in two-stage variable selection

Numerous variable selection methods rely on a two-stage procedure, where a sparsity-inducing penalty is used in the first stage to predict the support, which is then conveyed to the second stage for estimation or inference purposes. In this framework, the first stage screens variables to find a set of possibly relevant variables and the second stage operates on this set of candidate variables, to improve estimation accuracy or to assess the uncertainty associated to the selection of variables. We advocate that more information can be conveyed from the first stage to the second one: we use the magnitude of the coefficients estimated in the first stage to define an adaptive penalty that is applied at the second stage. We give the example of an inference procedure that highly benefits from the proposed transfer of information. The procedure is precisely analyzed in a simple setting, and our large-scale experiments empirically demonstrate that actual benefits can be expected in much more general situations, with sensitivity gains ranging from 50 to 100 % compared to state-of-the-art.

Variable selection in high-dimensional quantile varying coefficient models

In this paper, we propose a two-stage variable selection procedure for high dimensional quantile varying coefficient models. The proposed method is based on basis function approximation and LASSO-type penalties. We show that the first stage penalized estimator with LASSO penalty reduces the model from ultra-high dimensional to a model that has size close to the true model, but contains the true model as a valid sub model. By applying adaptive LASSO penalty to the reduced model, the second stage excludes the remained irrelevant covariates, leading to an estimator consistent in variable selection. A simulation study and the analysis of a real data demonstrate that the proposed method performs quite well in finite samples, with regard to dimension reduction and variable selection.

A two-stage variable selection strategy for supersaturated designs with multiple responses

A supersaturated design (SSD), whose run size is not enough for estimating all the main effects, is commonly used in screening experiments. It offers a potential useful tool to investigate a large number of factors with only a few experimental runs. The associated analysis methods have been proposed by many authors to identify active effects in situations where only one response is considered. However, there are often situations where two or more responses are observed simultaneously in one screening experiment, and the analysis of SSDs with multiple responses is thus needed. In this paper, we propose a two-stage variable selection strategy, called the multivariate partial least squares-stepwise regression (MPLS-SR) method, which uses the multivariate partial least squares regression in conjunction with the stepwise regression procedure to select true active effects in SSDs with multiple responses. Simulation studies show that the MPLS-SR method performs pretty good and is easy to understand and implement.

Two-Stage Procedures for High-Dimensional Data

In this article, we consider a variety of inference problems for high-dimensional data. The purpose of this article is to suggest directions for future research and possible solutions about p ≫ n problems by using new types of two-stage estimation methodologies. This is the first attempt to apply sequential analysis to high-dimensional statistical inference ensuring prespecified accuracy. We offer the sample size determination for inference problems by creating new types of multivariate two-stage procedures. To develop theory and methodologies, the most important and basic idea is the asymptotic normality when p → ∞. By developing asymptotic normality when p → ∞, we first give (a) a given-bandwidth confidence region for the square loss. In addition, we give (b) a two-sample test to assure prespecified size and power simultaneously together with (c) an equality-test procedure for two covariance matrices. We also give (d) a two-stage discriminant procedure that controls misclassification rates being no more than a prespecified value. Moreover, we propose (e) a two-stage variable selection procedure that provides screening of variables in the first stage and selects a significant set of associated variables from among a set of candidate variables in the second stage. Following the variable selection procedure, we consider (f) variable selection for high-dimensional regression to compare favorably with the lasso in terms of the assurance of accuracy and the computational cost. Further, we consider variable selection for classification and propose (g) a two-stage discriminant procedure after screening some variables. Finally, we consider (h) pathway analysis for high-dimensional data by constructing a multiple test of correlation coefficients.

Two-Stage Variable Selection Using the Wavelet Transform of Batch Trajectories for Data Interpretation and Construction of Parsimonious Quality-Estimation Models

We propose a two-stage variable-selection strategy performed in the wavelet domain in order to extract quality-related information from batch trajectories of process variables and to build parsimonious quality-estimation models. The proposed variable-selection method proceeds in two stages and uses the discrete wavelet transform of the batch trajectories. This approach greatly reduces the computation time required for finding those wavelet coefficients related to product quality. A quality-estimation model built with the selected wavelet coefficient subset is shown for the case study we discuss to exhibit satisfactory estimation accuracy. The application of the method to an industrial PVC polymerization process was found to achieve the same prediction accuracy as a multiway partial least-squares (MPLS) based model that uses all variables in the time domain.

Asymptotics of forced-in variables from screening processes

Freedman, Navidi and Peters (1988) had shown that when a two-stage variable selection procedure is used to screen linear regression equations, the type I error rate of the t-test for a forced-in-first estimator is inflated if the ratio of the number of parameters to number of data points is large, In this paper, we investigate the asymptotic distribution of the “force-in-last” estimator, and show that this estimator is bias toward zero asymptotically and its asymptotic standard error is close to the nominal standard error under certain conditions. To remedy the bias and reduce the inflation of the type I error rate, two estimators involved with propensity score (Rubin and Rosenbaum, 1983) were introduced.

Some variable selection procedures in multivariate linear regression models

In this paper, we propose a two-stage variable selection procedure for multivariate linear regression models. We select appropriate models under a guaranteed probability by using the summation of noncentralities in the first stage. In the second stage, we exclude those models with large individual noncentrality, and then select the best model with the minimum Akaike's information criterion (AIC). Empirical study is provided to show how to achieve our goal in variable selection and to demonstrate the efficiency and usefulness of the procedure in practical applications. In addition, we have built a reasonable model to explain and predict the earnings and productivity in Taiwan area.