New Variable Selection Method Research Articles

A modified Nadaraya–Watson procedure for variable selection and nonparametric prediction with missing data

We propose a new method for variable selection and prediction under a nonparametric regression setting, where a covariate may be missing either because its value is hidden from the observer or because it is inapplicable to the particular subject being observed. Despite its practical relevance, the problem has received little attention in the literature and its solutions are largely non-existent. Our proposal hinges on the construction of a modified Nadaraya–Watson estimator of the conditional mean regression function, with its bandwidths regularised to select variables and its weights adapted to accommodate different types of missingness. The method allows for information sharing across different missing data patterns without affecting consistency of the estimator. Unlike other conventional methods such as those based on imputations or likelihoods, our method requires only mild assumptions on the model and the missingness mechanism. For prediction we focus on finding relevant variables for predicting mean responses, conditional on covariate vectors subject to a given type of missingness. Our theoretical and numerical results show that the new method is consistent in variable selection and yields better prediction accuracy compared to existing methods.

Variable Selection Using aModified Gibbs Sampler Algorithm with Application on Rock Strength Dataset

Variable selection is an essential and necessary task in the statistical modeling field. Several studies have triedto develop and standardize the process of variable selection, but it isdifficultto do so. The first question a researcher needs to ask himself/herself what are the most significant variables that should be used to describe a given dataset’s response. In thispaper, a new method for variable selection using Gibbs sampler techniqueshas beendeveloped.First, the model is defined, and the posterior distributions for all the parameters are derived.The new variable selection methodis tested usingfour simulation datasets. The new approachiscompared with some existingtechniques: Ordinary Least Squared (OLS), Least Absolute Shrinkage and Selection Operator (Lasso), and Tikhonov Regularization (Ridge). The simulation studiesshow that the performance of our method is better than the othersaccording to the error and the time complexity. Thesemethodsare applied to a real dataset, which is called Rock StrengthDataset.The new approach implemented using the Gibbs sampler is more powerful and effective than other approaches.All the statistical computations conducted for this paper are done using R version 4.0.3 on a single processor computer.

Variable selection in nonlinear function-on-scalar regression.

We develop a new method for variable selection in a nonlinear additive function-on-scalar regression (FOSR) model. Existing methods for variable selection in FOSR have focused on the linear effects of scalar predictors, which can be a restrictive assumption in the presence of multiple continuously measured covariates. We propose a computationally efficient approach for variable selection in existing linear FOSR using functional principal component scores of the functional response and extend this framework to a nonlinear additive function-on-scalar model. The proposed method provides a unified and flexible framework for variable selection in FOSR, allowing nonlinear effects of the covariates. Numerical analysis using simulation study illustrates the advantages of the proposed method over existing variable selection methods in FOSR even when the underlying covariate effects are all linear. The proposed procedure is demonstrated on accelerometer data from the 2003-2004 cohorts of the National Health and Nutrition Examination Survey (NHANES) in understanding the association between diurnal patterns of physical activity and demographic, lifestyle, and health characteristics of the participants.

Variable selection in nonparametric functional concurrent regression

We develop a new method for variable selection in nonparametric functional concurrent regression. The commonly used functional linear concurrent model (FLCM) is far too restrictive in assuming linearity of the covariate effects, which is not necessarily true in many real‐world applications. The nonparametric functional concurrent model (NPFCM), on the other hand, is much more flexible and can capture complex dynamic relationships present between the response and the covariates. We extend the classically used variable selection methods, e.g., group LASSO, group SCAD and group MCP, to perform variable selection in NPFCM. We show via numerical simulations that the proposed variable selection method with the non‐convex penalties can identify the true functional predictors with minimal false‐positive rate and negligible false‐negative rate. The proposed method also provides better out‐of‐sample prediction accuracy compared to the FLCM in the presence of nonlinear effects of the functional predictors. The proposed method's application is demonstrated by identifying the influential predictor variables in two real data studies: a dietary calcium absorption study, and some bike‐sharing data.

SuRF: A new method for sparse variable selection, with application in microbiome data analysis.

In this article, we present a new variable selection method for regression and classification purposes, particularly for microbiome analysis. Our method, called subsampling ranking forward selection (SuRF), is based on LASSO penalized regression, subsampling and forward-selection methods. SuRF offers major advantages over existing variable selection methods in terms of both sparsity of selected models and model inference. We provide an R package that can implement our method for generalized linear models. We apply our method to classification problems from microbiome data, using a novel agglomeration approach to deal with the special tree-like correlation structure of the variables. Existing methods arbitrarily choose a taxonomic level a priori before performing the analysis, whereas by combining SuRF with these aggregated variables, we are able to identify the key biomarkers at the appropriate taxonomic level, as suggested by the data. We present simulations in multiple sparse settings to demonstrate that our approach performs better than several other popularly used existing approaches in recovering the true variables. We apply SuRF to two microbiome datasets: one about prediction of pouchitis and another for identifying samples from two healthy individuals. We find that SuRF can provide a better or comparable prediction with other methods while controlling the false positive rate of variable selection.

Variable selection from a feature representing protein sequences: a case of classification on bacterial type IV secreted effectors

BackgroundClassification of certain proteins with specific functions is momentous for biological research. Encoding approaches of protein sequences for feature extraction play an important role in protein classification. Many computational methods (namely classifiers) are used for classification on protein sequences according to various encoding approaches. Commonly, protein sequences keep certain labels corresponding to different categories of biological functions (e.g., bacterial type IV secreted effectors or not), which makes protein prediction a fantasy. As to protein prediction, a kernel set of protein sequences keeping certain labels certified by biological experiments should be existent in advance. However, it has been hardly ever seen in prevailing researches. Therefore, unsupervised learning rather than supervised learning (e.g. classification) should be considered. As to protein classification, various classifiers may help to evaluate the effectiveness of different encoding approaches. Besides, variable selection from an encoded feature representing protein sequences is an important issue that also needs to be considered.ResultsFocusing on the latter problem, we propose a new method for variable selection from an encoded feature representing protein sequences. Taking a benchmark dataset containing 1947 protein sequences as a case, experiments are made to identify bacterial type IV secreted effectors (T4SE) from protein sequences, which are composed of 399 T4SE and 1548 non-T4SE. Comparable and quantified results are obtained only using certain components of the encoded feature, i.e., position-specific scoring matix, and that indicates the effectiveness of our method.ConclusionsCertain variables other than an encoded feature they belong to do work for discrimination between different types of proteins. In addition, ensemble classifiers with an automatic assignment of different base classifiers do achieve a better classification result.

Graph Structured Sparse Subset Selection

We propose a new method for variable subset selection and regression coefficient estimation in linear regression models that incorporates a graph structure of the predictor variables. The proposed method is based on the cardinality constraint that controls the number of selected variables and the graph structured subset constraint that encourages the predictor variables adjacent in the graph to be simultaneously selected or eliminated from the model. Moreover, we develop an efficient discrete projected gradient descent method to handle the NP-hardness of the problem originating from the discrete constraints. Numerical experiments on simulated and real-world data are conducted to demonstrate the usefulness and applicability of the proposed method by comparing it with existing graph regularization methods in terms of the predictive accuracy and variable selection performance. The results confirm that the proposed method outperforms the existing methods.

An Variable Selection Method of the Significance Multivariate Correlation Competitive Population Analysis for Near-Infrared Spectroscopy in Chemical Modeling

The high dimensionality of spectral datasets makes it difficult to select the optimal subset of variables. This paper presents a new method for variable selection called the significant multivariate competitive population analysis (SMCPA), Which combines ideas of significant multivariate correlation (SMC) and model population analysis, and employs weighted bootstrap sampling (WBS) and exponential decline function (EDF) competition methods. In this study, the values of SMC distributions are used as an index for evaluating the importance of each wavelength. Then, based on the importance level of each wavelength. SMCPA sequentially selects N subsets of spectral wavelengths by N Monte Carlo sampling in an iterative and competitive procedure. In each sampling run, a fixed ratio of samples is used to build a calibrated partial least-squares model, and then SMC is performed to obtain the score and threshold values. Next, based on the significant multivariate correlation scores, the key variables are selected by two steps: the compulsory selection of exponential decline function and the competitive selection of adaptive weighted sampling. Finally, cross-validation(CV) is applied to select the optimal subset with the lowest root mean square error. This method is tested on three NIR spectral datasets and compared against three high-performance variable selection methods. The experimental results show that the proposed algorithm has the highest efficiency and the best selection effect, and can usually locate the optimal combination of key wavelength variables in a dataset. The evaluation result after PLS modeling is also the best.

A new variable selection method applied to credit scoring

Credit scoring (CS) is an important process in both banking and finance. Lenders or creditors have to use CS to predict the probability that a borrower will default or become delinquent. CS is usually based on variables related to the applicant such as: his age, his historical payments, his behavio r, etc. This paper first proposes a new method for variable selection. The proposed method (VS-VNS) is based on the variable neighborhood search meta-heuristic. VS-VNS allows us to select a set of significant variables for the data classification task. The VS-VNS is combined then with a Bayesian network (BN) to build models for CS and select counterparties. Further, six search methods are studied for BN on different sets of variables. The different techniques and combinations are evaluated on some well-known financial datasets. The numerical results are promising and show the benefits of the new proposed approach (VS-VNS) for data classification and credit scoring.

Binary quantile regression and variable selection: A new approach

ABSTRACTIn this paper, we propose a new estimation method for binary quantile regression and variable selection which can be implemented by an iteratively reweighted least square approach. In contrast to existing approaches, this method is computationally simple, guaranteed to converge to a unique solution and implemented with standard software packages. We demonstrate our methods using Monte-Carlo experiments and then we apply the proposed method to the widely used work trip mode choice dataset. The results indicate that the proposed estimators work well in finite samples.

A simple method for forward variable selection and calibration: evaluation for compact and low-cost laser-induced breakdown spectroscopy system.

This work presents a new method for forward variable selection and calibration and its evaluation for manganese determination in steel by laser-induced breakdown spectroscopy (LIBS). A compact and low-cost LIBS instrument was used, based on a microchip laser and a grating mini-spectrometer containing a non-intensified, non-gated, and non-cooled linear sensor array. Sixty steel samples were analyzed, with known manganese concentrations from 0.106 to 1.696wt%. The spectra (1757 variables between 200 and 850nm) were acquired under the continuous application of laser pulses at 100Hz and using 80, 400, and 1000ms integration times. The new method generated a mathematic combination of the selected variables and the results were calibrated against the manganese content by linear or quadratic regression. The best results were obtained using the spectra from all integration times together, with 31 selected variables and root mean square errors of cross-validation and prediction of 0.015 and 0.033, respectively. Compared to Jack-knife partial least squares regression, the new method presented lower prediction errors and numbers of selected variables, with the advantages of no data pretreatment and a simpler mathematic calculation. Graphical abstract New method for forward variable selection and calibration applied to manganese determination in steel by laser induced breakdown spectroscopy.

The model adaptive space shrinkage (MASS) approach: a new method for simultaneous variable selection and outlier detection based on model population analysis.

Variable selection and outlier detection are important processes in chemical modeling. Usually, they affect each other. Their performing orders also strongly affect the modeling results. Currently, many studies perform these processes separately and in different orders. In this study, we examined the interaction between outliers and variables and compared the modeling procedures performed with different orders of variable selection and outlier detection. Because the order of outlier detection and variable selection can affect the interpretation of the model, it is difficult to decide which order is preferable when the predictabilities (prediction error) of the different orders are relatively close. To address this problem, a simultaneous variable selection and outlier detection approach called Model Adaptive Space Shrinkage (MASS) was developed. This proposed approach is based on model population analysis (MPA). Through weighted binary matrix sampling (WBMS) from model space, a large number of partial least square (PLS) regression models were built, and the elite parts of the models were selected to statistically reassign the weight of each variable and sample. Then, the whole process was repeated until the weights of the variables and samples converged. Finally, MASS adaptively found a high performance model which consisted of the optimized variable subset and sample subset. The combination of these two subsets could be considered as the cleaned dataset used for chemical modeling. In the proposed approach, the problem of the order of variable selection and outlier detection is avoided. One near infrared spectroscopy (NIR) dataset and one quantitative structure-activity relationship (QSAR) dataset were used to test this approach. The result demonstrated that MASS is a useful method for data cleaning before building a predictive model.

A Shrinkage-Thresholding Metropolis Adjusted Langevin Algorithm for Bayesian Variable Selection

This paper introduces a new Markov Chain Monte Carlo method for Bayesian variable selection in high dimensional settings. The algorithm is a Hastings-Metropolis sampler with a proposal mechanism which combines a Metropolis Adjusted Langevin (MALA) step to propose local moves associated with a shrinkage-thresholding step allowing to propose new models. The geometric ergodicity of this new trans-dimensional Markov Chain Monte Carlo sampler is established. An extensive numerical experiment, on simulated and real data, is presented to illustrate the performance of the proposed algorithm in comparison with some more classical trans-dimensional algorithms.

Variable Selection for Sparse High-Dimensional Nonlinear Regression Models by Combining Nonnegative Garrote and Sure Independence Screening.

In many regression problems, the relations between the covariates and the response may be nonlinear. Motivated by the application of reconstructing a gene regulatory network, we consider a sparse high-dimensional additive model with the additive components being some known nonlinear functions with unknown parameters. To identify the subset of important covariates, we propose a new method for simultaneous variable selection and parameter estimation by iteratively combining a large-scale variable screening (the nonlinear independence screening, NLIS) and a moderate-scale model selection (the nonnegative garrote, NNG) for the nonlinear additive regressions. We have shown that the NLIS procedure possesses the sure screening property and it is able to handle problems with non-polynomial dimensionality; and for finite dimension problems, the NNG for the nonlinear additive regressions has selection consistency for the unimportant covariates and also estimation consistency for the parameter estimates of the important covariates. The proposed method is applied to simulated data and a real data example for identifying gene regulations to illustrate its numerical performance.

A multilayer-perceptron based method for variable selection in soft sensor design

The paper proposes a new method for variable selection for prediction settings and soft sensors applications. The new variable selection method is based on the multi-layer perceptron (MLP) neural network model, where the network is trained a single time, maintaining low computational cost. The proposed method was successfully applied, and compared with four state-of-the-art methods in one artificial dataset and three real-world datasets, two publicly available datasets (Box–Jenkins gas furnace and gas mileage), and a dataset of a problem where the objective is to estimate the fluoride concentration in the effluent of a real urban water treatment plant (WTP). The proposed method presents similar or better approximation performance when compared to the other four methods. In the experiments, among all the five methods, the proposed method selects the lowest number of variables and variables-delays pairs to achieve the best solution. In soft sensors applications having a lower number of variables is a positive factor for decreasing implementation costs, or even making the soft sensor feasible at all.

Incorporating group correlations in genome-wide association studies using smoothed group Lasso.

In genome-wide association studies, penalization is an important approach for identifying genetic markers associated with disease. Motivated by the fact that there exists natural grouping structure in single nucleotide polymorphisms and, more importantly, such groups are correlated, we propose a new penalization method for group variable selection which can properly accommodate the correlation between adjacent groups. This method is based on a combination of the group Lasso penalty and a quadratic penalty on the difference of regression coefficients of adjacent groups. The new method is referred to as smoothed group Lasso (SGL). It encourages group sparsity and smoothes regression coefficients for adjacent groups. Canonical correlations are applied to the weights between groups in the quadratic difference penalty. We first derive a GCD algorithm for computing the solution path with linear regression model. The SGL method is further extended to logistic regression for binary response. With the assistance of the majorize-minimization algorithm, the SGL penalized logistic regression turns out to be an iteratively penalized least-square problem. We also suggest conducting principal component analysis to reduce the dimensionality within groups. Simulation studies are used to evaluate the finite sample performance. Comparison with group Lasso shows that SGL is more effective in selecting true positives. Two datasets are analyzed using the SGL method.

SimSel: a new simulation method for variable selection

We propose a new simulation method, SimSel, for variable selection in linear and nonlinear modelling problems. SimSel works by disturbing the input data with pseudo-errors. We then study how this disturbance affects the quality of an approximative model fitted to the data. The main idea is that disturbing unimportant variables does not affect the quality of the model fit. The use of an approximative model has the advantage that the true underlying function does not need to be known and that the method becomes insensitive to model misspecifications. We demonstrate SimSel on simulated data from linear and nonlinear models and on two real data sets. The simulation studies suggest that SimSel works well in complicated situations, such as nonlinear errors-in-variable models.

Biomarker discovery in mass spectral profiles by means of selectivity ratio plot

This work presents a new method for variable selection in complex spectral profiles. The method is validated by comparing samples from cerebrospinal fluid (CSF) with the same samples spiked with peptide and protein standards at different concentration levels. Partial least squares discriminant analysis (PLS-DA) attempts to separate two groups of samples by regressing on a y -vector consisting of zeros and ones in the PLS decomposition. In most cases, several PLS components are needed to optimize the discrimination between groups. This creates difficulties for the interpretation of the model. By using the y -vector as a target, it is possible to transform the PLS components to obtain a single predictive target-projected component analogously to the predictive component in orthogonal partial least squares discriminant analysis (OPLS-DA). By calculating the ratio between explained and residual variance of the spectral variables on the target-projected component, a selectivity ratio plot is obtained that can be used for variable selection. Used on whole mass spectral profiles of pure and spiked CSF, we can detect peptide in the low molecular mass range (740–9000 Da) at least down to 400 pM level without severe problems with false biomarker candidates. Similarly, we detect added proteins at least down to 2 nM level in the medium mass range (6000–17,500 Da). Target projection represents the optimal way to fit a latent variable decomposition to a known target, but the selectivity ratio plot can be used for OPLS as well as other methods that produce a single predictive component. Comparison with some commonly used tools for variable selection shows that the selectivity ratio plot has the best performance. This observation is attributed to the fact that target projection utilizes both the predictive ability (regression coefficients) and the explanatory ability (spectral variance/covariance matrix) for the calculation of the selectivity ratio.

Exponential-Bound Property of Estimators and Variable Selection in Generalized Additive Models

We discuss the asymptotic property of estimators for generalized additive models based on a marginal integration estimation method introduced by Linton and Nielsen (1995). General results of the mean error exponential bound property are established. Applying these results, a new method for variable selection in generalized additive models is introduced. We directly test if a additive component is significant. We use the wild-bootstrap method (Wu, 1986) for choosing threshold of the test. This procedure is applied to each design variable one at one time. Comparisons are made to demonstrate the practical advantages of the proposed method over the other existing methods.

Parsimonious additive models

A new method for function estimation and variable selection, specifically designed for additive models fitted by cubic splines is proposed. This new method involves regularizing additive models using the l 1 -norm, which generalizes the lasso to the nonparametric setting. As in the linear case, it shrinks coefficients and produces some coefficients that are exactly zero. It gives parsimonious models, selects significant variables, and reveals nonlinearities in the effects of predictors. Two strategies for finding a parsimonious additive model solution are proposed. Both algorithms are based on a fixed point algorithm, combined with a singular value decomposition that considerably reduces computation. The empirical behavior of parsimonious additive models is compared to the adaptive backfitting BRUTO algorithm. The results allow to characterize the domains in which our approach is effective: it performs significantly better than BRUTO when model estimation is challenging. An implementation of this method is illustrated using real data from the Cophar 1 ANRS 102 trial. Parsimonious additive models are applied to predict the indinavir plasma concentration in HIV patients. Results suggest that this new method is a promising technique for the research and application areas.