Variable Selection Property Research Articles

Tabular deep learning: a comparative study applied to multi-task genome-wide prediction

PurposeMore accurate prediction of phenotype traits can increase the success of genomic selection in both plant and animal breeding studies and provide more reliable disease risk prediction in humans. Traditional approaches typically use regression models based on linear assumptions between the genetic markers and the traits of interest. Non-linear models have been considered as an alternative tool for modeling genomic interactions (i.e. non-additive effects) and other subtle non-linear patterns between markers and phenotype. Deep learning has become a state-of-the-art non-linear prediction method for sound, image and language data. However, genomic data is better represented in a tabular format. The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports successful results on various datasets. Tabular deep learning applications in genome-wide prediction (GWP) are still rare. In this work, we perform an overview of the main families of recent deep learning architectures for tabular data and apply them to multi-trait regression and multi-class classification for GWP on real gene datasets.MethodsThe study involves an extensive overview of recent deep learning architectures for tabular data learning: NODE, TabNet, TabR, TabTransformer, FT-Transformer, AutoInt, GANDALF, SAINT and LassoNet. These architectures are applied to multi-trait GWP. Comprehensive benchmarks of various tabular deep learning methods are conducted to identify best practices and determine their effectiveness compared to traditional methods.ResultsExtensive experimental results on several genomic datasets (three for multi-trait regression and two for multi-class classification) highlight LassoNet as a standout performer, surpassing both other tabular deep learning models and the highly efficient tree based LightGBM method in terms of both best prediction accuracy and computing efficiency.ConclusionThrough series of evaluations on real-world genomic datasets, the study identifies LassoNet as a standout performer, surpassing decision tree methods like LightGBM and other tabular deep learning architectures in terms of both predictive accuracy and computing efficiency. Moreover, the inherent variable selection property of LassoNet provides a systematic way to find important genetic markers that contribute to phenotype expression.

Prediction-based variable selection for component-wise gradient boosting.

Model-based component-wise gradient boosting is a popular tool for data-driven variable selection. In order to improve its prediction and selection qualities even further, several modifications of the original algorithm have been developed, that mainly focus on different stopping criteria, leaving the actual variable selection mechanism untouched. We investigate different prediction-based mechanisms for the variable selection step in model-based component-wise gradient boosting. These approaches include Akaikes Information Criterion (AIC) as well as a selection rule relying on the component-wise test error computed via cross-validation. We implemented the AIC and cross-validation routines for Generalized Linear Models and evaluated them regarding their variable selection properties and predictive performance. An extensive simulation study revealed improved selection properties whereas the prediction error could be lowered in a real world application with age-standardized COVID-19 incidence rates.

Robust adaptive variable selection in ultra-high dimensional linear regression models

We consider the problem of simultaneous variable selection and parameter estimation in an ultra-high dimensional linear regression model. The adaptive penalty functions are used in this regard to achieve the oracle variable selection property with simpler assumptions and lesser computational burden. Noting the non-robust nature of the usual adaptive procedures (e.g. adaptive LASSO) based on the squared error loss function against data contamination, quite frequent with modern large-scale data sets (e.g. noisy gene expression data, spectra and spectral data), in this paper, we present a new adaptive regularization procedure using a robust loss function based on the density power divergence (DPD) measure under a general class of error distributions. We theoretically prove that the proposed adaptive DPD-LASSO estimator of the regression coefficients is highly robust, consistent, asymptotically normal and leads to robust oracle-consistent variable selection under easily verifiable assumptions. Numerical illustrations are provided for the mostly used normal and heavy-tailed error densities. Finally, the proposal is applied to analyse an interesting spectral dataset, in the field of chemometrics, regarding the electron-probe X-ray microanalysis (EPXMA) of archaeological glass vessels from the 16th and 17th centuries.

Robust variable selection and estimation via adaptive elastic net S-estimators for linear regression

Heavy-tailed error distributions and predictors with anomalous values are ubiquitous in high-dimensional regression problems and can seriously jeopardize the validity of statistical analyses if not properly addressed. For more reliable variable selection and prediction under these adverse conditions, adaptive PENSE, a new robust regularized regression estimator, is proposed. Adaptive PENSE yields reliable variable selection and coefficient estimates even under aberrant contamination in the predictors or residuals. It is shown that the adaptive penalty leads to more robust and reliable variable selection than other penalties, particularly in the presence of gross outliers in the predictor space. It is further demonstrated that adaptive PENSE has strong variable selection properties and that it possesses the oracle property even under heavy-tailed errors and without the need to estimate the error scale. Numerical studies on simulated and real data sets highlight the superior finite-sample performance in a vast range of settings compared to other robust regularized estimators in the case of contaminated samples. An R package implementing a fast algorithm for computing the proposed method and additional simulation results are provided in the supplementary materials.

Lasso regression under stochastic restrictions in linear regression: An application to genomic data

Variable selection approaches are often employed in high-dimensionality and multicollinearity problems. Since lasso selects variables by shrinking the coefficients, it has extensive use in many fields. On the other, we may sometime have extra information on the model. In this case, the extra information should be considered in the estimation procedure. In this paper, we propose a stochastic restricted lasso estimator in linear regression model which uses the extra information as stochastic linear restrictions. The estimator is a generalization of mixed estimator with L 1 type penalization. We give the coordinate descent algorithm to estimate the coefficient vector of the proposed method and strong rules for the coordinate descent algorithm to discard variables from the model. Also, we propose a method to estimate the tuning parameter. We conduct two real data analyses and simulation studies to compare the new estimator with several estimators including the ridge, lasso and stochastic restricted ridge. The real data analyses and simulation studies show that the new estimator enjoys the automatic variable selection property of the lasso while outperforms standard methods, achieving lower test mean squared error.

Joint Modelling Approaches to Survival Analysis via Likelihood-Based Boosting Techniques.

Joint models are a powerful class of statistical models which apply to any data where event times are recorded alongside a longitudinal outcome by connecting longitudinal and time-to-event data within a joint likelihood allowing for quantification of the association between the two outcomes without possible bias. In order to make joint models feasible for regularization and variable selection, a statistical boosting algorithm has been proposed, which fits joint models using component-wise gradient boosting techniques. However, these methods have well-known limitations, i.e., they provide no balanced updating procedure for random effects in longitudinal analysis and tend to return biased effect estimation for time-dependent covariates in survival analysis. In this manuscript, we adapt likelihood-based boosting techniques to the framework of joint models and propose a novel algorithm in order to improve inference where gradient boosting has said limitations. The algorithm represents a novel boosting approach allowing for time-dependent covariates in survival analysis and in addition offers variable selection for joint models, which is evaluated via simulations and real world application modelling CD4 cell counts of patients infected with human immunodeficiency virus (HIV). Overall, the method stands out with respect to variable selection properties and represents an accessible way to boosting for time-dependent covariates in survival analysis, which lays a foundation for all kinds of possible extensions.

More than one way: exploring the capabilities of different estimation approaches to joint models for longitudinal and time-to-event outcomes.

The development of physical functioning after a caesura in an aged population is still widely unexplored. Analysis of this topic would need to model the longitudinal trajectories of physical functioning and simultaneously take terminal events (deaths) into account. Separate analysis of both results in biased estimates, since it neglects the inherent connection between the two outcomes. Thus, this type of data generating process is best modelled jointly. To facilitate this several software applications were made available. They differ in model formulation, estimation technique (likelihood-based, Bayesian inference, statistical boosting) and a comparison of the different approaches is necessary to identify their capabilities and limitations. Therefore, we compared the performance of the packages JM, joineRML, JMbayes and JMboost of the R software environment with respect to estimation accuracy, variable selection properties and prediction precision. With these findings we then illustrate the topic of physical functioning after a caesura with data from the German ageing survey (DEAS). The results suggest that in smaller data sets and theory driven modelling likelihood-based methods (expectation maximation, JM, joineRML) or Bayesian inference (JMbayes) are preferable, whereas statistical boosting (JMboost) is a better choice with high-dimensional data and data exploration settings.

Expectation propagation on the diluted Bayesian classifier.

Efficient feature selection from high-dimensional datasets is a very important challenge in many data-driven fields of science and engineering. We introduce a statistical mechanics inspired strategy that addresses the problem of sparse feature selection in the context of binary classification by leveraging a computational scheme known as expectation propagation (EP). The algorithm is used in order to train a continuous-weights perceptron learning a classification rule from a set of (possibly partly mislabeled) examples provided by a teacher perceptron with diluted continuous weights. We test the method in the Bayes optimal setting under a variety of conditions and compare it to other state-of-the-art algorithms based on message passing and on expectation maximization approximate inference schemes. Overall, our simulations show that EP is a robust and competitive algorithm in terms of variable selection properties, estimation accuracy, and computational complexity, especially when the student perceptron is trained from correlated patterns that prevent other iterative methods from converging. Furthermore, our numerical tests demonstrate that the algorithm is capable of learning online the unknown values of prior parameters, such as the dilution level of the weights of the teacher perceptron and the fraction of mislabeled examples, quite accurately. This is achieved by means of a simple maximum likelihood strategy that consists in minimizing the free energy associated with the EP algorithm.

Classification of high‐dimensional electroencephalography data with location selection using structured spike‐and‐slab prior

AbstractWith the advent of modern technologies, it is increasingly common to deal with data of large dimensions in various scientific fields of study. In this paper, we develop a Bayesian approach for the classification of multi‐subject high‐dimensional electroencephalography (EEG) data. In this EEG data, we have a matrix of covariates corresponding to each subject from either the alcoholic or control group. The matrix covariates have a natural spatial correlation based on the locations of the brain, and temporal correlation as the measurements are taken over time. We employ a divide and conquer strategy by building multiple local Bayesian models at each time point separately. We incorporate the spatial structure through the structured spike‐and‐slab prior, which has inherent variable selection properties. The temporal structure is incorporated within the prior by learning from the local model from the previous time point. We pool the information from the local models and use a weighted average to design a prediction method. We perform simulation studies to show the efficiency of our approach and demonstrate the local Bayesian modeling with a case study on EEG data.

Logistic regression with missing covariates—Parameter estimation, model selection and prediction within a joint-modeling framework

Logistic regression is a common classification method in supervised learning. Surprisingly, there are very few solutions for performing logistic regression with missing values in the covariates. A complete approach based on a stochastic approximation version of the EM algorithm is proposed in order to perform statistical inference with missing values, including the estimation of the parameters and their variance, derivation of confidence intervals, and also a model selection procedure. The problem of prediction for new observations on a test set with missing covariate data is also tackled. Supported by a simulation study in which the method is compared to previous ones, it has proved to be computationally efficient, and has good coverage and variable selection properties. The approach is then illustrated on a dataset of severely traumatized patients from Paris hospitals by predicting the occurrence of hemorrhagic shock, a leading cause of early preventable death in severe trauma cases. The aim is to improve the current red flag procedure, a binary alert identifying patients with a high risk of severe hemorrhage. The method is implemented in the R package misaem.

Bayesian Adaptive Lasso binary regression with ridge parameter

The variable selection (VS) characteristic was considered very important in the data analysis. Regularization technique is one gorgeous way that has proven effective for dealing with high dimensional data. In previous years, statistical researchers have made great efforts in developing procedures of regularization to solved problems of VS. In this paper, we have proposed a new technique for model selection in Binary regression. This technique is Bayesian adaptive Lasso Binary regression (BALBR). It has many features that give optimum estimation and VS property. Specifically, we introduced a new hierarchal model. Then, a new Gibbs sampler method is introduced. We also extend the new approach by adding the ridge parameter inside the variance-covariance matrix to avoid the singularity in case of multicollinearity or in case the number of observations less than the number of predictors. A comparison was made with other previous techniques applying the simulation examples and real data. It is worth mentioning, that the obtained results were promising and encouraging, giving better results compared to the previous methods.

Bayesian Lasso Tobit regression

In the present research, we have proposed a new approach for model selection in Tobit regression. The new technique uses Bayesian Lasso in Tobit regression (BLTR). It has many features that give optimum estimation and variable selection property. Specifically, we introduced a new hierarchal model. Then, a new Gibbs sampler is introduced.We also extend the new approach by adding the ridge parameter inside the variance covariance matrix to avoid the singularity in the case of multicollinearity or in case the number of predictors greater than the number of observations. A comparison was made with other previous techniques applying the simulation examples and real data. It is worth mentioning, that the obtained results were promising and encouraging, giving better results compared to the previous methods.

Additive monotone regression in high and lower dimensions

In numerous problems where the aim is to estimate the effect of a predictor variable on a response, one can assume a monotone relationship. For example, dose-effect models in medicine are of this type. In a multiple regression setting, additive monotone regression models assume that each predictor has a monotone effect on the response. In this paper, we present an overview and comparison of very recent frequentist methods for fitting additive monotone regression models. Three of the methods we present can be used both in the high dimensional setting, where the number of parameters $p$ exceeds the number of observations $n$, and in the classical multiple setting where $1<p\leq n$. However, many of the most recent methods only apply to the classical setting. The methods are compared through simulation experiments in terms of efficiency, prediction error and variable selection properties in both settings, and they are applied to the Boston housing data. We conclude with some recommendations on when the various methods perform best.

Boosting the discriminatory power of sparse survival models via optimization of the concordance index and stability selection.

BackgroundWhen constructing new biomarker or gene signature scores for time-to-event outcomes, the underlying aims are to develop a discrimination model that helps to predict whether patients have a poor or good prognosis and to identify the most influential variables for this task. In practice, this is often done fitting Cox models. Those are, however, not necessarily optimal with respect to the resulting discriminatory power and are based on restrictive assumptions. We present a combined approach to automatically select and fit sparse discrimination models for potentially high-dimensional survival data based on boosting a smooth version of the concordance index (C-index). Due to this objective function, the resulting prediction models are optimal with respect to their ability to discriminate between patients with longer and shorter survival times. The gradient boosting algorithm is combined with the stability selection approach to enhance and control its variable selection properties.ResultsThe resulting algorithm fits prediction models based on the rankings of the survival times and automatically selects only the most stable predictors. The performance of the approach, which works best for small numbers of informative predictors, is demonstrated in a large scale simulation study: C-index boosting in combination with stability selection is able to identify a small subset of informative predictors from a much larger set of non-informative ones while controlling the per-family error rate. In an application to discover biomarkers for breast cancer patients based on gene expression data, stability selection yielded sparser models and the resulting discriminatory power was higher than with lasso penalized Cox regression models.ConclusionThe combination of stability selection and C-index boosting can be used to select small numbers of informative biomarkers and to derive new prediction rules that are optimal with respect to their discriminatory power. Stability selection controls the per-family error rate which makes the new approach also appealing from an inferential point of view, as it provides an alternative to classical hypothesis tests for single predictor effects. Due to the shrinkage and variable selection properties of statistical boosting algorithms, the latter tests are typically unfeasible for prediction models fitted by boosting.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-016-1149-8) contains supplementary material, which is available to authorized users.

Variable selection of linear programming discriminant estimator

ABSTRACTIn this paper, the variable selection property will be studied for the linear programming discriminant (LPD) estimator, denoted by with n being the sample size. The LPD estimator is used in high-dimensional linear discriminant analysis under the assumption that the Bayes direction is sparse which has support T. More exactly, we will study the property as n → ∞, which means sign consistency. A sufficient condition will be proposed under which the sign consistency property holds as log (p) ⩽ cn for small enough c > 0. The result is also non asymptotic. Our result gives optimal bounds on n and min a ∈ T|βa| and an optimal bound on .

Prediction consistency of forward iterated regression and selection technique

Recently, Hwang et al. (2009) introduced a penalized forward selection technique for high dimensional linear regression which appears to possess excellent prediction and variable selection properties. In this article, we show that the procedure is prediction consistent.

Variable Selection in Nonparametric Classification Via Measurement Error Model Selection Likelihoods

Using the relationships among ridge regression, LASSO estimation, and measurement error attenuation as motivation, a new measurement-error-model-based approach to variable selection is developed. After describing the approach in the familiar context of linear regression, we apply it to the problem of variable selection in nonparametric classification, resulting in a new kernel-based classifier with LASSO-like shrinkage and variable-selection properties. Finite-sample performance of the new classification method is studied via simulation and real data examples, and consistency of the method is studied theoretically. Supplementary materials for the article are available online.

Auxiliary Marker-Assisted Classification in the Absence of Class Identifiers

Constructing classification rules for accurate diagnosis of a disorder is an important goal in medical practice. In many clinical applications, there is no clinically significant anatomical or physiological deviation that exists to identify the gold standard disease status to inform development of classification algorithms. Despite the absence of perfect disease class identifiers, there are usually one or more disease-informative auxiliary markers along with feature variables that comprise known symptoms. Existing statistical learning approaches do not effectively draw information from auxiliary prognostic markers. We propose a large margin classification method, with particular emphasis on the support vector machine, assisted by available informative markers to classify disease without knowing a subject's true disease status. We view this task as statistical learning in the presence of missing data, and introduce a pseudo-Expectation-Maximization (EM) algorithm to the classification. A major difference between a regular EM algorithm and the algorithm proposed here is that we do not model the distribution of missing data given the observed feature variables either parametrically or semiparametrically. We also propose a sparse variable selection method embedded in the pseudo-EM algorithm. Theoretical examination shows that the proposed classification rule is Fisher consistent, and that under a linear rule, the proposed selection has an oracle variable selection property and the estimated coefficients are asymptotically normal. We apply the methods to build decision rules for including subjects in clinical trials of a new psychiatric disorder and present four applications to data available at the University of California, Irvine Machine Learning Repository.

Prediction intervals for future BMI values of individual children - a non-parametric approach by quantile boosting

BackgroundThe construction of prediction intervals (PIs) for future body mass index (BMI) values of individual children based on a recent German birth cohort study with n = 2007 children is problematic for standard parametric approaches, as the BMI distribution in childhood is typically skewed depending on age.MethodsWe avoid distributional assumptions by directly modelling the borders of PIs by additive quantile regression, estimated by boosting. We point out the concept of conditional coverage to prove the accuracy of PIs. As conditional coverage can hardly be evaluated in practical applications, we conduct a simulation study before fitting child- and covariate-specific PIs for future BMI values and BMI patterns for the present data.ResultsThe results of our simulation study suggest that PIs fitted by quantile boosting cover future observations with the predefined coverage probability and outperform the benchmark approach. For the prediction of future BMI values, quantile boosting automatically selects informative covariates and adapts to the age-specific skewness of the BMI distribution. The lengths of the estimated PIs are child-specific and increase, as expected, with the age of the child.ConclusionsQuantile boosting is a promising approach to construct PIs with correct conditional coverage in a non-parametric way. It is in particular suitable for the prediction of BMI patterns depending on covariates, since it provides an interpretable predictor structure, inherent variable selection properties and can even account for longitudinal data structures.

Identifying Risk Factors for Severe Childhood Malnutrition by Boosting Additive Quantile Regression

We investigated the risk factors for childhood malnutrition in India based on the 2005/2006 Demographic and Health Survey by applying a novel estimation technique for additive quantile regression. Ordinary linear and generalized linear regression models relate the mean of a response variable to a linear combination of covariate effects, and, as a consequence, focus on average properties of the response. The use of such a regression model for analyzing childhood malnutrition in developing or transition countries implies that the estimated effects describe the average nutritional status. However, it is of even greater interest to analyze quantiles of the response distribution, such as the 5% or 10% quantile, which relate to the risk of extreme malnutrition. Our investigation is based on a semiparametric extension of quantile regression models where different types of nonlinear effects are included in the model equation, leading to additive quantile regression. We addressed the variable selection and model choice problems associated with estimating such an additive quantile regression model using a novel boosting approach. Our proposal allows for data-driven determination of the amount of smoothness required for the nonlinear effects and combines model choice with an automatic variable selection property. In an empirical evaluation, we compared our boosting approach with state-of-the-art methods for additive quantile regression. The results suggest that boosting is an appropriate tool for estimation and variable selection in additive quantile regression models and helps to identify yet unknown risk factors for childhood malnutrition. This article has supplementary material online.