Framework For Variable Selection Research Articles

A novel variable selection algorithm based on neural network for near-infrared spectral modeling

BackgroundPartial least squares (PLS) is a widely used technique for modeling spectral data. Researchers have developed numerous PLS-based variable selection algorithms to enhance model predictive ability and interpretability. In recent years, as neural network technology has advanced, these algorithms have been increasingly applied to spectral data modeling. However, current research on neural network modeling tends to prioritize network structure over variable selection. ResultsOur study introduces a neural network-based variable selection algorithm called VSNN (Variable Selection based on Neural Network). By iteratively eliminating unimportant variables using an exponentially decreasing function (EDF), the algorithm achieves the selection of variables in spectral data. VSNN can easily integrate different types of neural networks. In this study, we analyzed the impact of neural network types, activation functions, and variable importance vectors on algorithm performance. We tested the algorithm on four datasets: corn moisture, corn oil, tablets, and meat. The results indicate that VSNN significantly enhances the predictive ability of the model compared to partial least squares (PLS), neural networks (NN), and Joint Mutual Information Maximisation (JMIM). Specifically, non-linear activation functions markedly improve performance on non-linear meat datasets. Compared to PLS, the Root Mean Square Error of Prediction (RMSEP) values for the four datasets—corn moisture, corn oil, tablets, and meat—decreased from 0.0409, 0.0728, 3.97, and 3.2 to 0.002, 0.0236, 3.12, and 0.36, respectively, after applying the VSNN variable selection algorithm. SignificanceVSNN can serve as a versatile framework to enhance variable selection, modeling, and predictive performance by adapting neural network types and variable importance evaluation indicators. As machine learning technology advances, the strength of VSNN is poised to increase. This study highlights the potential of VSNN as an effective algorithmic framework for variable selection in spectroscopy applications.

Discussion of the 2023 Hansen Lecture: “Model Selection and Its Important Roles in Surveys”

Abstract The 31st Morris Hansen Lecture gave a broad overview of some fundamental issues regarding the roles of statistical models in surveys, with an emphasis on a method of model selection known as the fence. This discussion provides some general lecture remarks and expands on the possible applicability of the fence methods in small area estimation. Unit-level and area-level small area estimation modeling illustrations are provided. A general variable selection framework for small area estimation is presented, along with recent application studies and ideas for expansion to include the fence methods.

A Bayesian Multiplex Graph Classifier of Functional Brain Connectivity Across Diverse Tasks of Cognitive Control.

This article seeks to investigate the impact of aging on functional connectivity across different cognitive control scenarios, particularly emphasizing the identification of brain regions significantly associated with early aging. By conceptualizing functional connectivity within each cognitive control scenario as a graph, with brain regions as nodes, the statistical challenge revolves around devising a regression framework to predict a binary scalar outcome (aging or normal) using multiple graph predictors. Popular regression methods utilizing multiplex graph predictors often face limitations in effectively harnessing information within and across graph layers, leading to potentially less accurate inference and predictive accuracy, especially for smaller sample sizes. To address this challenge, we propose the Bayesian Multiplex Graph Classifier (BMGC). Accounting for multiplex graph topology, our method models edge coefficients at each graph layer using bilinear interactions between the latent effects associated with the two nodes connected by the edge. This approach also employs a variable selection framework on node-specific latent effects from all graph layers to identify influential nodes linked to observed outcomes. Crucially, the proposed framework is computationally efficient and quantifies the uncertainty in node identification, coefficient estimation, and binary outcome prediction. BMGC outperforms alternative methods in terms of the aforementioned metrics in simulation studies. An additional BMGC validation was completed using an fMRI study of brain networks in adults. The proposed BMGC technique identified that sensory motor brain network obeys certain lateral symmetries, whereas the default mode network exhibits significant brain asymmetries associated with early aging.

Automated Bayesian variable selection methods for binary regression models with missing covariate data

Data collection and the availability of large data sets has increased over the last decades. In both statistical and machine learning frameworks, two methodological issues typically arise when performing regression analysis on large data sets. First, variable selection is crucial in regression modeling, as it helps to identify an appropriate model with respect to the considered set of conditioning variables. Second, especially in the context of survey data, handling of missing values is important for estimation, which occur even with state-of-the-art data collection and processing methods. Within this paper, we provide an Bayesian approach based on a spike-and-slab prior for the regression coefficients, which allows for simultaneous handling of variable selection and estimation in combination with handling of missing values in covariate data. The paper also discusses the implementation of the approach using Markov chain Monte Carlo techniques and provides results for simulated data sets and an empirical illustration based on data from the German National Educational Panel Study. The suggested Bayesian approach is compared to other statistical and machine learning frameworks such as Lasso, ridge regression, and Elastic net, and is shown to perform well in terms of estimation performance and variable selection accuracy. The simulation results demonstrate that ignoring the handling of missing values in data sets can lead to the generation of biased selection results. Overall, the proposed Bayesian method offers a holistic, flexible, and powerful framework for variable selection in the presence of missing covariate data.

Structured learning in time-dependent Cox models.

Cox models with time-dependent coefficients and covariates are widely used in survival analysis. In high-dimensional settings, sparse regularization techniques are employed for variable selection, but existing methods for time-dependent Cox models lack flexibility in enforcing specific sparsity patterns (ie, covariate structures). We propose a flexible framework for variable selection in time-dependent Cox models, accommodating complex selection rules. Our method can adapt to arbitrary grouping structures, including interaction selection, temporal, spatial, tree, and directed acyclic graph structures. It achieves accurate estimation with low false alarm rates. We develop the sox package, implementing a network flow algorithm for efficiently solving models with complex covariate structures. sox offers a user-friendly interface for specifying grouping structures and delivers fast computation. Through examples, including a case study on identifying predictors of time to all-cause death in atrial fibrillation patients, we demonstrate the practical application of our method with specific selection rules.

Enhancing credit scoring accuracy with a comprehensive evaluation of alternative data.

This study explores the potential of utilizing alternative data sources to enhance the accuracy of credit scoring models, compared to relying solely on traditional data sources, such as credit bureau data. A comprehensive dataset from the Home Credit Group's home loan portfolio is analysed. The research examines the impact of incorporating alternative predictors that are typically overlooked, such as an applicant's social network default status, regional economic ratings, and local population characteristics. The modelling approach applies the model-X knockoffs framework for systematic variable selection. By including these alternative data sources, the credit scoring models demonstrate improved predictive performance, achieving an area under the curve metric of 0.79360 on the Kaggle Home Credit default risk competition dataset, outperforming models that relied solely on traditional data sources, such as credit bureau data. The findings highlight the significance of leveraging diverse, non-traditional data sources to augment credit risk assessment capabilities and overall model accuracy.

All that Glitters Is not Gold: Type-I Error Controlled Variable Selection from Clinical Trial Data.

Clinical trials are primarily conducted to estimate causal effects, but the data collected can also be invaluable for additional research, such as identifying prognostic measures of disease or biomarkers that predict treatment efficacy. However, these exploratory settings are prone to false discoveries (type-I errors) due to the multiple comparisons they entail. Unfortunately, many methods fail to address this issue, in part because the algorithms used are generally designed to optimize predictions and often only provide the measures used for variable selection, such as machine learning model importance scores, as a byproduct. To address the resulting unclear uncertainty in the selection sets, the knockoff framework offers a model-agnostic, robust approach to variable selection with guaranteed type-I error control. Here, we review the knockoff framework in the setting of clinical data, highlighting main considerations using simulation studies. We also extend the framework by introducing a novel knockoff generation method that addresses two main limitations of previously suggested methods relevant for clinical development settings. With this new method, we empirically obtain tighter bounds on type-I error control and gain an order of magnitude in computational efficiency in mixed data settings. We demonstrate comparable selections to those of the competing method for identifying prognostic biomarkers for C-reactive protein levels in patients with psoriatic arthritis in four clinical trials. Our work increases access to the knockoff framework for variable selection from clinical trial data. Hereby, this paper helps to address the current replicability crisis which can result in unnecessary research efforts, increased patient burden, and avoidable costs.

Incorporating novel input variable selection method for in the different water basins of Thailand

Selecting appropriate input variables for developing a rainfall prediction model is significantly difficult. The present study proposed an innovative framework for input variable selection (IVS) model bootstrapped long short-term recurrent neural network (BTSP-LSTM-RNN) to identify relevant variables for monthly rainfall forecasting. Monthly meteorological and large-scale climatic variables (LCVs) from 1993 to 2022 at two selected river basins in the northern region of Thailand were used for model development. The proposed BTSP-LSTM-RNN model results were compared with the support vector regression with recursive feature elimination (SVR-RFE) and Gradient boosting (GB) by statistical metrics such as coefficient of determination (R2), mean absolute error (MAE), relative root mean squared error (RRMSE), Pearson’s correlation coefficient (r) and mean absolute percentage error (MAPE). BTSP-LSTM-RNN demonstrated exceptional performance, boasting a higher R2 (0.84), MAE (92.28), RRMSE (10.36) in the Wang basin, and R2 (0.83), MAE (242.60), RRMSE (9.93) in the Nan basin. BTSP-LSTM-RNN also achieved the lowest MAPE of 29.82%. Based on this IVS model results, two input variable combinations (IVCs) were designed. IVC-1 is based on BTSP-LSTM-RNN selection, and IVC-2 is an original set of variables. LSTM-RNN, multi-layer perceptron artificial neural network (MLP-ANN), and ensemble model with bootstrapping on random forest (RF) were employed for monthly prediction. When BTSP-LSTM-RNN's selected input variables from IVC-1 are utilized, the BSTP-RF model demonstrates robust performance. It achieves a high R2 (0.82), a low RRMSE (10.13%) suggests accurate predictions, and the r of 0.91 further supports the model's strong linear relationship with observed rainfall data. Based on prediction model results, the BTSP-LSTM-RNN (IVS) model plays a pivotal role in the selection of input variables for rainfall forecasting and its impact on the performance of prediction models (BSTP-RF, MLP-ANN, and LSTM-RNN). These results consistently underscored the pivotal role of the BTSP-LSTM-RNN IVS model in enhancing the precision and reliability of rainfall predictions.

Variable selection with the knockoffs: Composite null hypotheses

The fixed-X knockoff filter is a flexible framework for variable selection with false discovery rate (FDR) control in linear models with arbitrary design matrices (of full column rank) and it allows for finite-sample selective inference via the Lasso estimates. In this paper, we extend the theory of the knockoff procedure to tests with composite null hypotheses, which are usually more relevant to real-world problems. The main technical challenge lies in handling composite nulls in tandem with dependent features from arbitrary designs. We develop two methods for composite inference with the knockoffs, namely, shifted ordinary least-squares (S-OLS) and feature-response product perturbation (FRPP), building on new structural properties of test statistics under composite nulls. We also propose two heuristic variants of S-OLS method that outperform the celebrated Benjamini–Hochberg (BH) procedure for composite nulls, which serves as a heuristic baseline under dependent test statistics. Finally, we analyze the loss in FDR when the original knockoff procedure is naively applied on composite tests.

A Bayesian genomic selection approach incorporating prior feature ordering and population structures with application to coronary artery disease.

Coronary artery disease is one of the most common types of cardiovascular disease. Death from coronary heart disease is influenced by genetic factors in both women and men. In this article, we propose a novel Bayesian variable selection framework for the identification of important genetic variants associated with coronary artery disease disease status. Instead of treating each feature independently as in conventional Bayesian variable selection methods, we propose an innovative prior for the inclusion probabilities of genetic variants that accounts for their ordering structure. We assume that neighboring variants are more likely to be selected together as they tend to be highly correlated and have similar biological functions. Additionally, we propose to group participating subjects based on underlying population structure and fit separate regressions, so that the regression coefficients can better reflect different disease risks in different population groups. Our approach borrows strength across regression models through an innovative prior inspired by the Markov random fields. The proposed framework can improve variable selection and prediction performances as demonstrated in the simulation studies. We also apply the proposed framework to the CATHeterization GENetics data with binary Coronary artery disease disease status.

How important are teacher characteristics to predict mathematics achievement in European countries?

This study focused on the relationships between teacher characteristics and students’ mathematics achievement in EU countries, including Hungary, Italy, Lithuania, Malta, Slovenia, Sweden, and Turkey, which are the only EU countries participated in TIMSS 2015 at the eighth-grade level. The data consisted of the sample of 31,969 eighth-grade students and their teachers ( n = 1687) in TIMSS 2015. The Qualities of Effective Teachers were used as a conceptual framework for variable selection and 24 teacher characteristics were tested across these EU countries through Hierarchical Linear Modeling technique (HLM). As a result, 17 characteristics suggested as effective by previous researchers were found to have significant relationships with students’ mathematics achievement in the pooled sample while there were discrepancies when investigating countries individually. The characteristics found to have non-significant, or negatively-significant relationships as well as the discrepancies in the individual countries raised concerns regarding the implementation of these characteristics in the classroom environments, along with the scope of the study.

Penalized weighted least-squares estimate for variable selection on correlated multiply imputed data

Abstract Considering the inevitable correlation among different datasets within the same subject, we propose a framework of variable selection on multiply imputed data with penalized weighted least squares (PWLS–MI). The methodological development is motivated by an epidemiological study of A/H7N9 patients from Zhejiang province in China, where nearly half of the variables are not fully observed. Multiple imputation is commonly adopted as a missing data processing method. However, it generates correlations among imputed values within the same subject across datasets. Recent work on variable selection for multiply imputed data does not fully address such similarities. We propose PWLS–MI to incorporate the correlation when performing the variable selection. PWLS–MI can be considered as a framework for variable selection on multiply imputed data since it allows various penalties. We use adaptive LASSO as an illustrating example. Extensive simulation studies are conducted to compare PWLS–MI with recently developed methods and the results suggest that the proposed approach outperforms in terms of both selection accuracy and deletion accuracy. PWLS–MI is shown to select variables with clinical relevance when applied to the A/H7N9 database.

Machine learning models for predicting steroid-resistant of nephrotic syndrome.

In the absence of effective measures to predict steroid responsiveness, patients with nonhereditary steroid-resistant nephrotic syndrome (SRNS) have a significantly increased risk of progression to end-stage renal disease. In view of the poor outcomes of SRNS, it is urgent to identify the steroid responsiveness of idiopathic nephrotic syndrome (INS) early. To build a prediction model for SRNS, we collected 91 subjects; 57 of them had steroid-sensitive nephrotic syndrome, and the others had SRNS. For each subject, 87 clinical variables were measured. In general, only a small part of these variables is informative to SRNS. Thus, we proposed a new variable selection framework including a penalized regression approach (named MLR+TLP) to select variables having a linear effect on the SRNS and a nonparametric screening method (MAC) to select variables having a nonlinear marginal (joint) effect on the SRNS. Thereafter, considering the correlation between selected clinical variables, we used a stepwise method to build our final model for predicting SRNS. In addition, a statistical testing procedure is proposed to test the overfitting of the proposed model. Twenty-six clinical variables were selected to be informative to SRNS, and an SVM model was built to predict SRNS with a leave-one-out cross-validation (LOO-CV) accuracy of 95.2% (overfitting p value<0.005). To make the model more useful, we incorporate prior medical information into the model and consider the correlation between selected variables. Then, a reduced SVM model including only eight clinical variables (erythrocyte sedimentation rate, urine occult blood, percentage of neutrophils, immunoglobulin A, cholesterol, vinculin autoantibody, aspartate aminotransferase, and prolonged prothrombin time) was built to have a LOO-CV accuracy of 92.8% (overfitting p value<0.005). The validation cohort showed that the reduced model obtained an accuracy of 94.0% (overfitting p value<0.005), with a sensitivity of 90.0% and a specificity of 96.7%. Notably, vinculin autoantibody is the only podocyte autoantibody included in this model. It is linearly related to steroid responsiveness. Finally, our model is freely available as a user-friendly web tool at https://datalinkx.shinyapps.io/srns/. The SRNS prediction model constructed in this study comprehensively and objectively evaluates the internal conditions and disease status of INS patients and will provide scientific guidance for selecting treatment methods for children with nonhereditary SRNS.

Revisiting vulnerable growth in the Euro Area: Identifying the role of financial conditions in the distribution

Growth-at-Risk modelling has been a cornerstone for research and policymaking recently as a way to model tail risk in the macroeconomy. However, the majority of the research has been almost exclusively been done on US data. The aim of this paper is to utilise a variable selection framework to identify which variables are key in capturing the different parts of the GDP distribution for the Euro Area. Importantly this paper uses a methodology that can handle variable selection task in small sample settings.

An automated exact solution framework towards solving the logistic regression best subset selection problem

An automated logistic regression solution framework (ALRSF) is proposed to solve a mixed integer programming (MIP) formulation of the well known logistic regression best subset selection problem. The solution framework firstly determines the optimal number of independent variables that should be included in the model using an automated cardinality parameter selection procedure. The cardinality parameter dictates the size of the subset of variables and can be problem-specific. A novel regression parameter fixing heuristic that utilises a Benders decomposition algorithm is applied to prune the solution search space such that the optimal regression parameter values are found faster. An optimality gap is subsequently calculated to quantify the quality of the final regression model by considering the distance between the best possible log-likelihood value and a log-likelihood value that is calculated using the current parameter values. Attempts are then made to reduce the optimality gap by adjusting regression parameter values. The ALRSF serves as a holistic variable selection framework that enables the user to consider larger datasets when solving the best subset selection logistic regression problem by significantly reducing the memory requirements associated with its mixed integer programming formulation. Furthermore, the automated framework requires minimal user intervention during model training and hyperparameter tuning. Improvements in quality of the final model (when considering both the optimality gap and computing resources required to achieve a result) are observed when the ALRSF is applied to well-known real-world UCI machine learning datasets. Keywords: Best subset selection, Independent variable selection, Logistic regression, Mixed integer programming

Unified reciprocal LASSO estimation via least squares approximation

The primary goal of this article is to extend the reciprocal LASSO for applications to binary and survival outcomes. We consider the least squares approximation (LSA) as a solver for the reciprocal LASSO problem. The LSA is a general theoretical framework that includes generalized linear models, Cox regression, and many others as special cases. By applying LSA to reciprocal LASSO regularization, we transfer the original reciprocal LASSO problem into an asymptotically equivalent least squares problem. While the existing literature on reciprocal LASSO has mostly focused on linear models, our algorithm can be easily implemented for general likelihoods, providing a flexible framework for variable selection using reciprocal penalties. To handle the computational burden of implementing the resulting procedure, we employ a scalable stochastic search method called Simplified Shotgun Stochastic Search with Screening (S5), which is easy to implement, without requiring any sophisticated optimization package other than a linear equation solver. We examine the effectiveness of our procedure through Monte Carlo simulations and real data analyses.

Ensemble classification based feature selection: a case of identification on plant pentatricopeptide repeat proteins.

In order to identify plant pentatricopeptide repeat (PPR) proteins, a framework of variable selection has been proposed. In fact, it is an effective feature selection strategy that focuses on the performance of classification. Random forest has been used as the classifier with certain variables automatically selected for discrimination between PPR functional and non-functional proteins. However, it is found that samples regarded as PPR functional proteins are wrongly classified in a high rate. In this paper, we plan to improve the framework in order to achieve better classification results. Modifications are made on the framework for better identifying PPR functional proteins. Instead of random forest, a hybrid ensemble classifier is built with its base classifiers derived from six different classification methods. Besides, an incremental strategy and a clustering by search in descending order are alternatively used for feature selection, which can effectively select the most representative variables for identification on PPR proteins. In addition, it can be found that different base classifiers alternately play an important role in the ensemble classifier with feature dimension increasing. The experimental results demonstrate the effectiveness of our improvements.

Domain knowledge-enhanced variable selection for biomedical data analysis

Machine learning has achieved impressive results in biomedical data analysis. To cope with high-dimensional data, variable selection is proposed to identify patterns in the feature space and select informative and predictive variables. However, due to the scarce cases and expensive sampling costs in the biomedical area, the lack of samples has become the main obstacle to the performance improvement of traditional variable selection algorithms in the biomedical data. In this paper, we solve this problem by the feat of domain knowledge, which seems to be a unique method in the biomedicine area due to the abundant and reliable domain knowledge in it. Nevertheless, the empirical study demonstrated that the brute-force implantation of domain knowledge may be counterproductive, especially when unfaithful knowledge exists. To elegantly incorporate domain knowledge into the variable selection framework, this paper starts from the joint likelihood function of the discriminative model and derives the extended form of prior knowledge term based on the existing variable selection framework. Based on this, a novel method is presented. We prove and substantiate with both synthetic and real-world biomedical data that, the proposal could effectively utilize the information from domain knowledge to assist variable selection and simultaneously reject incorrect knowledge.

Unobserved classes and extra variables in high-dimensional discriminant analysis

In supervised classification problems, the test set may contain data points belonging to classes not observed in the learning phase. Moreover, the same units in the test data may be measured on a set of additional variables recorded at a subsequent stage with respect to when the learning sample was collected. In this situation, the classifier built in the learning phase needs to adapt to handle potential unknown classes and the extra dimensions. We introduce a model-based discriminant approach, Dimension-Adaptive Mixture Discriminant Analysis (D-AMDA), which can detect unobserved classes and adapt to the increasing dimensionality. Model estimation is carried out via a full inductive approach based on an EM algorithm. The method is then embedded in a more general framework for adaptive variable selection and classification suitable for data of large dimensions. A simulation study and an artificial experiment related to classification of adulterated honey samples are used to validate the ability of the proposed framework to deal with complex situations.

Selection of predictor variables for species distribution models: a case study with an invasive marine bryozoan.

Species distribution models (SDMs) are important tools for predicting the occurrence and abundance of organisms in space and time, with numerous applications in ecology. However, the accuracy and utility of SDMs can be compromised when predictor variables are selected without careful consideration of their ecophysiological relevance to the focal organism. We conducted an in-depth examination of the variable selection process by evaluating predictors to be used in SDMs for Membranipora membranacea, an ecologically significant marine invasive species with a complex lifecycle, as a case study. Using an information-theoretic and multi-model inference approach based on generalized linear mixed models, we assessed multiple environmental variables (depth, kelp density, kelp substrate, temperature, and wave exposure) as predictors of the abundance of multiple life stages of M. membranacea, investigating species-environment relationships and relative and absolute variable importance. We found that the relative importance of a predictor, the metric calculated to represent a predictor, and whether a predictor was proximal or distal were important considerations in the variable selection process. Data constraints (e.g. sample size, characteristics of available predictor data) may inhibit accurate assessment of predictor variables during variable selection. Importantly, our results suggest that species-environment relationships derived from small-scale studies can inform variable selection for SDMs at larger spatiotemporal scales. We developed a conceptual framework for variable selection for SDMs which can be applied to most contexts of species distribution modelling, but particularly those with several candidate predictors and a large dataset.