Variable Selection Performance Research Articles

Bayesian effect selection in structured additive quantile regression

Bayesian structured additive quantile regression is an established tool for regressing outcomes with unknown distributions on a set of explanatory variables and/or when interest lies with effects on the more extreme values of the outcome. Even though variable selection for quantile regression exists, its scope is limited. We propose the use of the Normal Beta Prime Spike and Slab (NBPSS) prior in Bayesian quantile regression to aid the researcher in not only variable but also effect selection. We compare the Bayesian NBPSS approach to statistical boosting for quantile regression, a current standard in automated variable selection in quantile regression, in a simulation study with varying degrees of model complexity and illustrate both methods on an example of childhood malnutrition in Nigeria. The NBPSS prior shows good performance in variable and effect selection as well as prediction compared to boosting and can thus be recommended as an additional tool for quantile regression model building.

Multivariate Functional Clustering with Variable Selection and Application to Sensor Data from Engineering Systems

Multisensor data that track system operating behaviors are widely available nowadays from various engineering systems. Measurements from each sensor over time form a curve and can be viewed as functional data. Clustering of these multivariate functional curves is important for studying the operating patterns of systems. One complication in such applications is the possible presence of sensors whose data do not contain relevant information. Hence, it is desirable for the clustering method to equip with an automatic sensor selection procedure. Motivated by a real engineering application, we propose a functional data clustering method that simultaneously removes noninformative sensors and groups functional curves into clusters using informative sensors. Functional principal component analysis is used to transform multivariate functional data into a coefficient matrix for data reduction. We then model the transformed data by a Gaussian mixture distribution to perform model-based clustering with variable selection. Three types of penalties, the individual, variable, and group penalties, are considered to achieve automatic variable selection. Extensive simulations are conducted to assess the clustering and variable selection performance of the proposed methods. The application of the proposed methods to an engineering system with multiple sensors shows the promise of the methods and reveals interesting patterns in the sensor data. History: Kwok Tsui served as the senior editor for this article. Funding: The research by J. Min and Y. Hong was partially supported by the National Science Foundation [Grant CMMI-1904165] to Virginia Tech. The work by Y. Hong was partially supported by the Virginia Tech College of Science Research Equipment Fund. Data Ethics & Reproducibility Note: The original dataset is proprietary and cannot be shared. The full code to replicate the results in this paper, based on summary statistics of the original data, is available at https://github.com/jiem3/MultiFuncClustering . The code applied to a simplified version is available at https://codeocean.com/capsule/4041000/tree/v1 , which covers the data analysis and part of the simulation scenarios with a single dataset under each scenario using a fixed set of hyperparameters, for reducing computation time, and at https://doi.org/10.1287/ijds.2022.0034 .

Penalized Bayesian forward continuation ratio model with application to high-dimensional data with discrete survival outcomes.

While time-to-event data are often continuous, there are several instances where discrete survival data, which are inherently ordinal, may be available or are more appropriate or useful. Several discrete survival models exist, but the forward continuation ratio model with a complementary log-log link has a survival interpretation and is closely related to the Cox proportional hazards model, despite being an ordinal model. This model has previously been implemented in the high-dimensional setting using the ordinal generalized monotone incremental forward stagewise algorithm. Here, we propose a Bayesian penalized forward continuation ratio model with a complementary log-log link and explore different priors to perform variable selection and regularization. Through simulations, we show that our Bayesian model outperformed the existing frequentist method in terms of variable selection performance, and that a 10% prior inclusion probability performed better than 1% or 50%. We also illustrate our model on a publicly available acute myeloid leukemia dataset to identify genomic features associated with discrete survival. We identified nine features that map to ten unique genes, five of which have been previously associated with leukemia in the literature. In conclusion, our proposed Bayesian model is flexible, allows simultaneous variable selection and uncertainty quantification, and performed well in simulation studies and application to real data.

Flexible variable selection in the presence of missing data.

In many applications, it is of interest to identify a parsimonious set of features, or panel, from multiple candidates that achieves a desired level of performance in predicting a response. This task is often complicated in practice by missing data arising from the sampling design or other random mechanisms. Most recent work on variable selection in missing data contexts relies in some part on a finite-dimensional statistical model, e.g., a generalized or penalized linear model. In cases where this model is misspecified, the selected variables may not all be truly scientifically relevant and can result in panels with suboptimal classification performance. To address this limitation, we propose a nonparametric variable selection algorithm combined with multiple imputation to develop flexible panels in the presence of missing-at-random data. We outline strategies based on the proposed algorithm that achieve control of commonly used error rates. Through simulations, we show that our proposal has good operating characteristics and results in panels with higher classification and variable selection performance compared to several existing penalized regression approaches in cases where a generalized linear model is misspecified. Finally, we use the proposed method to develop biomarker panels for separating pancreatic cysts with differing malignancy potential in a setting where complicated missingness in the biomarkers arose due to limited specimen volumes.

The Effect of the Second Stage Estimator on Model Performance in Post-LASSO Method

Penalized linear regression methods are used for the accurate prediction of new observations and to obtain interpretable models. The performance of these methods depends on the properties of the true coefficient vector. The LASSO method is a penalized regression method that can simultaneously perform coefficient shrinkage and variable selection in a continuous process. Depending on the structure of the dataset, different estimators have been proposed to overcome the problems faced by LASSO. The estimation method used in the second stage of the post-LASSO two-stage regression method proposed as an alternative to LASSO has a considerable effect on model performance.
 In this study, the performance of the post-LASSO is compared with classical penalized regression methods ridge, LASSO, elastic net, adaptive LASSO and Post-LASSO by using different estimation methods in the second stage of the post-LASSO. In addition, the effect of the magnitude and position of the signal values in the real coefficient vector on the performance of the models obtained by these methods is analyzed. The mean squared error and standard deviation of the predictions calculated on the test set are used to compare the prediction performance of the models, while the active set sizes are used to compare their performance in variable selection. According to the findings obtained from the simulation studies, the choice of the second-stage estimator and the structure of the true coefficient vector significantly affect the success of the post-LASSO method compared to other methods.

Bayesian continuous-time hidden Markov models with covariate selection for intensive longitudinal data with measurement error.

Intensive longitudinal data collected with ecological momentary assessment methods capture information on participants' behaviors, feelings, and environment in near real-time. While these methods can reduce recall biases typically present in survey data, they may still suffer from other biases commonly found in self-reported data (e.g., measurement error and social desirability bias). To accommodate potential biases, we develop a Bayesian hidden Markov model to simultaneously identify risk factors for subjects transitioning between discrete latent states as well as risk factors potentially associated with them misreporting their true behaviors. We use simulated data to demonstrate how ignoring potential measurement error can negatively affect variable selection performance and estimation accuracy. We apply our proposed model to smartphone-based ecological momentary assessment data collected within a randomized controlled trial that evaluated the impact of incentivizing abstinence from cigarette smoking among socioeconomically disadvantaged adults. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Analytical approaches for food authentication using LIBS fingerprinting

Laser-induced breakdown spectroscopy (LIBS) is a widely acknowledged elemental analysis approach used in various study domains due to its rapid measurement capability and minimal sample-preparation requirements. Recently, there has been an increase in interest in the applications of LIBS in the realm of food safety and quality. Given that the majority of commonly consumed foods exhibit only modest trace-element variations, discovering predictive spectral patterns through multivariate analysis is crucial for the data-analysis pipeline. The efficacy of multivariate analysis and machine-learning algorithms to identify the most predictive spectral features, conduct class recognition and classification was evaluated in this paper, utilizing both a custom-developed benchtop LIBS system and a commercially available portable one. Specifically, this study's objective was to evaluate the performance of spectral variable selection using elastic-net multinomial logistic regression. The data processing pipeline and the LIBS hardware were evaluated in the context of food authentication and identification, a rising field of research addressing the issue of food fraud. Our findings indicated that classifying food samples with carefully selected fewer variables reduces model overfitting and improves the accuracy of LIBS pattern classification. In a broader sense, the results support the continued development of field-deployable, portable LIBS equipment designed for food authentication and fingerprinting activities.

Ecpc: an R-package for generic co-data models for high-dimensional prediction

BackgroundHigh-dimensional prediction considers data with more variables than samples. Generic research goals are to find the best predictor or to select variables. Results may be improved by exploiting prior information in the form of co-data, providing complementary data not on the samples, but on the variables. We consider adaptive ridge penalised generalised linear and Cox models, in which the variable-specific ridge penalties are adapted to the co-data to give a priori more weight to more important variables. The R-package ecpc originally accommodated various and possibly multiple co-data sources, including categorical co-data, i.e. groups of variables, and continuous co-data. Continuous co-data, however, were handled by adaptive discretisation, potentially inefficiently modelling and losing information. As continuous co-data such as external p values or correlations often arise in practice, more generic co-data models are needed.ResultsHere, we present an extension to the method and software for generic co-data models, particularly for continuous co-data. At the basis lies a classical linear regression model, regressing prior variance weights on the co-data. Co-data variables are then estimated with empirical Bayes moment estimation. After placing the estimation procedure in the classical regression framework, extension to generalised additive and shape constrained co-data models is straightforward. Besides, we show how ridge penalties may be transformed to elastic net penalties. In simulation studies we first compare various co-data models for continuous co-data from the extension to the original method. Secondly, we compare variable selection performance to other variable selection methods. The extension is faster than the original method and shows improved prediction and variable selection performance for non-linear co-data relations. Moreover, we demonstrate use of the package in several genomics examples throughout the paper.ConclusionsThe R-package ecpc accommodates linear, generalised additive and shape constrained additive co-data models for the purpose of improved high-dimensional prediction and variable selection. The extended version of the package as presented here (version number 3.1.1 and higher) is available on (https://cran.r-project.org/web/packages/ecpc/).

Partial least squares regression with conditional orthogonal projection for variable selection

Partial least squares regression (PLS) is a popular multivariate statistical analysis method. It not only can deal with high-dimensional variables but also can effectively select variables. However, the traditional PLS variable selection approaches cannot deal with some prior important variables. In this article, we propose two filter PLS variable selection algorithms (CPLS-BETA, CPLS-VIP) that combine with the prior variables by the conditional orthogonal projection. The performance of variable selection can be improved by projecting the other variables and response orthogonally on some prior active variables. Moreover, we introduce a kind of data-driven conditional method named forward projection PLS (FPPLS), which is suitable for the situation of unknown prior information. Finally, the validity of our method is verified by simulated datasets and real datasets.

Local linear approximation with Laplacian smoothing penalty and application in biology.

Highly correlated structures appear in various fields, such as biology, biochemistry, and finance, with challenges of dimensionality and sparse estimation. To solve this problem, we propose an algorithm called local linear approximation with the Laplacian smoothing penalty (LLA-LSP). This method produces an accurate and smooth estimate that incorporates the correlation structure among predictors. We compare and discuss the difference between the Laplacian smoothing penalty and the total variance penalty. We prove that this algorithm converges to the oracle solution in a few iterations with a large probability. Numerical results show that the LLA-LSP has good performance in both variable selection and estimation. We apply the proposed algorithm to two biological datasets, a gene expression dataset and a chemical protein dataset, and provide meaningful insights.

Recovery of partly sparse and dense signals

In high-dimensional data analysis, we often encounter partly sparse and dense signals or parameters. Considering an lq-penalization with different qs for each sub-vector of the signals, we formularize an optimal solution for q=1 or 2 in a linear regression model to well represent such signals or parameters. We also provide an algorithm to derive it. Furthermore, we provide the consistency result of the variable selection in this optimal solution under a fixed design. Simulation study and real-data analysis illustrate its improved variable selection performance relative to the conventional methods.

Improving the Robustness of Variable Selection and Predictive Performance of Regularized Generalized Linear Models and Cox Proportional Hazard Models.

High-dimensional data applications often entail the use of various statistical and machine-learning algorithms to identify an optimal signature based on biomarkers and other patient characteristics that predicts the desired clinical outcome in biomedical research. Both the composition and predictive performance of such biomarker signatures are critical in various biomedical research applications. In the presence of a large number of features, however, a conventional regression analysis approach fails to yield a good prediction model. A widely used remedy is to introduce regularization in fitting the relevant regression model. In particular, a penalty on the regression coefficients is extremely useful, and very efficient numerical algorithms have been developed for fitting such models with different types of responses. This -based regularization tends to generate a parsimonious prediction model with promising prediction performance, i.e., feature selection is achieved along with construction of the prediction model. The variable selection, and hence the composition of the signature, as well as the prediction performance of the model depend on the choice of the penalty parameter used in the regularization. The penalty parameter is often chosen by K-fold cross-validation. However, such an algorithm tends to be unstable and may yield very different choices of the penalty parameter across multiple runs on the same dataset. In addition, the predictive performance estimates from the internal cross-validation procedure in this algorithm tend to be inflated. In this paper, we propose a Monte Carlo approach to improve the robustness of regularization parameter selection, along with an additional cross-validation wrapper for objectively evaluating the predictive performance of the final model. We demonstrate the improvements via simulations and illustrate the application via a real dataset.

Envelope-based sparse reduced-rank regression for multivariate linear model

Envelope models were first proposed by Cook et al. (2010) as a method to reduce estimative and predictive variations in multivariate regression. Sparse reduced-rank regression, introduced by Chen and Huang (2012), is a widely used technique that performs dimension reduction and variable selection simultaneously in multivariate regression. In this work, we combine envelope models and sparse reduced-rank regression method to propose an envelope-based sparse reduced-rank regression estimator, and then establish its consistency, asymptotic normality and oracle property in high-dimensional data. We carry out some Monte Carlo simulation studies and also analyze two datasets to demonstrate that the proposed envelope-based sparse reduced-rank regression method displays good variable selection and prediction performance.

Characterization of spatio-temporal patterns of grassland utilization intensity in the Selinco watershed of the Qinghai-Tibetan Plateau from 2001 to 2019 based on multisource remote sensing and artificial intelligence algorithms

ABSTRACT Due to the limitations of spatial quantification methods, the spatio-temporal patterns of grassland utilization intensity (GUI) in the Selinco watershed (SLCW), the core region of ecological security on the Qinghai-Tibetan Plateau, is unclear under multiple utilization modes. This paper quantified GUI by constructing the association between the potential and actual Enhanced Vegetation Index (EVI) of grasslands in terms of interannual variability. To obtain an accurate spatio-temporal dataset of potential EVI, the following two components were considered on. Firstly, the temporal lag effects of each raw climate factor were investigated to determine the optimal climate variables affecting vegetation productivity. Secondly, four machine learning (ML) algorithms, including an artificial neural network, random forest, support vector machine, and gradient boosting regression tree combined with the Bayesian model average, were used to construct grassland potential EVI models involving EVI, grassland type, and environmental factors (topography, soil, raw climate, and bioclimatic). Meanwhile, to maximize the performance of ML models, variable selection, variable transformation, and hyperparameter optimization were systematically implemented, where the hyperparameter optimization algorithms employ the grid search algorithm, Bayesian optimization, genetic algorithm, and particle swarm optimization. Then, the spatio-temporal dataset of GUI in the SLCW from 2001 to 2019 was established by using the above quantification method based on multisource remote sensing and artificial intelligence algorithms. The analysis of spatio-temporal variation in GUI showed that the implementation of ecological restoration projects leads to a significant and rapid decline in the overall GUI of the SLCW after 2010 (declining by 4.8%), which is more obvious in the non-nature reserve (declining by 9.3%). In the Qiangtang Nature Reserve within the SLCW, although the GUI shows a declining trend after 2010 because of the implementation of ecological restoration projects, it shows an insignificant increase from 2001 to 2019 due to the recovery increase of wildlife populations in recent decades. Besides, by exploring the effects of elevation and slope on the GUI, it is found that grasslands on higher slopes at lower elevations are at a greater risk of degradation due to more intensive grassland utilization.

Gradient-induced Model-free Variable Selection Based on Composite Quantile Regression in Reproducing Kernel Hilbert Space

Variable selection plays an important role to identify truly informative variables in high-dimensional data analysis. In his paper, we propose a variable selection method with composite quantile regression in reproducing kernel Hilbert space (RKHS), which has two main advantages. The first is that our method requires no special model structure assumption and no independence of error term. It is suitable for general non-parametric models and even heteroscedastic models. The second is that the calculation is simple and fast. So, it can also work in high-dimensional situations. Finally, the numerical experiments and real data analysis demonstrate its superior performance in variable selection.

Sparse sliced inverse regression for high dimensional data analysis

BackgroundDimension reduction and variable selection play a critical role in the analysis of contemporary high-dimensional data. The semi-parametric multi-index model often serves as a reasonable model for analysis of such high-dimensional data. The sliced inverse regression (SIR) method, which can be formulated as a generalized eigenvalue decomposition problem, offers a model-free estimation approach for the indices in the semi-parametric multi-index model. Obtaining sparse estimates of the eigenvectors that constitute the basis matrix that is used to construct the indices is desirable to facilitate variable selection, which in turn facilitates interpretability and model parsimony.ResultsTo this end, we propose a group-Dantzig selector type formulation that induces row-sparsity to the sliced inverse regression dimension reduction vectors. Extensive simulation studies are carried out to assess the performance of the proposed method, and compare it with other state of the art methods in the literature.ConclusionThe proposed method is shown to yield competitive estimation, prediction, and variable selection performance. Three real data applications, including a metabolomics depression study, are presented to demonstrate the method’s effectiveness in practice.

Quantitative methods for descriptive intersectional analysis with binary health outcomes

Intersectionality recognizes that in the context of sociohistorically shaped structural power relations, an individual's multiple social positions or identities (e.g., gender, ethnicity) can interact to affect health-related outcomes. Despite limited methodological guidance, intersectionality frameworks have increasingly been incorporated into epidemiological studies, both to describe health disparities and to examine their causes. This study aimed to advance methods in intersectional estimation of binary outcomes in descriptive health disparities research through evaluation of 7 potentially intersectional data analysis methods: cross-classification, regression with interactions, multilevel analysis of individual heterogeneity (MAIHDA), and decision trees (CART, CTree, CHAID, random forest). Accuracy of estimated intersection-specific outcome prevalence was evaluated across 192 intersections using simulated data scenarios. For comparison we included a non-intersectional main effects regression. We additionally assessed variable selection performance amongst decision trees. Example analyses using National Health and Nutrition Examination Study data illustrated differences in results between methods. At larger sample sizes, all methods except for CART performed better than non-intersectional main effects regression. In smaller samples, MAIHDA was the most accurate method but showed no advantage over main effects regression, while random forest, cross-classification, and saturated regression were the least accurate, and CTree and CHAID performed moderately well. CART performed poorly for estimation and variable selection. Sensitivity analyses examining the bias-variance tradeoff suggest MAIHDA as the preferred unbiased method for accurate estimation of high-dimensional intersections at smaller sample sizes. Larger sample sizes are more imperative for other methods. Results support the adoption of an intersectional approach to descriptive epidemiology.

Two-Stage Robust and Sparse Distributed Statistical Inference for Large-Scale Data

In this paper, we address the problem of conducting statistical inference in settings involving large-scale data that may be high-dimensional and contaminated by outliers. The high volume and dimensionality of the data require distributed processing and storage solutions. We propose a two-stage distributed and robust statistical inference procedures coping with high-dimensional models by promoting sparsity. In the first stage, known as model selection, relevant predictors are locally selected by applying robust Lasso estimators to the distinct subsets of data. The variable selections from each computation node are then fused by a voting scheme to find the sparse basis for the complete data set. It identifies the relevant variables in a robust manner. In the second stage, the developed statistically robust and computationally efficient bootstrap methods are employed. The actual inference constructs confidence intervals, finds parameter estimates and quantifies standard deviation. Similar to stage 1, the results of local inference are communicated to the fusion center and combined there. By using analytical methods, we establish the favorable statistical properties of the robust and computationally efficient bootstrap methods including consistency for a fixed number of predictors, and robustness. The proposed two-stage robust and distributed inference procedures demonstrate reliable performance and robustness in variable selection, finding confidence intervals and bootstrap approximations of standard deviations even when data is high-dimensional and contaminated by outliers.

Performance Assessment of High-dimensional Variable Identification

Since model selection is ubiquitous in data analysis, reproducibility of statistical results demands a serious evaluation of reliability of the employed model selection method, no matter what label it may have in terms of good properties. Instability measures have been proposed for evaluating model selection uncertainty. However, low instability does not necessarily indicate that the selected model is trustworthy, since low instability can also arise when a certain method tends to select an overly parsimonious model. F- and G-measures have become increasingly popular for assessing variable selection performance in theoretical studies and simulation results. However, they are not computable in practice. In this work, we propose an estimation method for F- and G-measures and prove their desirable properties of uniform consistency. This gives the data analyst a valuable tool to compare different variable selection methods based on the data at hand. Extensive simulations are conducted to show the very good finite sample performance of our approach. We further demonstrate the application of our methods using several micro-array gene expression data sets, with intriguing findings.

Deselection of base-learners for statistical boosting-with an application to distributional regression.

We present a new procedure for enhanced variable selection for component-wise gradient boosting. Statistical boosting is a computational approach that emerged from machine learning, which allows to fit regression models in the presence of high-dimensional data. Furthermore, the algorithm can lead to data-driven variable selection. In practice, however, the final models typically tend to include too many variables in some situations. This occurs particularly for low-dimensional data (), where we observe a slow overfitting behavior of boosting. As a result, more variables get included into the final model without altering the prediction accuracy. Many of these false positives are incorporated with a small coefficient and therefore have a small impact, but lead to a larger model. We try to overcome this issue by giving the algorithm the chance to deselect base-learners with minor importance. We analyze the impact of the new approach on variable selection and prediction performance in comparison to alternative methods including boosting with earlier stopping as well as twin boosting. We illustrate our approach with data of an ongoing cohort study for chronic kidney disease patients, where the most influential predictors for the health-related quality of life measure are selected in a distributional regression approach based on beta regression.