Variable Selection Tool Research Articles

Newton-Raphson Meets Sparsity: Sparse Learning Via a Novel Penalty and a Fast Solver.

In machine learning and statistics, the penalized regression methods are the main tools for variable selection (or feature selection) in high-dimensional sparse data analysis. Due to the nonsmoothness of the associated thresholding operators of commonly used penalties such as the least absolute shrinkage and selection operator (LASSO), the smoothly clipped absolute deviation (SCAD), and the minimax concave penalty (MCP), the classical Newton-Raphson algorithm cannot be used. In this article, we propose a cubic Hermite interpolation penalty (CHIP) with a smoothing thresholding operator. Theoretically, we establish the nonasymptotic estimation error bounds for the global minimizer of the CHIP penalized high-dimensional linear regression. Moreover, we show that the estimated support coincides with the target support with a high probability. We derive the Karush-Kuhn-Tucker (KKT) condition for the CHIP penalized estimator and then develop a support detection-based Newton-Raphson (SDNR) algorithm to solve it. Simulation studies demonstrate that the proposed method performs well in a wide range of finite sample situations. We also illustrate the application of our method with a real data example.

Sparse kernel k-means clustering

Clustering is an essential technique that groups similar data points to uncover the underlying structure and features of the data. Although traditional clustering methods such as k-means are widely utilized, they have limitations in identifying nonlinear clusters. Thus, alternative techniques, such as kernel k-means and spectral clustering, have been developed to address this issue. However, another challenge arises when irrelevant variables are present in the data; this can be mitigated by employing variable selection methods such as the filter, wrapper, and embedded approaches. In this study, with a particular focus on kernel k-means clustering, we propose an embedded variable selection method using a tensor product space along with a general analysis of variance kernel for nonlinear clustering. Comprehensive experiments involving simulations and real data analysis demonstrated that the proposed method achieves competitive performance compared to existing approaches. Thus, the proposed method may serve as a reliable tool for accurate cluster identification and variable selection to gain insights into complex datasets.

Consistent skinny Gibbs in probit regression

Spike and slab priors have emerged as effective and computationally scalable tools for Bayesian variable selection in high-dimensional linear regression. However, the crucial model selection consistency and efficient computational strategies using spike and slab priors in probit regression have rarely been investigated. A hierarchical probit model with continuous spike and slab priors over regression coefficients is considered, and a highly scalable Gibbs sampler with a computational complexity that grows only linearly in the dimension of predictors is proposed. Specifically, the “Skinny Gibbs” algorithm is adapted to the setting of probit and negative binomial regression and model selection consistency for the proposed method under probit model is established, when the number of covariates is allowed to grow much larger than the sample size. Through simulation studies, the method is shown to achieve superior empirical performance compared with other state-of-the art methods. Gene expression data from 51 asthmatic and 44 non-asthmatic samples are analyzed and the performance for predicting asthma using the proposed approach is compared with existing approaches.

Prediction-based variable selection for component-wise gradient boosting.

Model-based component-wise gradient boosting is a popular tool for data-driven variable selection. In order to improve its prediction and selection qualities even further, several modifications of the original algorithm have been developed, that mainly focus on different stopping criteria, leaving the actual variable selection mechanism untouched. We investigate different prediction-based mechanisms for the variable selection step in model-based component-wise gradient boosting. These approaches include Akaikes Information Criterion (AIC) as well as a selection rule relying on the component-wise test error computed via cross-validation. We implemented the AIC and cross-validation routines for Generalized Linear Models and evaluated them regarding their variable selection properties and predictive performance. An extensive simulation study revealed improved selection properties whereas the prediction error could be lowered in a real world application with age-standardized COVID-19 incidence rates.

Variable selection using inverse probability of censoring weighting.

In this article, we propose two variable selection methods for adjusting the censoring information for survival times, such as the restricted mean survival time. To adjust for the influence of censoring, we consider an inverse probability of censoring weighted for subjects with events. We derive a least absolute shrinkage and selection operator (lasso)-type variable selection method, which considers an inverse weighting for of the squared losses, and an information criterion-type variable selection method, which applies an inverse weighting of the survival probability to the power of each density function in the likelihood function. We prove the consistency of the inverse probability of censoring weighted lasso estimator and the maximum inverse probability of censoring weighted likelihood estimator. The performance of the inverse probability of censoring weighted lasso and inverse probability of censoring weighted information criterion are evaluated via a simulation study with six scenarios, and then their variable selection ability is demonstrated using data from two clinical studies. The results confirm that inverse probability of censoring weighted lasso and the inverse probability of censoring weighted likelihood function produce good estimation accuracy and consistent variable selection. We conclude that our two proposed methods are useful variable selection tools for adjusting the censoring information for survival time analyses.

An application on forecasting for stock market prices: hybrid of some metaheuristic algorithms with multivariate adaptive regression splines

PurposeBorsa Istanbul 100 Index, known as BIST100, is the main indicator to measure the performance of the 100 highest stocks publicly traded in Borsa Istanbul concerning market and trading volume. BIST 100 index prediction is a popular research domain for its complex data structure caused by stock price, commodity, interest rate and exchange rate effects. The study proposed hybrid models using both Genetic, Particle Swarm Optimization, Harmony Search and Greedy algorithms from metaheuristic algorithms approach for dimension reduction, and MARS for prediction.Design/methodology/approachThis paper aims to model in the simplest way through metaheuristic algorithms hybridized with the MARS model the effects of stock, commodity, interest and exchange rate variables on BIST 100 during the Covid-19 pandemic period (in the process of closing) between January 2020 and June 2021.FindingsThe most suitable hybrid model was chosen as PSO & MARS by calculating the RMSE, MSE, GCV, MAE, MAD, MAPE and R2 measurements of training, test and overall dataset to check every model's efficiency. Empirical results demonstrated that the proposed PSO & MARS hybrid modeling procedure gave results both as good as the MARS model and a simpler and non-complex model structure.Originality/valueUsing metaheuristic algorithms as a supporting tool for variable selection can help to identify important independent variables and contribute to the establishment of more non-complex models.ing, test and overall dataset to check every model's efficiency.

Fast optimization methods for high-dimensional row-sparse multivariate quantile linear regression

Sparse quantile regression is a useful tool for variable selection, robust estimation and heteroscedasticity treatment in high-dimensional data analysis. Due to the non-smooth of quantile loss, the computation is heavier than the least square models. In the literature, there are various numerical methods for linear quantile regression, such as ADMM developed in Gu et al. [Technometrics. 2017;60(3):319–331]. However, the computation of multivariate quantile regression has not yet been fully resolved, especially when the dimension is high. Motivated by this, we will focus on the design of fast numerical algorithms for row-sparse multivariate quantile regression model. By virtue of proximal operator and Majorize–Minimization, four smoothed algorithms are designed. For all the obtained algorithms, we analyse their convergence and the parameter selection. We conduct plenty of simulations and four real data sets analysis. Finally, we conclude that the smoothed method is faster than the non-smooth method, especially when the number of predictors is large.

Comparing Factors Associated with Increased Stimulant Use in Relation to HIV Status Using a Machine Learning and Prediction Modeling Approach.

Stimulant use is an important driver of HIV/STI transmission among men who have sex with men (MSM). Evaluating factors associated with increased stimulant use is critical to inform HIV prevention programming efforts. This study seeks to use machine learning variable selection techniques to determine characteristics associated with increased stimulant use and whether these factors differ by HIV status. Data from a longitudinal cohort of predominantly Black/Latinx MSM in Los Angeles, CA was used. Every 6months from 8/2014-12/2020, participants underwent STI testing and completed surveys evaluating the following: demographics, substance use, sexual risk behaviors, and last partnership characteristics. Least absolute shrinkage and selection operator (lasso) was used to select variables and create predictive models for an interval increase in self-reported stimulant use across study visits. Mixed-effects logistic regression was then used to describe associations between selected variables and the same outcome. Models were also stratified based on HIV status to evaluate differences in predictors associated with increased stimulant use. Among 2095 study visits from 467 MSM, increased stimulant use was reported at 20.9% (n = 438) visits. Increased stimulant use was positively associated with unstable housing (adjusted [a]OR 1.81; 95% CI 1.27-2.57), STI diagnosis (1.59; 1.14-2.21), transactional sex (2.30; 1.60-3.30), and last partner stimulant use (2.21; 1.62-3.00). Among MSM living with HIV, increased stimulant use was associated with binge drinking, vaping/cigarette use (aOR 1.99; 95% CI 1.36-2.92), and regular use of poppers (2.28; 1.38-3.76). Among HIV-negative MSM, increased stimulant use was associated with participating in group sex while intoxicated (aOR 1.81; 95% CI 1.04-3.18), transactional sex (2.53; 1.40-2.55), and last partner injection drug use (1.96; 1.02-3.74). Our findings demonstrate that lasso can be a useful tool for variable selection and creation of predictive models. These results indicate that risk behaviors associated with increased stimulant use may differ based on HIV status and suggest that co-substance use and partnership contexts should be considered in the development of HIV prevention/treatment interventions.

Variable selection wrapper in presence of correlated input variables for random forest models

AbstractIn most data analytic applications in manufacturing, understanding the data‐driven models plays a crucial role in complementing the engineering knowledge about the production process. Identifying relevant input variables, rather than only predicting the response through some “black‐box” model, is of great interest in many applications. There is, therefore, a growing focus on describing the contributions of the input variables to the model in the form of “variable importance”, which is readily available in certain machine learning methods such as random forest (RF). Once a ranking based on the importance measure of the variables is established, the question of how many variables are truly relevant in predicting the output variable rises. In this study, we focus on the Boruta algorithm, which is a wrapper around the RF model. It is a variable selection tool that assesses the variable importance measure for the RF model. It has been previously shown in the literature that the correlation among the input variables, which is often a common occurrence in high dimensional data, distorts and overestimates the importance of variables. The Boruta algorithm is also affected by this resulting in a larger set of input variables deemed important. To overcome this issue, in this study, we propose an extension of the Boruta algorithm for the correlated data by exploiting the conditional importance measure. This extension greatly improves the Boruta algorithm in the case of high correlation among variables and provides a more precise ranking of the variables that significantly contribute to the response. We believe this approach can be used in many industrial applications by providing more transparency and understanding of the process.

Modeling water age in a full‐scale residential plumbing system

AbstractThe residence time of water in residential building water systems is a critical factor regarding water quality at end use. Published literature has highlighted the importance of water age in these systems and its relationship with pathogenic bacteria such as Legionella pneumophila. However, tools to measure water age in such plumbing systems are typically repurposed from other applications and include limitations that make them inappropriate for some plumbing systems. This work presents a novel means of estimating water age by assuming these systems operate without mixing. Data for this study was collected from a full‐scale home equipped with an extensive array of flow meters to monitor water use. Further, 408 individual water quality samples were collected to ascertain water quality changes that take place in the plumbing. Model results show weak correlation with EPANET 2.2 (ρ = 0.666), a commonly used hydraulic modeling software. The results of the water age model were also evaluated with several variable selection tools. These analyses indicate that this method's water age results are a statistically significant (p < .05) predictor of Legionella concentrations. Model results from this approach could be used in plumbing design and/or operation to assist in managing Legionella risks.

The influence of key success factors on business growth for tourism business along Thailand’s Andaman coast: moderating role of location

Purpose – This research aimed to examine the influences of key success factors on business growth. Additionally, it aimed to examine the moderating role of the location of the tourism enterprise. Design – Multiple regression was used and a variable selection tool was employed to test a statistical model both for the aggregate and in the context of individual provinces. Methodology – Quantitative research methods were used in this research. Approach – The convenience sampling method was used. Data was collected from fourhundred and nineteen tourism enterprises along the Andaman Coast in Thailand using online and in-person questionnaires. Findings – The results showed a positive influence of some dimensions of key success factors on business growth during the pandemic crisis COVID -19. In addition, the results showed that business location played a positive moderating role. Originality of the research – Key success factors are important concepts for business models to improve and increase the growth of a company. However, there are few studies that attempt to find out the key success factors in the context of tourism businesses that have been significantly affected by the COVID -19 pandemic. Therefore, this research aims to fill a gap in academic and practical discussions and implications.

Grey Wolf Optimizer for Variable Selection in Quantification of Quaternary Edible Blend Oil by Ultraviolet-Visible Spectroscopy.

A novel swarm intelligence algorithm, discretized grey wolf optimizer (GWO), was introduced as a variable selection tool in edible blend oil analysis for the first time. In the approach, positions of wolves were updated and then discretized by logical function. The performance of a wolf pack, the iteration number and the number of wolves were investigated. The partial least squares (PLS) method was used to establish and predict single oil contents in samples. To validate the method, 102 edible blend oil samples containing soybean oil, sunflower oil, peanut oil and sesame oil were measured by an ultraviolet-visible (UV-Vis) spectrophotometer. The results demonstrated that GWO-PLS models can provide best prediction accuracy with least variables compared with full-spectrum PLS, Monte Carlo uninformative variable elimination-PLS (MCUVE-PLS) and randomization test-PLS (RT-PLS). The determination coefficients (R2) of GWO-PLS were all above 0.95. Therefore, the research indicates the feasibility of using discretized GWO for variable selection in rapid determination of quaternary edible blend oil.

Small-Variance Priors in Bayesian Factor Analysis with Ordinal Data

To evaluate multidimensional factor structure, a popular method that combines features of confirmatory and exploratory factor analysis is Bayesian structural equation modeling with small-variance normal priors (BSEM-N). This simulation study evaluated BSEM-N as a variable selection and parameter estimation tool in factor analysis with sparse cross-loading structures, focusing on ordered categorical data. A sensitivity analysis was conducted by assigning eight choices of small-variance priors on all potential cross-loadings. Results indicated that variable selection was performed well in a sparse loading structure in which the number of essential cross-loadings was small and the magnitudes were relatively large. Characteristics of ordinal items did not seem to have a sizable impact on parameter estimation. If the number of cross-loading estimates were small and centered around zero, BSEM-N may serve more efficiently as a tool for parameter estimation.

Bayesian spatiotemporal modeling on complex-valued fMRI signals via kernel convolutions.

We propose a model-based approach that combines Bayesian variable selection tools, a novel spatial kernel convolution structure, and autoregressive processes for detecting a subject's brain activation at the voxel level in complex-valued functional magnetic resonance imaging (CV-fMRI) data. A computationally efficient Markov chain Monte Carlo algorithm for posterior inference is developed by taking advantage of the dimension reduction of the kernel-based structure. The proposed spatiotemporal model leads to more accurate posterior probability activation maps and less false positives than alternative spatial approaches based on Gaussian process models, and other complex-valued models that do not incorporate spatial and/or temporal structure. This is illustrated in the analysis of simulated data and human task-related CV-fMRI data. In addition, we show that complex-valued approaches dominate magnitude-only approaches and that the kernel structure in our proposed model considerably improves sensitivity rates when detecting activation at the voxellevel.

Comparison of carbohydrate composition in lignocellulosic biomass by high performance liquid chromatography and gas chromatography analysis

The carbohydrate composition (glucose, xylose, mannose, galactose, and arabinose) of lignocellulosic biomass Liriodendron tulipifera, Populus nigra × Populus maximowiczii, Populus alba × Populus glandulosa, Populus euramericana, Salix alba, Quercus variabilis, Robinia pseudoacacia, Zelkova serrata, Abies holophylla, Pinus rigida, rice straw, and peanut hull was investigated based on high-performance liquid chromatography (HPLC) and gas chromatography (GC) analyses derived from ASTM and NREL methods. The glucose content was higher in HPLC than in GC analysis, and the xylose, mannose, galactose, and arabinose contents were higher in GC than in HPLC analysis. The difference in carbohydrate composition was noticeable in the glucose, mannose, and arabinose contents of Abies holophylla and Pinus rigida, and this was affected by the species. A decision tree, as a data mining and artificial intelligence method, is a reliable and simple variable selection tool. This technique was used for carbohydrate analysis classification. Accordingly, 432 monosaccharide content reading data and analysis methods were used for model checking. It was found that arabinose was the most important splitting variable in carbohydrate analysis, and other monosaccharides did not influence the assay decision. However, the selection of a determination method for each sample should be considered comprehensively in future studies.

Bayesian networks as a novel tool to enhance interpretability and predictive power of ecological models

In today's world, it is becoming increasingly important to have the tools to understand, and ultimately to predict, the response of ecosystems to disturbance. However, understanding such dynamics is not simple. Ecosystems are a complex network of species interactions, and therefore any change to a population of one species will have some degree of community level effect. In recent years, the use of Bayesian networks (BNs) has seen successful applications in molecular biology and ecology, where they were able to recover plausible links in the respective systems they were applied to. The recovered network also comes with a quantifiable metric of interaction strength between variables. While the latter is an invaluable piece of information in ecology, an unexplored application of BNs would be using them as a novel variable selection tool in the training of predictive models. To this end, we evaluate the potential usefulness of BNs in two aspects: (1) we apply BN inference on species abundance data from a rocky shore ecosystem, a system with well documented links, to test the ecological validity of the revealed network; and (2) we evaluate BNs as a novel variable selection method to guide the training of an artificial neural network (ANN). Here, we demonstrate that not only was this approach able to recover meaningful species interactions networks from ecological data, but it also served as a meaningful tool to inform the training of predictive models, where there was an improvement in predictive performance in models with BN variable selection. Combining these results, we demonstrate the potential of this novel application of BNs in enhancing the interpretability and predictive power of ecological models; this has general applicability beyond the studied system, to ecosystems where existing relationships between species and other functional components are unknown.

Advanced chemometric methods as powerful tools for impurity profiling of drug substances and drug products: Application on bisoprolol and perindopril binary mixture

Impurity profiling has a rising importance nowadays due to the increased health problems associated with impurities and degradation products found in several drug substances and formulations. Three advanced, accurate and precise chemometric methods were developed as impurity profiling methods for a mixture of bisoprolol fumarate (BIS) and perindopril arginine (PER) with their degradation products which represent drug impurity or a precursor to such impurity. The methods applied were Partial Least Squares (PLS-1), Multivariate Curve Resolution-Alternating Least Squares (MCR-ALS) and Artificial Neural Networks (ANN). Genetic Algorithm (GA) was used as a variable selection tool to select the most significant wavelengths for the three chemometric models. For proper analysis, a 5-factor 5-level experimental design was used to establish a calibration set of 25 mixtures containing different ratios of the drugs and their degradation products (impurities). The validity of the proposed methods was assessed using an independent validation set. The designed models were able to predict the concentrations of the drugs and the degradation products/impurities in the validation set and pharmaceutical formulation. The proposed methods presented a powerful alternative to traditional and expensive chromatographic methods as impurity profiling tools.

Hormone concentrations mediate the associations between exposure to phthalate mixtures and preterm birth

BACKGROUND AND AIM: Mixtures of phthalate compounds can be found in many consumer products, resulting in widespread human exposure. Previous work has suggested that phthalates display endocrine disrupting capabilities that may differ based on molecular weight of the parent compound, and are associated with preterm birth (PTB). Given the importance of hormone regulation during pregnancy, we hypothesized that phthalates may affect the timing of labor via disruption of hormone concentrations. We aimed to assess the mediating effects of hormone concentrations on the associations between phthalate mixtures and timing of delivery. METHODS: Urinary phthalate metabolite measurements were repeated up to 3 times, and serum hormone measurements up to 2 times, during mid-gestation, among 1011 women in the PROTECT (Puerto Rico Testsite for Exploring Contamination Threats) longitudinal birth cohort. We utilized ridge regression as a shrinkage and variable selection tool to create phthalate environmental risk scores (ERS) at each study visit for both low and high molecular weight (LMW, HMW) compounds, which represent a weighted sum of each individual’s exposure to the mixture of metabolites. Causal mediation analyses were then conducted on a subset of 705 women for whom hormone data was available. RESULTS:First visit testosterone and fT4 mediated 17.2% and 10.2%, respectively, of the association between LMW phthalate ERS and spontaneous PTB. When analyses were stratified by fetal sex, the association between LMW phthalate ERS and spontaneous PTB was mediated by CRH (28.8%), progesterone (17.6%), and testosterone (30.6%) among only pregnancies with a male fetus. Third visit CRH also mediated 38.3% of the association between HMW phthalate ERS and reduced gestational age at birth among only male fetuses. There were no significant mediating results observed for mothers carrying a female fetus. CONCLUSIONS:These results provide introductory evidence of hormone disruption on the causal pathway between phthalate exposure and preterm birth. KEYWORDS: endocrine disrupting chemicals, birth outcomes, mixtures analysis, mediation analysis

Selection of projects for automotive assembly structures using a hybrid method composed of the group-input compatible, best-worst method for criteria weighting and TrBF-TOPSIS

The constant search for the optimal outcome, and its associated decision-making schemes within automotive assembly projects, can be greatly enhanced by utilizing informed decision methods that access relevant information. When simulated project alternatives are developed and benchmarks established, decision makers must select the appropriate project. When differences in opinion arise between decision makers, or when selection criteria are numerous, it can be difficult to be sure the correct choice was made. This highlights the need for a method that assists decision makers, when presented with such circumstances, so that they can be sure their choice is informed and optimal. Therefore, this paper proposes the use of a hybrid best-worst method (BWM) for weighting group decision making; and a Trapezoidal Bipolar Fuzzy TOPSIS (TrBF-TOPSIS) decision making strategy. The proposed method is used in ranking twenty-one design alternatives for automotive door assembly structures. The observed results indicate that the proposed method can be a relevant tool for variable selection in automotive assembly projects that involve multiple criteria and alternative designs.

Bayesian variable selection for understanding mixtures in environmental exposures.

Social and environmental stressors are crucial factors in child development. However, there exists a multitude of measurable social and environmental factors-the effects of which may be cumulative, interactive, or null. Using a comprehensive cohort of children in North Carolina, we study the impact of social and environmental variables on 4th end-of-grade exam scores in reading and mathematics. To identify the essential factors that predict these educational outcomes, we design new tools for Bayesian linear variable selection using decision analysis. We extract a predictive optimal subset of explanatory variables by coupling a loss function with a novel model-based penalization scheme, which leads to coherent Bayesian decision analysis and empirically improves variable selection, estimation, and prediction on simulated data. The Bayesian linear model propagates uncertainty quantification to all predictive evaluations, which is important for interpretable and robust model comparisons. These predictive comparisons are conducted out-of-sample with a customized approximation algorithm that avoids computationally intensive model refitting. We apply our variable selection techniques to identify the joint collection of social and environmental stressors-and their interactions-that offer clear and quantifiable improvements in prediction of reading and mathematics exam scores.