A Physics-Informed Spectral-Structure Synergy Optimization (SSSO) Method for Consistent and Interpretable Spectral Variable Selection.

Nature create variables using its character component, and variables are sharing characters from a vary small to relatively large scale. This results, variables to have from a vary different to a more similar character, and leads to have a relation ship. Literature suggested different relation measures based on the nature of variable and type of relation ship exist. Today, due to having high variety of frequently produced large data size, currently suggested variable filtering and selection methods have gaps to full fill the need. This research desires to fill this gap by comparing literature suggested methods to finding out a better variable selection and dimension reduction methods. The result from regression analysis using all literature suggested factors shows that none of the predictors for development status of enterprise are significant, and only 10 predictors for number of employer in an enterprise are significant out of 81 factors. Since, variable selection and dimension reduction methods are applied to find out predictors of a response by removing variable redundancy, and complexity of incorporating large number variable. Based on statistical power, for the results from variable selection methods, specially association and correlation methods showed that, CANOVA more efficiently detects non-linear or non-monotonic correlation between a continuous–continuous and a continuous-categorical variables. Spearman’s correlation coefficient more efficiently detects a monotonic correlation between a continuous with a continuous, and a continuous with a categorical variable. Pearson correlation coefficient more efficiently detects the linear correlation between continuous variables. MIC efficiently detects non-linear or non-monotonic relation between continuous variables. Chi-square test of independence efficiently detects relation between a continuous with a continuous, and categorical with categorical variables, but the non linear or non monotonic relation between a continuous with a categorical are not well detected. On the other hand, the result from lasso and stepwise methods reveals that, the relation between the predictor and response due to interaction effect not detected by correlation and association methods are detected by stepwise variable selection method, and the multicollinearity is detected and removed by lasso method. Regressing the response variable “number of employer in an enterprise” based on variables selected by lasso and stepwise method does bring greater model fitness (based on adjusted R-squared value) than variables selected by association and correlation methods. Similarly, regressing the response variable “development status of an enterprise” based on variables selected by association and correlation methods does bring 12 significant variables, where none of variables are significant from variables selected by lasso and stepwise methods. As a result, 51 predictors for number of employment in an enterprise, and 40 predictors for development status of an enterprise are detected as significantly related variables. And, lasso and stepwise methods are preferred to select predictors of a continuous response variable “number of employers in an enterprise”, and association and correlation methods are preferred to select predictors of a categorical response variable “development status of an enterprise”. Finally, the reduced regression models result reveals that, 20 predictors have causal relation with number of employment in an enterprise, and 12 predictors have causal relation with development status of an enterprise. On the other hand, based on model fitness, information lost, and number of significant factors, principal factor is preferred and applied in dimension reduction for a categorical response variable “development status of an enterprise”, and factor score based regression is preferred and applied for a continuous response variable “number of employers in an enterprise”. However, the comparison of the results in variable selection and dimension reduction indicates that, variable selection methods gave more gain in model fitness than dimension reduction methods. Hence, the suggested variable selection methods are more preferred than dimension reduction methods, and applied to find out predictors. In general, the suggested procedure for variable selection methods are recommended when small number of variables are studied, and the suggested dimension reduction methods are recommended for large number of variant variables (Big data case).

A Monte Carlo resampling based multiple feature-spaces ensemble (MFE) strategy for consistency-enhanced spectral variable selection

Sparse representation using over-complete dictionaries have shown to produce good quality results in various image processing tasks. Dictionary learning algorithms have made it possible to engineer data adaptive dictionaries which have promising applications in image compression and image enhancement. The most common sparse dictionary learning algorithms use the techniques of matching pursuit and K-SVD iteratively for sparse coding and dictionary learning respectively. While this technique produces good results, it requires a large number of iterations to converge to an optimal solution. In this article, we propose a closed form convex optimization technique for both sparse coding and dictionary learning. The approach results in providing the best possible dictionary and the sparsest representation resulting in minimum reconstruction error which in turn results in compression. It is clearly seen from the results that the proposed algorithm provides much better reconstruction results than conventional sparse dictionary techniques for a fixed number of iterations. Depending upon the amount of details present in the image, the proposed algorithm is seen to reach the optimal solution with significantly lower number of iterations. Consequently low mean squared error is obtained using the proposed algorithm. We demonstrate the results with standard image.

Sparse representation using over-complete dictionaries have shown to produce good quality results in various image processing tasks. Dictionary learning algorithms have made it possible to engineer data-adaptive dictionaries that have promising applications in image compression and image enhancement. The most common sparse dictionary learning algorithms use the techniques of matching pursuit and K-SVD iteratively for sparse coding and dictionary learning respectively. While this technique produces good results, it requires a large number of iterations to converge to an optimal solution. In this article, we use a closed-form stabilized convex optimization technique for both sparse coding and dictionary learning. The approach results in providing the best possible dictionary and the sparsest representation resulting in minimum reconstruction error. We have used the proposed algorithm for compressed sensing of satellite images. Once the image is reconstructed from the compressively sensed samples, we use adaptive spatial and frequency domain filtering techniques to move towards exact image recovery. It is seen from the results that the proposed algorithm provides much better reconstruction results than conventional sparse dictionary techniques for a fixed number of iterations. Depending upon the number of details present in the image, the proposed algorithm is seen to reach the optimal solution with a significantly lower number of iterations. Consequently, high PSNR and low MSE is obtained using the proposed algorithm for our compressive sensing framework.

This paper investigates variable selection (VS) and classification for biomedical datasets with a small sample size and a very high input dimension. The sequential sparse Bayesian learning methods with linear bases are used as the basic VS algorithm. Selected variables are fed to the kernel-based probabilistic classifiers: Bayesian least squares support vector machines (BayLS-SVMs) and relevance vector machines (RVMs). We employ the bagging techniques for both VS and model building in order to improve the reliability of the selected variables and the predictive performance. This modeling strategy is applied to real-life medical classification problems, including two binary cancer diagnosis problems based on microarray data and a brain tumor multiclass classification problem using spectra acquired via magnetic resonance spectroscopy. The work is experimentally compared to other VS methods. It is shown that the use of bagging can improve the reliability and stability of both VS and model prediction.

The selection of a descriptor, X, is crucial for improving the interpretation and prediction accuracy of a regression model. In this study, the prediction accuracy of models constructed using the selected X was determined and the results of variable selection, according to the number of selected X and number of selected variables that are unrelated to an objective variable, such as activities and properties (y), were investigated to evaluate the variable or feature selection methods. Variable selection methods include least absolute shrinkage and selection operator, genetic algorithm-based partial least squares, genetic algorithm-based support vector regression, and Boruta. Several regression analysis methods were used to test the prediction accuracy of the model constructed using the selected X. The characteristics of each variable selection method were analyzed using eight datasets. The results showed that even when variables unrelated to y were selected by variable selection and the number of unrelated variables was the same as the number of the original variables, a regression model with good accuracy, which ignores the influence of such noise variables, can be constructed by applying various regression analysis methods. Additionally, the variables related to y must not to be deleted. These findings provide a basis for improving the variable selection methods.

In regression analysis, variable selection is a challenging task. Over the last decade, the lasso-type methods have become popular method for variable selection due to their property of shrinking some of the model coefficients to exactly zero. Theory says that lasso-type methods are able to do consistent variable selection but it is hard to achieve this property in practice. This consistent variable selection highly depends on the right choice of the tuning parameter. In this paper, we show that selection of tuning parameter by cross validation almost always fail to achieve consistent variable selection. We have also shown that lasso-type methods with a BIC-type tuning parameter selector, under certain conditions, can do the consistent variable selection. We have also made a novel suggestion for choosing the value of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , a weight on estimated model size, in BIC. Our results show that with this choice of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , the lasso-type methods can do consistent variable selection.In regression analysis, variable selection is a challenging task. Over the last decade, the lasso-type methods have become popular method for variable selection due to their property of shrinking some of the model coefficients to exactly zero. Theory says that lasso-type methods are able to do consistent variable selection but it is hard to achieve this property in practice. This consistent variable selection highly depends on the right choice of the tuning parameter. In this paper, we show that selection of tuning parameter by cross validation almost always fail to achieve consistent variable selection. We have also shown that lasso-type methods with a BIC-type tuning parameter selector, under certain conditions, can do the consistent variable selection. We have also made a novel suggestion for choosing the value of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , a weight on estimated model size, in BIC. Our results show that with this choice of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , the lasso-type methods can do consistent variable selection.

INTRODUCTION Variable selection is a common step in clinical research, where large datasets often include many, potentially highly correlated, variables. The main objective is to identify the most relevant predictors for an outcome, thereby enhancing model interpretability, simplicity, and predictive performance [1]. However, data-driven variable selection also carries several underappreciated risks. These include the potential exclusion of important predictors, inclusion of irrelevant ones, biased coefficient estimates, underestimated standard errors, invalid confidence intervals, and overall model instability [2]. Simulation studies are a valuable approach for evaluating statistical methods, provided they are carefully designed. Yet, many such studies exhibit bias in favor of the newly proposed methods [3]. To address this, we developed a neutral comparison simulation study to fairly evaluate the performance of several variable selection techniques. OBJECTIVE To systematically evaluate and compare different variable selection methods across multiple simulated scenarios. METHODS To improve the design and reporting of our simulation study, we followed the ADEMP structure [4], this involves specifying the aim (A), the data-generating process (D), the estimand or target of inference (E), the analytical methods (M), and the criteria used to evaluate performance (P). We designed different simulation scenarios by varying the number of observations, total variables, and number of true predictors. Predictor correlations were modeled to decay exponentially with increasing distance between variables, and effect sizes for true predictors were varied [5, 6]. Noise was introduced into the correlation structures to better mimic real-world data. We focused on a binary classification setting, evaluating each method on two key outcomes: model selection accuracy (i.e. whether the true model is selected) and predictive performance. Five methods for selecting variables were compared: stepwise logistic regression, LASSO logistic regression, Elastic net logistic regression, Random Forest Classifier with OOB error based backward elimination [7] and Genetic Algorithm (GA) [8, 9]. Performance metrics included the Area Under the Curve (AUC), number of variables selected, and True Positive Rate (TPR). All the analyses were performed using Python 3.12. RESULTS We ran 1,000 Monte Carlo simulations per scenario, varying key factors such as sample size, number of predictors, true signal strength, and correlation strength. Elastic Net consistently achieved the highest mean AUC and TPR, particularly in high-dimensional or strong-signal settings (e.g., Scenarios 5–8), showing robust performance across conditions. Random Forest and Genetic Algorithm performed comparably in some scenarios but incurred substantially higher computational costs. LASSO achieved competitive AUC with significantly lower runtime, though it tended to underselect in weaker signal scenarios. Stepwise selection, while the fastest method, had the lowest overall predictive performance and true positive rates (Table 1). CONCLUSION Among the five evaluated methods, Elastic Net provided the best trade-off between predictive performance and model stability, particularly in realistic, high-dimensional settings. Our results reinforce the importance of carefully considering the variable selection method in the context of the data structure and research goals. This neutral comparison contributes to evidence-based guidance for method selection in clinical research and similar applied settings.

In this article, we propose two variable selection methods for adjusting the censoring information for survival times, such as the restricted mean survival time. To adjust for the influence of censoring, we consider an inverse probability of censoring weighted for subjects with events. We derive a least absolute shrinkage and selection operator (lasso)-type variable selection method, which considers an inverse weighting for of the squared losses, and an information criterion-type variable selection method, which applies an inverse weighting of the survival probability to the power of each density function in the likelihood function. We prove the consistency of the inverse probability of censoring weighted lasso estimator and the maximum inverse probability of censoring weighted likelihood estimator. The performance of the inverse probability of censoring weighted lasso and inverse probability of censoring weighted information criterion are evaluated via a simulation study with six scenarios, and then their variable selection ability is demonstrated using data from two clinical studies. The results confirm that inverse probability of censoring weighted lasso and the inverse probability of censoring weighted likelihood function produce good estimation accuracy and consistent variable selection. We conclude that our two proposed methods are useful variable selection tools for adjusting the censoring information for survival time analyses.

Multivariate data such as spectra frequently contain measured variables that are uninformative, and removal of such variables requires the use of methods that can be used to select informative variables. Partial least squares (PLS) regression may incorporate information from uninformative measured variables, and so it is important to select variables before performing the PLS regression. Elastic net (EN) regression can be used to perform variable selection automatically. An EN regression can be used to select groups of correlated variables or to select either sparse or nonsparse sets of variables. However, the predictive performance of the EN regression can be significantly worse than competing 1‐step variable selection methods such as variable importance in projection (VIP). In the present work, the use of the EN to select variables, followed by conventional PLS regression on the selected variables (EN‐PLS), has been investigated. Variable selection by using EN‐PLS was compared with that from EN regression, sparse PLS regression, VIP, and from selectivity ratio selection on 2 data sets of visible/near‐infrared spectra. In all cases, the wavelengths selected were compared with reference data. The variables selected by using EN‐PLS offered advantages in interpretability and gave more robust prediction performance as compared with those obtained from full‐spectrum PLS and the other variable selection methods. This paper reports a method for variable selection by using an EN regression prior to a second regression by using PLS, a 2‐step method termed EN‐PLS. Variables selected by using EN‐PLS are compared with variables selected from the EN regression, as well as VIP, selectivity ratio, and the sparse PLS regression, 3 commonly used methods for variable selection in chemometrics. The EN‐PLS is shown to select variables that were more easily interpreted. In addition, EN‐PLS performed more robustly than a PLS regression performed on all variables, as well as reduced PLS regressions by using variables selected from either the sparse PLS regression algorithm or a VIP variable selection followed by PLS modeling.

Compressive sensing is a sub-Nyquist sampling technique for efficient signal acquisition and reconstruction of sparse or compressible signals. In order to account for the sparsity of the underlying signal of interest, it is common to use sparsifying priors such as Bernoulli–Gaussian-inverse Gamma (BGiG) and Gaussian-inverse Gamma (GiG) priors on the components of the signal. With the introduction of variational Bayesian inference, the sparse Bayesian learning (SBL) methods for solving the inverse problem of compressive sensing have received significant interest as the SBL methods become more efficient in terms of execution time. In this paper, we consider the sparse signal recovery problem using compressive sensing and the variational Bayesian (VB) inference framework. More specifically, we consider two widely used Bayesian models of BGiG and GiG for modeling the underlying sparse signal for this problem. Although these two models have been widely used for sparse recovery problems under various signal structures, the question of which model can outperform the other for sparse signal recovery under no specific structure has yet to be fully addressed under the VB inference setting. Here, we study these two models specifically under VB inference in detail, provide some motivating examples regarding the issues in signal reconstruction that may occur under each model, perform comparisons and provide suggestions on how to improve the performance of each model.

Variable selection is crucial for the accuracy of spectral analysis and is typically formulated as an optimization problem using regression techniques. However, these data-driven methods may overlook physical laws or mechanisms, leading to the deselection of physically relevant variables. To address this, we propose a hierarchical clustering and optimal interval combination (HCIC) strategy, guided by domain knowledge, in which physical principles and mechanisms inform algorithm design to capture more physically relevant feature structures. In the first step, spectral variable hierarchical clustering (SVHC) is employed to determine correlations between adjacent variables, generating non-uniform intervals. Each interval corresponds to distinct patterns that reflect underlying molecular interactions, such as peak shifts, functional group contributions, and even non-reaction background signals. Secondly, a Bayesian linear regression-based optimal interval combination (BLR-OIC) strategy is applied to identify the most effective interval combinations, capturing and exploiting the synergistic effects among functional bands or functional groups. We conduct extensive experiments on publicly available and proprietary databases to validate the efficacy of the proposed algorithm. The results demonstrate not only improved predictive performance compared to benchmarks but also greater interpretability and consistent variable selection.

The forest stock volume (FSV) is one of the crucial indicators to reflect the quality of forest resources. Variable selection methods are usually used for FSV estimated models. However, few studies have explored which variable selection methods can make the selected data set have better explanatory and robustness for the same dominant tree species in different regions after the feature variables were filtered by the feature selection methods. In this study, we chose six dominant tree species from Lin’an District, Anji County, and a part of Longquan City. The tree species include broad-leaved, coniferous, Masson pine, Chinese fir, coniferous and broad-leaved mixed forest, and all tree species which include the above five groups of tree species. The last two tree species were represented by mixed and all, respectively. Then, the satellite images, terrain factors, and forest inventory data were selected by six variable selection methods (least absolute shrinkage and selection operator (LASSO), recursive feature elimination (RFE), stepwise regression (Step-Reg), permutation importance (PI), mean decrease impurity (MDI), and SelectFromModel based on LightGBM (SFM)), according to different dominant tree types in different regions. The selected variables were formed into a new dataset divided by different dominant trees. Besides, extreme gradient boosting (XGBoost) was used, combined with variable selection methods to estimate the FSV. The performed results are as follows: In the feature selection of coniferous, RFE performed better both in the average and in the separate regions. In the feature selection of Chinese fir and all, PI performed better both in the average and in the separate regions. In the feature selection of Masson pine, MDI performed better both in the average and in the separate regions. In the feature selection of mixed, MDI performed better in the average while RFE performed better in the separate regions comprehensively. The results showed that not only in separate regions, but the average result two factors, RFE, MDI, and PI all performed well to select variables to estimate the FSV. Furthermore, we selected the top five high feature-importance factors of different tree types, and the results showed that tree age and canopy density were both of great importance to the estimation of FSV. Besides, in the exhibited results of feature selection methods, compared with no variable selection, the research also found that variable selection can improve the performance of the model. Additionally, from the results of different tree types in different regions, we also found that small-scale and diversity of dominant tree types may lead to the instability and unreliability of experimental results. The study provides some insight into the application the optimal variable selection methods of the same dominant tree type in different regions. This study will help the development of variable selection methods to estimate FSV.

In the data-rich environment, using many economic predictors to forecast a few key variables has become a new trend in econometrics. The commonly used approach is factor augment (FA) approach. This paper pursues another direction, variable selection (VS) approach, to handle high-dimensional predictors. VS is an active topic in statistics and computer science. However, it does not receive as much attention as FA in economics. This paper introduces several cutting-edge VS methods to economic forecasting, which includes: (1) classical greedy procedures; (2) l1 regularization; (3) false-discovery-rate control methods, (4) gradient descent with sparsification and (5) meta-heuristic algorithms. Comprehensive simulation studies are conducted to compare their variable selection accuracy and prediction performance under different scenarios. Among the reviewed methods, a meta-heuristic algorithm called sequential Monte Carlo algorithm performs the best. Surprisingly the classical forward selection is comparable to it and better than other more sophisticated algorithms. In addition, these VS methods are applied on economic forecasting and compared with the popular FA approach. It turns out for employment rate and CPI inflation, some VS methods can achieve considerable improvement over FA, and the selected predictors can be well explained by economic theories.

Sparse deep dictionary learning identifies differences of time-varying functional connectivity in brain neuro-developmental study