Human-in-The-Loop Feature Selection Using Interpretable Kolmogorov-Arnold Network-Based Double Deep Q-Network

A fast dual-module hybrid high-dimensional feature selection algorithm

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

Polynomial-SHAP analysis of liver disease markers for capturing of complex feature interactions in machine learning models

Investigating the performance of machine learning models combined with different feature selection methods to estimate the energy consumption of buildings

The study presents a novel framework integrating feature selection (FS) and machine learning (ML) techniques to forecast inland national energy consumption (EC) in the United Kingdom across all energy sources. This innovative framework strategically combines three FS approaches with five interpretable ML models using Shapley Additive Explanations (SHAP), with the dual goal of enhancing accuracy and transparency in EC predictions. By meticulously selecting the most pertinent features from diverse features—including meteorological conditions, socioeconomic parameters, and historical consumption patterns of different primary fuels—the proposed framework enhances the robustness of the forecasting model. This is achieved through benchmarking three FS approaches: ensemble filter, wrapper, and a hybrid ensemble filter-wrapper. In addition, we introduce a novel ensemble filter FS, synthesizing outcomes from multiple base FS methods to make well-informed decisions about feature retention. Experimental results underscore the efficacy of integrating both wrapper and ensemble filter-wrapper FS approaches with interpretable ML models, ensuring the forecasting process remains comprehensible and interpretable while utilizing a manageable number of features (four to eight). In addition, experimental results indicate that different feature subsets are usually selected for each combined FS approach and ML model. This study not only demonstrates the framework's capability to provide accurate forecasts but also establishes it as a valuable tool for policymakers and energy analysts.

<p><b>The quality of the data space, which is often represented by a set of features, is one of the most critical aspects affecting the classification performance of a machine learning algorithm. The existence of noisy, irrelevant, and/or redundant features often reduces the classification accuracy of the learning algorithms. To improve the quality of the data, feature selection is proposed to choose a small set of a given feature set that captures the properties of the data. However, very few studies pay attention on that there are usually multiple best feature subsets for a feature selection task. In other words, due to complex feature interactions, multiple feature subsets with different features selected can have very similar or the same classification performance. Many existing feature selection methods output only one feature subset since they discard the other candidate feature subsets with the same classification performances during the feature selection process, and therefore some good feature subsets are lost. Thanks to the potential global search ability, evolutionary computation (EC) methods, especially multimodal optimization techniques and differential evolution (DE), have been widely and successfully applied to address different tasks including feature selection. However, using these techniques to search for multiple best feature subsets in a feature selection task has not been systematically investigated.</b></p> <p>The overall goal of this thesis is to investigate and improve the capability of evolutionary multimodal optimization techniques mainly niching-based DE for feature selection to search for multiple best feature subsets. The obtained multiple feature subsets are expected to have small subset sizes while maintaining or even improving the classification performance over using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective and multi-objective), the encoding schemes (binary and real-valued), and the search operators.</p> <p>This thesis introduces a new multimodal DE method with duplication analysis for feature selection. The results show that the proposed method can find different feature subsets with very similar or the same classification performance, and therefore users can choose one based on their preferences. More importantly, the selected features achieve better classification performance than using the original whole feature set and features selected by some conventional methods and the compared niching-based feature selection methods.</p> <p>This thesis develops a novel niching-based DE method for multi-objective feature selection, which aims to reduce the redundancy rate among the obtained feature subsets by considering the characteristics that different feature subsets with the same subset size can achieve very similar or the same classification performance. The experimental results show that the proposed feature selection method evolves a rich and diverse set of non-dominated solutions for different feature selection tasks.</p> <p>This thesis proposes a new binary DE method for multi-objective feature selection. To improve the population diversity, during the environmental selection process, solutions (feature subsets) with large diversity scores in the candidate pool are preferred. The results show that the proposed method achieves significantly better feature selection performance than the current popular multi-objective feature selection methods.</p> <p>This thesis introduces a new feature clustering-assisted feature selection method. The proposed method incorporates the correlation knowledge obtained by filter measures into the encoding and search process to search for multiple best feature subsets. The results show that the classification performance can be significantly improved by integrating the obtained feature subsets on some of the used high-dimensional datasets.</p>

This paper investigates the impact of reducing feature-vector dimensionality on the performance of machine learning (ML) models. Dimensionality reduction and feature selection techniques can improve computational efficiency, accuracy, robustness, transparency, and interpretability of ML models. In high-dimensional data, where features outnumber training instances, redundant or irrelevant features introduce noise, hindering model generalization and accuracy. This study explores the effects of dimensionality reduction methods on binary classifier performance using network traffic data for cybersecurity applications. The paper examines how dimensionality reduction techniques influence classifier operation and performance across diverse performance metrics for seven ML models. Four dimensionality reduction methods are evaluated: principal component analysis (PCA), singular value decomposition (SVD), univariate feature selection (UFS) using chi-square statistics, and feature selection based on mutual information (MI). The results suggest that direct feature selection can be more effective than data projection methods in some applications. Direct selection offers lower computational complexity and, in some cases, superior classifier performance. This study emphasizes that evaluation and comparison of binary classifiers depend on specific performance metrics, each providing insights into different aspects of ML model operation. Using open-source network traffic data, this paper demonstrates that dimensionality reduction can be a valuable tool. It reduces computational overhead, enhances model interpretability and transparency, and maintains or even improves the performance of trained classifiers. The study also reveals that direct feature selection can be a more effective strategy when compared to feature engineering in specific scenarios.

This study investigates the use of machine learning (ML) models for wastewater treatment plant (WWTP) sludge predictions and explainable artificial intelligence (XAI) techniques for understanding the impact of variables behind the prediction. Three ML models, random forest (RF), gradient boosting machine (GBM), and gradient boosting tree (GBT), were evaluated for their performance using statistical indicators. Input variable combinations were selected through different feature selection (FS) methods. XAI techniques were employed to enhance the interpretability and transparency of ML models. The results suggest that prediction accuracy depends on the choice of model and the number of variables. XAI techniques were found to be effective in interpreting the decisions made by each ML model. This study provides an example of using ML models in sludge production prediction and interpreting models applying XAI to understand the factors influencing it. Understandable interpretation of ML model prediction can facilitate targeted interventions for process optimization and improve the efficiency and sustainability of wastewater treatment processes. PRACTITIONER POINTS: Explainable artificial intelligence can play a crucial role in promoting trust between machine learning models and their real-world applications. Widely practiced machine learning models were used to predict sludge production of a United States wastewater treatment plant. Feature selection methods can reduce the required number of input variables without compromising model accuracy. Explainable artificial intelligence techniques can explain driving variables behind machine learning prediction.

Soil salinization endangers the development of ecological agriculture. As soil salinization is often heavily affected by regional environments, difficulties arise when constructing an adaptive multi-regional soil salinity estimation model. In this study, we proposed an extreme gradient boosting (XGBoost) model based on the Tree-structure Parzen Estimator (TPE) optimization algorithm to apply to four study areas with different environments (TPE-XGBoost). The four areas are the Weigan-Kuqa Oasis (Weiku), the Sangong River Basin (Sgr) and the Qitai Oasis in Xinjiang, China, and the middle and lower reaches of the Syr Darya Basin in Kazakhstan. Most previous soil salinity studies did not pay much attention to the impact of feature selection and hyper-parameter tuning on the performance of machine learning models, and the complex dependence and interaction between input features and hyper-parameters. In order to improve the performance of XGBoost model in estimating soil salinity, we proposed for the first time to use TPE algorithm to jointly optimize feature selection and hyper-parameter tuning, and verified it in four areas. Coefficient of determination (R2) and Root Mean Square Error (RMSE) were used to evaluate the model performance. First, we calculated 55 environmental features from Landsat and terrain data. Then, in order to reduce the computational complexity of the TPE-XGBoost model, we used Pearson correlation analysis between surface soil salinity content (SSC) and features to initially filter out the features that were not significantly related (P > 0.05). Finally, the TPE algorithm was used to jointly optimize the parameter space composed of features and hyper-parameters. The results showed that (1) TPE joint optimization algorithm significantly improved the performance of the XGBoost model, achieving high accuracy in the four areas, and had powerful generalization. R2 values of test sets for Weiku Oasis, Qitai Oasis, Sgr Basin, and the Syr Basin were 0.95, 0.95, 0.80, and 0.81, respectively. (2) There is no universal feature can be applied to soil salinity inversion in different environments. TPE algorithm adaptively selected different types and numbers of features for four areas, 19, 11, 25, and 15 features were selected in Weiku Oasis, Qitai Oasis, Sgr Basin, and the Syr Basin, respectively. This showed that the optimal model parameters should not be fixed parameters, but should be re-determined locally according to different environmental conditions. The TPE algorithm can capture the features that reflect environmental differences. (3) The XGBoost model can provide feature importance ranking, which improves the interpretability of machine learning model. The importance analysis results showed that the features had different contributions in different areas. The TPE-XGBoost model proposed in this study has great potential in multi-regional soil salt estimation research.

HeFS: helper-enhanced feature selection via Pareto-optimized genetic search.

BackgroundThis study aimed to develop an interpretable machine learning (ML) model for predicting plaque-induced gingivitis risk in schoolchildren using questionnaire data. To enhance the model’s interpretation, SHapley Additive exPlanations (SHAP) method was applied to analyze and explain the risk factors associated with plaque-gingivitis.Materials and methodsUsing multi-stage cluster random sampling, 1755 children aged 6–12 in Lanzhou were enrolled. Participants completed a 22-item questionnaire and underwent clinical dental examinations. The collected data were stratified and randomly divided into a training set (70%) and a testing set (30%), with an independent external validation cohort (n = 120) prospectively collected for generalizability assessment. Feature selection was performed using Least Absolute Shrinkage and Selection Operator (LASSO) regression. Six ML algorithms—Light Gradient Boosting Machine (LightGBM), random forest (RF), logistic regression (LR), eXtreme Gradient Boosting (XGBoost), decision tree (DT), and K-nearest neighbors (KNN)—were employed to process the data. The efficacy of each algorithm was evaluated using area under the curve (AUC), sensitivity (recall), specificity, accuracy, precision, F1–score and decision curve analysis. Using the SHAP method, all predictors of gingivitis prevalence in children were ranked by importance.Results51.3% (901/1755) of the children were clinically diagnosed with plaque‑induced gingivitis. 11 key predictors were selected using LASSO regression to build the ML models. Among all models, the RF achieved the highest discrimination (training AUC: 0.991; testing AUC: 0.909), followed closely by LightGBM (training AUC: 0.970; testing AUC: 0.904). The RF model was selected as the optimal model and maintained generalizability (external validation AUC: 0.824). SHAP analysis identified key predictors ranked by importance, including brushing frequency, age, regular dental checkups, brushing time, gingival bleeding during brushing, and annual income.ConclusionAn interpretable RF model accurately stratified gingivitis risk using self-reported factors. This ML-driven strategy may reduce reliance on resource-intensive clinical examinations, supporting scalable pediatric gingivitis prevention in resource-limited settings.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12903-025-07245-y.

BackgroundAlthough neoadjuvant immunochemotherapy (nICT) has revolutionized the management of locally advanced esophageal squamous cell carcinoma (ESCC), the inability to accurately predict pathological complete response (pCR) remains a major barrier to treatment personalization. We aimed to develop and validate an interpretable machine learning (ML) model using pretreatment multimodal features to predict pCR prior to nICT initiation.MethodsIn this retrospective study, 114 ESCC patients receiving nICT were randomly allocated into training (n=81) and validation (n=33) cohorts (7:3 ratio). Predictors of pCR were identified from pretreatment clinical variables, endoscopic ultrasonography, and hematological biomarkers via least absolute shrinkage and selection operator (LASSO) regression. Eight machine learning algorithms were implemented to construct prediction models. Model performance was assessed by area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Shapley Additive Explanations (SHAP) provided feature importance and model interpretability.ResultsFollowing feature selection, 17 variables were incorporated into model construction. The Random Forest (RF) model demonstrated perfect discrimination in the training cohort (AUC = 1.000, sensitivity = 1.000, specificity = 1.000, PPV = 1.000, NPV = 1.000), while maintaining robust predictive ability in the independent validation cohort (AUC = 0.913, sensitivity = 0.733, specificity = 0.889, PPV = 0.846, NPV = 0.800). Decision curve analysis (DCA) confirmed favorable clinical utility. SHAP analysis identified alcohol consumption, circumferential involvement ≥50%, elevated neutrophil-to-lymphocyte ratio (NLR), C-reactive protein (CRP), and alanine aminotransferase (ALT) as the key contributors to pCR prediction.ConclusionsWe established a clinically applicable, interpretable ML model that accurately predicts pCR to nICT in ESCC by integrating multimodal pretreatment data. This tool may optimize patient selection for nICT and advance precision therapy paradigms.

ObjectiveThe subset of breast cancer patients with pathologic node-negative status (ypN0) after neoadjuvant chemotherapy (NAC) who benefit from postoperative radiotherapy (PORT) through a reduced risk of recurrence remains poorly defined. This study aimed to develop and validate an interpretable machine learning (ML) model to perform risk stratification for recurrence among ypN0 patients who all received PORT.MethodsWe conducted a retrospective analysis of 1450 breast cancer patients treated between January 2017 and January 2024. All patients received NAC, underwent radical surgery confirming ypN0 status, and subsequently received PORT. Based on follow-up outcomes after PORT, patients were classified as ‘recurrence’ or ‘no-recurrence’. From 20 initial clinicopathological variables, feature selection was performed using LASSO regression, stepwise logistic regression, and the Boruta algorithm, retaining 16 key features identified by at least two methods. Predictive models were constructed using ten machine learning algorithms, with hyperparameters optimized via particle swarm optimization. Model performance was assessed using the area under the receiver operating characteristic curve (AUC), calibration curves, decision curve analysis (DCA), and standard metrics including accuracy, sensitivity, specificity, and F1-score. The optimal model was interpreted using the SHapley Additive exPlanations (SHAP) framework.ResultsThe Gradient Boosting Machine (GBM) model demonstrated superior predictive performance, achieving an AUC of 0.945 (95% CI: 0.917–0.972) on the test set. SHAP analysis identified breast reconstruction, perineural invasion, and the chemotherapy-to-surgery interval as the three most influential predictors. These features primarily exerted independent effects on recurrence risk.ConclusionWe developed and validated a highly accurate and interpretable ML model to stratify recurrence risk in ypN0 breast cancer patients after PORT. Functioning as a risk-stratification tool within this uniformly treated cohort, it may help identify patients at high risk of recurrence despite receiving PORT, who could be candidates for intensified surveillance or consideration of adjuvant therapy escalation. Its role in personalizing radiotherapy decisions requires prospective validation in studies including untreated control groups.

Interpretable Machine Learning Models for Severe Esophagitis Prediction in LA-NSCLC Patients Treated with Chemoradiation Therapy