Feature Selection Research Articles

Embedded feature selection for robust probability learning machines

Methods:Feature selection is essential for building effective machine learning models in binary classification. Eliminating unnecessary features can reduce the risk of overfitting and improve classification performance. Moreover, the data we handle typically contains a stochastic component, making it important to develop robust models that are insensitive to data perturbations. Although there are numerous methods and tools for feature selection, relatively few studies address embedded feature selection within robust classification models using penalization techniques. Objective:In this work, we introduce robust classifiers with integrated feature selection capabilities, utilizing probability machines based on different penalization techniques, such as the ℓ1-norm or the elastic-net, combined with a novel Direct Feature Elimination process to improve model resilience and efficiency. Findings:Numerical experiments on standard datasets demonstrate the effectiveness and robustness of the proposed models in classification tasks even when using a reduced number of features. These experiments were evaluated using original performance indicators, highlighting the models’ ability to maintain high performance with fewer features. Novelty:The study discusses the trade-offs involved in combining different penalties to select the most relevant features while minimizing empirical risk. In particular, the integration of elastic-net and ℓ1-norm penalties within a unified framework, combined with the original Direct Feature Elimination approach, presents a novel method for improving both model accuracy and robustness.

A robust multi-label feature selection based on label significance and fuzzy entropy

Multi-label feature selection is one of the key steps in dealing with multi-label classification problems in high-dimensional data. In this step, label enhancement techniques play an important role. However, it is worth noting that many current methods tend to ignore the intrinsic connection between inter-sample similarity and inter-label correlation when implementing label enhancement learning. The neglect may prevent the process of label enhancement from accurately revealing the complex structure and underlying patterns within data. For this reason, a fuzzy multi-label feature selection method based on label significance and fuzzy entropy is proposed. An innovative label enhancement technique that considers not only the intrinsic connection between features and labels, but also the correlation between labels was first devised. Based on this enhanced label representation, the concept of fuzzy entropy is further defined to quantify the uncertainty of features for multi-label classification tasks. Subsequently, a feature selection algorithm based on feature importance and label importance was developed. The algorithm guides the feature selection process by evaluating how much each feature contributes to multi-label classification and how important each label is to the overall classification task. Finally, through a series of experimental validation, the proposed algorithm is proved to have better performance for multi-label classification tasks.

A Two-Stage Feature Selection Method to Enhance Prediction of Daily PM2.5 Concentration Air Pollution

In recent decades, air pollution has negatively affected human health and the environment. One of the important features contributing to air pollution is called PM2.5. However, daily prediction of PM2.5 is still lacking, especially using feature selection infused into the model. Hence, the main objective of this research is to utilize the feature selection procedures by proposing two stages feature selection methods namely adjusted correlation sharing t-test (adjcorT) and radial basis function neural network (RBFNN) in identifying the important features. This consequently also helps enhance the prediction of daily PM2.5 concentrations. Secondary data were obtained from the Department of Environment Malaysia (DOE) from 2018 until 2022 that consists of 5 years of air pollutant daily data. The results found that adjcorT-RBFNN identified the NO2, PM2.5, PM10, CO, O3, wind speed and SO2 as important features. The finding revealed that the accuracy, sensitivity, specificity, precision, F1 score and AUROC value, for a day-ahead prediction in Shah Alam are 0.756, 0.801, 0.717, 0.717, 0.757, and 0.758 respectively. Additionally, the predicted model may serve as an instrument for an early warning system, providing local authorities with information on air quality for formulation of strategies of air quality improvement.

Sandpiper optimization with hybrid deep learning model for blockchain-assisted intrusion detection in iot environment

Intrusion detection in the Internet of Things (IoTs) is a vital unit of IoT safety. IoT devices face diverse kinds of attacks, and intrusion detection systems (IDSs) play a significant role in detecting and responding to these threats. A typical IDS solution can be utilized from the IoT networks for monitoring traffic, device behaviour, and system logs for signs of intrusion or abnormal movement. Deep learning (DL) approaches are exposed to promise in enhancing the accuracy and effectiveness of IDS for IoT devices. Blockchain (BC) aided intrusion detection from IoT platforms provides many benefits, including better data integrity, transparency, and resistance to tampering. This paper projects a novel sandpiper optimizer with hybrid deep learning-based intrusion detection (SPOHDL-ID) from the BC-assisted IoT platform. The key contribution of the SPOHDL-ID model is to accomplish security via the intrusion detection and classification process from the IoT platform. In this case, the BC technology can be used for a secure data-sharing process. In the presented SPOHDL-ID technique, the selection of features from the network traffic data takes place using the SPO model. Besides, the SPOHDL-ID technique employs the HDL model for intrusion detection, which involves the design of a convolutional neural network with a stacked autoencoder (CNN-SAE) model. The beetle search optimizer algorithm (BSOA) method is used for the hyperparameter tuning procedure to increase the recognition outcomes of the CNN-SAE technique. An extensive simulation outcome is created to exhibit a better solution to the SPOHDL-ID method. The experimental validation of the SPOHDL-ID method portrayed a superior accuracy value of 99.59 % and 99.54 % over recent techniques under the ToN-IoT and CICIDS-2017 datasets.

Microwave spectroscopy in a free-space arrangement for nondestructive quality assessment of chicken eggs: Comparing different measurement modes and feature selection approaches

A free-space dielectric arrangement with X-band coaxial-to-waveguide adapters was used to non-destructively assess shell egg quality. Scattering parameters in the microwave spectral range (8–12 GHz) were measured in reflectance and transmittance modes from eggs placed in three orientations. Partial least squares regression was applied to predict egg quality indices, including air cell height (ACH), yolk coefficient (YC), thick albumen height (TAH), Haugh unit (HU), and albumen pH. Prioritizing PR_S22 spectrum in horizontal 1 orientation, several feature selection methods were employed to identify the most effective frequencies. Artificial neural networks (ANNs) were then used to develop predictive models based on influential frequencies. The competitive adaptive reweighted sampling method consistently outperformed others, yielding robust ANN models with excellent residual predictive deviation values of 4.80, 4.00, 3.27, 3.03, and 3.72 for ACH, YC, TAH, HU, and albumen pH, respectively. This study demonstrates the effectiveness of free-space dielectric arrangements in predicting egg quality.

Optimized Deep Learning Model for Medical Image Diagnosis

Deep learning models, particularly convolutional neural networks (CNNs), have excelled in pattern recognition tasks like face recognition, pedestrian detection, and medical diagnosis. CNNs, known for their end-to-end feature extraction and classification, are widely used in computer vision. Pre-trained CNN models such as AlexNet, VGG-19, ResNet, and Inception can serve as feature extractors for various problems, though they often produce a large number of features. To address this challenge and reduce the number of features, this research employs an efficient two-layer feature selection algorithm. The first layer, a global search optimizer, guides the swarm in the search space to identify the optimal feature subset. The second layer refines and fine-tunes the solution obtained from the global search. The main contributions of this work encompass several aspects: (i) the introduction of a lightweight classification model that utilizes a subset of features, (ii) the use of an efficient optimizer to perform simultaneous feature and instance selection, and (iii) the direct implications for the radiology field, where it could serve as a supportive tool for diagnosis. To evaluate the proposed approach, it was tested on skin lesion diagnosis using 1113 abnormal and 1099 normal lesions. The pre-trained VGG-19 model provided a 1-D feature vector of size 4096, with a linear SVM used for classification. Results showed the optimizer reduced the feature vector by up to 86% while achieving over 70% accuracy. The proposed optimizer outperformed the Reinforcement Learning-based Memetic Particle Swarm Optimization (RLMPSO).

Integrating temporal deep learning models for predicting screen-out risk levels in hydraulic fracturing

Amid the transformative shift in global energy structures, the exploitation and utilization of shale gas, an essential unconventional natural gas resource, have drawn widespread attention from both industrial and academic circles. However, screen-out incidents during hydraulic fracturing operations pose significant obstacles to extraction efficiency and safety. Traditional prediction methods, which rely on empirical estimations and simplified models, are deficient in accuracy and real-time applicability. Addressing this, our study introduces a novel deep learning ensemble integrating Gated Recurrent Units (GRU), Transformer, and One-Dimensional Convolutional Neural Networks (1D-CNN) for precise screen-out prediction. This approach markedly improves predictive accuracy by efficiently processing time-series data and capturing the complex dynamics of fracturing processes. Furthermore, the application of the correlation coefficient method and random forest algorithm for feature selection optimizes model input and further enhances prediction accuracy and operational efficiency. Our comparative analysis demonstrates the model’s superiority, achieving an F1 score of 0.951 and a loss of 0.430, clearly surpassing traditional and other deep learning methods. This integration of advanced neural architectures and feature selection techniques not only advances screen-out prediction but also yields practical insights for optimizing shale gas extraction strategies and enhancing safety.

Feature Selection With Discernibility and Independence Criteria

Feature Selection With Discernibility and Independence Criteria

HFS-CSR: A hierarchical feature selection method based on correlation and structural redundancy

Feature selection (FS) is a crucial step in data engineering, as it not only saves time and resources for model training but also improves the prediction performance. In many realistic data sets, the feature space is formed by binary features which are related via generalization-specialization relationships (a.k.a., hierarchical feature space). These interdependencies among features, which can be described via a directed acyclic graph, present new challenges for classical FS methods. In this paper, a Hierarchical Feature Selection method based on Correlation and Structural Redundancy (HFS-CSR) is proposed to address it. In HFS-CSR, the correlation between features and the class label is measured based on Fisher Score whereas the structural redundancy among features are assessed quantitatively via hierarchical redundancy and depth penalties. The importance of features are determined by combining them adaptively for feature filtering. Experiments are conducted based on two kinds of typical data sets with hierarchical feature space: aging gene data sets and NLP data sets. The results demonstrate that HFS-CSR can achieve better performance compared with seven existing FS methods.

Joint feature selection and classification of low-resolution satellite images using the SAT-6 dataset

The modern industries of today demand the classification of satellite images, and to use the information obtained from it for their advantage and growth. The extracted information also plays a crucial role in national security and the mapping of geographical locations. The conventional methods often fail to handle the complexities of this process. so, an effective method is required with high accuracy and stability. In this paper, a new methodology named RankEnsembleFS is proposed that addresses the crucial issues of stability and feature aggregation in the context of the SAT-6 dataset. RankEnsembleFS makes use of a two-step process that consists of ranking the features and then selecting the optimal feature subset from the top-ranked features. RankEnsembleFS achieved comparable accuracy results to state-of-the-art models for the SAT-6 dataset while significantly reducing the feature space. This reduction in feature space is important because it reduces computational complexity and enhances the interpretability of the model. Moreover, the proposed method demonstrated good stability in handling changes in data characteristics, which is critical for reliable performance over time and surpasses existing ML ensemble methods in terms of stability, threshold setting, and feature aggregation. In summary, this paper provides compelling evidence that this RankEnsembleFS methodology presents excellent performance and overcomes key issues in feature selection and image classification for the SAT-6 dataset.

Inherently interpretable machine learning for credit scoring: Optimal classification tree with hyperplane splits

An accurate and interpretable credit scoring model plays a crucial role in helping financial institutions reduce losses by promptly detecting, containing, and preventing defaulters. However, existing models often face a trade-off between interpretability and predictive accuracy. Traditional models like Logistic Regression (LR) offer high interpretability but may have limited predictive performance, while more complex models may improve accuracy at the expense of interpretability. In this paper, we tackle the credit scoring problem with imbalanced data by proposing two new classification models based on the optimal classification tree with hyperplane splits (OCT-H). OCT-H provides transparency and easy interpretation with ‘if-then’ decision tree rules. The first model, the cost-sensitive optimal classification tree with hyperplane splits (CSOCT-H). The second model, the optimal classification tree with hyperplane splits based on maximizing F1-Score (OCT-H-F1), aims to directly maximize the F1-score. To enhance model scalability, we introduce a data sample reduction method using data binning and feature selection. We then propose two solution methods: a heuristic approach and a method utilizing warm-start techniques to accelerate the solving process. We evaluated the proposed models on four public datasets. The results show that OCT-H significantly outperforms traditional interpretable models, such as Decision Trees (DT) and Logistic Regression (LR), in both predictive performance and interpretability. On certain datasets, OCT-H performs as well as or better than advanced ensemble tree models, effectively narrowing the gap between interpretable models and black-box models.

Split tensile strength of fiber-reinforced coral aggregate concrete: Deep learning model and experimental validation

Machine learning methods are increasingly becoming an important research field for solving the complex mechanical behavior of concrete materials. A reliable deep learning model was developed based on a dataset consisting of 319 experimental data points to predict the split tensile strength of fiber-reinforced coral aggregate concrete (FRCAC). The feature selection was carried out using correlation analysis, and the neural network structure was optimized by adjusting hyperparameters based on the 10-fold cross-validation (CV). Subsequently, six statistical indicators were employed to evaluate the accuracy and generalization ability of the model. The deep neural network (DNN) model demonstrates excellent performance, with R2, MAE, and MAPE of 0.98, 0.181 MPa, and 0.037 for the training dataset and 0.95, 0.286 MPa, and 0.074 for the test dataset, respectively. Additionally, 24 sets of concrete proportions are designed using the trained DNN model, and the results indicate that the absolute error and relative error between predicted and experimental values are less than 0.4 MPa and 6.8 %, respectively. The feature analysis based on SHapley Additive exPlanations (SHAP) reveals that volume fraction, curing age, water-binder ratio, and the tensile strength of fiber have the highest contribution in estimating the split tensile strength of FRCAC. This research can provide a reference for optimizing the design of FRCAC.

Fitness and historical success information-assisted binary particle swarm optimization for feature selection

Feature selection is a critical preprocessing step in machine learning aimed at identifying the most relevant features or variables from a dataset. Although conventional particle swarm optimization (PSO) has shown efficiency for feature selection tasks, developing an effective PSO algorithm for this task is still challenging. This study proposes a fitness and historical success information-assisted binary particle swarm optimization, denoted by FPSO. The FPSO is developed by integrating different search strategies, including a weighted center-based approach, historical information-based acceleration coefficients, and selection operation. These strategies are embedded into the FPSO to enhance the levels of exploration and exploitation based on the fitness value of particles and their historical search status. In the FPSO, the transfer function is also added to transform the continuous search space into binary search space. Experimental validation and comparison with seven other metaheuristic algorithms on 24 datasets verify the effectiveness of the FPSO in eliminating irrelevant and redundant features.

Research and performance analysis of random forest-based feature selection algorithm in sports effectiveness evaluation

The rapid progress in fields such as data mining and machine learning, as well as the explosive growth of sports big data, have posed new challenges to the research of sports big data. Most of the available sports data mining techniques concentrates on extracting and constructing effective features for basic sports data, which cannot be achieved simply by using data statistics. Especially in the targeted mining of sports data, traditional mining techniques still have shortcomings such as low classification accuracy and insufficient refinement. In order to solve the problem of low accuracy in traditional mining methods, the study combines the random forest algorithm with the artificial raindrop algorithm, and adopts a sports data mining method based on feature selection to achieve effective analysis of sports big data. This study is based on the evaluation method of motion effects using random forests, and uses feature extraction algorithms to study the motion effect impacts. It uses the information gain index to rank the importance of features and accurately gain the degree of influence of exercise on various indicators of the human body. Through simulation verification, the algorithm proposed by the research institute performs the best in accuracy and FI scores on the training and testing sets, with accuracies of 0.849 ± 0.021 and 0.819 ± 0.022, respectively, and F1 scores of 0.837 ± 0.020 and 0.864 ± 0.021, respectively. This indicates that the algorithm proposed by the research institute has high classification accuracy and performance proves that the Random Forest-based feature selection algorithm established in this study is superior to the existing traditional feature extraction and extraction methods in terms of both performance and accuracy. The proposal of this data analysis method has achieved accurate and efficient utilization of sports big data, which is of great significance for the development of the sports education industry.

ALR-HT: A fast and efficient Lasso regression without hyperparameter tuning

Lasso regression, known for its efficacy in high-dimensional data analysis and feature selection, stands as a cornerstone in the realm of supervised learning for regression estimation. However, hyperparameter tuning for Lasso regression is often time-consuming and susceptible to noisy data in big data scenarios. In this paper we introduce a new additive Lasso regression without Hyperparameter Tuning (ALR-HT) by integrating Markov resampling with additive models. We estimate the generalization bounds of the proposed ALR-HT and establish the fast learning rate. The experimental results for benchmark datasets confirm that the proposed ALR-HT algorithm has better performance in terms of sampling and training total time, mean squared error (MSE) compared to other algorithms. We present some discussions on the ALR-HT algorithm and apply it to Ridge regression, to show its versatility and effectiveness in regularized regression scenarios.

WS-BiTM: Integrating White Shark Optimization with Bi-LSTM for Enhanced Autism Spectrum Disorder Diagnosis

Autism Spectrum Disorder (ASD) is a multifaceted neurodevelopmental condition marked by challenges in social communication, sensory processing, and behavioral regulation. The delayed diagnosis of ASD significantly impedes timely interventions, which can exacerbate symptom severity. With approximately 62 million individuals affected worldwide, the demand for efficient diagnostic tools is critical. This study introduces a novel framework that combines a White Shark Optimization (WSO)-based feature selection method with a Bidirectional Long Short-Term Memory (Bi-LSTM) classifier for enhanced autism classification. Utilizing the WSO technique, we identify key features from autism screening datasets, which markedly improves the model's predictive capabilities. The optimized feature set is then processed by the Bi-LSTM classifier, enhancing its efficiency in handling sequential data. We comprehensively address methodological challenges, including overfitting, generalization, interpretability, and computational efficiency. Furthermore, we conduct a comparative analysis against baseline algorithms such as Neural Networks, Convolutional Neural Networks (CNN), and Long Short-Term Memory (LSTM) networks, while also employing Particle Swarm Optimization (PSO) for feature selection validation. We evaluate performance metrics, including accuracy, F1-score, specificity, precision, and sensitivity across three ASD datasets: Toddlers, Adults, and Children. Our results demonstrate that the WS-BiTM model significantly outperforms baseline methods, achieving accuracies of 97.6%, 96.2%, and 96.4% on the respective datasets. Additionally, we implemented leave-one-dataset cross-validation and confirmed the statistical significance of our findings through a paired t-test, supplemented by an ablation study to detail the contributions of individual model components. These findings highlight the potential of the WS-BiTM model as a robust tool for ASD classification.

Multi-level discrimination index for intuitionistic fuzzy coverings and its applications in feature selection

Multi-level discrimination index for intuitionistic fuzzy coverings and its applications in feature selection

Research on Data Feature Selection of Sharing Platform Based on Particle Swarm Optimization Algorithm

Abstract. This paper is based on the optimization of machine learning models such as SVM (Support Vector Machine) and LightGBM through the application of PSO (Particle Swarm Optimization) and combines shared platform recovery data to provide training material for machine models. The goal is to accelerate the parameter configuration of machine learning models and enhance the rigor of feature selection through the application of algorithmic models. The experimental results show that PSO performs well in optimizing parameters for machine learning models like SVM and LightGBM, achieving high levels in evaluation metrics such as accuracy, recall, precision, and F1 score. Based on this, a framework is proposed for embedding the PSO algorithm into a shared platform architecture, including layers for data collection and preprocessing, algorithm integration and optimization, decision support and service, and feedback and optimization. This framework allows for precise predictions of user behavior, market demand, and other factors, achieving automated scheduling of machine model parameters.

Prediction of Postoperative Mechanical Complications in ASD Patients Based on Total Sequence and Proportional Score of Spinal Sagittal Plane

This article aimed to predict the occurrence of postoperative mechanical complications in adult spinal deformity (ASD) patients through the total sequence and proportional score of the spinal sagittal plane, to improve the quality of life of patients after surgery. The study adopted a comprehensive evaluation and data analysis method, including data collection and preprocessing, feature selection, model construction and training, and constructed a prediction model based on the Random Forest (RF) algorithm. The experimental results showed that the model significantly reduced the risk of complications in randomized controlled trials. The incidence of mechanical complications in the experimental group was 10%, while that in the control group was 25%, with statistical significance (P<0.05). In addition, in retrospective data analysis, the accuracy of the article's model on five datasets ranged from 89% to 93%, outperforming logistic regression and support vector machine models, and performing well on other performance data. In prospective studies, the model's predictions showed good consistency with the actual occurrence of complications. Sensitivity analysis shows that the model has low sensitivity to changes in key parameters and exhibits stability, indicating that the model proposed in this article is suitable for uncertain medical environments. The expert rating further confirmed the effectiveness and practicality of the model in predicting postoperative mechanical complications in ASD patients, with the highest score reaching 4.9. These data demonstrate the high accuracy and clinical potential of the model in predicting postoperative complications of ASD.

Simulation-based Driver Scoring and Profiling System

This paper describes a rule-based Driver Scoring System model, derived from behavioral data collected using a driving simulator. It introduces a novel approach to establish driver profiles through feature engineering of acquired dataset, with features evaluating various aspects of driver behaviour. The research aims to provide employers and drivers with profile-specific feedback and recommendations to design training protocols. Principal Component Analysis is applied on preprocessed dataset from 412 drivers for dimensionality reduction and feature selection. The K-means clustering algorithm is used for data analysis, resulting in three distinct clusters. The Kruskal-Wallis test, supplemented by post hoc Dunn testing is employed to determine statistical significance between clusters. Clusters are portrayed using descriptive statistics, specifically the mean scores and overall driver performance averages. Our method delineates three driver profiles, with two driving styles reflecting desirable driving skills and good overall performance, while the third represents unacceptable driving skills and bad overall performance.