Feature Selection Techniques Research Articles

Robust machine learning based Intrusion detection system using simple statistical techniques in feature selection

There are serious security issues with the quick growth of IoT devices, which are increasingly essential to Industry 4.0. These gadgets frequently function in challenging environments with little energy and processing power, leaving them open to cyberattacks and making it more difficult to implement intrusion detection systems (IDS) that work. In order to address this issue, this study presents a unique feature selection algorithm based on basic statistical methods and a lightweight intrusion detection system. This methodology improves performance and cuts training time by 27–63% for a variety of classifiers. By utilizing the most discriminative features, the suggested methods lower the computational overhead and improve the detection accuracy. The IDS achieved over 99.9% accuracy, precision, recall, and F1-Score on the dataset IoTID20, with consistent performance on the NSLKDD dataset.

Predicting personality traits from Arabic text: an investigation of textual and demographic features with feature selection analysis

Automatic personality recognition (APR) utilizes machine learning to predict personality traits from various data sources. This study aims to predict the big five personality traits from modern standard Arabic (MSA) texts, using both textual and demographic features. The “MSAPersonality” dataset is employed to conduct a comprehensive analysis of features and feature selection methods to evaluate their impact on APR model performance. We compared feature selection algorithms from the filter, wrapper, and embedded-based categories through a systematic experimental design that consisted of feature engineering, feature selection, and regression. This study showed that each trait was more accurately predicted using a distinct set of features. However, age and study level were the most common features among the five traits. Moreover, although there were no statistically significant differences in performance between the feature selection techniques, embedded-based methods offered the best compromise between performance, time, and interpretability. These findings contribute to the understanding of APR in general and among Arabic speakers.

Implementing cloud computing in drug discovery and telemedicine for quantitative structure-activity relationship analysis

This work aims to use cutting-edge machine learning methods to improve quantitative structure-activity relationship (QSAR) analysis, which is used in drug development and telemedicine. The major goal is to examine the performance of several predictive modeling approaches, including random forest, deep learning-based QSAR models, and support vector machines (SVM). It investigates the potential of feature selection techniques developed in chemoinformatics for enhancing model accuracy. The innovative aspect is using cloud computing resources to strengthen computational skills, allowing for managing massive amounts of chemical information. This strategy produces accurate and generalizable QSAR models. By using the cloud's scalability and constant availability, remote healthcare apps have a workable answer. The goal is to show how these methods may improve telemedicine and the drug development process. Utilizing cloud computing equips researchers with a flexible set of tools for precise and timely QSAR analysis, speeding up the discovery of bioactive chemicals for therapeutic use. This new method fits well with the dynamic nature of pharmaceutical study and has the potential to transform the way drugs are developed and delivered to patients via telemedicine.

Student Performance Prediction with Decision Tree Ensembles and Feature Selection Techniques

Student Performance Prediction with Decision Tree Ensembles and Feature Selection Techniques

A Survey on Online Payment Fraud Detection Techniques using Machine Learning Algorithms

Online transaction fraud detection has become a critical challenge with the rise of digital payment systems. This paper surveys various machine learning techniques employed in fraud detection, including Support Vector Machines (SVM-QUBO), Logistic Regression, K-Nearest Neighbors (KNN), Naive Bayes, Decision Trees, and Random Forest. The performance of each method is evaluated in terms of accuracy, precision, recall, and computational efficiency. The study also explores how different algorithms handle high-dimensional, imbalanced datasets and the impact of feature selection techniques. The results provide insights into the strengths and limitations of these algorithms, offering a comprehensive comparison for fraud detection in digital transactions

Review of filtering based feature selection for Botnet detection in the Internet of Things

Botnets are a major security threat in the Internet of Things (IoT), posing significant risks to user privacy, network availability, and the integrity of IoT devices. With the increasing availability of large datasets that contain hundreds or even thousands of variables, selecting the right set of features can be a challenging task. Feature selection is a critical step in developing effective machine learning-based botnet detection systems, as it enables the selection of a subset of features that are most relevant for detection. This paper provides a comprehensive review of filtering based feature selection techniques for botnet detection in IoT. It examines a range of filtering based techniques and evaluates their effectiveness in addressing the challenges and limitations of botnet detection in IoT. It aims to identify the gaps in the literature and areas for future research, and discuss the broader implications of findings for the field of IoT botnet detection. This review provides valuable insights and guidance for researchers and practitioners working on botnet detection in IoT, and highlights the importance of effective feature selection in developing robust and reliable detection systems.

An Analytical Benchmark of Feature Selection Techniques for Industrial Fault Classification Leveraging Time-Domain Features

An Analytical Benchmark of Feature Selection Techniques for Industrial Fault Classification Leveraging Time-Domain Features

A Robust Framework for Detecting Brute-Force Attacks through Deep Learning Techniques

A considerable concern arises with the precise identification of brute-force threats within a networked environment. It emphasizes the need for new methods, as existing ones often lead to many false alarms, as well as delays in real-time threat detection. To tackle these issues, this study proposes a novel intrusion detection framework that utilizes deep learning models for more accurate and efficient detection of brute-force attacks. The framework’s structure includes data collection and preprocessing components performed at the outset of the study using the CSE-CICIDS2018 dataset. The design architecture includes data collection and preprocessing steps. Feature extraction and selection techniques are employed to optimize data for model training. Further, after building the model, various attributes are extracted from the data from feature selection to be used in the training. Then, the construction of multiple architectures of deep learning algorithms, which include Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM) models. Evaluation results show CNN and LSTM achieved the highest accuracy of 99.995 Parsant and 99.99 Parsant respectively. It showcases its ability to detect complex attack patterns in network traffic. It indicates that the CNN network got the best optimum results with a test time of 9.94 seconds. This establishes CNN as an effective method, achieving high accuracy quickly. In comparison, we have surpassed the accuracy of current methods while addressing their weaknesses. The findings are consistent with the effectiveness of CNN in brute-force attack detection frameworks as a more accurate and faster alternative, increasing the capability of detecting intrusions on a network in real-time.

Predicting COPD Readmission: An Intelligent Clinical Decision Support System

Background: COPD is a chronic disease characterized by frequent exacerbations that require hospitalization, significantly increasing the care burden. In recent years, the use of artificial intelligence-based tools to improve the management of patients with COPD has progressed, but the prediction of readmission has been less explored. In fact, in the state of the art, no models specifically designed to make medium-term readmission predictions (2–3 months after admission) have been found. This work presents a new intelligent clinical decision support system to predict the risk of hospital readmission in 90 days in patients with COPD after an episode of acute exacerbation. Methods: The system is structured in two levels: the first one consists of three machine learning algorithms —Random Forest, Naïve Bayes, and Multilayer Perceptron—that operate concurrently to predict the risk of readmission; the second level, an expert system based on a fuzzy inference engine that combines the generated risks, determining the final prediction. The employed database includes more than five hundred patients with demographic, clinical, and social variables. Prior to building the model, the initial dataset was divided into training and test subsets. In order to reduce the high dimensionality of the problem, filter-based feature selection techniques were employed, followed by recursive feature selection supported by the use of the Random Forest algorithm, guaranteeing the usability of the system and its potential integration into the clinical environment. After training the models in the first level, the knowledge base of the expert system was determined on the training data subset using the Wang–Mendel automatic rule generation algorithm. Results: Preliminary results obtained on the test set are promising, with an AUC of approximately 0.8. At the selected cutoff point, a sensitivity of 0.67 and a specificity of 0.75 were achieved. Conclusions: This highlights the system’s future potential for the early identification of patients at risk of readmission. For future implementation in clinical practice, an extensive clinical validation process will be required, along with the expansion of the database, which will likely contribute to improving the system’s robustness and generalization capacity.

Leveraging Explainable AI for Dementia Classification: A Machine Learning Approach

In last two decades the growing life expectancy has spiralled the prevalence of neurogenerative disorder such as Dementia among elderly population posing a significant challenge to mental health globally. The dementia is generally characterized by progressive cognitive decline which it affects memory, thinking, behaviour, and the ability to perform everyday activities. Early diagnosis and intervention are crucial for improving patient outcomes and managing the societal burden of dementia. The primary aim of this study is to deploy various machine learning models such as Logistic Regression, K-Nearest Neighbour (KNN), Support Vector Machine (SVM), Decision Tree, Random Forest (RF). Extreme Gradient Boosting (XGBoost) for the classification of Dementia. The suitable features required for the classification is determined through efficient filter-based and wrapper-based feature selection techniques. The experimental evaluation of machine learning models is performed using standard metrics such as accuracy, precision, recall and F1 score. Furthermore, Explainable AI (XAI) techniques such as SHAP and LIME are employed to interpret the black-box nature of these models, offering transparency and insights into the contribution of individual features to the model predictions. The thorough evaluation of machine learning models exhibited that Random Forest outperforms other models with an accuracy of 96%, with CDR identified as a key predictor through XAI analysis.

A comparison between penalized logistic regressions and classification tree approaches

. The penalized models have been established to be necessary for classifying the emerging complex, high-dimensional datasets. One of these penalized models is the “modified elastic-net” (M-Enet). The special cases of M-Enet (ridge, bridge, Lasso, and elastic-net) can be produced based on its penalty function. By using the fact that the Fisher-Scoring (FS) and Newton-Raphson (NR) algorithms use different approaches to solve numerical problems but are the same in the canonical link, in order to ease implementation, we developed an FS algorithm for the computation of the parameter estimates of the M-Enet. A simulation study of the microbiome as complex, high-dimensional data shows that each special case of the M-Enet penalty function has superiority in the sense of model accuracy, sensitivity, and feature selection technique. A comparison study is performed between the classification of binary responses using M-Enet as a parametric method and the classification tree via “GUIDE” as a non parametric method. The statistical analysis of the numerical results reveals that the special cases of M-Enet models outperform other M-Enet models and GUIDE in terms of average prediction accuracy and sensitivity, with p-values of 0. 0362 and 0. 0214, respectively. Consequently, these models exhibit superior classification performance compared to other classifier models.

Advancing lung cancer diagnosis: Combining 3D auto-encoders and attention mechanisms for CT scan analysis

Objective The goal of this study is to assess the effectiveness of a hybrid deep learning model that combines 3D Auto-encoders with attention mechanisms to detect lung cancer early from CT scan images. The study aims to improve diagnostic accuracy, sensitivity, and specificity by focusing on key features in the scans. Materials and methods A hybrid model was developed that combines feature extraction using 3D Auto-encoder networks with attention mechanisms. First, the 3D Auto-encoder model was tested without attention, using feature selection techniques such as RFE, LASSO, and ANOVA. This was followed by evaluation using several classifiers: SVM, RF, GBM, MLP, LightGBM, XGBoost, Stacking, and Voting. The model's performance was evaluated based on accuracy, sensitivity, F1-Score, and AUC-ROC. After that, attention mechanisms were added to help the model focus on important areas in the CT scans, and the performance was re-assessed. Results The 3D Auto-encoder model without attention achieved an accuracy of 93% and sensitivity of 89%. When attention mechanisms were added, the performance improved across all metrics. For example, the accuracy of SVM increased to 94%, sensitivity to 91%, and AUC-ROC to 0.96. Random Forest (RF) also showed improvements, with accuracy rising to 94% and AUC-ROC to 0.93. The final model with attention improved the overall accuracy to 93.4%, sensitivity to 90.2%, and AUC-ROC to 94.1%. These results highlight the important role of attention in identifying the most relevant features for accurate classification. Conclusions The proposed hybrid deep learning model, especially with the addition of attention mechanisms, significantly improves the early detection of lung cancer. By focusing on key features in the CT scans, the attention mechanism helps reduce false negatives and boosts overall diagnostic accuracy. This approach has great potential for use in clinical applications, particularly in the early-stage detection of lung cancer.

An Agglomerative Clustering Combined with an Unsupervised Feature Selection Approach for Structural Health Monitoring

Structural health monitoring (SHM) is critical for ensuring the safety and longevity of structures, yet existing methodologies often face challenges such as high data dimensionality, lack of interpretability, and reliance on extensive label datasets. Current research in SHM has primarily focused on supervised approaches, which require significant manual effort for data labeling and are less adaptable to new environments. Additionally, the large volume of data generated from dynamic structural monitoring campaigns often includes irrelevant or redundant features, further complicating the analysis and reducing computational efficiency. This study addresses these issues by introducing an unsupervised learning approach for SHM, employing an agglomerative clustering model alongside an unsupervised feature selection technique utilizing box-plot statistics. The proposed method is assessed through raw acceleration signals obtained from four dynamic structural monitoring campaigns, including 44 features with temporal, statistical, and spectral information. In addition, these features are also evaluated in terms of their relevance, and the most important ones are selected for a new execution of the computational procedure. The proposed feature selection not only reduces data dimensionality but also enhances model interpretability, improving the clustering performance in terms of homogeneity, completeness, V-measure, and adjusted Rand score. The results obtained for the four analyzed cases provide clear insights into the patterns of behavior and structural anomalies.

A multi-model machine learning framework for breast cancer risk stratification using clinical and imaging data

Purpose This study presents a comprehensive machine learning framework for assessing breast cancer malignancy by integrating clinical features with imaging features derived from deep learning. Methods The dataset included 1668 patients with documented breast lesions, incorporating clinical data (e.g., age, BI-RADS category, lesion size, margins, and calcifications) alongside mammographic images processed using four CNN architectures: EfficientNet, ResNet, DenseNet, and InceptionNet. Three predictive configurations were developed: an imaging-only model, a hybrid model combining imaging and clinical data, and a stacking-based ensemble model that aggregates both data types to enhance predictive accuracy. Twelve feature selection techniques, including ReliefF and Fisher Score, were applied to identify key predictive features. Model performance was evaluated using accuracy and AUC, with 5-fold cross-valida tion and hyperparameter tuning to ensure robustness. Results The imaging-only models demonstrated strong predictive performance, with EfficientNet achieving an AUC of 0.76. The hybrid model combining imaging and clinical data reached the highest accuracy of 83% and an AUC of 0.87, underscoring the benefits of data integration. The stacking-based ensemble model further optimized accuracy, reaching a peak AUC of 0.94, demonstrating its potential as a reliable tool for malignancy risk assessment. Conclusion This study highlights the importance of integrating clinical and deep imaging features for breast cancer risk stratification, with the stacking-based model.

Distributed Denial of Services (DDoS) attack detection in SDN using Optimizer-equipped CNN-MLP.

Software-Defined Networks (SDN) provides more control and network operation over a network infrastructure as an emerging and revolutionary paradigm in networking. Operating the many network applications and preserving the network services and functions, the SDN controller is regarded as the operating system of the SDN-based network architecture. The SDN has several security problems because of its intricate design, even with all its amazing features. Denial-of-service (DoS) attacks continuously impact users and Internet service providers (ISPs). Because of its centralized design, distributed denial of service (DDoS) attacks on SDN are frequent and may have a widespread effect on the network, particularly at the control layer. We propose to implement both MLP (Multilayer Perceptron) and CNN (Convolutional Neural Networks) based on conventional methods to detect the Denial of Services (DDoS) attack. These models have got a complex optimizer installed on them to decrease the false positive or DDoS case detection efficiency. We use the SHAP feature selection technique to improve the detection procedure. By assisting in the identification of which features are most essential to spot the incidents, the approach aids in the process of enhancing precision and flammability. Fine-tuning the hyperparameters with the help of Bayesian optimization to obtain the best model performance is another important thing that we do in our model. Two datasets, InSDN and CICDDoS-2019, are utilized to assess the effectiveness of the proposed method, 99.95% for the true positive (TP) of the CICDDoS-2019 dataset and 99.98% for the InSDN dataset, the results show that the model is highly accurate.

Efficient diagnosis of diabetes mellitus using an improved ensemble method

Diabetes is a growing health concern in developing countries, causing considerable mortality rates. While machine learning (ML) approaches have been widely used to improve early detection and treatment, several studies have shown low classification accuracies due to overfitting, underfitting, and data noise. This research employs parallel and sequential ensemble ML approaches paired with feature selection techniques to boost classification accuracy. The Pima India Diabetes Data from the UCI ML Repository served as the dataset. Data preprocessing included cleaning the dataset by replacing missing values with column means and selecting highly correlated features using forward and backward selection methods. The dataset was split into two parts: training (70%), and testing (30%). Python was used for classification in Jupyter Notebook, and there were two design phases. The first phase utilized J48, Classification and Regression Tree (CART), and Decision Stump (DS) to create a random forest model. The second phase employed the same algorithms alongside sequential ensemble methods—XG Boost, AdaBoostM1, and Gradient Boosting—using an average voting algorithm for binary classification. Evaluation revealed that XG Boost, AdaBoostM1, and Gradient Boosting achieved classification accuracies of 100%, with performance metrics including F1 score, MCC, Precision, Recall, AUC-ROC, and AUC-PR all equal to 1.00, indicating reliable predictions of diabetes presence. Researchers and practitioners can leverage the predictive model developed in this work to make quick predictions of diabetes mellitus, which could save many lives.

Enhancing Laser-Induced Breakdown Spectroscopy Spectral Quantification Through Minimum Redundancy and Maximum Relevance-Based Feature Selection

Laser-induced breakdown spectroscopy (LIBS) is a rapid, non-contact analytical technique that is widely applied in various fields. However, the high dimensionality and information redundancy of LIBS spectral data present challenges for effective model development. This study aims to assess the effectiveness of the minimum redundancy and maximum relevance (mRMR) method for feature selection in LIBS spectral data and to explore its adaptability across different predictive modeling approaches. Using the ChemCam LIBS dataset, we constructed predictive models with four quantitative methods: random forest (RF), support vector regression (SVR), back propagation neural network (BPNN), and partial least squares regression (PLSR). We compared the performance of mRMR-based feature selection with that of full-spectrum data and three other feature selection methods: competitive adaptive re-weighted sampling (CARS), Regressional ReliefF (RReliefF), and neighborhood component analysis (NCA). Our results demonstrate that the mRMR method significantly reduces the number of selected features while improving model performance. This study validates the effectiveness of the mRMR algorithm for LIBS feature extraction and highlights the potential of feature selection techniques to enhance predictive accuracy. The findings provide a valuable strategy for feature selection in LIBS data analysis and offer significant implications for the practical application of LIBS in predicting elemental content in geological samples.

A problem-agnostic approach to feature selection and analysis using SHAP

Feature selection is an effective data reduction technique. SHapley Additive exPlanations (SHAP) can be used to provide a feature importance ranking for models built with labeled or unlabeled data. Thus, one may use the SHAP feature importance ranking in a feature selection technique by selecting the k highest ranking features. Furthermore, this SHAP-based feature selection technique is applicable regardless of the availability of labels for data. We use the Kaggle Credit Card Fraud detection dataset to simulate three label availability scenarios. When no labeled data is available, unsupervised learners should be used. We explore feature selection for data reduction with Isolation Forest and SHAP for this case. When data of one class is available, a one-class classifier, such as Gaussian Mixture Model (GMM) can be used in combination with SHAP for determining feature importance, and for feature selection. Finally, if labeled data from both classes is available a binary-class classifier can be used in conjunction with SHAP for data reduction. Our contribution is to provide a comparative analysis of features selected in the three label availability scenarios. Our primary conclusion is that feature sets may be reduced with SHAP without compromising performance. To the best of our knowledge, this is the first study to explore a feature analysis technique, applicable in the three label availability scenarios.

Deep Learning for Bond Yield Forecasting: The LSTM‐LagLasso

ABSTRACTWe present long short‐term memory (LSTM)‐LagLasso, a novel explainable deep learning approach applied to bond yield forecasting. Our method involves feature selection from a large universe of potential features and forecasts bond yields using dynamic LSTM networks. It examines the internal gating signals of a trained LSTM and explains their dynamics through exogenous variables that may influence bond price formation. By considering these variables at various lags and using the Lasso technique for feature selection, we demonstrate how different hidden units within the LSTM dynamically adjust to make predictions across different temporal regimes and how their evolution is shaped by various external factors. In an empirical study on government bond yield forecasting, we demonstrate the statistical accuracy of LSTM‐LagLasso compared to a multilayer perceptron (MLP) and highlight its explainability.

Integrating Ensemble Learning and Information Gain for Malware Detection based on Static and Dynamic Features

The rapid advancement of malware poses a significant threat to devices, like personal computers and mobile phones. One of the most serious threats commonly faced is malicious software, including viruses, worms, trojan horses, and ransomware. Conventional antivirus software is becoming ineffective against the ever-evolving nature of malware, which can now take on various forms like polymorphic, metamorphic, and oligomorphic variants. These advanced malware types can not only replicate and distribute themselves, but also create unique fingerprints for each offspring. To address this challenge, a new generation of antivirus software based on machine learning is needed. This intelligent approach can detect malware based on its behavior, rather than relying on outdated fingerprint-based methods. This study explored the integration of machine learning models for malware detection using various ensemble algorithms and feature selection techniques. The study compared three ensemble algorithms: Gradient Boosting, Random Forest, and AdaBoost. It used Information Gain for feature selection, analyzing 21 features. Additionally, the study employed a public dataset called ‘Malware Static and Dynamic Features VxHeaven and VirusTotal Data Set’, which encompasses both static and dynamic malware features. The results demonstrate that the Gradient Boosting algorithm combined with Information Gain feature selection achieved the highest performance, reaching an accuracy and F1-Score of 99.2%.