Ensemble Feature Selection Method Research Articles

Hybrid Machine Learning Approach for Early Diagnosis of Polycystic Ovary Syndrome with Stable Features

Polycystic Ovary Syndrome (PCOS) is a hormonal condition that typically affects female during the time of their reproduction. It is identified by the disruptions in hormonal balance, particularly an increase in levels of androgen (male hormone) in the female body. PCOS can lead to various symptoms and health complications including irregular menstrual cycles, ovarian cysts, fertility issues, insulin resistance, weight gain, acne, and excess hair growth. The real-world PCOS detection is a challenging task whilst PCOS specific cause is unknown and its symptoms are unclear. Thus, accurate and timely diagnosis of PCOS is crucial for effective management and prevention of long-term complications. In such cases, Machine learning based PCOS prediction model support diagnostic process, address potential errors and time constraints. Machine learning algorithms can analyze large set of patient data, including medical history, hormonal profiles, and imaging results, to assist in the diagnosis of PCOS. In particular, the performance of data analysis chore and prediction model is improved by ensemble feature selection strategies. These methods concentrate on selecting a subset of pertinent features from a broader range of features. The unstable nature of the outcome of feature selection algorithm is a frequent issue in practical applications, when it is applied multiple times on similar dataset or with slight modifications in the data. Thus, evaluating the robustness of feature selection algorithm is most important. To address these issues and quantify the robustness, this study uses Jenson-Shannon divergence, an information theoretic approach with ensemble feature selection method to handle the various findings, such as complete ranking, half ranking and top-k lists (without ranking). Furthermore, this article proposes a hybrid machine learning classifier with SMOTE – SVM for the prompt detection of PCOS and the performance of the model is compared with a number of other individual classifiers including KNN (K-Nearest Neighbour), Support Vector Machine (SVM), AdaBoost, LR –Logistic Regression, NB –Nave Bayes, RF –Random Forest, Decision Tree. The proposed SWISS-AdaBoost classifier surpassed other models with 97.81% of accuracy and AUC of 99.08%.

Ensemble learning-based stability improvement method for feature selection towards performance prediction

The uncertainty and complexity of real data collected in the industrial production process increase the difficulty in data-based knowledge discovering. Feature selection is an important step to remove redundant and irrelevant data, and thus it is essential to construct an efficient feature selection method. In this paper, an ensemble learning-driven stable feature selection method is proposed to improve the stability and accuracy of the feature selection. Firstly, datasets of different characteristics are generated to increase the diversity of data segments for feature selection. Secondly, two criteria (stability and prediction accuracy) are adopted to evaluate the performance weight of each feature selection algorithm, to ensure that the results of high-performance selectors have high priority in the algorithm aggregation process. Thirdly, the feature subsets are weighted and filtered based on expert experience to further ensure its stability. Finally, comparative experiments are conducted to show the effectiveness of the proposed method. Comparing with other methods, the proposed one can achieve the highest overall stability for feature selection (namely 0.936 measured by the Spearman rank correlation coefficient), and select the reasonable feature subset for data-driven prediction with the low mean absolute error (namely 0.315 as the average level).

Machine learning-based prediction of in-hospital mortality for critically ill patients with sepsis-associated acute kidney injury

Objectives This study aims to develop and validate a prediction model in-hospital mortality in critically ill patients with sepsis-associated acute kidney injury (SA-AKI) based on machine learning algorithms. Methods Patients who met the criteria for inclusion were identified in the Medical Information Mart for Intensive Care-IV (MIMIC-IV) database and divided according to the validation (n = 2440) and development (n = 9756, 80%) queues. Ensemble stepwise feature selection method was used to screen for effective features. The prediction models of short-term mortality were developed by seven machine learning algorithms. Ten-fold cross-validation was used to verify the performance of the algorithm in the development queue. The area under the receiver operating characteristic curve (ROC-AUC) was used to evaluate the differentiation accuracy and performance of the prediction model in the validation queue. The best-performing model was interpreted by Shapley additive explanations (SHAP). Results A total of 12,196 patients were enrolled in this study. Eleven variables were finally chosen to develop the prediction model. The AUC of the random forest (RF) model was the highest value both in the Ten-fold cross-validation and evaluation (AUC: 0.798, 95% CI: 0.774–0.821). According to the SHAP plots, old age, low Glasgow Coma Scale (GCS) score, high AKI stage, reduced urine output, high Simplified Acute Physiology Score (SAPS II), high respiratory rate, low temperature, low absolute lymphocyte count, high creatinine level, dysnatremia, and low body mass index (BMI) increased the risk of poor prognosis. Conclusions The RF model developed in this study is a good predictor of in-hospital mortality for patients with SA-AKI in the intensive care unit (ICU), which may have potential applications in mortality prediction.

Performance Analysis of Ensemble Learning and Feature Selection Methods in Loan Approval Prediction at Banks

Applying for a loan at a bank has a series of relevant assessments based on data and credit scores in determining a borrower's eligibility to receive a loan from the bank. Machine learning is the basis for evaluating whether an individual is worthy of obtaining a loan, in order to reduce the potential risks faced by banks. This research aims to obtain the best accuracy value from the Loan Approval Prediction dataset which is sourced from the open dataset provider website, namely Kaggle. This Loan Approval Prediction dataset has 14 features with 4,269 data. The results of dataset analysis carried out on 4,269 data showed that the amount of data that could be studied was 4,173 data (2,599 data were approved and 1,574 data were rejected). The results of the feature importance evaluation on 14 features show that loan amount is the most important feature compared to other features, while bank asset value is the feature that has the lowest influence. Research on the Loan Approval Prediction dataset was also carried out by testing several Decision Tree ensemble models, including Extreme Gradient Boosting or XGBoost, Light Gradient Boosting Machine (Light GBM), Gradient Boosting, Random Forest, Adaptive Boosting (Adaboost) and Extra Trees. The comparison results show that the XGBoost (Extreme Gradient Boosting) model is the best model, with Accuracy 0.9974, AUC 0.9998, Recall 0.9963, Prec 0.9969, F1 0.9966.

Weighted Rank Difference Ensemble: A New Form of Ensemble Feature Selection Method for Medical Datasets

Background: Feature selection (FS), a crucial preprocessing step in machine learning, greatly reduces the dimension of data and improves model performance. This paper focuses on selecting features for medical data classification. Methods: In this work, a new form of ensemble FS method called weighted rank difference ensemble (WRD-Ensemble) has been put forth. It combines three FS methods to produce a stable and diverse subset of features. The three base FS approaches are Pearson’s correlation coefficient (PCC), reliefF, and gain ratio (GR). These three FS approaches produce three distinct lists of features, and then they order each feature by importance or weight. The final subset of features in this study is chosen using the average weight of each feature and the rank difference of a feature across three ranked lists. Using the average weight and rank difference of each feature, unstable and less significant features are eliminated from the feature space. The WRD-Ensemble method is applied to three medical datasets: chronic kidney disease (CKD), lung cancer, and heart disease. These data samples are classified using logistic regression (LR). Results: The experimental results show that compared to the base FS methods and other ensemble FS methods, the proposed WRD-Ensemble method leads to obtaining the highest accuracy value of 98.97% for CKD, 93.24% for lung cancer, and 83.84% for heart disease. Conclusion: The results indicate that the proposed WRD-Ensemble method can potentially improve the accuracy of disease diagnosis models, contributing to advances in clinical decision-making.

Feature selection and its combination with data over-sampling for multi-class imbalanced datasets

Feature selection aims at filtering out some unrepresentative features from a given dataset in order to construct more effective learning models. Furthermore, ensemble feature selection by combining multiple feature selection methods has shown its outperformance over single feature selection. However, the performances of different (ensemble) feature selection methods have not been fully examined over multi-class imbalanced datasets. On the other hand, for class imbalanced datasets, one widely considered solution is to re-balance the datasets by data over-sampling, which generates some synthetic examples for the minority classes. However, the effect of performing (ensemble) feature selection on over-sampling multi-class imbalanced datasets has not been investigated. Therefore, the first research objective is to examine the performances of single and ensemble feature selection methods by fifteen well-known filter, wrapper, and embedded algorithms in terms of classification accuracy. For the second research objective, two orders of combining the feature selection and over-sampling steps are compared in order to find out the best combination procedure as well as the best combined algorithms. The experimental results based on ten different domain datasets containing low to very high feature dimensions show that ensemble feature selection methods slightly perform better than single ones. However, their performance differences are not big. To combine with the Synthetic Minority Oversampling Technique (SMOTE) over-sampling algorithm, performing feature selection first and over-sampling second outperforms the other procedure. Although the best combined algorithms are based on ensemble feature selection, eXtreme Gradient Boosting (XGBoost), as the single best feature selection algorithm, combined with SMOTE provides very similar classification performance to the best combined algorithms. To consider the issues of classification performance and compactional cost, the optimal solution is based on the combined XGBoost and SMOTE.

An Ensemble Feature Selection Approach-Based Machine Learning Classifiers for Prediction of COVID-19 Disease.

The respiratory disease of coronavirus disease 2019 (COVID-19) has wreaked havoc on the economy of every nation by infecting and killing millions of people. This deadly disease has taken a toll on the life of the entire human race, and an exact cure for it is still not developed. Thus, the control and cure of this disease mainly depend on restricting its transmission rate through early detection. The detection of coronavirus infection facilitates the isolation and exclusive care of infected patients. This research paper proposes a novel data mining system that combines the ensemble feature selection method and machine learning classifier for the effective identification of COVID-19 infection. Different feature selection approaches including chi-square test, recursive feature elimination (RFE), genetic algorithm (GA), particle swarm optimization (PSO), and random forest are evaluated for their effectiveness in enhancing the classification accuracy of the machine learning classifiers. The classifiers that are considered in this research work are decision tree, naïve Bayes, K-nearest neighbor (KNN), multilayer perceptron (MLP), and support vector machine (SVM). Two COVID-19 datasets were used for testing from which the best features supporting the dataset were extracted by the proposed system. The performance of the machine learning classifiers based on the ensemble feature selection methods is analyzed.

Ensemble Feature Selection Method for Single Pulse Classification

Affected by a large number of radio frequency interference signals, it has become an important task for astronomical data processing to quickly and accurately identify single pulse signals from massive observation data. Designing and extracting effective data features is the key issue for efficient identification of single pulse signals using machine learning. This paper proposes an ensemble feature selection method for single pulse signal classification. The method first mixed three types of features, including the parametric features, statistical features, and abstract features of single pulse signals, and then used five individual feature selection methods to select the corresponding optimal feature set, respectively. At last, the features selected by the five individual methods are mixed, and the greedy strategy was used to select the optimal ensemble feature set. The experimental results show that the ensemble feature set can improve F1-score by a value of 1.8% at most, and can obtain higher accuracy than the features selected by individual methods. Under the background of high-speed and large-scale sky survey, the ensemble feature selection method plays an important role in reducing the number of features, improving classification performance, and speeding up data processing.

A Learning Approach for The Identification of Network Intrusions Based on Ensemble XGBoost Classifier

The limitation of signature-based Intrusion Detection systems has given rise for the popularity of Machine learning (ML) approaches r for building such intrusion detection systems (IDSs). ML is a sub-filed of Artificial Intelligence that enables algorithms to learn from data and its applications have been widely accepted and used in many domains. To achieve a promising ML-based model that can identify attacks and intrusions in networks and the cyber space, different stages of machine learning approach like pre-processing, attribute selection, model building, hyper parameter tuning can be very important. CICIDS2017 intrusion dataset was used for all the experimentations. This study focuses on building cyber threat detection model based on the ensemble feature selection and classification method. Innovative approaches were used for the analysis and pre-processing of the dataset. Thereafter, XGboost algorithm was used for selecting relevant features from the default input attributes in each of the captures. Thereafter, the reduced features were employed in the identification of cyber intrusions. The average accuracy achieved in the 8 captures of the dataset is 98% while precision is 0.98. Also, recall is 0.98, f1-score is 0.98 while AUC ROC score is 0.99. The study concluded that XGBoost-based model was able to achieve promising results based on the proper dataset encoding, feature importance-based feature selection and tuning of the algorithm for intrusion identification.

MCDM-EFS: A novel ensemble feature selection method for software defect prediction using multi-criteria decision making

Software defect prediction models are used for predicting high risk software components. Feature selection has significant impact on the prediction performance of the software defect prediction models since redundant and unimportant features make the prediction model more difficult to learn. Ensemble feature selection has recently emerged as a new methodology for enhancing feature selection performance. This paper proposes a new multi-criteria-decision-making (MCDM) based ensemble feature selection (EFS) method. This new method is termed as MCDM-EFS. The proposed method, MCDM-EFS, first generates the decision matrix signifying the feature’s importance score with respect to various existing feature selection methods. Next, the decision matrix is used as the input to well-known MCDM method TOPSIS for assigning a final rank to each feature. The proposed approach is validated by an experimental study for predicting software defects using two classifiers K-nearest neighbor (KNN) and naïve bayes (NB) over five open-source datasets. The predictive performance of the proposed approach is compared with existing feature selection algorithms. Two evaluation metrics – nMCC and G-measure are used to compare predictive performance. The experimental results show that the MCDM-EFS significantly improves the predictive performance of software defect prediction models against other feature selection methods in terms of nMCC as well as G-measure.

A feature selection method based on multiple feature subsets extraction and result fusion for improving classification performance

Directly applying high-dimensional data to machine learning leads to dimensionality disasters and may induce model overfitting. Feature selection can effectively reduce feature size. However, a single feature selection algorithm has instability and poor generalization ability problems. The ensemble feature selection algorithm is complex to find a suitable feature subset aggregation strategy. To solve these two problems, we propose a feature selection method based on multiple feature subsets extraction and result fusion (FSM). Generate multiple feature subsets to improve stability. This method uses mutual information to mine the relationship between features and categories. Fast non-dominated sorting combines this correlation to distribute similar features in the same layer. A layer optimization algorithm is proposed to combine the layered features to generate multiple different feature subsets. To evaluate the excellence of feature subsets, FSM uses precision, recall, and F-Score comprehensively to assess and remove ineffective feature subsets. The idea of fusion is put on the output of the results. Multiple superior feature subsets train various classifiers. The results of numerous classifiers are fused as the final output according to the voting method, simplifying the ensemble feature selection method’s aggregation process. Experiments on 20 well-known datasets show that FSM can effectively reduce the data dimension and improve the classification performance compared with the original datasets. FSM performs well in classification performance and efficiency compared with other dimensionality reduction algorithms.

An explainable machine learning based prediction model for Alzheimer's disease in China longitudinal aging study.

Alzheimer's disease (AD) is the most common cause of dementia. Accurate prediction and diagnosis of AD and its prodromal stage, i.e., mild cognitive impairment (MCI), is essential for the possible delay and early treatment for the disease. In this paper, we adopt the data from the China Longitudinal Aging Study (CLAS), which was launched in 2011, and includes a joint effort of 15 institutions all over the country. Four thousand four hundred and eleven people who are at least 60 years old participated in the project, where 3,514 people completed the baseline survey. The survey collected data including demographic information, daily lifestyle, medical history, and routine physical examination. In particular, we employ ensemble learning and feature selection methods to develop an explainable prediction model for AD and MCI. Five feature selection methods and nine machine learning classifiers are applied for comparison to find the most dominant features on AD/MCI prediction. The resulting model achieves accuracy of 89.2%, sensitivity of 87.7%, and specificity of 90.7% for MCI prediction, and accuracy of 99.2%, sensitivity of 99.7%, and specificity of 98.7% for AD prediction. We further utilize the SHapley Additive exPlanations (SHAP) algorithm to visualize the specific contribution of each feature to AD/MCI prediction at both global and individual levels. Consequently, our model not only provides the prediction outcome, but also helps to understand the relationship between lifestyle/physical disease history and cognitive function, and enables clinicians to make appropriate recommendations for the elderly. Therefore, our approach provides a new perspective for the design of a computer-aided diagnosis system for AD and MCI, and has potential high clinical application value.

An Optimized Bagging Learning with Ensemble Feature Selection Method for URL Phishing Detection

An Optimized Bagging Learning with Ensemble Feature Selection Method for URL Phishing Detection

Filter and Wrapper Stacking Ensemble (FWSE): a robust approach for reliable biomarker discovery in high-dimensional omics data.

Selecting informative features, such as accurate biomarkers for disease diagnosis, prognosis and response to treatment, is an essential task in the field of bioinformatics. Medical data often contain thousands of features and identifying potential biomarkers is challenging due to small number of samples in the data, method dependence and non-reproducibility. This paper proposes a novel ensemble feature selection method, named Filter and Wrapper Stacking Ensemble (FWSE), to identify reproducible biomarkers from high-dimensional omics data. In FWSE, filter feature selection methods are run on numerous subsets of the data to eliminate irrelevant features, and then wrapper feature selection methods are applied to rank the top features. The method was validated on four high-dimensional medical datasets related to mental illnesses and cancer. The results indicate that the features selected by FWSE are stable and statistically more significant than the ones obtained by existing methods while also demonstrating biological relevance. Furthermore, FWSE is a generic method, applicable to various high-dimensional datasets in the fields of machine intelligence and bioinformatics.

Cancer Classification Utilizing Voting Classifier with Ensemble Feature Selection Method and Transcriptomic Data.

Biomarker-based cancer identification and classification tools are widely used in bioinformatics and machine learning fields. However, the high dimensionality of microarray gene expression data poses a challenge for identifying important genes in cancer diagnosis. Many feature selection algorithms optimize cancer diagnosis by selecting optimal features. This article proposes an ensemble rank-based feature selection method (EFSM) and an ensemble weighted average voting classifier (VT) to overcome this challenge. The EFSM uses a ranking method that aggregates features from individual selection methods to efficiently discover the most relevant and useful features. The VT combines support vector machine, k-nearest neighbor, and decision tree algorithms to create an ensemble model. The proposed method was tested on three benchmark datasets and compared to existing built-in ensemble models. The results show that our model achieved higher accuracy, with 100% for leukaemia, 94.74% for colon cancer, and 94.34% for the 11-tumor dataset. This study concludes by identifying a subset of the most important cancer-causing genes and demonstrating their significance compared to the original data. The proposed approach surpasses existing strategies in accuracy and stability, significantly impacting the development of ML-based gene analysis. It detects vital genes with higher precision and stability than other existing methods.

ADHD diagnosis using structural brain MRI and personal characteristic data with machine learning framework.

An essential yet challenging task is an automatic diagnosis of attention-deficit/hyperactivity disorder (ADHD) without manual intervention. The present study emphasises utilizing structural MRI and personal characteristic (PC) data for developing an automated diagnostic system for ADHD classification. Here, an age-balanced dataset of 316 ADHD and 316 Typically Developing Children (TDC) was prepared from the publicly available dataset. We extracted volumetric features from gray matter (GM) volumes from brain regions defined by Automated Anatomical Labelling (AAL3) atlas and cortical thickness-based (CT) features using the Destrieux atlas. A set of salient features were selected independently using minimum redundancy and maximum relevance (mRMR) and ensemble feature selection (EFS) methods. Decision models were trained using five well-known classifiers: K-nearest neighbours, logistic regression, linear Support Vector Machine (SVM), radial-based SVM (RBSVM), and Random Forest. The performance of the proposed system was evaluated using accuracy, recall, and specificity with ten runs of a ten-fold cross-validation scheme. We run seven experiments by considering different combinations of features. The maximum classification accuracy of 75% was obtained with CT and PC features with RBSVM and SVM with the EFS. An increase in GM volume in fifteen brain regions and loss of cortical thickness in twenty-seven brain regions were observed.

MIC-SHAP: An ensemble feature selection method for materials machine learning

Feature selection has kept playing a significant role in the workflow of materials machine learning, but currently most of works of materials machine learning tend to use single or stepwise feature selection methods. A new ensemble feature selection method named MIC-SHAP was proposed in this work, which combines the SHapley Additive exPlanations (SHAP) method and the maximal information coefficient (MIC) method. The effectiveness of the ensemble feature selection method was evaluated with three different material datasets collected from publications. The results have demonstrated that MIC-SHAP method outperforms the commonly used feature selection methods, guaranteeing the prediction accuracy and greatly reducing the model complexity. The highest feature reduction rate is 91.67%, while the R2 of the 10-fold cross-validation reaches 0.98. The MIC-SHAP method could quickly select the optimal feature subset effectively, avoiding repeated attempts of different feature selection methods. Moreover, the MIC-SHAP method could increase the stability and interpretability of feature selection to help the subsequent process of materials design and discovery.

Vibration signal-based defect detection method for railway signal relay using parameter-optimized VMD and ensemble feature selection

Railway signal relay (RSR) plays a critical role in the railway signaling system and its failure may bring serious consequences. At present, the manual-based RSR defect detection method has many shortcomings. A vibration-based defect detection method for RSR is proposed to solve this problem. First, since most of the variational mode decomposition (VMD) optimization strategies do not apply to the vibration signal of RSR, a parameter-optimized VMD based on a hybrid criterion of sample entropy (SampEn) and Pearson correlation coefficient (PCC) is proposed. Second, an ensemble feature selection method combining Filter method and Embedded method is proposed to select defect-sensitive features. The ensemble is considered as a multi-criteria decision-making (MCDM) problem and solved using the VIKOR method, which can integrate the advantages of each feature selection method to select the optimal feature subset. Finally, the effectiveness of the proposed method is verified by real vibration signals of RSR.

Heterogeneous Ensemble Variable Selection To Improve Customer Prediction Using Naive Bayes Model

The analysis of customer patterns and behaviours is essential for all businesses, as the customer is the sole source of revenue. Understanding customer patterns and behavior enables businesses to enhance their business processes and customer happiness. The availability of voluminous client datasets within organizations facilitates efficient customer analysis. Yet, the inclusion of interrelated, irrelevant, as well as missing factors leads to a poor forecast of the dataset. Feature selection techniques are investigated in order to handle the problem. Objective of feature selection is to pick the pertinent variables from a complete set of associated, irrelevant, and missing variables. In general, FS is classified into 3 types: filter, wrapper, & hybrid method. The filter method is a quick one, but the variables used are ineffective. Similarly, a wrapper method is effective yet computationally inefficient. In this study, an ensemble feature selection strategy is presented and tested to circumvent the issue with these feature selections. There are two techniques to ensemble FS: one is homogenous and the other is heterogeneous. This study employs a heterogeneous ensemble feature selection method. In the suggested method, the learning dataset is applied to five distinct filter FS approaches, and the ranked attributes that result are aggregated using two distinct methods: the mean method and the min method. Relevant variables are chosen to further build the final sorted qualities using the cut off value as a guide. As the HEVS technique's filter approach simply ranks the variables, it is necessary to select the variable subset cut off value. The experimental technique is conducted from two distinct vantage points: Heterogeneous ensemble variable selection with Naive Bayes and Naive Bayes without variable selection. In the end, the outcomes that were obtained via the use of the two different approaches are compared using different factors. The experimental results demonstrate that the suggested HEVS method outperforms the usual Naive Bayes model. As relevant variables are included when modeling using NB, the computational complexity of this proposed methodology is also minimized.

A two-stage hybrid biomarker selection method based on ensemble filter and binary differential evolution incorporating binary African vultures optimization

BackgroundIn the field of genomics and personalized medicine, it is a key issue to find biomarkers directly related to the diagnosis of specific diseases from high-throughput gene microarray data. Feature selection technology can discover biomarkers with disease classification information.ResultsWe use support vector machines as classifiers and use the five-fold cross-validation average classification accuracy, recall, precision and F1 score as evaluation metrics to evaluate the identified biomarkers. Experimental results show classification accuracy above 0.93, recall above 0.92, precision above 0.91, and F1 score above 0.94 on eight microarray datasets.MethodThis paper proposes a two-stage hybrid biomarker selection method based on ensemble filter and binary differential evolution incorporating binary African vultures optimization (EF-BDBA), which can effectively reduce the dimension of microarray data and obtain optimal biomarkers. In the first stage, we propose an ensemble filter feature selection method. The method combines an improved fast correlation-based filter algorithm with Fisher score. obviously redundant and irrelevant features can be filtered out to initially reduce the dimensionality of the microarray data. In the second stage, the optimal feature subset is selected using an improved binary differential evolution incorporating an improved binary African vultures optimization algorithm. The African vultures optimization algorithm has excellent global optimization ability. It has not been systematically applied to feature selection problems, especially for gene microarray data. We combine it with a differential evolution algorithm to improve population diversity.ConclusionCompared with traditional feature selection methods and advanced hybrid methods, the proposed method achieves higher classification accuracy and identifies excellent biomarkers while retaining fewer features. The experimental results demonstrate the effectiveness and advancement of our proposed algorithmic model.