Ensemble Feature Selection Technique Research Articles

Evolutionary-based ensemble feature selection technique for dynamic application-specific credit risk optimization in FinTech lending

Evolutionary-based ensemble feature selection technique for dynamic application-specific credit risk optimization in FinTech lending

DNN-BP: a novel framework for cuffless blood pressure measurement from optimal PPG features using deep learning model.

Continuous blood pressure (BP) provides essential information for monitoring one's health condition. However, BP is currently monitored using uncomfortable cuff-based devices, which does not support continuous BP monitoring. This paper aims to introduce a blood pressure monitoring algorithm based on only photoplethysmography (PPG) signals using the deep neural network (DNN). The PPG signals are obtained from 125 unique subjects with 218 records and filtered using signal processing algorithms to reduce the effects of noise, such as baseline wandering, and motion artifacts. The proposed algorithm is based on pulse wave analysis of PPG signals, extracted various domain features from PPG signals, and mapped them to BP values. Four feature selection methods are applied and yielded four feature subsets. Therefore, an ensemble feature selection technique is proposed to obtain the optimal feature set based on major voting scores from four feature subsets. DNN models, along with the ensemble feature selection technique, outperformed in estimating the systolic blood pressure (SBP) and diastolic blood pressure (DBP) compared to previously reported approaches that rely only on the PPG signal. The coefficient of determination ( ) and mean absolute error (MAE) of the proposed algorithm are 0.962 and 2.480 mmHg, respectively, for SBP and 0.955 and 1.499 mmHg, respectively, for DBP. The proposed approach meets the Advancement of Medical Instrumentation standard for SBP and DBP estimations. Additionally, according to the British Hypertension Society standard, the results attained Grade A for both SBP and DBP estimations. It concludes that BP can be estimated more accurately using the optimal feature set and DNN models. The proposed algorithm has the potential ability to facilitate mobile healthcare devices to monitor continuous BP.

Improving the performance and interpretability on medical datasets using graphical ensemble feature selection.

A major hindrance towards using Machine Learning (ML) on medical datasets is the discrepancy between a large number of variables and small sample sizes. While multiple feature selection techniques have been proposed to avoid the resulting overfitting, overall ensemble techniques offer the best selection robustness. Yet, current methods designed to combine different algorithms generally fail to leverage the dependencies identified by their components. Here, we propose Graphical Ensembling (GE), a graph-theory-based ensemble feature selection technique designed to improve the stability and relevance of the selected features. Relying on four datasets, we show that GE increases classification performance with fewer selected features. For example, on rheumatoid arthritis patient stratification, GE outperforms the baseline methods by 9% Balanced Accuracy while relying on fewer features. We use data on sub-cellular networks to show that the selected features (proteins) are closer to the known disease genes, and the uncovered biological mechanisms are more diversified. By successfully tackling the complex correlations between biological variables, we anticipate that GE will improve the medical applications of ML. https://github.com/ebattistella/auto_machine_learning.

A Novel Ensemble Feature Selection Technique for Cancer Classification Using Logarithmic Rank Aggregation Method

Recent studies have shown that ensemble feature selection (EFS) has achieved outstanding performance in microarray data classification. However, some issues remain partially resolved, such as suboptimal aggregation methods and non-optimised underlying FS techniques. This study proposed the logarithmic rank aggregate (LRA) method to improve feature aggregation in EFS. Additionally, a hybrid aggregation framework was presented to improve the performance of the proposed method by combining it with several methods. Furthermore, the proposed method was applied to the feature rank lists obtained from the optimised FS technique to investigate the impact of FS technique optimisation. The experimental setup was performed on five binary microarray datasets. The experimental results showed that LRA provides a comparable classification performance to mean rank aggregation (MRA) and outperforms MRA in terms of gene selection stability. In addition, hybrid techniques provided the same or better classification accuracy as MRA and significantly improved stability. Moreover, some proposed configurations had better accuracy, sensitivity, and specificity performance than MRA. Furthermore, the optimised LRA drastically improved the FS stability compared to the unoptimised LRA and MRA. Finally, When the results were compared with other studies, it was shown that optimised LRA provided a remarkable stability performance, which can help domain experts diagnose cancer diseases with a relatively smaller subset of genes.

Application of Stacked Ensemble Techniques on Ensemble Feature Selection Techniques for Classifying Recurrent Head and Neck Squamous Cell Carcinoma Prognosis

This study aimed to identify the optimal combination of the stacked ensemble (SE) and the heterogeneous ensemble feature selection (HETR-EFS) technique for classifying HNSCC recurrence patterns. Four SE classification models were developed based on various EFS techniques, using GBM meta-classifiers in each case. The results showed that implementing the SE technique consisting of five base classifiers on the heterogeneous ensemble feature (HETR-EF) subset achieved better performance than other EF subsets and HETR-EFs. Thus, learning SE technique having five base classifiers on HETR-EFs is clinically appropriate as a prognostic model for classifying and predicting HNSCC patients' recurrence data. The SE technique, which combines base classifier models, is clinically appropriate for classifying and predicting HNSCC patients' recurrence data. The study highlights the importance of finding a machine learning algorithm that performs best given varied distributions, as not all algorithms are equally created.

Classification of Breast Cancer using Ensemble Filter Feature Selection with Triplet Attention Based Efficient Net Classifier

In medical imaging, the effective detection and classification of Breast Cancer (BC) is a current research important task because of the still existing difficulty to distinguish abnormalities from normal breast tissues due to their subtle appearance and ambiguous margins and distinguish abnormalities from the normal breast. Moreover, BC detection based on an automated detection model is needed, because manual diagnosis faces problems due to cost and shortage of skilled manpower, and also takes a very long time. Using deep learning and ensemble feature selection techniques, in this paper, a novel framework is introduced for classifying BC from histopathology images. The five primary steps of the suggested framework are as follows: 1) to make the largest original dataset and then deep learning model with data augmentation to improve the learning. 2) The best features are selected by an Ensemble Filter Feature selection Method (EFFM) which combines the best feature subsets to produce the final feature subsets. 3) Then the pruned Convolution Neural Network (CNN) model is utilized to extract the optimal features. 4) Finally, the classification is done through the Triplet Attention based Efficient Network (TAENet) classifier. The suggested model produces a 98% accuracy rate after being trained and tested on two different histopathology imaging datasets including images from four different data cohorts. Subsequently, the suggested strategy outperforms the conventional ones since the ensemble filter habitually acquires the best features, and experimental results demonstrate the importance of the proposed approach

UNMASKING FRAUDSTERS: Ensemble Features Selection to Enhance Random Forest Fraud Detection

Fraud detection is used in various industries, including banking institutes, finance, insurance, government agencies, etc. Recent increases in the number of fraud attempts make fraud detection crucial for safeguarding financial information that is confidential or personal. Many types of fraud problems exist, including card-not-present fraud, fake Marchant, counterfeit checks, stolen credit cards, and others. An ensemble feature selection technique based on Recursive feature elimination (RFE), Information gain (IG), and Chi-Squared (X2) in concurrence with the Random Forest algorithm, was proposed to give research findings and results on fraud detection and prevention. The objective was to choose the essential features for training the model. The Receiver Operating Characteristic (ROC) Score, Accuracy, F1 Score, and Precision are used to evaluate the model's performance. The findings show that the model can differentiate between fraudulent transactions and those that are not, with an ROC Score of 95.83% and an Accuracy of 99.6%. The F1 Score of 99.6%% and precision of 100% further sustain the model's ability to detect fraudulent transactions with the least false positives correctly. The ensemble feature selection technique reduced training time and did not compromise the model's performance, making it a valuable tool for businesses in preventing fraudulent transactions.

Novel ensemble feature selection techniques applied to high-grade gastroenteropancreatic neuroendocrine neoplasms for the prediction of survival

Background and ObjectiveDetermining the most informative features for predicting the overall survival of patients diagnosed with high-grade gastroenteropancreatic neuroendocrine neoplasms is crucial to improve individual treatment plans for patients, as well as the biological understanding of the disease. The main objective of this study is to evaluate the use of modern ensemble feature selection techniques for this purpose with respect to (a) quantitative performance measures such as predictive performance, (b) clinical interpretability, and (c) the effect of integrating prior expert knowledge. MethodsThe Repeated Elastic Net Technique for Feature Selection (RENT) and the User-Guided Bayesian Framework for Feature Selection (UBayFS) are recently developed ensemble feature selectors investigated in this work. Both allow the user to identify informative features in datasets with low sample sizes and focus on model interpretability. While RENT is purely data-driven, UBayFS can integrate expert knowledge a priori in the feature selection process. In this work, we compare both feature selectors on a dataset comprising 63 patients and 110 features from multiple sources, including baseline patient characteristics, baseline blood values, tumor histology, imaging, and treatment information. ResultsOur experiments involve data-driven and expert-driven setups, as well as combinations of both. In a five-fold cross-validated experiment without expert knowledge, our results demonstrate that both feature selectors allow accurate predictions: A reduction from 110 to approximately 20 features (around 82%) delivers near-optimal predictive performances with minor variations according to the choice of the feature selector, the predictive model, and the fold. Thereafter, we use findings from clinical literature as a source of expert knowledge. In addition, expert knowledge has a stabilizing effect on the feature set (an increase in stability of approximately 40%), while the impact on predictive performance is limited. ConclusionsThe features WHO Performance Status, Albumin, Platelets, Ki-67, Tumor Morphology, Total MTV, Total TLG, and SUVmax are the most stable and predictive features in our study. Overall, this study demonstrated the practical value of feature selection in medical applications not only to improve quantitative performance but also to deliver potentially new insights to experts.

Using Random Undersampling and Ensemble Feature Selection for IoT Attack Prediction

One consequence of the widespread use of (IoT) devices is an increase in the volume of attacks on (IoT) networks. In this study, we focus on the Bot-IoT dataset, with the aim of classifying its four types of attacks: Denial-of-Service (DoS), Distributed Denial-of-Service (DDoS), Reconnaissance, and Information Theft. Our contribution is based on the evaluation of the Random Undersampling (RUS) technique and ensemble Feature Selection Techniques (FSTs). Our results indicate that RUS has a positive impact on overall classification performance. Furthermore, our results show that the FSTs are beneficial for DoS, Reconnaissance, and Information Theft classification but not for DDoS classification. Finally, we note that the ensemble classifiers have generally outperformed the nonensemble classifiers in our study.

Microarray Gene Expression Data Classification via Wilcoxon Sign Rank Sum and Novel Grey Wolf Optimized Ensemble Learning Models.

Cancer is a deadly disease that affects the lives of people all over the world. Finding a few genes relevant to a single cancer disease can lead to effective treatments. The difficulty with microarray datasets is their high dimensionality; they have a large number of features in comparison to the small number of samples in these datasets. Additionally, microarray data typically exhibit significant asymmetry in dimensionality as well as high levels of redundancy and noise. It is widely held that the majority of genes lack informative value about the classes under study. Recent research has attempted to reduce this high dimensionality by employing various feature selection techniques. This paper presents new ensemble feature selection techniques via the Wilcoxon Sign Rank Sum test (WCSRS) and the Fisher's test (F-test). In the first phase of the experiment, data preprocessing was performed; subsequently, feature selection was performed via the WCSRS and F-test in such a way that the (probability values) p-values of the WCRSR and F-test were adopted for cancerous gene identification. The extracted gene set was used to classify cancer patients using ensemble learning models (ELM), random forest (RF), extreme gradient boosting (Xgboost), cat boost, and Adaboost. To boost the performance of the ELM, we optimized the parameters of all the ELMs using the Grey Wolf optimizer (GWO). The experimental analysis was performed on colon cancer, which included 2000 genes from 62 patients (40 malignant and 22 benign). Using a WCSRS test for feature selection, the optimized Xgboost demonstrated 100% accuracy. The optimized cat boost, on the other hand, demonstrated 100% accuracy using the F-test for feature selection. This represents a 15% improvement over previously reported values in the literature.

Efficient prediction of evaporation using ensemble feature selection techniques

For the timely planning and management of water resources, evaporation prediction must be estimated properly, especially in regions that are prone to drought and where evaporation directly affects the pest population. Changes in meteorological variables such as temperature, relative humidity, solar radiation, rainfall have a great impact on the evaporation process. In order to forecast the variable, ensemble feature selection techniques along with various machine learning techniques were investigated. Meteorological weekly weather data were collected from the ICRISAT location over a period from 1974 to 2021. The reliability of these developed models was based on statistical approaches namely Mean Absolute Error, Root Mean Square Error, Coefficient of Determination, Nash–Sutcliffe Efficiency coefficient, and Willmott’s Index of agreement along with several graphical aids. The results indicate that lasso regression outperforms all other machine learning approaches and the results are validated using current data (2020-2021). For a better understanding of the results, these validated results were also compared with results obtained from the established linear regression method and artificial neural network. It was further found that lasso regression shows an improved performance (R2 = 0.929) over linear regression (R2 = 0.871) and artificial neural network (R2 = 0.889).

A new feature popularity framework for detecting cyberattacks using popular features

We propose a novel feature popularity framework, and introduce this new framework to the cybersecurity domain. Feature popularity has not yet been used in machine learning or data mining, and we implement it with three web attacks from the CSE-CIC-IDS2018 dataset: Brute Force, SQL Injection, and XSS web attacks. Feature popularity is based upon ensemble Feature Selection Techniques (FSTs) and allows us to more easily understand common and important features between different cyberattacks. Three filter-based and four supervised learning-based FSTs are used to generate feature subsets for each of our three different web attack datasets, and then our feature popularity frameworks are applied. Classification performance for feature popularity is mostly similar as compared to when “all features” are evaluated (with feature popularity subsets having better performance in 5 out of 15 experiments). Our feature popularity technique effectively builds an ensemble of ensembles by first building an ensemble of FSTs for each dataset, and then building another ensemble across a dataset agreement dimension. The Jaccard similarity is also employed with our feature popularity framework in order to better identify which attack classes should (or should not) be grouped together when applying feature popularity. The four most popular features across all three web attacks from this experiment are: Flow_Bytes_s, Flow_IAT_Max, Fwd_IAT_Std, and Fwd_IAT_Total. When only using these four features as input to our models, classification performance is not seriously degraded. This feature popularity framework granted us new and previously unseen insights into the web attack detection process with CSE-CIC-IDS2018 big data, even though we had intensely studied it previously. We realized these four particular features cannot properly identify our three web attacks, as they operate mainly from the time dimension and NetFlow features from layers 3 and 4 of the OSI model. Conversely, our three web attacks operate in the application layer (7) of the OSI model and should not leave signatures in these four features. Feature popularity produces easier to explain models which provide domain experts better visibility into the problem, and can also reduce the complexity of implementing models in real-world systems.

Predictive factors for allergy at 4–6 years of age based on machine learning: A pilot study

BackgroundIn Europe, allergic diseases are the most common chronic childhood illnesses and the result of a complex interplay between genetics and environmental factors. A new approach for analyzing this complex data is to employ machine learning (ML) algorithms. Therefore, the aim of this pilot study was to find predictors for the presence of parental-reported allergy at 4–6 years of age by using feature selection in ML. MethodsA recursive ensemble feature selection (REFS) was used, with a 20% step reduction and with eight different classifiers in the ensemble, and resampling given the class unbalance. Thereafter, the Receiver Operating Characteristic Curves for five different classifiers, not included in the original ensemble feature selection technique, were calculated. ResultsIn total, 130 children (14 with and 116 without parental-reported allergy) and 248 features were included in the ML analyses. The REFS algorithm showed a result of 20 features and particularly, the Multi-layer Perceptron Classifier had an area under the curve (AUC) of 0.86 (SD 0.08). The features predictive for allergy were: tobacco exposure during pregnancy, atopic parents, gestational age, days of: diarrhea, cough, rash, and fever during first year of life, ever being exposed to antibiotics, Resistin, IL-27, MMP9, CXCL8, CCL13, Vimentin, IL-4, CCL22, GAL1, IL-6, LIGHT, and GMCSF. ConclusionsThis ML model shows that a combination of environmental exposures and cytokines can predict later allergy with an AUC of 0.86 despite the small sample size. In the future, our ML model still needs to be externally validated.

A user-guided Bayesian framework for ensemble feature selection in life science applications (UBayFS)

Feature selection reduces the complexity of high-dimensional datasets and helps to gain insights into systematic variation in the data. These aspects are essential in domains that rely on model interpretability, such as life sciences. We propose a (U)ser-Guided (Bay)esian Framework for (F)eature (S)election, UBayFS, an ensemble feature selection technique embedded in a Bayesian statistical framework. Our generic approach considers two sources of information: data and domain knowledge. From data, we build an ensemble of feature selectors, described by a multinomial likelihood model. Using domain knowledge, the user guides UBayFS by weighting features and penalizing feature blocks or combinations, implemented via a Dirichlet-type prior distribution. Hence, the framework combines three main aspects: ensemble feature selection, expert knowledge, and side constraints. Our experiments demonstrate that UBayFS (a) allows for a balanced trade-off between user knowledge and data observations and (b) achieves accurate and robust results.

Deep-Learning-Based Approach to Detect ICMPv6 Flooding DDoS Attacks on IPv6 Networks

Internet Protocol version six (IPv6) is more secure than its forerunner, Internet Protocol version four (IPv4). IPv6 introduces several new protocols, such as the Internet Control Message Protocol version six (ICMPv6), an essential protocol to the IPv6 networks. However, it exposes IPv6 networks to some security threats since ICMPv6 messages are not verified or authenticated, and they are mandatory messages that cannot be blocked or disabled. One of the threats currently facing IPv6 networks is the exploitation of ICMPv6 messages by malicious actors to execute distributed denial of service (DDoS) attacks. Therefore, this paper proposes a deep-learning-based approach to detect ICMPv6 flooding DDoS attacks on IPv6 networks by introducing an ensemble feature selection technique that utilizes chi-square and information gain ratio methods to select significant features for attack detection with high accuracy. In addition, a long short-term memory (LSTM) is employed to train the detection model on the selected features. The proposed approach was evaluated using a synthetic dataset for false-positive rate (FPR), detection accuracy, F-measure, recall, and precision, achieving 0.55%, 98.41%, 98.39%, 97.3%, and 99.4%, respectively. Additionally, the results reveal that the proposed approach outperforms the existing approaches.

A robust ensemble feature selection technique for high‐dimensional datasets based on minimum weight threshold method

AbstractEnsemble feature selection (EFS) is a valuable technique for developing accurate and robust machine‐learning (ML) models. Data variation plays a crucial role in the success of EFS models; however, it also causes some outliers in the ranked lists. In this study, we proposed the minimum weight threshold method‐based EFS (MWT‐EFS) to address the outlier problem and use the true power of EFS. The proposed method employs the support vector classifier to assign weights for features, and the MWT method handles outliers in the ranked feature lists while creating the ensemble list. First, a threshold value is determined. After that, the feature weights below the threshold are replaced with this value. This approach eliminates the negative effect of outliers. After the new feature weights are assigned, the average of the feature weights is calculated (mean aggregation) for all features, and the ensemble (final) feature list is created accordingly. The experiment results showed that the proposed method significantly improves gene selection stability while maintaining classification performance and reducing computational complexity. In conclusion, the proposed method led to an accurate and robust classification that can help domain experts to make more confident decisions with less effort, resources, and time.

A Novel Extra Tree Ensemble Optimized DL Framework (ETEODL) for Early Detection of Diabetes.

Diabetes has been recognized as a global medical problem for more than half a century. Patients with diabetes can benefit from the Internet of Things (IoT) devices such as continuous glucose monitoring (CGM), intelligent pens, and similar devices. Smart devices generate continuous data streams that must be processed in real-time to benefit the users. The amount of medical data collected is vast and heterogeneous since it is gathered from various sources. An accurate diagnosis can be achieved through a variety of scientific and medical techniques. It is necessary to process this streaming data faster to obtain relevant and significant knowledge. Recently, the research has concentrated on improving the prediction model's performance by using ensemble-based and Deep Learning (DL) approaches. However, the performance of the DL model can degrade due to overfitting. This paper proposes the Extra-Tree Ensemble feature selection technique to reduce the input feature space with DL (ETEODL), a predictive framework to predict the likelihood of diabetes. In the proposed work, dropout layers follow the hidden layers of the DL model to prevent overfitting. This research utilized a dataset from the UCI Machine learning (ML) repository for an Early-stage prediction of diabetes. The proposed scheme results have been compared with state-of-the-art ML algorithms, and the comparison validates the effectiveness of the predictive framework. This proposed work, which outperforms the other selected classifiers, achieves a 97.38 per cent accuracy rate. F1-Score, precision, and recall percent are 96, 97.7, and 97.7, respectively. The comparison unveils the superiority of the suggested approach. Thus, the proposed method effectively improves the performance against the earlier ML techniques and recent DL approaches and avoids overfitting.

IoT information theft prediction using ensemble feature selection

The recent years have seen a proliferation of Internet of Things (IoT) devices and an associated security risk from an increasing volume of malicious traffic worldwide. For this reason, datasets such as Bot-IoT were created to train machine learning classifiers to identify attack traffic in IoT networks. In this study, we build predictive models with Bot-IoT to detect attacks represented by dataset instances from the Information Theft category, as well as dataset instances from the data exfiltration and keylogging subcategories. Our contribution is centered on the evaluation of ensemble feature selection techniques (FSTs) on classification performance for these specific attack instances. A group or ensemble of FSTs will often perform better than the best individual technique. The classifiers that we use are a diverse set of four ensemble learners (Light GBM, CatBoost, XGBoost, and random forest (RF)) and four non-ensemble learners (logistic regression (LR), decision tree (DT), Naive Bayes (NB), and a multi-layer perceptron (MLP)). The metrics used for evaluating classification performance are area under the receiver operating characteristic curve (AUC) and Area Under the precision-recall curve (AUPRC). For the most part, we determined that our ensemble FSTs do not affect classification performance but are beneficial because feature reduction eases computational burden and provides insight through improved data visualization.

Automatic ensemble feature selection using fast non-dominated sorting

Feature selection refers to selecting optimal feature subset with effective data preprocessing policy in making high dimensional data for diverse pattern recognition problems. The aims of feature selection are enhancing accuracy, improving the evaluation performance, and finding the smallest effective feature subset. In this study, ensemble feature selection method is adopted based on an assumption indicating that a combination of several feature selection methods obtains more robust results than any individual feature selection method. Accordingly, when carrying out ensemble feature selection, a combinational method should be used to combine rankings of features from diverse algorithms into an individual rank for each feature. It is also required to set a threshold to acquire a functional subset of features. In this work, a three-step ensemble feature selection technique called Automatic Thresholding Feature Selection (ATFS) is proposed. The first step involves diversity generation where multiple rankers are applied to each dataset to generate different feature rankings. Second, output rankings of individual selectors are combined using fast non-dominated sorting that is a combinational method empowering the proposed ensemble with automatic thresholding capability. Third, feature sets are generated to obtain the optimal feature set. Additionally, a new filter method called Sorted Label Interference (SLI) is proposed based on interference between class labels. Both SLI and ATFS are applicable to binary datasets. The performance of SLI and ATFS is at least comparable and often better than the performance of individual rankers and existing ensemble methods. The obtained results also show that the use of ATFS-generated threshold improves not only the performance of ATFS and SLI, but also the performance of other filters and combinational methods.

Detecting cybersecurity attacks across different network features and learners

Machine learning algorithms efficiently trained on intrusion detection datasets can detect network traffic capable of jeopardizing an information system. In this study, we use the CSE-CIC-IDS2018 dataset to investigate ensemble feature selection on the performance of seven classifiers. CSE-CIC-IDS2018 is big data (about 16,000,000 instances), publicly available, modern, and covers a wide range of realistic attack types. Our contribution is centered around answers to three research questions. The first question is, “Does feature selection impact performance of classifiers in terms of Area Under the Receiver Operating Characteristic Curve (AUC) and F1-score?” The second question is, “Does including the Destination_Port categorical feature significantly impact performance of LightGBM and Catboost in terms of AUC and F1-score?” The third question is, “Does the choice of classifier: Decision Tree (DT), Random Forest (RF), Naive Bayes (NB), Logistic Regression (LR), Catboost, LightGBM, or XGBoost, significantly impact performance in terms of AUC and F1-score?” These research questions are all answered in the affirmative and provide valuable, practical information for the development of an efficient intrusion detection model. To the best of our knowledge, we are the first to use an ensemble feature selection technique with the CSE-CIC-IDS2018 dataset.