Ensemble Selection Research Articles

A penalized variable selection ensemble algorithm for high-dimensional group-structured data.

This paper presents a multi-algorithm fusion model (StackingGroup) based on the Stacking ensemble learning framework to address the variable selection problem in high-dimensional group structure data. The proposed algorithm takes into account the differences in data observation and training principles of different algorithms. It leverages the strengths of each model and incorporates Stacking ensemble learning with multiple group structure regularization methods. The main approach involves dividing the data set into K parts on average, using more than 10 algorithms as basic learning models, and selecting the base learner based on low correlation, strong prediction ability, and small model error. Finally, we selected the grSubset + grLasso, grLasso, and grSCAD algorithms as the base learners for the Stacking algorithm. The Lasso algorithm was used as the meta-learner to create a comprehensive algorithm called StackingGroup. This algorithm is designed to handle high-dimensional group structure data. Simulation experiments showed that the proposed method outperformed other R2, RMSE, and MAE prediction methods. Lastly, we applied the proposed algorithm to investigate the risk factors of low birth weight in infants and young children. The final results demonstrate that the proposed method achieves a mean absolute error (MAE) of 0.508 and a root mean square error (RMSE) of 0.668. The obtained values are smaller compared to those obtained from a single model, indicating that the proposed method surpasses other algorithms in terms of prediction accuracy.

Ensemble learning classifiers hybrid feature selection for enhancing performance of intrusion detection system

Feature selection (FS) plays an important role in the construction of efficient ensemble classifiers; particularly for intrusion detection system (IDS). An IDS is a utilized in a network architecture to protect the availability of sensitive information. However, existing IDSs suffer from redundancy, high dimensionality, and high false alarm rate (FAR). Also, lots of models are constructed for outdated datasets, which makes them less flexible to deal with new assaults. Therefore, this paper proposes a new IDS relies on hybrid FS and ensemble classifiers. A hybrid FS approach consists of two techniques, hard-voting and mean. In contrast to recent papers, we use three different FS approaches: extra tree classifier importance as an embedded FS, recursive feature elimination (RFE) as a wrapper FS, and mutual information (MI) as a filter FS. Then, a hard-voting technique has been used to fuse output of these approaches and obtain a reduced subset of features. Since each feature has three weights, a mean technique has been utilized to assign one weight to each feature and obtain an optimal subset of features. The experimental outcomes, utilizing the modern InSDN dataset, confirm that the proposed hybrid FS with ensemble soft voting classifier achieves better results than other ensemble and individual classifiers due to several measures.

Evolutionary shift detection with ensemble variable selection

Abrupt environmental changes can lead to evolutionary shifts in trait evolution. Identifying these shifts is an important step in understanding the evolutionary history of phenotypes. The detection performances of different methods are influenced by many factors, including different numbers of shifts, shift sizes, where a shift occurs on a tree, and the types of phylogenetic structure. Furthermore, the model assumptions are oversimplified, so are likely to be violated in real data, which could cause the methods to fail. We perform simulations to assess the effect of these factors on the performance of shift detection methods. To make the comparisons more complete, we also propose an ensemble variable selection method (R package ELPASO) and compare it with existing methods (R packages ℓ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell$$\\end{document}1ou and PhylogeneticEM). The performances of methods are highly dependent on the selection criterion. ℓ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell$$\\end{document}1ou+pBIC is usually the most conservative method and it performs well when signal sizes are large. ℓ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell$$\\end{document}1ou+BIC is the least conservative method and it performs well when signal sizes are small. The ensemble method provides more balanced choices between those two methods. Moreover, the performances of all methods are heavily impacted by measurement error, tree reconstruction error and shifts in variance.

Feature selection and its combination with data over-sampling for multi-class imbalanced datasets

Feature selection aims at filtering out some unrepresentative features from a given dataset in order to construct more effective learning models. Furthermore, ensemble feature selection by combining multiple feature selection methods has shown its outperformance over single feature selection. However, the performances of different (ensemble) feature selection methods have not been fully examined over multi-class imbalanced datasets. On the other hand, for class imbalanced datasets, one widely considered solution is to re-balance the datasets by data over-sampling, which generates some synthetic examples for the minority classes. However, the effect of performing (ensemble) feature selection on over-sampling multi-class imbalanced datasets has not been investigated. Therefore, the first research objective is to examine the performances of single and ensemble feature selection methods by fifteen well-known filter, wrapper, and embedded algorithms in terms of classification accuracy. For the second research objective, two orders of combining the feature selection and over-sampling steps are compared in order to find out the best combination procedure as well as the best combined algorithms. The experimental results based on ten different domain datasets containing low to very high feature dimensions show that ensemble feature selection methods slightly perform better than single ones. However, their performance differences are not big. To combine with the Synthetic Minority Oversampling Technique (SMOTE) over-sampling algorithm, performing feature selection first and over-sampling second outperforms the other procedure. Although the best combined algorithms are based on ensemble feature selection, eXtreme Gradient Boosting (XGBoost), as the single best feature selection algorithm, combined with SMOTE provides very similar classification performance to the best combined algorithms. To consider the issues of classification performance and compactional cost, the optimal solution is based on the combined XGBoost and SMOTE.

Selective ensemble method for anomaly detection based on parallel learning

Anomaly detection is a highly important task in the field of data analysis. Traditional anomaly detection approaches often strongly depend on data size, structure and features, while introducing the idea of ensemble into anomaly detection can greatly improve the generalization ability. Ensemble-based anomaly detection methods still face some challenges, however, such as data imbalance, time and space demand and the selection of base detectors. To this end, we propose a selective ensemble method for anomaly detection based on parallel learning (SEAD-PL). First, a differentiated stratified sampling method is designed to alleviate the problem of data imbalance. Then, a distributed parallel training frame is built to address the problem of excessive time and space consumption for base detector training. Finally, a clustering-based ensemble selection strategy is introduced to balance the accuracy and diversity of base detectors. Experiments are performed on six datasets, which demonstrate that the proposed method has obvious advantages over four selected methods.

Cluster ensemble selection based on maximum quality-maximum diversity

Diversity and quality are two important factors that affect clustering ensemble performance. Some base clusterings are irrelevant and redundant, which decreases the performance of the clustering ensemble. By removing irrelevant and redundant clusterings, diversity and quality are increased simultaneously, which often leads to a more accurate ensemble solution. The relationship between quality and diversity as an optimization problem is a challenge. Based on the minimum redundancy-maximum relevance (mRMR) criterion, pair-wise and non-pair-wise methods are proposed. In the pair-wise method, each clustering is weighted in contrast to other base clusterings, whereas in the non-pair-wise method, virtual labeling is obtained using a consensus function, and then based on this labeling, each clustering is weighted. To evaluate the performance of these methods, several experiments were conducted on 10 real datasets, and the obtained results were compared to those of full ensembles. The results showed that the proposed methods led to a more significant performance improvement compared with full ensembles and other clustering ensemble selection methods.

A cross-domain intelligent fault diagnosis method based on multi-source domain feature adaptation and selection

Although fault diagnosis methods integrating transfer learning are research hotspots, their ability to handle industrial fault diagnosis problems with large domain differences still needs to be improved. A multi-source domain feature adaptation and selection method is presented to address the issues of domain mismatch and domain negative transfer. The method integrates the top-level network parameter transfer strategy with the 2D convolutional neural network backbone network to acquire the target domain feature extractor quickly. Multiple feature adaptive extractors (FAEs) are constructed using a multi-branch structure to align the source and target domain’s feature distributions, respectively. The inter-domain distance computed by multi-kernel maximum mean discrepancy is embedded in the FAEs loss function to improve the inter-domain matching degree. Based on the information gain of the adaptively integrated features, the ensemble adaptive selection is performed on the extracted feature matrices to exclude the negative transfer feature. Finally, the effective feature matrix is input into the diagnosis classifier for classification. Cross-domain fault diagnosis experiments are developed based on the data set gathered from several types of rotating machinery operated under varied working conditions. The experimental results show that the proposed method outperforms the existing intelligent fault diagnosis methods in terms of fault detection accuracy, generalization, and stability.

The Credit Card Anti-fraud Detection Model in the Context of Dynamic Integration Selection Algorithm

Because of its close relationship with information technology, every change in the field of information technology will have an important impact on the means, time and space distribution of the final fraud of the credit card network. The rapid development and popularization of artificial intelligence (AI) software based on Deepfake technology has greatly promoted the intelligent transformation of credit card network fraud. Through the use of advanced hardware and software equipment, criminals have iterated on their own fraud tools, from the use of traditional phone calls and text messages to the use of the latest AI software. Therefore, for the problem of credit card fraud detection with missing data set labels and highly unbalanced category distribution, A credit card approval anomaly detection model DES-HBOS (Dynamic Ensemble Selection based on Histogram-based Outlier Score) is proposed. Firstly, the unsupervised anomaly detection algorithm is used to construct the false label of the training set customer. Then, the customer capability area to be measured is determined, and the classifier performance is evaluated according to Pearson correlation coefficient. Finally, a set of optimal classifiers is selected to integrate the test customers. Experiments on real credit card customer data sets show that DES-HBOS has a higher Recall and can identify more fraudulent customers than other 6 classical anomaly detection models. Comparative experiments were carried out on 4 unbalanced data sets, and the experimental results showed that DES-HBOS had stronger anomaly detection ability than HBOS.

An ensemble learning-based feature selection algorithm for identification of biomarkers of renal cell carcinoma.

Feature selection plays a crucial role in classification tasks as part of the data preprocessing process. Effective feature selection can improve the robustness and interpretability of learning algorithms, and accelerate model learning. However, traditional statistical methods for feature selection are no longer practical in the context of high-dimensional data due to the computationally complex. Ensemble learning, a prominent learning method in machine learning, has demonstrated exceptional performance, particularly in classification problems. To address the issue, we propose a three-stage feature selection algorithm framework for high-dimensional data based on ensemble learning (EFS-GINI). Firstly, highly linearly correlated features are eliminated using the Spearman coefficient. Then, a feature selector based on the F-test is employed for the first stage selection. For the second stage, four feature subsets are formed using mutual information (MI), ReliefF, SURF, and SURF* filters in parallel. The third stage involves feature selection using a combinator based on GINI coefficient. Finally, a soft voting approach is proposed to employ for classification, including decision tree, naive Bayes, support vector machine (SVM), k-nearest neighbors (KNN) and random forest classifiers. To demonstrate the effectiveness and efficiency of the proposed algorithm, eight high-dimensional datasets are used and five feature selection methods are employed to compare with our proposed algorithm. Experimental results show that our method effectively enhances the accuracy and speed of feature selection. Moreover, to explore the biological significance of the proposed algorithm, we apply it on the renal cell carcinoma dataset GSE40435 from the Gene Expression Omnibus database. Two feature genes, NOP2 and NSUN5, are selected by our proposed algorithm. They are directly involved in regulating m5c RNA modification, which reveals the biological importance of EFS-GINI. Through bioinformatics analysis, we shows that m5C-related genes play an important role in the occurrence and progression of renal cell carcinoma, and are expected to become an important marker to predict the prognosis of patients.

Ensemble Feature Selection Method for Single Pulse Classification

Affected by a large number of radio frequency interference signals, it has become an important task for astronomical data processing to quickly and accurately identify single pulse signals from massive observation data. Designing and extracting effective data features is the key issue for efficient identification of single pulse signals using machine learning. This paper proposes an ensemble feature selection method for single pulse signal classification. The method first mixed three types of features, including the parametric features, statistical features, and abstract features of single pulse signals, and then used five individual feature selection methods to select the corresponding optimal feature set, respectively. At last, the features selected by the five individual methods are mixed, and the greedy strategy was used to select the optimal ensemble feature set. The experimental results show that the ensemble feature set can improve F1-score by a value of 1.8% at most, and can obtain higher accuracy than the features selected by individual methods. Under the background of high-speed and large-scale sky survey, the ensemble feature selection method plays an important role in reducing the number of features, improving classification performance, and speeding up data processing.

Design optimization of a multi-source renewable energy system using a novel method based on selective ensemble learning

This study conducts a comparative assessment to optimize the design of a Hybrid Renewable Energy System (HRES) consisting of PV panels, wind turbines (WTs), and hydrogen storage (PV/WT/FC). The study focuses on Dakhla City, leveraging its favorable weather conditions to evaluate the potential of renewable energy sources. A novel approach, Selective Ensemble Marine Predators (SEMPA), is introduced. SEMPA utilizes a selective ensemble learning strategy to enhance the performance of the Marine Predators Algorithm (MPA) and determine the optimal system size with the lowest Total Net Present Cost (TNPC) and improved reliability. To benchmark SEMPA's performance, we compare it against established methods, including MPA, Particle Swarm Optimization (PSO), and Harmony Search (HS). Results demonstrate that the integrated PV/WT/FC system provides a reliable and cost-effective energy solution for the southwest region of Morocco, particularly in Dakhla. SEMPA outperforms HS, PSO, and traditional MPA, achieving stable convergence after 30 iterations. The system exhibits a superior Cost of Energy (COE) at 0.3127 $/kWh, a TNPC of 0.9689 M$, and a low Loss of Power Supply Probability (LPSP) at 0.04747.

Classifier Ensemble Based on Multiview Optimization for High-Dimensional Imbalanced Data Classification.

High-dimensional class imbalanced data have plagued the performance of classification algorithms seriously. Because of a large number of redundant/invalid features and the class imbalanced issue, it is difficult to construct an optimal classifier for high-dimensional imbalanced data. Classifier ensemble has attracted intensive attention since it can achieve better performance than an individual classifier. In this work, we propose a multiview optimization (MVO) to learn more effective and robust features from high-dimensional imbalanced data, based on which an accurate and robust ensemble system is designed. Specifically, an optimized subview generation (OSG) in MVO is first proposed to generate multiple optimized subviews from different scenarios, which can strengthen the classification ability of features and increase the diversity of ensemble members simultaneously. Second, a new evaluation criterion that considers the distribution of data in each optimized subview is developed based on which a selective ensemble of optimized subviews (SEOS) is designed to perform the subview selective ensemble. Finally, an oversampling approach is executed on the optimized view to obtain a new class rebalanced subset for the classifier. Experimental results on 25 high-dimensional class imbalanced datasets indicate that the proposed method outperforms other mainstream classifier ensemble methods.

A Learning Approach for The Identification of Network Intrusions Based on Ensemble XGBoost Classifier

The limitation of signature-based Intrusion Detection systems has given rise for the popularity of Machine learning (ML) approaches r for building such intrusion detection systems (IDSs). ML is a sub-filed of Artificial Intelligence that enables algorithms to learn from data and its applications have been widely accepted and used in many domains. To achieve a promising ML-based model that can identify attacks and intrusions in networks and the cyber space, different stages of machine learning approach like pre-processing, attribute selection, model building, hyper parameter tuning can be very important. CICIDS2017 intrusion dataset was used for all the experimentations. This study focuses on building cyber threat detection model based on the ensemble feature selection and classification method. Innovative approaches were used for the analysis and pre-processing of the dataset. Thereafter, XGboost algorithm was used for selecting relevant features from the default input attributes in each of the captures. Thereafter, the reduced features were employed in the identification of cyber intrusions. The average accuracy achieved in the 8 captures of the dataset is 98% while precision is 0.98. Also, recall is 0.98, f1-score is 0.98 while AUC ROC score is 0.99. The study concluded that XGBoost-based model was able to achieve promising results based on the proper dataset encoding, feature importance-based feature selection and tuning of the algorithm for intrusion identification.

Ensemble Feature Selection Using Neighbourhood Rough Set–Based Multicriterion Fusion

Feature selection refers to finding a compact feature subset in which redundant and irrelevant information is reduced. It is generally aimed at simplifying the learning model and improving the learning performance simultaneously. To further enhance the traditional feature selection, ensemble feature selection has been developed vigorously in recent years. In this study, a novel ensemble feature selection algorithm that fuses multiple scores induced by multiple neighbourhood rough set–based feature evaluation criteria is presented. In detail, three different neighbourhood rough set–based feature evaluation criteria are applied, including neighbourhood approximation quality, neighbourhood recognition rate, and neighbourhood discrimination index. These representative measures serve as ensemble bases to induce multiple feature scores. Then, an aggregation method is proposed to fuse these scores for quantifying the feature importance quality. Finally, a searching algorithm based on this neighbourhood score by multicriterion fusion is devised. Experimental results on 16 UCI data sets prove that the proposed method outperforms other feature selection algorithms in classification task.

Classification of Breast Cancer using Ensemble Filter Feature Selection with Triplet Attention Based Efficient Net Classifier

In medical imaging, the effective detection and classification of Breast Cancer (BC) is a current research important task because of the still existing difficulty to distinguish abnormalities from normal breast tissues due to their subtle appearance and ambiguous margins and distinguish abnormalities from the normal breast. Moreover, BC detection based on an automated detection model is needed, because manual diagnosis faces problems due to cost and shortage of skilled manpower, and also takes a very long time. Using deep learning and ensemble feature selection techniques, in this paper, a novel framework is introduced for classifying BC from histopathology images. The five primary steps of the suggested framework are as follows: 1) to make the largest original dataset and then deep learning model with data augmentation to improve the learning. 2) The best features are selected by an Ensemble Filter Feature selection Method (EFFM) which combines the best feature subsets to produce the final feature subsets. 3) Then the pruned Convolution Neural Network (CNN) model is utilized to extract the optimal features. 4) Finally, the classification is done through the Triplet Attention based Efficient Network (TAENet) classifier. The suggested model produces a 98% accuracy rate after being trained and tested on two different histopathology imaging datasets including images from four different data cohorts. Subsequently, the suggested strategy outperforms the conventional ones since the ensemble filter habitually acquires the best features, and experimental results demonstrate the importance of the proposed approach

Dynamic Ensemble Selection for Imbalanced Data Streams With Concept Drift.

Ensemble learning, as a popular method to tackle concept drift in data stream, forms a combination of base classifiers according to their global performances. However, concept drift generally occurs in local data space, causing significantly different performances of a base classifier at different locations. Thus, employing global performance as a criterion to select base classifier is inappropriate. Moreover, data stream is often accompanied by class imbalance problem, which affects the classification accuracy of ensemble learning on minority instances. To drawback these problems, a dynamic ensemble selection for imbalanced data streams with concept drift (DES-ICD) is proposed. For data arrived in chunk-by-chunk, a novel synthetic minority oversampling technique with adaptive nearest neighbors (AnnSMOTE) is developed to generate new minority instances that conform to the new concept. Following that, DES-ICD creates a base classifier on newly arrived data chunk balanced by AnnSMOTE and merges it with historical base classifiers to form a candidate classifier pool. For each query instance, the optimal combination is constructed in terms of the performance of candidate classifiers in its neighborhood. Experimental results for nine synthetic and five real-world datasets show that the proposed method outperforms seven comparative methods on classification accuracy and tracks new concepts in an imbalanced data stream more preciously.

Road Traffic Crash Severity Analysis: A Bayesian-Optimized Dynamic Ensemble Selection Guided by Instance Hardness and Region of Competence Strategy

Road Traffic Crash Severity Analysis: A Bayesian-Optimized Dynamic Ensemble Selection Guided by Instance Hardness and Region of Competence Strategy

A Novel Heuristic-Based Selective Ensemble Prediction Method for Digital Financial Fraud Risk

A Novel Heuristic-Based Selective Ensemble Prediction Method for Digital Financial Fraud Risk

Ensemble Feature Selection for Age Estimation from Speech

The voice signal carries a wide range of data about the speaker, including their physical characteristics, feelings, and level of health. There are several uses for the estimate of these physical characteristics from the speech in forensics, security, surveillance, marketing, and customer service. The primary goal of this research is to identify the auditory characteristics that aid in estimating a speaker’s age. To this end, an ensemble feature selection model is proposed that selects the best features from a baseline acoustic feature vector for age estimation from speech. Using a feature vector that covers various spectral, temporal, and prosodic aspects of speech, an ensemble-based automatic feature selection is performed by, first calculating the feature importance or ranks based on individual feature selection methods, then voting is applied to the resulting feature ranks to attain the top-ranked subset by all feature selection methods. The proposed method is evaluated on the TIMIT dataset and achieved a mean absolute error (MAE) of 5.58 years and 5.12 years for male and female age estimation. Index Terms— Age Estimation, Feature Selection, Ensemble Selection, TIMIT dataset.

UNMASKING FRAUDSTERS: Ensemble Features Selection to Enhance Random Forest Fraud Detection

Fraud detection is used in various industries, including banking institutes, finance, insurance, government agencies, etc. Recent increases in the number of fraud attempts make fraud detection crucial for safeguarding financial information that is confidential or personal. Many types of fraud problems exist, including card-not-present fraud, fake Marchant, counterfeit checks, stolen credit cards, and others. An ensemble feature selection technique based on Recursive feature elimination (RFE), Information gain (IG), and Chi-Squared (X2) in concurrence with the Random Forest algorithm, was proposed to give research findings and results on fraud detection and prevention. The objective was to choose the essential features for training the model. The Receiver Operating Characteristic (ROC) Score, Accuracy, F1 Score, and Precision are used to evaluate the model's performance. The findings show that the model can differentiate between fraudulent transactions and those that are not, with an ROC Score of 95.83% and an Accuracy of 99.6%. The F1 Score of 99.6%% and precision of 100% further sustain the model's ability to detect fraudulent transactions with the least false positives correctly. The ensemble feature selection technique reduced training time and did not compromise the model's performance, making it a valuable tool for businesses in preventing fraudulent transactions.