Selective Ensemble Algorithm Research Articles

An efficient ensemble learning method based on multi-objective feature selection

Ensemble learning (EL) boosts model prediction performance across various domains through two main steps: generating individual classifiers (ICs) and combining them. Creating accurate and diverse ICs is crucial for a strong ensemble, while selecting the best ICs, known as ensemble selection (ES), is critical yet challenging due to the accuracy-diversity trade-off and the lack of agreed-upon diversity metrics. This paper introduces an EL strategy that uses multi-objective feature selection (MOFS) and a feature relevance-guided selection to tackle these challenges. Our approach uses a hybrid MOFS algorithm to produce accurate and diverse ICs, and then it employs a novel knowledge-based feature-relevance-guided metric for precise diversity assessment during ES. The ES issue is cast as an optimization problem, aiming to maximize both diversity and accuracy, and an efficient ES algorithm is developed to select optimal ICs. Extensive tests on public datasets and a real-world prediction task demonstrate the effectiveness of our method, especially in achieving high accuracy.

An air combat maneuver pattern extraction based on time series segmentation and clustering analysis

Target maneuver recognition is a prerequisite for air combat situation awareness, trajectory prediction, threat assessment and maneuver decision. To get rid of the dependence of the current target maneuver recognition method on empirical criteria and sample data, and automatically and adaptively complete the task of extracting the target maneuver pattern, in this paper, an air combat maneuver pattern extraction based on time series segmentation and clustering analysis is proposed by combining autoencoder, G-G clustering algorithm and the selective ensemble clustering analysis algorithm. Firstly, the autoencoder is used to extract key features of maneuvering trajectory to remove the impacts of redundant variables and reduce the data dimension; Then, taking the time information into account, the segmentation of Maneuver characteristic time series is realized with the improved FSTS-AEGG algorithm, and a large number of maneuver primitives are extracted; Finally, the maneuver primitives are grouped into some categories by using the selective ensemble multiple time series clustering algorithm, which can prove that each class represents a maneuver action. The maneuver pattern extraction method is applied to small scale air combat trajectory and can recognize and correctly partition at least 71.3% of maneuver actions，indicating that the method is effective and satisfies the requirements for engineering accuracy. In addition, this method can provide data support for various target maneuvering recognition methods proposed in the literature, greatly reduce the workload and improve the recognition accuracy.

Detecting financial statement fraud using dynamic ensemble machine learning

Our study uses Machine learning to develop an advanced fraud detection model that can detect fraudulent firms. We build our model using raw financial and non-financial variables following prior literature. In addition, we introduce the Dynamic Ensemble Selection algorithm to the fraud detection literature, which combines individual classifiers dynamically to make a final prediction. Using several performance evaluation metrics, we find that our model can outperform several machine learning models used in recent studies.

An adaptive selective ensemble algorithm for fault classification

With the aim of improving the classification performance of a single extreme learning machine (ELM) for fault diagnosis, an adaptive selective ensemble classification algorithm is proposed based on the idea of ensemble learning. The proposed algorithm achieves better classification performance than a single model and the selected ELM subnetworks are selected adaptively and the ensemble weights of selected ELM subnetworks are calculated adaptively, which avoids the complex process of manually selecting subnetworks and calculating ensemble weights. First, a regularized error weight adjustment ELM is constructed by introducing regularization parameters and error weight adjustment matrix to a standard ELM, where the error weight adjustment matrix is obtained by the method of adaptive Gaussian kernel density estimation. Then, discrepancy subnetworks are constructed using six different activation functionsand the ensemble weights of subnetworks are obtained adaptively according to the normalized ratio of mean and variance of subnetwork F-scores to achieve the ensemble of subnetworks. Finally, the adaptive selective ensemble classification algorithm is validated using the UCI dataset and experimental acoustic emission signals of gearbox faults. The results show that the adaptive selective ensemble method can improve the stability and accuracy of classification algorithms and the achieved classification accuracy for experimental acoustic emission was 0.9773.

A classification algorithm based on dynamic ensemble selection to predict mutational patterns of the envelope protein in HIV-infected patients

BackgroundTherapeutics against the envelope (Env) proteins of human immunodeficiency virus type 1 (HIV-1) effectively reduce viral loads in patients. However, due to mutations, new therapy-resistant Env variants frequently emerge. The sites of mutations on Env that appear in each patient are considered random and unpredictable. Here we developed an algorithm to estimate for each patient the mutational state of each position based on the mutational state of adjacent positions on the three-dimensional structure of the protein.MethodsWe developed a dynamic ensemble selection algorithm designated k-best classifiers. It identifies the best classifiers within the neighborhood of a new observation and applies them to predict the variability state of each observation. To evaluate the algorithm, we applied amino acid sequences of Envs from 300 HIV-1-infected individuals (at least six sequences per patient). For each patient, amino acid variability values at all Env positions were mapped onto the three-dimensional structure of the protein. Then, the variability state of each position was estimated by the variability at adjacent positions of the protein.ResultsThe proposed algorithm showed higher performance than the base learner and a panel of classification algorithms. The mutational state of positions in the high-mannose patch and CD4-binding site of Env, which are targeted by multiple therapeutics, was predicted well. Importantly, the algorithm outperformed other classification techniques for predicting the variability state at multi-position footprints of therapeutics on Env.ConclusionsThe proposed algorithm applies a dynamic classifier-scoring approach that increases its performance relative to other classification methods. Better understanding of the spatiotemporal patterns of variability across Env may lead to new treatment strategies that are tailored to the unique mutational patterns of each patient. More generally, we propose the algorithm as a new high-performance dynamic ensemble selection technique.

Inverse halftoning based on sparse representation with boosted dictionary

Under the imbalanced dataset, the performance of the base-classifier, the computing method of weight of base-classifier and the selection method of the base-classifier have a great impact on the performance of the ensemble classifier. In order to solve above problem to improve the generalization performance of ensemble classifier, a selective ensemble learning algorithm based on under-sampling for imbalanced dataset is proposed. First, the proposed algorithm calculates the number K of under-sampling samples according to the relationship between class sample density. Then, we use the improved K-means clustering algorithm to under-sample the majority class samples and obtain K cluster centers. Then, all cluster centers (or the sample of the nearest cluster center) are regarded as new majority samples to construct a new balanced training subset combine with the minority class?s samples. Repeat those processes to generate multiple training subsets and get multiple base-classifiers. However, with the increasing of iterations, the number of base-classifiers increase, and the similarity among the base-classifiers will also increase. Therefore, it is necessary to select some base-classifier with good classification performance and large difference for ensemble. In the stage of selecting base-classifiers, according to the difference and performance of base-classifiers, we use the idea of maximum correlation and minimum redundancy to select base-classifiers. In the ensemble stage, G-mean or F-mean is selected to evaluate the classification performance of base-classifier for imbalanced dataset. That is to say, it is selected to compute the weight of each base-classifier. And then the weighted voting method is used for ensemble. Finally, the simulation results on the artificial dataset, UCI dataset and KDDCUP dataset show that the algorithm has good generalization performance on imbalanced dataset, especially on the dataset with high imbalance degree.

Selective ensemble learning algorithm for imbalanced dataset

Under the imbalanced dataset, the performance of the base-classifier, the computing method of weight of base-classifier and the selection method of the base-classifier have a great impact on the performance of the ensemble classifier. In order to solve above problem to improve the generalization performance of ensemble classifier, a selective ensemble learning algorithm based on under-sampling for imbalanced dataset is proposed. First, the proposed algorithm calculates the number K of under-sampling samples according to the relationship between class sample density. Then, we use the improved K-means clustering algorithm to under-sample the majority class samples and obtain K cluster centers. Then, all cluster centers (or the sample of the nearest cluster center) are regarded as new majority samples to construct a new balanced training subset combine with the minority class?s samples. Repeat those processes to generate multiple training subsets and get multiple base-classifiers. However, with the increasing of iterations, the number of base-classifiers increase, and the similarity among the base-classifiers will also increase. Therefore, it is necessary to select some base-classifier with good classification performance and large difference for ensemble. In the stage of selecting base-classifiers, according to the difference and performance of base-classifiers, we use the idea of maximum correlation and minimum redundancy to select base-classifiers. In the ensemble stage, G-mean or F-mean is selected to evaluate the classification performance of base-classifier for imbalanced dataset. That is to say, it is selected to compute the weight of each base-classifier. And then the weighted voting method is used for ensemble. Finally, the simulation results on the artificial dataset, UCI dataset and KDDCUP dataset show that the algorithm has good generalization performance on imbalanced dataset, especially on the dataset with high imbalance degree.

Dioxin Emission Concentration Prediction Using the Selective Ensemble Algorithm Based on Bayesian Inference and Binary Tree

Dioxin Emission Concentration Prediction Using the Selective Ensemble Algorithm Based on Bayesian Inference and Binary Tree

A machine learning-based dynamic ensemble selection algorithm for microwave retrieval of surface soil freeze/thaw: A case study across China

ABSTRACT The surface soil freeze/thaw (FT) cycle serves as a “switch” for land surface processes; accurate retrieval of surface FT dynamics based on satellite passive microwave remote sensing is critical for studies on climate change and dynamics of the cryosphere. This study aims to improve FT retrieval accuracy by developing a new FT retrieval algorithm that applies the K-Nearest Oracle Union (KNORA-UNION) dynamic ensemble selection algorithm. This algorithm can optimally integrate three machine learning models on a grid cell scale, namely Random Forests, Extra-Trees, and Extreme Gradient Boosting. We applied our developed freeze/thaw dynamic ensemble selection retrieval algorithm (FT-DESA) to retrieve China’s daily surface FT states from 2009 to 2020 based on multiband Special Sensor Microwave Imager/Sounder (SSMIS) brightness temperatures. We then evaluated our FT-DESA results by comparing the observations of 2398 stations and three other existing FT algorithms, including the modified seasonal threshold algorithm (MSTA), decision tree algorithm (DTA), and dual-index algorithm (DIA) across China. Our results show that FT-DESA has the highest retrieval accuracy and the lowest biases across China among the four algorithms. The mean classification accuracy for the PM and AM overpasses of FT-DESA is 89% and 84%, respectively. The evaluations further indicate that some of the existing algorithms do not reflect the temporal and spatial heterogeneity in selecting thresholds for FT classification. This study demonstrates that the freeze/thaw dynamic ensemble selection algorithm can provide daily estimates of surface FT states across China, improve FT states’ retrieval accuracy, and provide a valuable multi-decadal record for daily FT states.

An ensemble active learning for a fluidized bed granulation in the pharmaceutical industry

Active learning (AL), allowing to query the labels by an oracle, can dwindle the generalization error from a few labeled samples, thus well-motivating in the scenario where there are fewer labeled samples and abundant unlabeled samples. However, the query samples cannot be exploited efficiently in the existing methods based on AL for regression tasks, and AL is typically applied for classification problems. In this paper, an active learning framework (CALF) is proposed to improve the efficiency with fewer query samples, aiming at the moisture content prediction in fluidized bed granulation. The proposed method, based on conditional variational auto-encoder (CVAE) and selective ensemble algorithm, can efficiently incorporate the information in query samples into the labeled sample space. The game of CVAE and selective ensemble algorithm improves the performance of the framework effectively, and its effectiveness is verified by the results obtained from the large batch of fluidized bed granulating experiments.

Automated Ensemble for Deep Learning Inference on Edge Computing Platforms

Advances in deep learning (DL) have triggered an explosion of mobile intelligence, posing a soaring demand for computing resources that cannot be satisfied by mobile devices. In this article, we employ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">edge computing</i> to deliver better DL inference services to end users. The key is to leverage deep neural network (DNN) ensemble techniques that provide state-of-the-art performance for many machine learning applications in terms of inference accuracy and robustness. Compared to end devices, the edge computing platform is endowed with more powerful computing resources, making it feasible to implement DNN ensembles for DL inferences. However, due to the constrained computing capacity of edge servers and the possible service response deadline, an edge server can only use a limited number of DNNs to construct DNN ensembles. This poses a unique problem, namely, DNN ensemble selection, for identifying the best-fit DNN ensembles. We propose a novel algorithm called automated DNN ensemble selection (AES) algorithm to solve this problem. Because DNNs exhibit performance variations over different distributions of input data, AES adaptively determines a DNN ensemble according to the features of admitted inference tasks. AES is an online learning algorithm that learns DNNs’ in-use performance over time. An ensemble selection rule is further designed as a subroutine of AES to recruit members into the DNN ensemble based on the accuracy and diversity of DNNs. In particular, we theoretically prove that AES can achieve asymptotic optimality. We carry out experiments on real-world data sets. The results show that using the DNN ensemble technique on edge computing platforms dramatically improves the DL inference quality, and AES outperforms other benchmark schemes.

Prediction of Daily Mean PM10 Concentrations Using Random Forest, CART Ensemble and Bagging Stacked by MARS

A novel framework for stacked regression based on machine learning was developed to predict the daily average concentrations of particulate matter (PM10), one of Bulgaria’s primary health concerns. The measurements of nine meteorological parameters were introduced as independent variables. The goal was to carefully study a limited number of initial predictors and extract stochastic information from them to build an extended set of data that allowed the creation of highly efficient predictive models. Four base models using random forest, CART ensemble and bagging, and their rotation variants, were built and evaluated. The heterogeneity of these base models was achieved by introducing five types of diversities, including a new simplified selective ensemble algorithm. The predictions from the four base models were then used as predictors in multivariate adaptive regression splines (MARS) models. All models were statistically tested using out-of-bag or with 5-fold and 10-fold cross-validation. In addition, a variable importance analysis was conducted. The proposed framework was used for short-term forecasting of out-of-sample data for seven days. It was shown that the stacked models outperformed all single base models. An index of agreement IA = 0.986 and a coefficient of determination of about 95% were achieved.

Research on a Risk Control Model Based on Selective Ensemble Algorithm of Hierarchical Clustering and Simulated Annealing

In order to solve the problems in the traditional ensemble learning model, selective ensemble algorithms are often used to optimize the ensemble learning model. To settle the shortcomings of the current selective ensemble algorithm existing in the selection efficiency and predict outcomes, this paper proposes selective ensemble algorithm basing on stacking ensemble framework. The algorithm mainly uses the agglomerated hierarchical clustering (AHC) algorithm and the metropolis criterion of simulated annealing to select the type and number of base learners. In terms of empirical analysis, Lending-Club data is used to build a multi-classification model. The experimental results show that compared with the single learner model, the AHC-Metropolis selective ensemble algorithm has better performance and stability in predicting multi-classification problems, and can provide a basis for improving the financial risk control system and ensuring national security.

An Ordered Aggregation-Based Ensemble Selection Method of Lightweight Deep Neural Networks With Random Initialization

Due to the popularity of 5G connectivity and The Internet of Things sensors, deep learning algorithms are being extended to edge devices. Compared with AI(Artificial Intelligence) cloud platforms, the deployment of deep neural networks on edge devices must focus on low power consumption, low latency, stability and reliability. In recent years, the development of lightweight deep neural network architecture has provided a basis for the deployment of deep neural networks on edge devices. However, the shortcomings of deep neural networks, such as overconfidence, vulnerability to adversarial attack, and easy over fitting when samples are insufficient, still limit their applications in many fields. One of the ways to compensate for these defects is to use deep ensemble. An ordered aggregation-based ensemble selection algorithm is proposed, which uses soft-margin as the importance assessment metric to take full advantage of the diversity and complementarity of lightweight deep neural networks obtained from different initialization training, so as to improve the overall performance of multiple edge devices. The experimental results show that this algorithm has a significant improvement in generalization performance compared with random ensemble and ordered aggregation algorithms based on accuracy or diversity, and provides a new complementary idea for the deployment of lightweight deep neural networks on edge devices.

Improving Remote Sensing Multiple Classification by Data and Ensemble Selection

In this article, margin theory is exploited to design better ensemble classifiers for remote sensing data. A semi-supervised version of the ensemble margin is at the core of this work. Some major challenges in ensemble learning are investigated using this paradigm in the difficult context of land cover classification: selecting the most informative instances to form an appropriate training set, and selecting the best ensemble members. The main contribution of this work lies in the explicit use of the ensemble margin as a decision method to select training data and base classifiers in an ensemble learning framework. The selection of training data is achieved through an innovative iterative guided bagging algorithm exploiting low-margin instances. The overall classification accuracy is improved by up to 3%, with more dramatic improvement in per-class accuracy (up to 12%). The selection of ensemble base classifiers is achieved by an ordering-based ensemble-selection algorithm relying on an original margin-based criterion that also targets low-margin instances. This method reduces the complexity (ensemble size under 30) but maintains performance.

Dynamic weighted selective ensemble learning algorithm for imbalanced data streams

Dynamic weighted selective ensemble learning algorithm for imbalanced data streams

A selective ensemble learning algorithm for imbalanced dataset

Network intrusion behaviour data is the imbalanced data. It includes a large amount of normal behavior data and a small amount of intrusion behavior data. The traditional selective ensemble learning algorithm will lead to high false negative rate. This paper proposes a selective ensemble learning algorithm for imbalanced data based on under sampling (SELAUS). First of all, the algorithm uses Bootstrap method to extract samples equal to the number of samples of a few classes from majority class samples to construct multiple balanced training subsets. Then, in order to ensure that the obtained base classifiers have great differences, several features are randomly selected on the training subset and a decision tree is constructed as the base classifier using CART algorithm. This method can also make some base classifiers have poor performance, so it can select and integrate base classifiers instead of all base classifiers. In order to accurately evaluate the generalization error of the classifier for imbalanced dataset, this paper defines the performance evaluation method in the imbalanced dataset and the difference evaluation method between the base classifiers. Then the generalization error of each base classifier is calculated, and the base classifier is selected according to the generalization error. In the integration of weighted voting, the weight of each base classifier is calculated by the weight calculation method for imbalanced data. Finally, the validity of the algorithm is verified by UCI dataset and applied to network intrusion detection. The simulation results show that the algorithm can improve the detection rate of minority class samples, that is to say, reduce the false negative rate.

Integrating Data and Model Space in Ensemble Learning by Visual Analytics

Ensembles of classifier models typically deliver superior performance and can outperform single classifier models given a dataset and classification task at hand. However, the gain in performance comes together with the lack of comprehensibility, posing a challenge to understand how each model affects the classification outputs and from where the errors come. We propose a tight visual integration of the data and the model space for exploring and combining classifier models. We introduce an interactive workflow that builds upon the visual integration and enables the effective exploration of classification outputs and models. The involvement of the user is key to our approach. Therefore, we elaborate on the role of the human and connect our approach to theoretical frameworks on human-centered machine learning. We showcase the usefulness of our approach and the integration of the user via binary and multiclass classification problems. Based on ensembles automatically selected by a standard ensemble selection algorithm, the user can manipulate models and alternative combinations.

Dual-granularity weighted ensemble clustering

Ensemble clustering is one of the research hotspots of data mining in recent years. The selection of high-quality and large-diversity base clustering results plays a key role in the quality of the final result. Traditional ensemble clustering selection algorithms usually treat each base clustering result as a whole which ignores the difference between the clusters in the same clustering result. It may cause the validity of the final clustering result to be affected. Aiming at this problem, inspired by the measurement method of uncertainty in the rough set theory, a dual-granularity weighted ensemble clustering model is proposed. The main contribution of this paper is shown as follows: (1) the evaluation of the reliability of clusters is transformed into an uncertainty measurement problem in the rough set; (2) in a finer-grained level, a sample local similarity measurement method is designed; (3) a weighted co-association matrix elements generation method based on global cluster reliability and local sample pair similarity is proposed, then the fusion function is used to get the final clustering result. Experimental results show that the proposed method is not sensitive to the size and diversity of base clustering members which has good robustness and stability. The final result obtained by this model is closer to the actual distribution of data sets.

The Research on Clustering Ensembles Selection Algorithm based on Semi-supervised K-means Clustering

Selective clustering ensemble algorithm can eliminate the inferior quality clustering member’s influence and can achieve a better clustering solution relative to the clustering ensemble algorithm. For high dimensional data clustering, in this paper, a novel selective ensemble algorithm based on semi-supervised K-means clustering is proposed. In this paper, through a large number of experiments to verify the validity of the proposed algorithm for dealing with high dimensional data clustering. The new algorithm can achieve statistically significant performance improvement over other clustering algorithms.