Dynamic Ensemble Selection Method Research Articles

Dynamic Ensemble Learning With Multi-View Kernel Collaborative Subspace Clustering for Hyperspectral Image Classification

Recently, a series of collaborative representation (CR) methods have attracted much attention for hyperspectral images classification. In this article, two CR-based dynamic ensemble selection (DES) methods using multiview kernel collaborative subspace clustering (MVKCSC) and random subspace MVKCSC (RSMVKCSC) are proposed. In order to combine spectral and spatial information to construct a region of competence (RoC), the multiview learning strategy is used in the general DES method. Compared with traditional clustering methods, the MVC can more effectively utilize multifeature information. Moreover, a new method of constructing the Laplacian matrix using kernel CR coefficients is proposed for clustering based on subspace clustering and CR theory. This method is called MVKCSC, which can obtain the clustering results by using kernel CR self-representation coefficients. In addition, to increase the diversity of samples, the random subspace method (RSM) and MVKCSC are combined for RMVKCSC. Moreover, the algorithm can obtain better clustering results by constraining samples and features simultaneously. The effectiveness of the proposed methods is validated using three hyperspectral data sets with few samples. The experimental results show that both DES-MVKCSC and DES-RSMVKCSC outperform their single classifier counterparts. In particular, the proposed methods provide superior performance compared with the state-of-the-art DES methods.

A dynamic ensemble selection method for bank telemarketing sales prediction

We propose a dynamic ensemble selection method, META-DES-AAP, to predict the success of bank telemarketing sales of time deposits. Unlike existing machine learning-based marketing sales prediction methods focusing only on prediction accuracy, META-DES-AAP considers the accuracy and average profit maximization. In META-DES-AAP, to consider both accuracy and average profit in the framework of dynamic ensemble selection using meta-training, a multi-objective optimization algorithm is designed to maximize the accuracy and average profit for base classifiers selection. Base classifiers suitable for each test telemarketing campaign are integrated according to the dynamic-based base classifiers integration method. Experimental results on bank telemarketing data show that META-DES-AAP achieves the best accuracy and the largest average profit when compared across several state-of-the-art machine learning methods. In addition, the factors influencing telemarketing on the average predicted probability of telemarketing success and average profit obtained by META-DES-AAP are analyzed.

A novel combined dynamic ensemble selection model for imbalanced data to detect COVID-19 from complete blood count

BackgroundAs blood testing is radiation-free, low-cost and simple to operate, some researchers use machine learning to detect COVID-19 from blood test data. However, few studies take into consideration the imbalanced data distribution, which can impair the performance of a classifier. MethodA novel combined dynamic ensemble selection (DES) method is proposed for imbalanced data to detect COVID-19 from complete blood count. This method combines data preprocessing and improved DES. Firstly, we use the hybrid synthetic minority over-sampling technique and edited nearest neighbor (SMOTE-ENN) to balance data and remove noise. Secondly, in order to improve the performance of DES, a novel hybrid multiple clustering and bagging classifier generation (HMCBCG) method is proposed to reinforce the diversity and local regional competence of candidate classifiers. ResultsThe experimental results based on three popular DES methods show that the performance of HMCBCG is better than only use bagging. HMCBCG+KNE obtains the best performance for COVID-19 screening with 99.81% accuracy, 99.86% F1, 99.78% G-mean and 99.81% AUC. ConclusionCompared to other advanced methods, our combined DES model can improve accuracy, G-mean, F1 and AUC of COVID-19 screening.

A Dynamic Ensemble Learning Algorithm based on K-means for ICU mortality prediction

This research proposes a Dynamic Ensemble Learning Algorithm based on K-means (DELAK) for intensive care unit (ICU) mortality prediction. Nowadays, the widely applied traditional scoring systems, which predict the mortality risk with some scores reflecting the severity of disease and physiological states of patients in ICU, have shown insufficient predictive performance when faced with large volume of data. Although taking advantage of large volume of data, single machine learning model and ensemble learning methods show an inadequate ability to make personalized predictions for each new patient. Dynamic ensemble selection (DES) methods which are widely studied in pattern recognition field can make personalized prediction but they only select a single classifier instead of ensemble of the results from a pool of classifiers. To overcome the limitations mentioned above, this research proposes an ensemble learning algorithm based on K-means sampling and distance-based dynamic ensemble. K-means sampling helps to achieve the diversity of base classifiers and the distance-based dynamic ensemble is a flexible fusion method which creates a personalized combination of results from base classifiers for each new test sample. To evaluate the performance for mortality prediction of the proposed algorithm, comprehensive experiments are conducted on MIMIC-III dataset. The experimental results show that DELAK achieves outstanding performance in term of AUROC and AUPRC compared with other six fusion strategies, traditional scoring systems, classical ensemble models, i.e. AdaBoost, Bagging and random forest, and dynamic ensemble selection methods in most mortality prediction tasks.

DDES: A Distribution-Based Dynamic Ensemble Selection Framework

Dynamic Ensemble Selection (DES) is a special type of ensemble modeling that selects different subsets of base classifiers for different test sample cases. In this process, multiple base classifiers are compared in terms of their competence as an ensemble to the best possible prediction for a given test sample case. Traditional DES methods rely on the Euclidean distance-based k-Nearest Neighbor (kNN) algorithm to identify the most relevant reference data points in such a way that the base classifiers correctly classifying them can be considered for an optimal ensemble. However, this approach is sensitive to the local structure of the data and the presence of noisy or irrelevant attributes. This study proposes a novel distribution-based DES (DDES) framework that takes the data structure into consideration when selecting reference data points. The experimental results for 30 classification problems indicate that the proposed approach yielded the best accuracy among the competing DES methods. Additionally, we discuss the correlation between the data complexity and the improvement in classification performance.

Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams

This work aims to connect two rarely combined research directions, i.e., non-stationary data stream classification and data analysis with skewed class distributions. We propose a novel framework employing stratified bagging for training base classifiers to integrate data preprocessing and dynamic ensemble selection methods for imbalanced data stream classification. The proposed approach has been evaluated based on computer experiments carried out on 135 artificially generated data streams with various imbalance ratios, label noise levels, and types of concept drift as well as on two selected real streams. Four preprocessing techniques and two dynamic selection methods, used on both bagging classifiers and base estimators levels, were considered. Experimentation results showed that, for highly imbalanced data streams, dynamic ensemble selection coupled with data preprocessing could outperform online and chunk-based state-of-art methods.

Dynamic ensemble selection based on hesitant fuzzy multiple criteria decision making

Among several extensions of fuzzy sets, hesitant fuzzy sets (HFSs) are interesting and practical. This paper proposes an application of HFSs in multiple classifier systems (MCSs). The MCSs have been proven as an effective and robust strategy for classification problems. These systems combine different classifiers and generally are composed of three steps: generation, selection (optional) and integration. This paper focuses on the selection step and proposes a novel dynamic ensemble selection method. In particular, the proposed method employs some selection criteria to determine the range of competency of the classifiers, and then, a HMCDM (hesitant fuzzy multiple criteria decision making) method is utilized to select the appropriate classifiers. Experimental results show that the proposed framework improves classification accuracy when compared against current state-of-the-art dynamic ensemble selection techniques. The Quade nonparametric statistical test confirms the capability of our proposed method.

Dynamic ensemble selection of learner-descriptor classifiers to assess curve types in adolescent idiopathic scoliosis.

While classification is important for assessing adolescent idiopathic scoliosis (AIS), it however suffers from low interobserver and intraobserver reliability. Classification using ensemble methods may contribute to improving reliability using the proper 2D and 3D images of spine curvature features. In this study, we present two new techniques to describe the spine, namely, leave-one-out and fan leave-one-out. Using these techniques, three descriptors are computed from a stereoradiographic 3D reconstruction to describe the relationship between a vertebra and its neighbors. A dynamic ensemble selection method is introduced for automatic spine classification. The performance of the method is evaluated on a dataset containing 962 3D spine models categorized according to three curve types. With a log loss of 0.5623, the dynamic ensemble selection outperforms voting and stacking ensemble learning techniques. This method can improve intraobserver and interobserver reliability, identify the best combination of descriptors for characterizing spine curve types, and provide assistance to clinicians in the form of information to classify borderline curvature types. Graphical abstract ᅟ.

Investigating the impact of selection criteria in dynamic ensemble selection methods

Ensemble of Classifiers are composed of parallel-organized components (individual classifiers) whose outputs are combined using a combination method that provides the final output for an ensemble. In this context, Dynamic Ensemble Systems (DES) is an ensemble-based system that, for each test pattern, a different ensemble structure is defined, in which a subset of classifiers is selected from an initial pool of classifiers. During the selection process of a DES, any criterion can be used, being the most important ones accuracy and distance. Distance measures are used to assess the distance of the classifier outputs within a validation set and the main examples of this measure are diversity and similarity. In this paper, we investigate the impact of selection criteria in DES methods. More specifically, we focus on the use of different distance measures (diversity and similarity) as selection criteria. In other to do this, an empirical analysis has been conducted using six different DES methods (three of them are existing methods and the remaining three are proposed in this paper) and with 20 different classification datasets. Our findings indicated that a distance measure improves the overall performance of the state-of-the-art ensemble generation methods.

Exploring the effectiveness of dynamic ensemble selection in the one-versus-one scheme

The One-versus-One (OVO) strategy is one of the most common and effective techniques to deal with multi-class classification problems. The basic idea of an OVO scheme is to divide a multi-class classification problem into several easier-to-solve binary classification problems with considering each possible pair of classes from the original problem, which is then built into a binary classifier by an independent base learner. In this study, we propose a novel methodology which attempts to select a group of base classifiers in each pairwise dataset for each unknown pattern. To implement this, the Dynamic Ensemble Selection (DES) method based on a competence measure is employed to select the most appropriate ensemble in each binary classification problem derived from the OVO decomposition. In order to verify the validity and effectiveness of our proposed method, we carry out a thorough experimental study. We first compare our proposal with several state-of-the-art approaches. Then, we perform the comparison of several well-known aggregation strategies to combine the binary ensemble obtained by Dynamic Ensemble Selection. Finally, we explore whether further improvement can be achieved by considering the competence-based method in OVO scheme. The extracted findings drawn from the empirical analysis are supported by the proper statistical analysis and indicate that there is a positive synergy between the DES method and the Distance-based Relative Competence Weighting (DRCW) approach for the OVO scheme.

Meta-learning recommendation of default size of classifier pool for META-DES

Dynamic ensemble selection (DES) is a mechanism for selecting an ensemble of competent classifiers from a pool of base classifiers, in order to classify a particular test sample. The size of this pool is user-defined, and yet is crucial for controlling the computational complexity and performance of a DES. An appropriate pool size depends on the choice of base classifiers, the underlying DES method used, and more importantly, the characteristics of the given problem. After the DES method and the base classifiers are selected, an appropriate pool size for a given problem can be obtained by the repetitive application of the DES with a variety of sizes, after which a selection is performed. Since this brute force approach is computationally expensive, researchers set the pool size to a pre-specified value. This strategy, may, however further complicate and reduce the performance of the DES method. Instead, we propose a framework that is akin to meta-learning, in order to predict a suitable pool size based on the intrinsic classification complexity of a problem. In our strategy, we collect meta-features corresponding to classification complexity from a number of data sets. Additionally, we obtain the best pool sizes for these data sets using the brute force approach. The association between these two pieces of information is captured using meta-regression models. Finally, for an unseen problem, we predict the pool size using this model and the classification complexity information. We carry out experiments on 65 two-class data sets and with a recent DES method, namely, META-DES. We also consider variants of meta-regression techniques and report prediction results, after which we carry out a statistical comparison among them. Moreover, we investigate the performance of META-DES and observe that it performs equivalently for both the predicted and the best pool sizes.

Improving a dynamic ensemble selection method based on oracle information

This work evaluates some strategies to approximate the performance of a dynamic ensemble selection method to the oracle performance of its pool of weak classifiers. For this purpose, we evaluated different distance metrics in the K-nearest-oracles (KNORA) method, the use of statistics related to the class accuracy of each classifier in the pool and some additional information calculated by using a clustering process in the validation dataset. Moreover, different strategies are also evaluated to combine the results of the KNORA dynamic ensemble selection method with the results of its built-in K-nearest neighbour (KNN) used to define the neighbourhood of a test pattern during the ensemble creation. A strong experimental protocol based on more than 60,000 samples of handwriting digits extracted from NIST-SD19 was used to evaluate each strategy. The experiments have shown that the fusion of the KNORA results with the results of its built-in KNN is a very promising strategy.