Dynamic Ensemble Selection Techniques Research Articles

An automated classification framework for glaucoma detection in fundus images using ensemble of dynamic selection methods

Glaucoma is an optic neuropathy, which leads to vision loss and is irreversible due to damage in the optic nerve head mainly caused by increased intra-ocular pressure. Retinal fundus photography facilitates ophthalmologist in detection of glaucoma but is subjective to human intervention and is time-consuming. Computational methods such as image processing and machine learning classifiers can aid in computer-based glaucoma detection which helps in mass screening of glaucoma. In this context, the proposed method develops an automated glaucoma detection system, in the following steps: (i) pre-processing by segmenting the blood vessels using directional filter; (ii) segmenting the region of interest by using statistical features; (iii) extracting the clinical and texture-based features; and (iv) developing ensemble of classifier models using dynamic selection techniques. The proposed method is evaluated on two publically available datasets and 300 fundus images collected from a hospital. The best results are obtained using ensemble of random forest using META-DES dynamic ensemble selection technique, and the average specificity, sensitivity and accuracy for glaucoma detection on hospital dataset are 100%, respectively. For RIM-ONE dataset, the average specificity, sensitivity and accuracy for glaucoma detection are 100%, 93.85% and 97.86%, respectively. For Drishti dataset, the average specificity, sensitivity and accuracy for glaucoma detection are 90%, 100% and 97%, respectively. The quantitative results and comparative study indicate the ability of the developed method, and thus, it can be deployed in mass screening and also as a second opinion in decision making by the ophthalmologist for glaucoma detection.

A classification algorithm based on dynamic ensemble selection to predict mutational patterns of the envelope protein in HIV-infected patients

BackgroundTherapeutics against the envelope (Env) proteins of human immunodeficiency virus type 1 (HIV-1) effectively reduce viral loads in patients. However, due to mutations, new therapy-resistant Env variants frequently emerge. The sites of mutations on Env that appear in each patient are considered random and unpredictable. Here we developed an algorithm to estimate for each patient the mutational state of each position based on the mutational state of adjacent positions on the three-dimensional structure of the protein.MethodsWe developed a dynamic ensemble selection algorithm designated k-best classifiers. It identifies the best classifiers within the neighborhood of a new observation and applies them to predict the variability state of each observation. To evaluate the algorithm, we applied amino acid sequences of Envs from 300 HIV-1-infected individuals (at least six sequences per patient). For each patient, amino acid variability values at all Env positions were mapped onto the three-dimensional structure of the protein. Then, the variability state of each position was estimated by the variability at adjacent positions of the protein.ResultsThe proposed algorithm showed higher performance than the base learner and a panel of classification algorithms. The mutational state of positions in the high-mannose patch and CD4-binding site of Env, which are targeted by multiple therapeutics, was predicted well. Importantly, the algorithm outperformed other classification techniques for predicting the variability state at multi-position footprints of therapeutics on Env.ConclusionsThe proposed algorithm applies a dynamic classifier-scoring approach that increases its performance relative to other classification methods. Better understanding of the spatiotemporal patterns of variability across Env may lead to new treatment strategies that are tailored to the unique mutational patterns of each patient. More generally, we propose the algorithm as a new high-performance dynamic ensemble selection technique.

An integrated approach explaining the detection of distributed denial of service attacks

In recent years, several machine learning and deep learning models have been designed to detect various DDoS attacks , but the presence of irrelevant features, lack of transparency and class imbalance make these models less efficient. In this paper, we developed a novel efficient model to address these issues in detecting DDOS attacks. To begin with, data preprocessing is performed to improve the quality of the training data. The minority class samples are then generated using the Adaptive Synthetic oversampling technique to overcome the class imbalance. Following that, feature selection is performed by embedding SHAP feature importance within recursive feature elimination with five base classifiers . In addition, the hyperparameter of these classifiers is tuned to determine the most contributed features. Furthermore, global and local explanations for extracted features are provided to ensure transparency. Finally, these features are fed to the dynamic ensemble selection techniques such as KNORA-E and KNORA-U for classification by varying k values. These evaluations are analyzed using the CICDDoS2019 dataset. The evaluations are carried out in balanced and imbalanced data scenarios. The results indicate that the balanced data scenario outperformed the imbalanced data scenario as well as existing approaches. An accuracy of 99.9878% using KNORA-E and 99.9886% using KNORA-U is obtained utilizing the five most contributed features.

OLP++: An online local classifier for high dimensional data

Ensemble diversity is an important characteristic of Multiple Classifier Systems (MCS), which aim at improving the overall performance of a classification system by combining the response of several models. While diversity may be introduced through various manipulations at the data level and the model level, some MCSs incorporate local information in order to increase it and/or take advantage of it, based on the idea that the different classifiers in the ensemble may have expertise in distinct areas of the feature space.11Code available at: https://github.com/marianaasouza/olp_plusplus.Following a similar reasoning, we introduced in a previous work an ensemble method which produces in test time a few experts in the local region where each given query sample is located. These local experts, which are generated with slightly differing views of the target area, are then used to label the corresponding unknown instance. While the framework was shown to perform well especially over imbalanced problems, the locality definition in the method is based on the nearest neighbors rule and Euclidian distance, as is the case of various local-based ensembles, which may suffer from the effects of the curse of dimensionality over high dimensional problems. Thus, in this work, we propose a local ensemble method in which we leverage the data partitions given by decision trees for locality definition. More specifically, the partitions defined at different levels of the decision path that a given query instance traverses in the tree(s) are used as the regions over which the local experts are produced. By using different node levels from the path, each classifier in the local pool has a moderately distinct view of the target region without resorting to a dissimilarity metric, which might be susceptible to high dimensional spaces. Experimental results over 39 high dimensional problems showed that the proposed approach was significantly superior to our previous, distance-based framework in balanced accuracy rate. Compared to other six local-based ensemble methods, including dynamic selection and weighting schemes, the proposed method achieved competitive results, outperforming the random forest baseline and two state-of-the-art dynamic ensemble selection techniques.

Novel ensemble of optimized CNN and dynamic selection techniques for accurate Covid-19 screening using chest CT images

The world is significantly affected by infectious coronavirus disease (covid-19). Timely prognosis and treatment are important to control the spread of this infection. Unreliable screening systems and limited number of clinical facilities are the major hurdles in controlling the spread of covid-19. Nowadays, many automated detection systems based on deep learning techniques using computed tomography (CT) images have been proposed to detect covid-19. However, these systems have the following drawbacks: (i) limited data problem poses a major hindrance to train the deep neural network model to provide accurate diagnosis, (ii) random choice of hyperparameters of Convolutional Neural Network (CNN) significantly affects the classification performance, since the hyperparameters have to be application dependent and, (iii) the generalization ability using CNN classification is usually not validated. To address the aforementioned issues, we propose two models: (i) based on a transfer learning approach, and (ii) using novel strategy to optimize the CNN hyperparameters using Whale optimization-based BAT algorithm + AdaBoost classifier built using dynamic ensemble selection techniques. According to our second method depending on the characteristics of test sample, the classifier is chosen, thereby reducing the risk of overfitting and simultaneously produced promising results. Our proposed methodologies are developed using 746 CT images. Our method obtained a sensitivity, specificity, accuracy, F-1 score, and precision of 0.98, 0.97, 0.98, 0.98, and 0.98, respectively with five-fold cross-validation strategy. Our developed prototype is ready to be tested with huge chest CT images database before its real-world application.

Multi-Layer Selector(MLS): Dynamic selection based on filtering some competence measures

Multi-Classifier Systems (MCSs) have been widely studied as committee methods for increasing accuracy in pattern recognition. Recently, dynamic selection (DS) techniques have attracted undivided attention as effective types of MCSs in application. In DS techniques, the selection process is accomplished in two steps including measurement and selection. Normally, most DS methods in the literature have tried to focus on approaches adopted based on only one of the two options that are to offer a new measurement or a new selection method. On the other hand, some other papers have proposed frameworks in which both options were applied in combination with each other making use of previously existing measurement methods. For the first time, the idea of combining DS methods via multi-layer selectors is offered in this paper. In specific, two main tasks are performed in each layer of the multi-layer selector. First, the competence-level of classifiers are calculated. Then, the classifiers with high competence-level are passed to the next layer, and this way, best classifiers will be passed to the output in the last layer. During this procedure, some competence criteria are combined to measure the competence-level of classifiers. Furthermore, this paper introduces a novel taxonomy for dynamic selection methods and a new technique, called Probability Based (PB) method, to find the competence level of classifier. Experimental results show that the suggested framework improves the classification accuracy when compared against current state-of-the-art dynamic ensemble selection techniques. The Quade non-parametric statistical test confirms the capability of our proposed method to deal with classification problems.

A novel dynamic multi-criteria ensemble selection mechanism applied to drinking water quality anomaly detection

The provision of clean and safe drinking water is a crucial task for water supply companies from all over the world. To this end, automatic anomaly detection plays a critical role in drinking water quality monitoring. Recent anomaly detection studies use techniques that focus on a single global objective. Yet, companies need solutions that better balance the trade-off between false positives (FPs), which lead to financial losses to water companies, and false negatives (FNs), which severely impact public health and damage the environment. This work proposes a novel dynamic multi-criteria ensemble selection mechanism to cope with both problems simultaneously: the non-dominated local class-specific accuracy (NLCA). Moreover, experiments rely on recent time series related classification metrics to assess the predictive performance. Results on data from a real-world water distribution system show that NLCA outperforms other ensemble learning and dynamic ensemble selection techniques by more than 15% in terms of time series related F1 scores. As a conclusion, NLCA enables the development of stronger anomaly detection systems for drinking water quality monitoring. The proposed technique also offers a new perspective on dynamic ensemble selection, which can be applied to different classification tasks to balance conflicting criteria.

Dynamic ensemble selection based on hesitant fuzzy multiple criteria decision making

Among several extensions of fuzzy sets, hesitant fuzzy sets (HFSs) are interesting and practical. This paper proposes an application of HFSs in multiple classifier systems (MCSs). The MCSs have been proven as an effective and robust strategy for classification problems. These systems combine different classifiers and generally are composed of three steps: generation, selection (optional) and integration. This paper focuses on the selection step and proposes a novel dynamic ensemble selection method. In particular, the proposed method employs some selection criteria to determine the range of competency of the classifiers, and then, a HMCDM (hesitant fuzzy multiple criteria decision making) method is utilized to select the appropriate classifiers. Experimental results show that the proposed framework improves classification accuracy when compared against current state-of-the-art dynamic ensemble selection techniques. The Quade nonparametric statistical test confirms the capability of our proposed method.

FIRE-DES++: Enhanced online pruning of base classifiers for dynamic ensemble selection

Dynamic Ensemble Selection (DES) techniques aim to select one or more competent classifiers for the classification of each new test sample. Most DES techniques estimate the competence of classifiers using a given criterion over the region of competence of the test sample, usually defined as the set of nearest neighbors of the test sample in the validation set. Despite being very effective in several classification tasks, DES techniques can select classifiers that classify all samples in the region of competence as being from the same class. The Frienemy Indecision REgion DES (FIRE-DES) tackles this problem by pre-selecting classifiers that correctly classify at least one pair of samples from different classes in the region of competence of the test sample. However, FIRE-DES applies the pre-selection for the classification of a test sample if and only if its region of competence is composed of samples from different classes (indecision region), even though this criterion is not reliable for determining if a test sample is located close to the borders of classes (true indecision region) when the region of competence is obtained using classical nearest neighbors approach. Because of that, FIRE-DES mistakes noisy regions for true indecision regions, leading to the pre-selection of incompetent classifiers, and mistakes true indecision regions for safe regions, leaving samples in such regions without any pre-selection. To tackle these issues, we propose the FIRE-DES++, an enhanced FIRE-DES that removes noise and reduces the overlap of classes in the validation set; and defines the region of competence using an equal number of samples of each class, avoiding selecting a region of competence with samples of a single class. Experiments are conducted using FIRE-DES++ with 8 different dynamic selection techniques on 64 classification datasets. Experimental results show that FIRE-DES++ increases the classification performance of all DES techniques considered in this work, outperforming FIRE-DES with 7 out of the 8 DES techniques, and outperforming state-of-the-art DES frameworks.

Online pruning of base classifiers for Dynamic Ensemble Selection

Dynamic Ensemble Selection (DES) techniques aim to select only the most competent classifiers for the classification of each test sample. The key issue in DES is how to estimate the competence of classifiers for the classification of each new test sample. Most DES techniques estimate the competence of classifiers using a given criterion over the set of nearest neighbors of the test sample in the validation set, these nearest neighbors compose the region of competence. However, using local accuracy criteria alone on the region of competence is not sufficient to accurately estimate the competence of classifiers for the classification of all test samples. When the test sample is located in a region with borderline samples of different classes (indecision region), DES techniques can select classifiers with decision boundaries that do not cross the region of competence, assigning all samples in the region of competence to the same class. In this paper, we propose a dynamic selection framework for two-class problems that detects if a test sample is located in an indecision region and, if so, prunes the pool of classifiers, pre-selecting classifiers with decision boundaries crossing the region of competence of the test sample (if such classifiers exist). After that, the proposed framework uses a DES technique to select the most competent classifiers from the set of pre-selected classifiers. Experiments are conducted using the proposed framework with 9 different dynamic selection approaches on 40 classification datasets. Experimental results show that for all DES techniques used in the framework, the proposed framework outperforms DES in classification accuracy, demonstrating that our proposal significantly improves the classification performance of DES techniques, achieving statistically equivalent classification performance to the current state-of-the-art DES frameworks.

META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection

Dynamic ensemble selection (DES) techniques work by estimating the competence level of each classifier from a pool of classifiers, and selecting only the most competent ones for the classification of a specific test sample. The key issue in DES is defining a suitable criterion for calculating the classifiers’ competence. There are several criteria available to measure the level of competence of base classifiers, such as local accuracy estimates and ranking. However, using only one criterion may lead to a poor estimation of the classifier’s competence. In order to deal with this issue, we have proposed a novel dynamic ensemble selection framework using meta-learning, called META-DES. A meta-classifier is trained, based on the meta-features extracted from the training data, to estimate the level of competence of a classifier for the classification of a given query sample. An important aspect of the META-DES framework is that multiple criteria can be embedded in the system encoded as different sets of meta-features. However, some DES criteria are not suitable for every classification problem. For instance, local accuracy estimates may produce poor results when there is a high degree of overlap between the classes. Moreover, a higher classification accuracy can be obtained if the performance of the meta-classifier is optimized for the corresponding data. In this paper, we propose a novel version of the META-DES framework based on the formal definition of the Oracle, called META-DES.Oracle. The Oracle is an abstract method that represents an ideal classifier selection scheme. A meta-feature selection scheme using an overfitting cautious Binary Particle Swarm Optimization (BPSO) is proposed for improving the performance of the meta-classifier. The difference between the outputs obtained by the meta-classifier and those presented by the Oracle is minimized. Thus, the meta-classifier is expected to obtain results that are similar to the Oracle. Experiments carried out using 30 classification problems demonstrate that the optimization procedure based on the Oracle definition leads to a significant improvement in classification accuracy when compared to previous versions of the META-DES framework and other state-of-the-art DES techniques.

Prototype selection for dynamic classifier and ensemble selection

In dynamic ensemble selection (DES) techniques, only the most competent classifiers, for the classification of a specific test sample, are selected to predict the sample’s class labels. The key in DES techniques is estimating the competence of the base classifiers for the classification of each specific test sample. The classifiers’ competence is usually estimated according to a given criterion, which is computed over the neighborhood of the test sample defined on the validation data, called the region of competence. A problem arises when there is a high degree of noise in the validation data, causing the samples belonging to the region of competence to not represent the query sample. In such cases, the dynamic selection technique might select the base classifier that overfitted the local region rather than the one with the best generalization performance. In this paper, we propose two modifications in order to improve the generalization performance of any DES technique. First, a prototype selection technique is applied over the validation data to reduce the amount of overlap between the classes, producing smoother decision borders. During generalization, a local adaptive K-Nearest Neighbor algorithm is used to minimize the influence of noisy samples in the region of competence. Thus, DES techniques can better estimate the classifiers’ competence. Experiments are conducted using 10 state-of-the-art DES techniques over 30 classification problems. The results demonstrate that the proposed scheme significantly improves the classification accuracy of dynamic selection techniques.

META-DES: A dynamic ensemble selection framework using meta-learning

Dynamic ensemble selection systems work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify a given test sample. This is achieved by defining a criterion to measure the level of competence of a base classifier, such as, its accuracy in local regions of the feature space around the query instance. However, using only one criterion about the behavior of a base classifier is not sufficient to accurately estimate its level of competence. In this paper, we present a novel dynamic ensemble selection framework using meta-learning. We propose five distinct sets of meta-features, each one corresponding to a different criterion to measure the level of competence of a classifier for the classification of input samples. The meta-features are extracted from the training data and used to train a meta-classifier to predict whether or not a base classifier is competent enough to classify an input instance. During the generalization phase, the meta-features are extracted from the query instance and passed down as input to the meta-classifier. The meta-classifier estimates, whether a base classifier is competent enough to be added to the ensemble. Experiments are conducted over several small sample size classification problems, i.e., problems with a high degree of uncertainty due to the lack of training data. Experimental results show that the proposed meta-learning framework greatly improves classification accuracy when compared against current state-of-the-art dynamic ensemble selection techniques.