Diversity Of Base Classifiers Research Articles

A Two-Stage Voting-Boosting Technique for Ensemble Learning in Social Network Sentiment Classification.

In recent years, social network sentiment classification has been extensively researched and applied in various fields, such as opinion monitoring, market analysis, and commodity feedback. The ensemble approach has achieved remarkable results in sentiment classification tasks due to its superior performance. The primary reason behind the success of ensemble methods is the enhanced diversity of the base classifiers. The boosting method employs a sequential ensemble structure to construct diverse data while also utilizing erroneous data by assigning higher weights to misclassified samples in the next training round. However, this method tends to use a sequential ensemble structure, resulting in a long computation time. Conversely, the voting method employs a concurrent ensemble structure to reduce computation time but neglects the utilization of erroneous data. To address this issue, this study combines the advantages of voting and boosting methods and proposes a new two-stage voting boosting (2SVB) concurrent ensemble learning method for social network sentiment classification. This novel method not only establishes a concurrent ensemble framework to decrease computation time but also optimizes the utilization of erroneous data and enhances ensemble performance. To optimize the utilization of erroneous data, a two-stage training approach is implemented. Stage-1 training is performed on the datasets by employing a 3-fold cross-segmentation approach. Stage-2 training is carried out on datasets that have been augmented with the erroneous data predicted by stage 1. To augment the diversity of base classifiers, the training stage employs five pre-trained deep learning (PDL) models with heterogeneous pre-training frameworks as base classifiers. To reduce the computation time, a two-stage concurrent ensemble framework was established. The experimental results demonstrate that the proposed method achieves an F1 score of 0.8942 on the coronavirus tweet sentiment dataset, surpassing other comparable ensemble methods.

EnHDC: Ensemble Learning for Brain-Inspired Hyperdimensional Computing

Recently, brain-inspired hyperdimensional computing (HDC) becomes an emerging computational scheme that has achieved success in various domains, such as human activity recognition, voice recognition, and bio-medical signal classification. HDC mimics the brain cognition and leverages high-dimensional vectors (e.g., 10 000 dimensions) with fully distributed holographic representation and (pseudo-)randomness. Ensemble learning is a classical learning method utilizing a group of weak learners to form a strong learner, which aims to increase the accuracy of the model. This letter presents a systematic effort in exploring ensemble learning in the context of HDC and proposes an ensemble HDC model referred to as <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EnHDC</monospace> . <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EnHDC</monospace> uses a majority voting-based mechanism to synergistically integrate the prediction outcomes of multiple base HDC classifiers. To enhance the diversity of base classifiers, we vary the encoding mechanisms, dimensions, and data width settings among base classifiers. By applying <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EnHDC</monospace> on a wide range of applications, results show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EnHDC</monospace> can achieve on average 3.2% accuracy improvement over a single HDC classifier. Further, we show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EnHDC</monospace> with reduced dimensionality can achieve similar or even surpass the accuracy of baseline HDC with higher dimensionality. This leads to a 20% reduction of storage requirement of the HDC model, which can enhance the efficiency of HDC enabled on low-power computing platforms.

A Weighted Ensemble Classification Algorithm Based on Nearest Neighbors for Multi-Label Data Stream

With the rapid development of data stream, multi-label algorithms for mining dynamic data become more and more important. At the same time, when data distribution changes, concept drift will occur, which will make the existing classification models lose effectiveness. Ensemble methods have been used for multi-label classification, but few methods consider both the accuracy and diversity of base classifiers. To address the above-mentioned problem, a Weighted Ensemble classification algorithm based on Nearest Neighbors for Multi-Label data stream (WENNML) is proposed. WENNML uses data blocks to train Active candidate Ensemble Classifiers (AEC) and Passive candidate Ensemble Classifiers (PEC). The base classifiers of AEC and PEC are dynamically updated using geometric and diversity weighting methods. When the difference value between the number of current instances and the number of warning instances reaches the passive warning value, the algorithm selects the optimal base classifiers from AEC and PEC according to the subset accuracy and hamming score and puts them into the predictive ensemble classifiers. Experiments are carried out on 12 kinds of datasets with 9 comparison algorithms. The results show that WENNML achieves the best average rankings among the four evaluation metrics.

Adaptive boosting for ordinal target variables using neural networks

AbstractBoosting has proven its superiority by increasing the diversity of base classifiers, mainly in various classification problems. In reality, target variables in classification often are formed by numerical variables, in possession of ordinal information. However, existing boosting algorithms for classification are unable to reflect such ordinal target variables, resulting in non‐optimal solutions. In this paper, we propose a novel algorithm of ordinal encoding adaptive boosting (AdaBoost) using a multi‐dimensional encoding scheme for ordinal target variables. Extending an original binary‐class AdaBoost, the proposed algorithm is equipped with a multi‐class exponential loss function. We show that it achieves the Bayes classifier and establishes forward stagewise additive modeling. We demonstrate the performance of the proposed algorithm with a base learner as a neural network. Our experiments show that it outperforms existing boosting algorithms in various ordinal datasets.

AERF: Adaptive ensemble random fuzzy algorithm for anomaly detection in cloud computing

The abnormality of system behavior is inevitable in cloud computing because of its complexity and scale. How to perform anomaly detection on the system’s operating data to discover abnormal behavior has become a popular research field. However, anomaly detection is a challenging research problem because data in cloud computing scenarios is continuous, imbalanced, and cannot be acquired at once. In this work, an adaptive ensemble random fuzzy (AERF) algorithm is proposed for anomaly detection in cloud computing systems. The random fuzzy rule-based method in the AERF selects samples randomly to enhance the diversity of base classifiers for dealing with disturbances caused by abnormal sample distribution effectively. A dynamic weighting strategy is used for fuzzy classifier ensembles to improve processing efficiency and anomaly detection accuracy. Experimental results show that the AERF yields 10% higher AUC and G-mean than existing methods on SMD and EMOSCloud datasets and 20% higher AUC and G-mean on five benchmarking datasets. The AERF yields 20% higher F1 than existing methods on EMOSCloud dataset. For all datasets, the AERF yields 50% faster training time than other methods.

A selective evolutionary heterogeneous ensemble algorithm for classifying imbalanced data

&lt;abstract&gt; &lt;p&gt;Learning from imbalanced data is a challenging task, as with this type of data, most conventional supervised learning algorithms tend to favor the majority class, which has significantly more instances than the other classes. Ensemble learning is a robust solution for addressing the imbalanced classification problem. To construct a successful ensemble classifier, the diversity of base classifiers should receive specific attention. In this paper, we present a novel ensemble learning algorithm called Selective Evolutionary Heterogeneous Ensemble (SEHE), which produces diversity by two ways, as follows: 1) adopting multiple different sampling strategies to generate diverse training subsets and 2) training multiple heterogeneous base classifiers to construct an ensemble. In addition, considering that some low-quality base classifiers may pull down the performance of an ensemble and that it is difficult to estimate the potential of each base classifier directly, we profit from the idea of a selective ensemble to adaptively select base classifiers for constructing an ensemble. In particular, an evolutionary algorithm is adopted to conduct the procedure of adaptive selection in SEHE. The experimental results on 42 imbalanced data sets show that the SEHE is significantly superior to some state-of-the-art ensemble learning algorithms which are specifically designed for addressing the class imbalance problem, indicating its effectiveness and superiority.&lt;/p&gt; &lt;/abstract&gt;

Group Decision Making-Based Fusion for Human Activity Recognition in Body Sensor Networks.

Ensemble learning systems (ELS) have been widely utilized for human activity recognition (HAR) with multiple homogeneous or heterogeneous sensors. However, traditional ensemble approaches for HAR cannot always work well due to insufficient accuracy and diversity of base classifiers, the absence of ensemble pruning, as well as the inefficiency of the fusion strategy. To overcome these problems, this paper proposes a novel selective ensemble approach with group decision-making (GDM) for decision-level fusion in HAR. As a result, the fusion process in the ELS is transformed into an abstract process that includes individual experts (base classifiers) making decisions with the GDM fusion strategy. Firstly, a set of diverse local base classifiers are constructed through the corresponding mechanism of the base classifier and the sensor. Secondly, the pruning methods and the number of selected base classifiers for the fusion phase are determined by considering the diversity among base classifiers and the accuracy of candidate classifiers. Two ensemble pruning methods are utilized: mixed diversity measure and complementarity measure. Thirdly, component decision information from the selected base classifiers is combined by using the GDM fusion strategy and the recognition results of the HAR approach can be obtained. Experimental results on two public activity recognition datasets (The OPPORTUNITY dataset; Daily and Sports Activity Dataset (DSAD)) suggest that the proposed GDM-based approach outperforms the well-known fusion techniques and other state-of-the-art approaches in the literature.

A multi-class classification framework for disease screening and disease diagnosis of COVID-19 from chest X-ray images.

To accurately diagnose multiple lung diseases from chest X-rays, the critical aspect is to identify lung diseases with high sensitivity and specificity. This study proposed a novel multi-class classification framework that minimises either false positives or false negatives that is useful in computer aided diagnosis or computer aided detection respectively. To minimise false positives or false negatives, we generated respective stacked ensemble from pre-trained models and fully connected layers using selection metric and systematic method. The diversity of base classifiers was based on diverse set of false positives or false negatives generated. The proposed multi-class framework was evaluated on two chest X-ray datasets, and the performance was compared with the existing models and base classifiers. Moreover, we used LIME (Local Interpretable Model-agnostic Explanations) to locate the regions focused by the multi-class classification framework.

Equalization ensemble for large scale highly imbalanced data classification

The class-imbalance problem has been widely distributed in various research fields. The larger the data scale and the higher the data imbalance, the more difficult the proper classification. For large-scale highly imbalanced data sets, the ensemble method based on under-sampling is one of the most competitive techniques among the existing techniques. However, it is susceptible to improperly sampling strategies, easy to lose the useful information of the majority class, and not easy to generalize the learning model. To overcome these limitations, we propose an equalization ensemble method (EASE) with two new schemes. First, we propose an equalization under-sampling scheme to generate a balanced data set for each base classifier, which can reduce the impact of class imbalance on the base classifiers; Second, we design a weighted integration scheme, where the G-mean scores obtained by base classifiers on the original imbalanced data set are used as the weights. These weights can not only make the better-performed base-classifiers dominate the final classification decision, but also adapt to a variety of imbalanced data sets with different scales while avoiding the occurrence of some extremely bad situations. Experimental results on three metrics show that EASE increases the diversity of base classifiers and outperforms twelve state-of-the-art methods on the imbalanced data sets with different scales.

Stacknet Based Decision Fusion Classifier for Network Intrusion Detection

Network intrusion is a subject of great concern to a variety of stakeholders. Decision fusion (ensemble) models that combine several base learners have been widely used to enhance detection rate of unauthorised network intrusion. However, the design of such an optimal decision fusion classifier is a challenging and open problem. The Matthews Correlation Coefficient (MCC) is an effective measure for detecting associations between variables in many fields; however, very few studies have applied it in selecting weak learners to the best of the authors’ knowledge. In this paper, we propose a decision fusion model with correlation-based MCC weak learner selection technique to augment the classification performance of the decision fusion model under a StackNet strategy. Specifically, the proposed model sought to improve the association between the prediction accuracy and diversity of base classifiers. We compare our proposed model with five other ensemble models, a deep neural model and two stand-alone state-of-the-art classifiers commonly used in network intrusion detection based on accuracy, AUC, recall, precision, F1-score and Kappa evaluation metrics. The experimental results using benchmark dataset KDDcup99 from Kaggle shows that the proposed model has a identified unauthorised network traffic at 99.8% accuracy, Extreme Gradient Boosting (Xgboost) (97.61%), Catboost (97.49%), Light Gradient Boosting Machine (LightGBM) (98.3%), Multilayer Perceptron (MLP) (97.7%), Random Forest (RF) (97.97%), Extra Trees Classifier (ET) (95.82%), Different decision (DT) (96.95%) and , K-Nearest Neighbor (KNN) (95.56), indicating that it is a more efficient and better intrusion detection system.

Diversity-Driven Multikernel Collaborative Representation Ensemble for Hyperspectral Image Classification

Recently, kernel collaborative representation classification (KCRC) has shown its outstanding performance in dealing with the problem of linear inseparability in hyperspectral remote sensing image classification. Meanwhile, ensemble learning has attracted great attention in improving the performance of a single classifier. Aiming at the limitation of a single classifier, bagging algorithm based on KCRC(KCRC-bagging) is presented in this article. The KCRC-bagging method uses bootstrap to increase the diversity of base classifiers, thus improving the classification accuracy and generalization performance. In order to reduce the scale of ensemble, a diversity-driven multikernel collaborative representation classifier ensemble approach (DIV-KCRC) is proposed. DIV-KCRC verifies the effectiveness of the representation classifier with the pair of diversity measures, and classifiers with high accuracy and diversity are selected to improve the classification performance and efficiency of the ensemble system. Three real hyperspectral data sets were applied to prove the validity of the proposed method. The experimental results demonstrate that both KCRC-bagging and DIV-KCRC can yield better classification performance than their corresponding base classifiers. In particular, DIV-KCRC provides more reliable classification results than KCRC-bagging.

Experimental Study and Comparison of Imbalance Ensemble Classifiers with Dynamic Selection Strategy.

Imbalance ensemble classification is one of the most essential and practical strategies for improving decision performance in data analysis. There is a growing body of literature about ensemble techniques for imbalance learning in recent years, the various extensions of imbalanced classification methods were established from different points of view. The present study is initiated in an attempt to review the state-of-the-art ensemble classification algorithms for dealing with imbalanced datasets, offering a comprehensive analysis for incorporating the dynamic selection of base classifiers in classification. By conducting 14 existing ensemble algorithms incorporating a dynamic selection on 56 datasets, the experimental results reveal that the classical algorithm with a dynamic selection strategy deliver a practical way to improve the classification performance for both a binary class and multi-class imbalanced datasets. In addition, by combining patch learning with a dynamic selection ensemble classification, a patch-ensemble classification method is designed, which utilizes the misclassified samples to train patch classifiers for increasing the diversity of base classifiers. The experiments’ results indicate that the designed method has a certain potential for the performance of multi-class imbalanced classification.

Prediction of credit risk with an ensemble model: a correlation-based classifier selection approach

PurposeEnsemble models that combine multiple base classifiers have been widely used to improve prediction performance in credit risk evaluation. However, an arbitrary selection of base classifiers is problematic. The purpose of this paper is to develop a framework for selecting base classifiers to improve the overall classification performance of an ensemble model.Design/methodology/approachIn this study, selecting base classifiers is treated as a feature selection problem, where the output from a base classifier can be considered a feature. The proposed correlation-based classifier selection using the maximum information coefficient (MIC-CCS), a correlation-based classifier selection under the maximum information coefficient method, selects the features (classifiers) using nonlinear optimization programming, which seeks to optimize the relationship between the accuracy and diversity of base classifiers, based on MIC.FindingsThe empirical results show that ensemble models perform better than stand-alone ones, whereas the ensemble model based on MIC-CCS outperforms the ensemble models with unselected base classifiers and other ensemble models based on traditional forward and backward selection methods. Additionally, the classification performance of the ensemble model in which correlation is measured with MIC is better than that measured with the Pearson correlation coefficient.Research limitations/implicationsThe study provides an alternate solution to effectively select base classifiers that are significantly different, so that they can provide complementary information and, as these selected classifiers have good predictive capabilities, the classification performance of the ensemble model is improved.Originality/valueThis paper introduces MIC to the correlation-based selection process to better capture nonlinear and nonfunctional relationships in a complex credit data structure and construct a novel nonlinear programming model for base classifiers selection that has not been used in other studies.

A Hybrid Ensemble Algorithm Combining AdaBoost and Genetic Algorithm for Cancer Classification with Gene Expression Data.

The diversity of base classifiers and integration of multiple classifiers are two key issues in the field of ensemble learning. This paper puts forward a hybrid ensemble algorithm combining AdaBoost and genetic algorithm(GA) for cancer classification with gene expression data. The decision group is designed to increase the diversity of base classifier pool, and the GA is used to assign weight to each base classifier, thus to improve the classification performance by avoiding local extrema. The decision groups composed by using base classifiers, including K-nearest neighbor (KNN), Naïve Bayes (NB), and Decision Tree (C4.5). Experimental results show that the proposed algorithm is superior to those existing ensemble learning methods, such as Bagging, Random Forest (RF), Rotation Forest (RoF), AdaBoost, AdaBoost-BPNN, AdaBoost-SVM, and AdaBoost-RF, especially it has better performance on small samples and unbalanced gene expression data processing.

A Dynamic Ensemble Learning Algorithm based on K-means for ICU mortality prediction

This research proposes a Dynamic Ensemble Learning Algorithm based on K-means (DELAK) for intensive care unit (ICU) mortality prediction. Nowadays, the widely applied traditional scoring systems, which predict the mortality risk with some scores reflecting the severity of disease and physiological states of patients in ICU, have shown insufficient predictive performance when faced with large volume of data. Although taking advantage of large volume of data, single machine learning model and ensemble learning methods show an inadequate ability to make personalized predictions for each new patient. Dynamic ensemble selection (DES) methods which are widely studied in pattern recognition field can make personalized prediction but they only select a single classifier instead of ensemble of the results from a pool of classifiers. To overcome the limitations mentioned above, this research proposes an ensemble learning algorithm based on K-means sampling and distance-based dynamic ensemble. K-means sampling helps to achieve the diversity of base classifiers and the distance-based dynamic ensemble is a flexible fusion method which creates a personalized combination of results from base classifiers for each new test sample. To evaluate the performance for mortality prediction of the proposed algorithm, comprehensive experiments are conducted on MIMIC-III dataset. The experimental results show that DELAK achieves outstanding performance in term of AUROC and AUPRC compared with other six fusion strategies, traditional scoring systems, classical ensemble models, i.e. AdaBoost, Bagging and random forest, and dynamic ensemble selection methods in most mortality prediction tasks.

Ensemble learning based on random super-reduct and resampling

Ensemble learning has been widely used for improving the performance of base classifiers. Diversity among base classifiers is considered as a key issue in ensemble learning. Recently, to promote the diversity of base classifiers, ensemble methods through multi-modal perturbation have been proposed. These methods simultaneously use two or more perturbation techniques when generating base classifiers. In this paper, from the perspective of multi-modal perturbation, we propose an ensemble approach (called ‘E $$\_$$ RSRR’) based on random super-reduct and resampling. To generate a set of accurate and diverse base classifiers, E $$\_$$ RSRR adopts a new multi-modal perturbation strategy. This strategy combines two perturbation techniques together, that is, resampling and random super-reduct. First, it perturbs the sample space via the resampling technique; Second, it perturbs the feature space via the random super-reduct technique, which is a combination of RSS (random subspace selection) technique and ADEFS (approximate decision entropy-based feature selection) method in rough sets. Experimental results show that E $$\_$$ RSRR can provide competitive solutions for ensemble learning.

Improving deep forest by ensemble pruning based on feature vectorization and quantum walks

Recently, a deep learning model, the deep forest (DF), was designed as an alternative to deep neural networks. Each cascade layer of the DF contains a set of random forests (RFs) with a large number of decision trees, some of which are of high redundancy and poor performance. To avoid the negative impacts of such decision trees, this paper proposes to optimize RFs in each cascade layer of the DF so as to realize a pruned deep forest (PDF) with higher performance and smaller ensemble size. In this paper, a new ordering-based ensemble pruning method is proposed based on feature vectorization and quantum walks. This method simultaneously considers the accuracy and the diversity of base classifiers, and it provides an integrated evaluation criterion for ordering base classifiers in the ensemble system. The effectiveness of the proposed method is verified by experiments and discussions.

An ensemble of shapelet-based classifiers on inter-class and intra-class imbalanced multivariate time series at the early stage

Early classification of time series will weaken the accuracy to some degree. If the time series data are imbalanced, it will be also challenging to accurately identify minority class examples. Up to now, these two problems have been intensively addressed separately on univariate time series data, but yet to be well studied when they occur together. Compared with univariate time series, multivariate time series (MTS) is more complex, which contains multiple variables, and the interconnections between variables are hidden. Therefore, it is even more challenging to handle the combination of both problems on multivariate time series. In this paper, we propose an adaptive classification ensemble method called early prediction on imbalanced MTS to deal with early classification on inter-class and intra-class imbalanced MTS data simultaneously. First, an adaptive ensemble framework is designed to learn an early classification model on imbalanced MTS data. Based on a multiple under-sampling approach and dynamical subspace generation method, the diversity of base classifiers is realized as well as all majority class examples being fully utilized. Second, to deal with the implicit issue of intra-class imbalance in the training data, a cluster-based shapelet selection method is introduced to obtain an optimal set of stable and robust shapelets. Finally, an associate-pattern mining approach is designed to efficiently learn base classifiers, which could enhance the interpretability of classification. Experimental results show that our proposed method can achieve effective early prediction on inter-class and intra-class imbalanced MTS data.

An Ensemble Learning Method for Text Classification Based on Heterogeneous Classifiers

Abstract Ensemble learning can improve the accuracy of the classification algorithm and it has been widely used. Traditional ensemble learning methods include bagging, boosting and other methods, both of which are ensemble learning methods based on homogenous base classifiers, and obtain a diversity of base classifiers only through sample perturbation. However, heterogenous base classifiers tend to be more diverse, and multi-angle disturbances tend to obtain a variety of base classifiers. This paper presents a text classification ensemble learning method based on multi-angle perturbation heterogeneous base classifier, and validates the effectiveness of the algorithm through experiments.

Imbalanced enterprise credit evaluation with DTE-SBD: Decision tree ensemble based on SMOTE and bagging with differentiated sampling rates

Enterprise credit evaluation model is an important tool for bank and enterprise risk management, but how to construct an effective decision tree (DT) ensemble model for imbalanced enterprise credit evaluation is seldom studied. This paper proposes a new DT ensemble model for imbalanced enterprise credit evaluation based on the synthetic minority over-sampling technique (SMOTE) and the Bagging ensemble learning algorithm with differentiated sampling rates (DSR), which is named as DTE-SBD (Decision Tree Ensemble based on SMOTE, Bagging and DSR). In different times of iteration for base DT classifier training, new positive (high risky) samples are produced to different degrees by SMOTE with DSR, and different numbers of negative (low risky) samples are drawn with replacement by Bagging with DSR. However, in the same time of iteration with certain sampling rate, the training positive samples including the original and the new are of the same number as the drawn training negative samples, and they are combined to train a DT base classifier. Therefore, DTE-SBD can not only dispose the class imbalance problem of enterprise credit evaluation, but also increase the diversity of base classifiers for DT ensemble. Empirical experiment is carried out for 100 times with the financial data of 552 Chinese listed companies, and the performance of imbalanced enterprise credit evaluation is compared among the six models of pure DT, over-sampling DT, over-under-sampling DT, SMOTE DT, Bagging DT, and DTE-SBD. The experimental results indicate that DTE-SBD significantly outperforms the other five models and is effective for imbalanced enterprise credit evaluation.