Multiple Base Classifiers Research Articles

A selective evolutionary heterogeneous ensemble algorithm for classifying imbalanced data

<abstract> <p>Learning from imbalanced data is a challenging task, as with this type of data, most conventional supervised learning algorithms tend to favor the majority class, which has significantly more instances than the other classes. Ensemble learning is a robust solution for addressing the imbalanced classification problem. To construct a successful ensemble classifier, the diversity of base classifiers should receive specific attention. In this paper, we present a novel ensemble learning algorithm called Selective Evolutionary Heterogeneous Ensemble (SEHE), which produces diversity by two ways, as follows: 1) adopting multiple different sampling strategies to generate diverse training subsets and 2) training multiple heterogeneous base classifiers to construct an ensemble. In addition, considering that some low-quality base classifiers may pull down the performance of an ensemble and that it is difficult to estimate the potential of each base classifier directly, we profit from the idea of a selective ensemble to adaptively select base classifiers for constructing an ensemble. In particular, an evolutionary algorithm is adopted to conduct the procedure of adaptive selection in SEHE. The experimental results on 42 imbalanced data sets show that the SEHE is significantly superior to some state-of-the-art ensemble learning algorithms which are specifically designed for addressing the class imbalance problem, indicating its effectiveness and superiority.</p> </abstract>

Chronic disease diagnosis model based on convolutional neural network and ensemble learning method.

Chronic diseases have become one of the main causes of premature death all around the world in recent years. The diagnosis of chronic diseases is time-consuming and costly. Therefore, timely diagnosis and prediction of chronic diseases are very necessary. In this paper, a new method for chronic disease diagnosis is proposed by combining convolutional neural network (CNN) and ensemble learning. This method utilizes random forest (RF) as the base classifier to improve classification performance and diagnostic accuracy, and then combines AdaBoost to successfully replace the Softmax layer of CNN to generate multiple accurate base classifiers while determining their optimal attributes, achieving high-quality classification and prediction of chronic diseases. To verify the effectiveness of the proposed method, real-world Electronic Medical Records dataset (C-EMRs) was used for experimental analysis. The results show that compared with other traditional machine learning methods such as CNN, K-Nearest Neighbor, and RF, the proposed method can effectively improve the accuracy of diagnosis and reduce the occurrence of missed diagnosis and misdiagnosis. This study will provide effective information for the diagnosis of chronic diseases, assist doctors in making clinical decisions, develop targeted intervention measures, and reduce the probability of misdiagnosis.

Distributed Boosting Classification Over Noisy Communication Channels

We address the design of inference-oriented communication systems where multiple transmitters send partial inference values through noisy communication channels, and the receiver aggregates these channel outputs to obtain a reliable final inference. Since large data items are replaced by compact inference values, these systems lead to significant savings of communication resources. In particular, we present a principled framework to optimize communication-resource allocation for distributed boosting classifiers. Boosting classification algorithms make a final decision via a weighted vote from the outputs of multiple base classifiers. Since these base classifiers transmit their partial inference values over noisy channels, communication errors would degrade the final classification accuracy. We formulate communication resource allocation problems to maximize the final classification accuracy by taking into account the importance of base classifiers and the resource budget. To solve these problems rigorously, we formulate convex optimization problems to optimize: 1) transmit-power allocations and 2) transmit-rate allocations. This framework departs from classical communication-systems optimizations in seeking to maximize the classification accuracy rather than the reliability of the individual communicated bits. Results from numerical experiments demonstrate the benefits of our approach.

Research on Network Traffic Anomaly Detection Method Based on Python

Abnormal traffic is the traffic that differs from the normal range of network services. Objective social and natural phenomena, network equipment failures on hardware, and man-made malicious attacks can all lead to abnormal network traffic. Python is a computer programming language that can realize cross-platform interaction, and it is also an object-oriented explanatory and interactive scripting language. Based on this, this paper studies the network traffic anomaly detection method based on Python. By sampling the data sets divided by each layer with different strategies, multiple balanced sub-data sets are obtained, and the feature selection fusion method proposed in the previous section is applied to each sub-data set to obtain the corresponding optimal feature subset, which is used to train multiple base classifiers to perform anomaly detection in this layer. The results show that Python-based network traffic anomaly detection method is superior to the traditional algorithm in accuracy and F1-Score.

LDM-satellite: A new scheme for packet loss classification over LEO satellite network

The packet loss classification has always been a hot and difficult issue in TCP congestion control research. Compared with the terrestrial network, the probability of packet loss in LEO satellite network increases dramatically. What's more, the problem of concept drifting is also more serious, which greatly affects the accuracy of the loss classification model. In this paper, we propose a new loss classification scheme based on concept drift detection and hybrid integration learning for LEO satellite networks, named LDM-Satellite, which consists of three modules: concept drift detection, lost packet cache and hybrid integration classification. As far, this is the first paper to consider the influence of concept drift on the loss classification model in satellite networks. We also innovatively use multiple base classifiers and a naive Bayes classifier as the final hybrid classifier. And a new weight algorithm for these classifiers is given. In ns-2 simulation, LDM-Satellite has a better AUC (0.9885) than the single-model machine learning classification algorithms. The accuracy of loss classification even exceeds 98%, higher than traditional TCP protocols. Moreover, compared with the existing protocols used for satellite networks, LDM-Satellite not only improves the throughput rate but also has good fairness.

Protein-DNA Binding Residue Prediction via Bagging Strategy and Sequence-Based Cube-Format Feature.

Protein-DNA interactions play an important role in diverse biological processes. Accurately identifying protein-DNA binding residues is a critical but challenging task for protein function annotations and drug design. Although wet-lab experimental methods are the most accurate way to identify protein-DNA binding residues, they are time consuming and labor intensive. There is an urgent need to develop computational methods to rapidly and accurately predict protein-DNA binding residues. In this study, we propose a novel sequence-based method, named PredDBR, for predicting DNA-binding residues. In PredDBR, for each query protein, its position-specific frequency matrix (PSFM), predicted secondary structure (PSS), and predicted probabilities of ligand-binding residues (PPLBR) are first generated as three feature sources. Secondly, for each feature source, the sliding window technique is employed to extract the matrix-format feature of each residue. Then, we design two strategies, i.e., square root (SR) and average (AVE), to separately transform PSFM-based and two predicted feature source-based, i.e., PSS-based and PPLBR-based, matrix-format features of each residue into three corresponding cube-format features. Finally, after serially combining the three cube-format features, the ensemble classifier is generated via applying bagging strategy to multiple base classifiers built by the framework of 2D convolutional neural network. The computational experimental results demonstrate that the proposed PredDBR achieves an average overall accuracy of 93.7% and a Mathew's correlation coefficient of 0.405 on two independent validation datasets and outperforms several state-of-the-art sequenced-based protein-DNA binding residue predictors. The PredDBR web-server is available at https://jun-csbio.github.io/PredDBR/.

Ship Classification in Synthetic Aperture Radar Images Based on Multiple Classifiers Ensemble Learning and Automatic Identification System Data Transfer Learning

With the continuous development of earth observation technology, space-based synthetic aperture radar (SAR) has become an important source of information for maritime surveillance, and ship classification in SAR images has also become a hot research direction in the field of maritime ship monitoring. In recent years, the remote sensing community has proposed several solutions to the problem of ship object classification in SAR images. However, it is difficult to obtain an adequate amount of labeled SAR samples for training classifiers, which limits the application of machine learning, particularly deep learning methods, in SAR image ship object classification. In contrast, as a real-time automatic tracking system for monitoring ships at sea, a ship automatic identification system (AIS) can provide a large amount of relatively easy-to-obtain labeled ship samples. Therefore, to solve the problem of SAR image ship classification and improve the classification performance of learning models with limited samples, we proposed a SAR image ship classification method based on multiple classifiers ensemble learning (MCEL) and AIS data transfer learning. The core idea of our method is to transfer the MCEL model trained on AIS data to SAR image ship classification, which mainly includes three steps: first, we use the acquired global space-based AIS data to build a dataset for ship object classification models training; then, the ensemble learning model is constructed by combining multiple base classifiers; and finally, the trained classification model is transferred to SAR images for ship type prediction. Experiments show that the proposed method achieves a classification accuracy of 85.00% for the SAR ship classification, which is better than the performance of each base classifier. This proves that AIS data transfer learning can effectively solve the problem of SAR ship classification with limited samples, and has important application value in maritime surveillance.

Towards blind detection of steganography in low‐bit‐rate speech streams

To prevent the abuse of low-rate speech-based steganography from threatening cyberspace security, the corresponding steganalysis approaches have been developed and received significant attention from research community. However, most existing steganalysis methods assume that steganography methods are known in advance, which in practice is impractical. That is why, in this paper, we present three blind detection schemes suitable for steganography in low-bit-rate speech streams. The first is based on mixed sample data augmentation. It randomly selects a certain proportion of steganographic samples from the sample set of each steganographic method to form a training set together with the original carrier samples for training to enhance the robustness of the model. The second relies on decision fusion where first step is to train a dedicated classification model for each steganography method and then use a majority voting mechanism in the detection stage to fuse the outputs of each model to give the final detection result. Compared to the other two steganalysis schemes, the third one design the detection model based on self-paced ensemble according to the distribution characteristics of speech samples. Its main idea is to fully train multiple base classifiers through multiple iterations as well as under-sampling processes, and organically fuse them to form a powerful ensemble classifier. In each iteration, differing from the traditional ensemble classifier solution, we put more attention to the steganographic samples at the decision boundary for the under-sampling process of the steganography set composed of multiple steganography methods, rather than randomly selecting steganographic samples. The steganographic samples at the decision boundary are searched using the classification hardness given by the ensemble classifier trained in the last iteration, which is more informative and more conducive to improve the performance of base classifiers. The experimental results show that the proposed three schemes can achieve efficient blind detection for low-bit-rate speech-based steganography, and the steganalysis scheme based on the self-paced ensemble has the best performance. Specifically, when the embedding rate is at 30%, the accuracy of the steganalysis scheme based on self-paced ensemble is more than 85%, while the accuracy of the other two steganalysis method is less than 80%. Additionally, the steganalysis scheme based on the self-paced ensemble learning even outperforms dedicated detectors for specific steganographic methods in terms of recall for steganographic sample detection.

Pattern classification based on regional models

In a supervised setting, the global classification paradigm leverages the whole training data to produce a single class discriminative model. Alternatively, the local classification approach builds multiple base classifiers, each of them using a small subset of the training data. In this paper, we take a path to stand in-between the global and local approaches. We introduce a two-level clustering-based method in which base classifiers operate on a larger portion of the input space than in the traditional local paradigm. In particular, we first obtain a grained input representation by employing a Self-Organizing Map (SOM) to the inputs. We then apply a clustering algorithm (e.g., K-Means) to the SOM units to define input regions — a subset of input samples associated with a specific cluster of SOM units. We refer to this approach as regional classification. We demonstrate the effectiveness of regional classification on several benchmarks. Also, we study the impact of (1) adopting linear and nonlinear base classifiers (e.g., least squares support vector machines) and (2) using cluster validation indexes to determine the optimal number of clusters. Based on the experiments, the regional classification approach achieves competitive performance compared to its global and local counterparts, especially when equipped with linear base classifiers.

A novel sample and feature dependent ensemble approach for Parkinson’s disease detection

Parkinson’s disease (PD) is a neurological disease that has been reported to have affected most people worldwide. Recent research pointed out that about 90% of PD patients possess voice disorders. Motivated by this fact, many researchers proposed methods based on multiple types of speech data for PD prediction. However, these methods either face the problem of low rate of accuracy or lack generalization. To develop an approach that will be free of these issues, in this paper we propose a novel ensemble approach. These paper contributions are two folds. First, investigating feature selection integration with deep neural network (DNN) and validating its effectiveness by comparing its performance with conventional DNN and other similar integrated systems. Second, development of a novel ensemble model namely EOFSC (Ensemble model with Optimal Features and Sample Dependant Base Classifiers) that exploits the findings of recently published studies. Recent research pointed out that for different types of voice data, different optimal models are obtained which are sensitive to different types of samples and subsets of features. In this paper, we further consolidate the findings by utilizing the proposed integrated system and propose the development of EOFSC. For multiple types of vowel phonations, multiple base classifiers are obtained which are sensitive to different subsets of features. These features and sample-dependent base classifiers are integrated, and the proposed EOFSC model is constructed. To evaluate the final prediction of the EOFSC model, the majority voting methodology is adopted. Experimental results point out that feature selection integration with neural networks improves the performance of conventional neural networks. Additionally, feature selection integration with DNN outperforms feature selection integration with conventional machine learning models. Finally, the newly developed ensemble model is observed to improve PD detection accuracy by 6.5%.

Information fusion for automated post-disaster building damage evaluation using deep neural network

This paper develops a hybrid neural network architecture named multi-class factorization machine with deep neural network (multi-FMDNN) to fuse multi-source information for the automatic post-earthquake building damage evaluation. The novel algorithm is a combination of the factorization machine (FM) and the deep neural network (DNN), which adopts the one-vs-all strategy to fuse results from multiple base classifiers. 39,352 buildings affected by the 2015 Nepal earthquake are taken as a case study to validate the effectiveness of the proposed multi-FMDNN. Experimental results confirm that the proposed model outperforms over many other popular machine learning methods due to the powerful feature learning ability, ultimately reaching an overall accuracy, macro F1-score, and weighted F1-score in the value of 0.703, 0.737, and 0.702, respectively. Features associated with building structural characteristics are found to contribute more to classifying damage grades precisely. Besides, data preprocessing for data cleaning, encoding, and transformation is a necessary step to bring additional performance enhancement. For significance in the knowledge aspect, a novel multi-FMDNN algorithm is developed, which is superior in extracting both low- and high-order feature representation automatically from large volumes of destroyed buildings-related data and learning the optimal feature interactions simultaneously to pursue more accurate classification. For significance in the application aspect, the predicted results provide deep insights into a better understanding of the building vulnerability in seismic areas and inform data-driven decisions in disaster relief efforts. A promising future scope is to make full use of the available pre-event data along with some post-event data, which is possible to return fairly promising predictions and reduce the burden in earthquake field investigations for rapid responses. In future work, advanced techniques associated with data augment, hyperparameter optimization, and others will be implemented to constantly improve the overall accuracy and generalizability of the prediction model.

Ensembling perturbation-based oversamplers for imbalanced datasets

Oversampling is effective to handle imbalanced data and is often combined with ensemble learning to boost its performance, for examples the synthetic minority oversampling technique (SMOTE) and its ensemble variants SMOTEBagging and SMOTEBoost. However, SMOTE-based methods often introduce unnecessary noises when classes are not well separated, which may impinge upon classifier training and hinder the classification performance. In addition, the aggressive generation of SMOTE-based methods would further complicate the learning process. Therefore, we propose a simple yet effective approach to generate diverse instances to balance the class distribution in a more conservative way in this work. Major differences among the proposed method and existing methods, which also highlight the originality of this work, are listed as follows. Firstly, unlike existing methods which utilize nearest neighbors-based methods to determine regions of interest, we analyze and compute the impact of class imbalance quantitatively for each minority instance in order to guide the data generation process. This ensures instances can be synthesized in a safe and informative region to avoid noise generation. Secondly, new instances are generated via perturbing a random subsets of input features of given seed instances. We show that this process generates instances that share the same asymptotic mean and covariance as given seed instances, thus classifiers with high generalization capability can be achieved. The generation process is repeated to train multiple base classifiers which are then fused via a majority voting to further improve its performance. Nonparametric Wilcoxon test and Friedman test confirm that the proposed method significantly outperforms most reference methods over thirty-five imbalanced datasets in terms of five metrics. The important finding of this work is that we show that ensembling simple perturbation-based oversamplers can yield better performance than many advanced ensemble methods for imbalanced datasets.

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion.

The rising use of online media has changed the social customs of the public. Users have become accustomed to sharing daily experiences and publishing personal opinions on social networks. Social data carrying emotion and attitude has provided significant decision support for numerous tasks in sentiment analysis. Conventional methods for sentiment classification only concern textual modality and are vulnerable to the multimodal scenario, while common multimodal approaches only focus on the interactive relationship among modalities without considering unique intra-modal information. A hybrid fusion network is proposed in this paper to capture both inter-modal and intra-modal features. Firstly, in the stage of representation fusion, a multi-head visual attention is proposed to extract accurate semantic and sentimental information from textual contents, with the guidance of visual features. Then, multiple base classifiers are trained to learn independent and diverse discriminative information from different modal representations in the stage of decision fusion. The final decision is determined based on fusing the decision supports from base classifiers via a decision fusion method. To improve the generalization of our hybrid fusion network, a similarity loss is employed to inject decision diversity into the whole model. Empiric results on five multimodal datasets have demonstrated that the proposed model achieves higher accuracy and better generalization capacity for multimodal sentiment analysis.

SentiXGboost: enhanced sentiment analysis in social media posts with ensemble XGBoost classifier

ABSTRACT Sentiment Analysis has emerged as one of the most challenging and significant research directions in text mining. The first milestone of Sentiment Analysis involves identifying the polarity of sentiments in social media and classifying them as positive or negative. Most of the previous work on sentiment analysis has focused on designing a classifier, for a specific dataset, using supervised machine learning approaches and feature extraction methods. Notably, choosing the most appropriate classification technique to correctly classify the sentiment polarity of text is crucial. Thus, this study proposes a novel ensemble classifier approach, which utilizes a combination of multiple feature sets with ensemble classification by combining multiple base classifiers, which are weak learners, into an ensemble classifier. These feature sets include Bag of Words, Term Frequency–Inverse Document Frequency, Part of Speech, N-gram, Opinion Lexicon, and Term Frequency. The experiments confirm that the proposed ensemble method outperforms all individual classifiers and significantly improves the overall sentiment classification performance on the most frequently used datasets in Sentiment Analysis.

CLASS-IMBALANCED CLASSIFIERS USING ENSEMBLES OF GAUSSIAN PROCESSES AND GAUSSIAN PROCESS LATENT VARIABLE MODELS.

Classification with imbalanced data is a common and challenging problem in many practical machine learning problems. Ensemble learning is a popular solution where the results from multiple base classifiers are synthesized to reduce the effect of a possibly skewed distribution of the training set. In this paper, binary classifiers based on Gaussian processes are chosen as bases for inferring the predictive distributions of test latent variables. We apply a Gaussian process latent variable model where the outputs of the Gaussian processes are used for making the final decision. The tests of the new method in both synthetic and real data sets show improved performance over standard approaches.

Prediction of credit risk with an ensemble model: a correlation-based classifier selection approach

PurposeEnsemble models that combine multiple base classifiers have been widely used to improve prediction performance in credit risk evaluation. However, an arbitrary selection of base classifiers is problematic. The purpose of this paper is to develop a framework for selecting base classifiers to improve the overall classification performance of an ensemble model.Design/methodology/approachIn this study, selecting base classifiers is treated as a feature selection problem, where the output from a base classifier can be considered a feature. The proposed correlation-based classifier selection using the maximum information coefficient (MIC-CCS), a correlation-based classifier selection under the maximum information coefficient method, selects the features (classifiers) using nonlinear optimization programming, which seeks to optimize the relationship between the accuracy and diversity of base classifiers, based on MIC.FindingsThe empirical results show that ensemble models perform better than stand-alone ones, whereas the ensemble model based on MIC-CCS outperforms the ensemble models with unselected base classifiers and other ensemble models based on traditional forward and backward selection methods. Additionally, the classification performance of the ensemble model in which correlation is measured with MIC is better than that measured with the Pearson correlation coefficient.Research limitations/implicationsThe study provides an alternate solution to effectively select base classifiers that are significantly different, so that they can provide complementary information and, as these selected classifiers have good predictive capabilities, the classification performance of the ensemble model is improved.Originality/valueThis paper introduces MIC to the correlation-based selection process to better capture nonlinear and nonfunctional relationships in a complex credit data structure and construct a novel nonlinear programming model for base classifiers selection that has not been used in other studies.

A hybrid ensemble learning method for the identification of gang-related arson cases

Arson is one of the most common crimes, and it has the characteristics of low cost and great harm. In addition to causing casualties and property damage, arson can often have huge social impacts and cause psychological panic in the public. Since arson is more harmful when conducted by a gang, how to effectively identify gang crimes in arson cases has become an important issue. In this paper, we propose a hybrid method that combines ensemble learning and intelligent optimization algorithms to solve this problem. First, we develop the recursive feature elimination (RFE)-based feature selection method to remove redundant features. Second, for the data imbalance problem, we determine the optimal processing algorithm from 18 candidate algorithms. Third, after trying a combination of multiple base classifiers, we obtain the optimal base classifier combination. Fourth, when integrating the prediction results of the base classifier, we propose a weighted ensemble strategy. Finally, we use the differential evolution (DE) algorithm to optimize the parameters of the base classifier and the weight of the combination, which further enhances the identification ability of the model. To verify the actual performance of the proposed method, we conducted experiments on the US National Fire Incident Reporting System (NFIRS) database. The results show that the proposed method is significantly superior to other popular machine learning methods, which proves that this method can provide a more reliable decision basis in the detection of arson cases.

A Novel Multi-Ensemble Method for Identifying Essential Proteins.

Essential proteins possess critical functions for cell survival. Identifying essential proteins improves our understanding of how a cell works and also plays a vital role in the research fields of disease treatment and drug development. Recently, some machine-learning methods and ensemble learning methods have been proposed to identify essential proteins by introducing effective protein features. However, the ensemble learning method only used to focus on the choice of base classifiers. In this article, we propose a novel ensemble learning framework called multi-ensemble to integrate different base classifiers. The multi-ensemble method adopts the idea of multi-view learning and selects multiple base classifiers and trains those classifiers by continually adding the samples that are predicted correctly by the other base classifiers. We applied multi-ensemble to Yeast data and Escherichia coli data. The results show that our approach achieved better performance than both individual classifiers and the other ensemble learning methods.

A Novel Method for Credit Scoring Based on Cost-Sensitive Neural Network Ensemble

Most existing studies on credit scoring adapted a concept of classifier ensemble for solving an imbalanced dataset. They apply resampling methods to generate multiple training subsets for constructing multiple base classifiers. However, this approach leads to several problems that degrade the classification performance, such as problems of information loss, model overfitting, and computational cost. Thus, we propose a novel ensemble approach for developing a credit scoring model based on a cost-sensitive neural network, called Cost-sensitive Neural Network Ensemble (CS-NNE). In the proposed approach, multiple class weights are adapted to original training data, enabling the multiple base neural networks to consider imbalanced classes. Following this approach, a high diversity of multiple base classifiers without consequent problems can be achieved. The approach's effectiveness is evaluated on five real-world credit datasets. Among them is a loan-requesting dataset provided by a financial institution in Thailand. The remaining datasets are publicly available and widely used by several existing studies. The experimental results showed that the proposed CS-NNE approach improves the predictive performance over a single neural network based on imbalanced credit datasets, e.g., Thai credit dataset, by achieving 1.36%, 15.67%, and 6.11% Area under the ROC Curve (AUC), Default Detection Rate (DDR), and G-Mean (GM), respectively, and achieving the best Misclassification Cost (MC). The proposed CS-NNE approach can effectively solve a class of imbalance problems and outperform many existing models. The prediction model can well compromise between classes of default (bad credit applicants) and non-default (good credit applicants), whereas existing approaches preferred a class of non-default over default loans (having high specificity and low DDR), resulting in NPL.

DDES: A Distribution-Based Dynamic Ensemble Selection Framework

Dynamic Ensemble Selection (DES) is a special type of ensemble modeling that selects different subsets of base classifiers for different test sample cases. In this process, multiple base classifiers are compared in terms of their competence as an ensemble to the best possible prediction for a given test sample case. Traditional DES methods rely on the Euclidean distance-based k-Nearest Neighbor (kNN) algorithm to identify the most relevant reference data points in such a way that the base classifiers correctly classifying them can be considered for an optimal ensemble. However, this approach is sensitive to the local structure of the data and the presence of noisy or irrelevant attributes. This study proposes a novel distribution-based DES (DDES) framework that takes the data structure into consideration when selecting reference data points. The experimental results for 30 classification problems indicate that the proposed approach yielded the best accuracy among the competing DES methods. Additionally, we discuss the correlation between the data complexity and the improvement in classification performance.