Train Base Classifiers Research Articles

Imbalanced ensemble learning leveraging a novel data-level diversity metric

Ensemble learning is one of the best solutions for imbalanced classification problems. Diversity is a key factor that affects the performance of ensemble learning. Most existing diversity metrics such as Q-statistics measure diversity based on the outputs of the base classifiers, incurring high complexity on model training due to the need to re-train base classifiers to achieve satisfactory diversity. We propose a new diversity measure, named Instance Euclidean Distance metric (IED), to evaluate diversity directly based on the training data without training base classifiers, which can significantly cut down the time costs of diversity measuring. A new imbalanced ensemble learning algorithm named P-EUSBagging is proposed to reduce training complexity and improve learning performance by combining IED with population-based incremental learning to generate training datasets with the maximal data-level diversity. Experimental results demonstrate that the diversities measured by IED and three classifier-based diversity measures exhibit a mean absolute correlation coefficient of 0.94, and P-EUSBagging significantly reduces training time and improves learning performance on both Geometric Mean (G-Mean) and Area Under the Curve (AUC).

Graph ensemble neural network

Ensemble methods have been shown to improve graph neural networks (GNNs). Existing ensemble methods on graphs determine a strong classifier by combining a set of trained base classifiers, i.e., combining the final outputs of base classifiers for prediction. However, these methods fail to promote many popular GNNs to perform well under heterophily (in graphs where many connected nodes have different class labels), which limits their applicability. Furthermore, they ignore the hierarchical nature of GNNs, which results in no interaction between base classifiers when neighbors are aggregated (during training). Two issues arise from this: low applicability and shallow ensemble. We propose Graph Ensemble Neural Network (GEN) for addressing above issues, which is not a simple ensemble of GNNs, but instead integrates ensemble into GNNs to fuse a set of graphs. GEN deepens single ensemble into multiple ensembles during training and applies to homophily and heterophily graphs. In GEN, we design structure augmentation to generate some graphs for training and design feature augmentation for attenuating errors brought by the initial features. Different from existing graph ensemble methods that execute only one ensemble, GEN executes multiple deep ensembles throughout the neighbor aggregation to fuse multiple graphs generated by structure augmentation. Extensive experiments show that GEN achieves new state-of-the-art performance on homophily and heterophily graphs for the semi- and full-supervised node classification. The source code of GEN is publicly available at https://github.com/graphNN/GEN1.

Interval-based sparse ensemble multi-class classification algorithm for terahertz data

Terahertz time-domain spectroscopy (THz-TDS) has been widely used for food and drug identification. The classification information of a THz spectrum usually does not exist in the whole spectral band but exists only in one or several small intervals. Therefore, feature selection is indispensable in THz-based substance identification. However, most THz-based identification methods empirically intercept the low-frequency band of the THz absorption coefficients for analysis. In order to adaptively find out important intervals of the THz spectra, an interval-based sparse ensemble multi-class classifier (ISEMCC) for THz spectral data classification is proposed. In ISEMCC, the THz spectra are first divided into several small intervals through window sliding. Then the data of training samples in each interval are extracted to train some base classifiers. Finally, a final robust classifier is obtained through a nonnegative sparse combination of these trained base classifiers. With l1 -norm, two objective functions that based on Mean Square Error (MSE) and Cross Entropy (CE) are established. For these two objective functions, two iterative algorithms based on the Alternating Direction Method of Multipliers (ADMM) and Gradient Descent (GD) are built respectively. ISEMCC transforms the problem of interval feature selection and decision-level fusion into a nonnegative sparse optimization problem. The sparse constraint ensures only a few important spectral segments are selected. In order to verify the performance of the proposed algorithm, comparative experiments on identifying the origin of Bupleurum and the harvesting year of Tangerine peel are carried out. The base classifiers used by ISEMCC are Support Vector Machine (SVM) and Decision Tree (DT). The experimental results demonstrate that the proposed algorithm outperforms six typical classifiers, including Random Forest (RF), AdaBoost, RUSBoost, ExtraTree, and the two base classifiers, in terms of classification accuracy.

A Mutual Guide Framework for Training Hyperspectral Image Classifiers With Small Data

This article develops a general yet effective hyperspectral image (HSI) classification framework that is trained with small data. To this end, two identically structured but differently initialized classifiers, which are referred to as two base classifiers, are trained in an iterative manner. Each iteration consists of three steps, that is: 1) the two base classifiers that are trained separately on guide data; 2) unclassified data that are processed by the two trained base classifiers; and 3) the classification results with high confidence that are explored as new guide data. In the first iteration, the guide data comprising the original small training data are the same for the two base classifiers. From the second iteration, the guide data for the two base classifiers start becoming different. Specifically, in each iteration, the guide data for training one base classifier keep being augmented by high confidence classification results provided by the other base classifier. It is in such an iterative manner that the two classifiers continuously provide different new guide data for each other, and thus increasingly augment labeled data from the original small training set to a reasonably larger amount of samples in a HSI. We refer to such a training strategy as mutual guide. We develop a mutual guide implementation scheme by exploiting extreme learning machines (ELMs) as base classifiers. Extensive experiments on four public HSI datasets, i.e., Indian Pines (IP), Kennedy Space Center (KSC), University of Pavia (UP), and Salinas (SA), validate the classification effectiveness of our mutual guide framework with small training data.

Remote Sensing Image Classification Using Deep–Shallow Learning

Recently, classification using multiple classifier system (MCS) has been reported as an effective method to improve remote sensing (RS) image classification. Such systems provide a complementary mechanism to use multiple classifiers, which have shallow architecture to solve the same classification problem; however, the system exhibits shortcomings due to complex ensemble strategy. Deep learning (DL) has been proven to be an advanced method for complex data classification; however, how to use its advantages to overcome the shortcomings of MCS in ensemble strategy for classification accuracy improvement is worthy of study. Thus, with the multiple classifier mechanism and DL architecture, we propose a novel RS image classification framework, namely, deep–shallow learning (DSL), to improve classification accuracy. The DSL framework consists of a shallow learning (SL) layer and a DL layer. The SL layer contains various classifiers with shallow architecture, which can output different classification results for a certain input, whereas the DL layer is formed by DL networks, which can continue learning from the outputs of the SL layer. DSL simulates a human thinking model that continuously learns from the existing learnings to improve learning efficiency. In our experiment, three shallow classification algorithms, i.e., C4.5, k-nearest neighbor, and naive Bayesian, are used to train base classifiers in the SL layer, whereas a deep belief network (DBN) is used to train the DL layer. The experiment results on three different datasets indicate that DSL outperforms other methods in terms of classification accuracy by using backpropagation neural network, bagging, AdaBoost, random forest, multilayer perceptron, and DBN.

Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams

This work aims to connect two rarely combined research directions, i.e., non-stationary data stream classification and data analysis with skewed class distributions. We propose a novel framework employing stratified bagging for training base classifiers to integrate data preprocessing and dynamic ensemble selection methods for imbalanced data stream classification. The proposed approach has been evaluated based on computer experiments carried out on 135 artificially generated data streams with various imbalance ratios, label noise levels, and types of concept drift as well as on two selected real streams. Four preprocessing techniques and two dynamic selection methods, used on both bagging classifiers and base estimators levels, were considered. Experimentation results showed that, for highly imbalanced data streams, dynamic ensemble selection coupled with data preprocessing could outperform online and chunk-based state-of-art methods.

Ensemble learning based on approximate reducts and bootstrap sampling

Ensemble learning is an effective approach for improving the generalization ability of base classifiers. To generate a set of accurate and diverse base classifiers, different data perturbation schemes have been proposed. For instance, Bagging perturbs the training data via bootstrap sampling. However, when a stable learning algorithm (e.g., KNN, Naive Bayes) is used to train base classifiers, the sole perturbation on the training data may not produce diverse base classifiers. In this paper, by using the attribute reduction technology in rough sets, a multi-modal perturbation-based algorithm (called ‘E_EARBS’) is proposed for the ensemble of base classifiers. E_EARBS simultaneously perturbs the feature space, training data and learning parameters, where the relative decision entropy(RDE)-based approximate reducts are used to perturb the feature space, and bootstrap sampling is used to perturb the training data. Experimental results show that E_EARBS can provide competitive solutions for ensemble learning.

A non-specialized ensemble classifier using multi-objective optimization

Ensemble classification algorithms are often designed for data with certain properties, such as imbalanced class labels, a large number of attributes, or continuous data. While high-performing, these algorithms sacrifice performance when applied to data outside the targeted domain. We propose a non-specific ensemble classification algorithm that uses multi-objective optimization instead of relying on heuristics and fragile user-defined parameters. Only two user-defined parameters are included, with both being found to have large windows of values that produce statistically indistinguishable results, indicating the low level of expertise required from the user to achieve good results. Additionally, when given a large initial set of trained base-classifiers, we demonstrate that a multi-objective genetic algorithm aiming to optimize prediction accuracy and diversity will prefer particular types of classifiers over others. The total number of chosen classifiers is also surprisingly small – only 10.14 classifiers on average, out of an initial pool of 900. This occurs without any explicit preference for small ensembles of classifiers. Even with these small ensembles, significantly lower empirical classification error is achieved compared to the current state-of-the-art.

Cost-sensitive semi-supervised selective ensemble model for customer credit scoring

Only a few customers can be labeled in realistic credit-scoring problems, while many other customers cannot. Further, satisfactory performance is difficult, as traditional supervised learning methods can only use labeled samples to build credit-scoring models. Semi-supervised learning (SSL) can use both labeled and unlabeled samples to solve this problem, but existing credit-scoring research has primarily constructed single semi-supervised models. This study introduces SSL, cost-sensitive learning, a group method of data handling (GMDH), and an ensemble learning technique to propose a GMDH-based cost-sensitive semi-supervised selective ensemble (GCSSE) model. This involves two stages: (1)First, train an ensemble model composed of N base classifiers on the initial training set L with class labels, use it to selectively label the samples from the dataset U without class labels, add them with their predicted labels to the training set, and update the N base classifiers on the new training set; (2)Second, classify L and the test set using the respective trained base classifiers, and construct a cost-sensitive GMDH neural network to obtain the selective ensemble classification results for the test set. Experimental comparisons of five public customer credit score datasets and an empirical analysis of a real customer credit score dataset suggest that this model exhibits the best overall credit-scoring performance compared with one supervised ensemble model and three semi-supervised ensemble models.

Protein Secondary Structure Prediction Based on Data Partition and Semi-Random Subspace Method

Protein secondary structure prediction is one of the most important and challenging problems in bioinformatics. Machine learning techniques have been applied to solve the problem and have gained substantial success in this research area. However there is still room for improvement toward the theoretical limit. In this paper, we present a novel method for protein secondary structure prediction based on a data partition and semi-random subspace method (PSRSM). Data partitioning is an important strategy for our method. First, the protein training dataset was partitioned into several subsets based on the length of the protein sequence. Then we trained base classifiers on the subspace data generated by the semi-random subspace method, and combined base classifiers by majority vote rule into ensemble classifiers on each subset. Multiple classifiers were trained on different subsets. These different classifiers were used to predict the secondary structures of different proteins according to the protein sequence length. Experiments are performed on 25PDB, CB513, CASP10, CASP11, CASP12, and T100 datasets, and the good performance of 86.38%, 84.53%, 85.51%, 85.89%, 85.55%, and 85.09% is achieved respectively. Experimental results showed that our method outperforms other state-of-the-art methods.

DroidFusion: A Novel Multilevel Classifier Fusion Approach for Android Malware Detection.

Android malware has continued to grow in volume and complexity posing significant threats to the security of mobile devices and the services they enable. This has prompted increasing interest in employing machine learning to improve Android malware detection. In this paper, we present a novel classifier fusion approach based on a multilevel architecture that enables effective combination of machine learning algorithms for improved accuracy. The framework (called DroidFusion), generates a model by training base classifiers at a lower level and then applies a set of ranking-based algorithms on their predictive accuracies at the higher level in order to derive a final classifier. The induced multilevel DroidFusion model can then be utilized as an improved accuracy predictor for Android malware detection. We present experimental results on four separate datasets to demonstrate the effectiveness of our proposed approach. Furthermore, we demonstrate that the DroidFusion method can also effectively enable the fusion of ensemble learning algorithms for improved accuracy. Finally, we show that the prediction accuracy of DroidFusion, despite only utilizing a computational approach in the higher level, can outperform stacked generalization, a well-known classifier fusion method that employs a meta-classifier approach in its higher level.

Classifier Ensemble by Exploring Supplementary Ordering Information

Supplementary information has been proven to be particularly useful in many machine learning tasks. In ensemble learning for a set of trained base classifiers, there also exists abundant implicit supplementary information about the performance orderings for the trained base classifiers in previous literature. However, few classifier ensemble studies consider exploring and utilizing supplementary information. The current study proposes a new learning method for stack classifier ensembles by considering the implicit supplementary ordering information regarding a set of trained classifiers. First, a new metric learning algorithm for measuring the similarities between two arbitrary learning tasks is introduced. Second, supplementary ordering information for the trained classifiers of a given learning task is inferred based on the learned similarities and related performance results reported in the previous literature. Third, a set of ordered soft constraints is generated based on the supplementary ordering information, and achieving the optimal combination weights of the trained classifiers is formalized into a goal programming problem. The optimal combination weights are then obtained. Finally, the experimental results verify the effectiveness of the proposed new classifier ensemble method.

A Comparative Study of The Ensemble and Base Classifiers Performance in Malay Text Categorization

Automatic text categorization (ATC) has attracted the attention of the research community over the last decade as it frees organizations from the need of manually organized documents. The ensemble techniques, which combine the results of a number of individually trained base classifiers, always improve classification performance better than base classifiers. This paper intends to compare the effectiveness of ensemble with that of base classifiers for Malay text classification. Two feature selection methods (being the Gini Index (GI) and Chi-square) with the ensemble methods are applied to examine Malay text classification, with the intention to efficiently integrate base classifiers algorithms into a more accurate classification procedure. Two types of ensemble methods, namely the voting combination and meta-classifier combination, are evaluated. A wide range of comparative experiments are conducted to assess manually classified Malay corpus. The applied experiments reveal that meta-classifier ensemble framework performed better than the best individual classifiers on the tested datasets.

Classifier ensemble selection based on affinity propagation clustering

A small number of features are significantly correlated with classification in high-dimensional data. An ensemble feature selection method based on cluster grouping is proposed in this paper. Classification-related features are chosen using a ranking aggregation technique. These features are divided into unrelated groups by an affinity propagation clustering algorithm with a bicor correlation coefficient. Some diversity and distinguishing feature subsets are constructed by randomly selecting a feature from each group and are used to train base classifiers. Finally, some base classifiers that have better classification performance are selected using a kappa coefficient and integrated using a majority voting strategy. The experimental results based on five gene expression datasets show that the proposed method has low classification error rates, stable classification performance and strong scalability in terms of sensitivity, specificity, accuracy and G-Mean criteria.

Polarity shift detection, elimination and ensemble: A three-stage model for document-level sentiment analysis

The polarity shift problem is a major factor that affects classification performance of machine-learning-based sentiment analysis systems. In this paper, we propose a three-stage cascade model to address the polarity shift problem in the context of document-level sentiment classification. We first split each document into a set of subsentences and build a hybrid model that employs rules and statistical methods to detect explicit and implicit polarity shifts, respectively. Secondly, we propose a polarity shift elimination method, to remove polarity shift in negations. Finally, we train base classifiers on training subsets divided by different types of polarity shifts, and use a weighted combination of the component classifiers for sentiment classification. The results on a range of experiments illustrate that our approach significantly outperforms several alternative methods for polarity shift detection and elimination.

Semi-Supervised Classification for Hyperspectral Image Based on Tri-Training

Hyperspectral image classification is difficult due to the high dimensional features but limited training samples. Tri-training learning is a widely used semi-supervised classification method that addresses the problem of lacking of labeled examples. In this paper, a novel semi-supervised learning algorithm based on tri-training method is proposed. The proposed algorithm combines margin sampling (MS) technique and differential evolution (DE) algorithm to select the most informative samples and perturb them randomly. Then the samples we obtained, which can fulfill the labeled data distribution and introduce diversity to multiple classifiers, are added to training set to train base classifiers for tri-training. The proposed algorithm is experimentally validated using real hyperspectral data sets, indicating that the combination of MS and DE can significantly reduce the need of labeled samples while achieving high accuracy compared with state-of-the-art algorithms.

A Novel Machine Learning Approach Toward Quality Assessment of Sensor Data

A novel machine learning approach to assess the quality of sensor data using an ensemble classification framework is presented in this paper. The quality of sensor data is indicated by discrete quality flags that indicate the level of uncertainty associated with a sensor reading. Depending on the domain and the problem under consideration, the level of uncertainty is different and thus unsupervised methods like outlier detection fails to match the expectation. The quality flags are normally assigned by domain experts. Considering the volume of sensor data, manual assignment is a laborious task and subject to human error. Given a representative set of labelled data, a supervised classification approach is thus a feasible alternative. The nature of sensor data, however, poses some challenges to the classification task. Data of dubious quality exists in such data sets with very small frequency leading to the class imbalance problem. We thus adopt a cluster oriented sampling approach to address the imbalance issue. In addition, it is beneficial to train multiple classifiers to improve the overall classification accuracy. We thus produce multiple under-sampled training sets using cluster oriented sampling and train base classifiers on each of them. Decisions produced by the base classifiers are fused into a single decision using majority voting. We have evaluated the proposed ensemble classification framework by assessing the quality of marine sensor data obtained from sensors situated at Sullivans Cove, Hobart, Australia. Experimental results reveal that the proposed framework agrees with expert judgement with high accuracy and achieves superior classification performance than other state-of-the-art approaches.

A hybrid classifier committee for analysing asymmetry features in breast thermograms

Breast cancer is the most commonly occurring form of cancer in women. While mammography is the standard modality for diagnosis, thermal imaging provides an interesting alternative as it can identify tumors of smaller size and hence lead to earlier detection. In this paper, we present an approach to analysing breast thermograms based on image features and a hybrid multiple classifier system. The employed image features provide indications of asymmetry between left and right breast regions that are encountered when a tumor is locally recruiting blood vessels on one side, leading to a change in the captured temperature distribution. The presented multiple classifier system is based on a hybridisation of three computational intelligence techniques: neural networks or support vector machines as base classifiers, a neural fuser to combine the individual classifiers, and a fuzzy measure for assessing the diversity of the ensemble and removal of individual classifiers from the ensemble. In addition, we address the problem of class imbalance that often occurs in medical data analysis, by training base classifiers on balanced object subspaces. Our experimental evaluation, on a large dataset of about 150 breast thermograms, convincingly shows our approach not only to provide excellent classification accuracy and sensitivity but also to outperform both canonical classification approaches as well as other classifier ensembles designed for imbalanced datasets.

An empirical evaluation of rotation-based ensemble classifiers for customer churn prediction

Several studies have demonstrated the superior performance of ensemble classification algorithms, whereby multiple member classifiers are combined into one aggregated and powerful classification model, over single models. In this paper, two rotation-based ensemble classifiers are proposed as modeling techniques for customer churn prediction. In Rotation Forests, feature extraction is applied to feature subsets in order to rotate the input data for training base classifiers, while RotBoost combines Rotation Forest with AdaBoost. In an experimental validation based on data sets from four real-life customer churn prediction projects, Rotation Forest and RotBoost are compared to a set of well-known benchmark classifiers. Moreover, variations of Rotation Forest and RotBoost are compared, implementing three alternative feature extraction algorithms: principal component analysis (PCA), independent component analysis (ICA) and sparse random projections (SRP). The performance of rotation-based ensemble classifier is found to depend upon: (i) the performance criterion used to measure classification performance, and (ii) the implemented feature extraction algorithm. In terms of accuracy, RotBoost outperforms Rotation Forest, but none of the considered variations offers a clear advantage over the benchmark algorithms. However, in terms of AUC and top-decile lift, results clearly demonstrate the competitive performance of Rotation Forests compared to the benchmark algorithms. Moreover, ICA-based Rotation Forests outperform all other considered classifiers and are therefore recommended as a well-suited alternative classification technique for the prediction of customer churn that allows for improved marketing decision making.

A novel method for constructing ensemble classifiers

This paper presents a novel ensemble classifier generation method by integrating the ideas of bootstrap aggregation and Principal Component Analysis (PCA). To create each individual member of an ensemble classifier, PCA is applied to every out-of-bag sample and the computed coefficients of all principal components are stored, and then the principal components calculated on the corresponding bootstrap sample are taken as additional elements of the original feature set. A classifier is trained with the bootstrap sample and some features randomly selected from the new feature set. The final ensemble classifier is constructed by majority voting of the trained base classifiers. The results obtained by empirical experiments and statistical tests demonstrate that the proposed method performs better than or as well as several other ensemble methods on some benchmark data sets publicly available from the UCI repository. Furthermore, the diversity-accuracy patterns of the ensemble classifiers are investigated by kappa-error diagrams.