Class Imbalance Learning Methods Research Articles

Graph-Based Class-Imbalance Learning With Label Enhancement.

Class imbalance is a common issue in the community of machine learning and data mining. The class-imbalance distribution can make most classical classification algorithms neglect the significance of the minority class and tend toward the majority class. In this article, we propose a label enhancement method to solve the class-imbalance problem in a graph manner, which estimates the numerical label and trains the inductive model simultaneously. It gives a new perspective on the class-imbalance learning based on the numerical label rather than the original logical label. We also present an iterative optimization algorithm and analyze the computation complexity and its convergence. To demonstrate the superiority of the proposed method, several single-label and multilabel datasets are applied in the experiments. The experimental results show that the proposed method achieves a promising performance and outperforms some state-of-the-art single-label and multilabel class-imbalance learning methods.

Generative-Adversarial Class-Imbalance Learning for Classifying Cyber-Attacks and Faults - A Cyber-Physical Power System

There has been an increasing interest in the use of data-driven techniques for classifying cyber-attacks and physical faults in cyber-physical systems. In real-world applications, the number of cyber-attack and faulty samples is usually far less than normal samples. This causes the skewed class distribution in data collected from cyber-physical systems. Training an accurate predictive model under skewed class conditions is not an easy task. In this work, we introduce a new generative adversarial framework for learning from skewed class distributions. This novel Adversarial Class-Imbalance Learning (ACIL) scheme has a novel loss function that is used during the adversarial training session. ACIL tries to iteratively adjust weights of an auxiliary multilayer perceptron to learn the minority class (i.e., cyber-attacks and physical faults) distributions along with the majority class (i.e., normal) distribution. Moreover, we devise an inclusive data-driven scheme for classifying cyber-attacks and faults, which includes four experiments of a baseline, nine state-of-the-art class-imbalance learning methods, two different generative-adversarial network-based approaches, and ACIL. These techniques are verified and compared through several experimental cyber-physical power scenarios. The obtained results show the effectiveness of ACIL for classifying samples of cyber-attacks and faults with skewed class distributions.

Evolutionary Dual-Ensemble Class Imbalance Learning for Human Activity Recognition

Human activity recognition is an imbalance classification problem in essence since various human actions may occur at different frequencies. Traditional ensemble class imbalance learning methods integrate resampling technique with multi-classifier models, which obtain better generalization than single ones. However, the number of base classifiers is often determined in advance, resulting in the redundant structure of a multi-classifier and larger computational cost. Moreover, various combinations of base classifiers may have similar recognition accuracy, forming a multimodel optimization problem. To address the issue, a dual-ensemble class imbalance learning method is presented, in which two ensemble learning models are nested each other. For each sub-dataset, three heterogeneous sub-classifiers compose of internal ensemble learning model, and the one with the highest recognition accuracy is selected as base classifier. In external ensemble learning model, an optimal combination that contains the smallest number of base classifiers and identifies human actions most exactly is found by multimodal evolutionary algorithms. Based on seven imbalanced activity datasets derived from localization data for person activity, statistical experimental results show that the proposed evolutionary dual-ensemble class imbalance learning method provides the simplest ensemble structure with the best accuracy for imbalanced data that outperforms other five widely-used ensemble classification methods on all metrics.

A transfer weighted extreme learning machine for imbalanced classification

Previous class imbalance learning methods are mostly grounded on the assumption that all training data have been labeled, however, is impractical in many real-world applications. The limited amount of labeled instances may produce a classifier with poor generalization. To address the issue, a transfer weighted extreme learning machine (TWELM) classifier is proposed, with the purpose of extracting knowledge from other domains to improve the classification performance of a classifier in a limited labeled target domain. To be specific, a well-tuned weighted extreme learning machine classifier is first learned from source data that has been completely labeled. Subsequently, another extreme learning machine classifier is obtained from the limited labeled target domain data to preserve the target domain structural knowledge and the decision boundary information. Finally, the target classifier is optimized by minimizing the outputs of the two classifiers on unlabeled target data. Experimental results on real-world data sets show that TWELM outperforms existing algorithms on classification accuracy and computation cost.

A Method for Class-Imbalance Learning in Android Malware Detection

More and more Android application developers are adopting many different methods against reverse engineering, such as adding a shell, resulting in certain features that cannot be obtained through decompilation, which causes a serious sample imbalance in Android malware detection based on machine learning. Hence, the researchers have focused on how to solve class-imbalance to improve the performance of Android malware detection. However, the disadvantages of the existing class-imbalance learning are mainly the loss of valuable samples and the computational cost. In this paper, we propose a method of Class-Imbalance Learning (CIL), which first selects representative features, uses the clustering K-Means algorithm and under-sampling to retain the important samples of the majority class while reducing the number of samples of the majority class. After that, we use the Synthetic Minority Over-Sampling Technique (SMOTE) algorithm to generate minority class samples for data balance, and finally use the Random Forest (RF) algorithm to build a malware detection model. The result of experiments indicates that CIL effectively improves the performance of Android malware detection based on machine learning, especially for class imbalance. Compared with existing class-imbalance learning methods, CIL is also effective for the Machine Learning Repository from the University of California, Irvine (UCI) and has better performance in some data sets.

Transfer synthetic over-sampling for class-imbalance learning with limited minority class data

The problem of limited minority class data is encountered in many class imbalanced applications, but has received little attention. Synthetic over-sampling, as popular class-imbalance learning methods, could introduce much noise when minority class has limited data since the synthetic samples are not i.i.d. samples of minority class. Most sophisticated synthetic sampling methods tackle this problem by denoising or generating samples more consistent with ground-truth data distribution. But their assumptions about true noise or ground-truth data distribution may not hold. To adapt synthetic sampling to the problem of limited minority class data, the proposed Traso framework treats synthetic minority class samples as an additional data source, and exploits transfer learning to transfer knowledge from them to minority class. As an implementation, TrasoBoost method firstly generates synthetic samples to balance class sizes. Then in each boosting iteration, the weights of synthetic samples and original data decrease and increase respectively when being misclassified, and remain unchanged otherwise. The misclassified synthetic samples are potential noise, and thus have smaller influence in the following iterations. Besides, the weights of minority class instances have greater change than those of majority class instances to be more influential. And only original data are used to estimate error rate to be immune from noise. Finally, since the synthetic samples are highly related to minority class, all of the weak learners are aggregated for prediction. Experimental results show TrasoBoost outperforms many popular class-imbalance learning methods.

Imputation-based Ensemble Techniques for Class Imbalance Learning

Correct classification of rare samples is a vital data mining task and of paramount importance in many research domains. This article mainly focuses on the development of the novel class-imbalance learning techniques, which make use of oversampling methods integrated with bagging and boosting ensembles. Two novel oversampling strategies based on the single and the multiple imputation methods are proposed. The proposed techniques aim to create useful synthetic minority class samples, similar to the original minority class samples, by estimation of missing values that are already induced in the minority class samples. The re-balanced datasets are then used to train base-learners of the ensemble algorithms. In addition, the proposed techniques are compared with the commonly used class imbalance learning methods in terms of three performance metrics including AUC, F-measure, and G-mean over several synthetic binary class datasets. The empirical results show that the proposed multiple imputation-based oversampling combined with bagging significantly outperforms other competitors.

Omni-Ensemble Learning (OEL): Utilizing Over-Bagging, Static and Dynamic Ensemble Selection Approaches for Software Defect Prediction

Machine learning methods in software engineering are becoming increasingly important as they can improve quality and testing efficiency by constructing models to predict defects in software modules. The existing datasets for software defect prediction suffer from an imbalance of class distribution which makes the learning problem in such a task harder. In this paper, we propose a novel approach by integrating Over-Bagging, static and dynamic ensemble selection strategies. The proposed method utilizes most of ensemble learning approaches called Omni-Ensemble Learning (OEL). This approach exploits a new Over-Bagging method for class imbalance learning in which the effect of three different methods of assigning weight to training samples is investigated. The proposed method first specifies the best classifiers along with their combiner for all test samples through Genetic Algorithm as the static ensemble selection approach. Then, a subset of the selected classifiers is chosen for each test sample as the dynamic ensemble selection. Our experiments confirm that the proposed OEL can provide better overall performance (in terms of G-mean, balance, and AUC measures) comparing with other six related works and six multiple classifier systems over seven NASA datasets. We generally recommend OEL to improve the performance of software defect prediction and the similar problem based on these experimental results.

An Improved SDA Based Defect Prediction Framework for Both Within-Project and Cross-Project Class-Imbalance Problems

Background. Solving the class-imbalance problem of within-project software defect prediction (SDP) is an important research topic. Although some class-imbalance learning methods have been presented, there exists room for improvement. For cross-project SDP, we found that the class-imbalanced source usually leads to misclassification of defective instances. However, only one work has paid attention to this cross-project class-imbalance problem. Objective. We aim to provide effective solutions for both within-project and cross-project class-imbalance problems. Method. Subclass discriminant analysis (SDA), an effective feature learning method, is introduced to solve the problems. It can learn features with more powerful classification ability from original metrics. For within-project prediction, we improve SDA for achieving balanced subclasses and propose the improved SDA (ISDA) approach. For cross-project prediction, we employ the semi-supervised transfer component analysis (SSTCA) method to make the distributions of source and target data consistent, and propose the SSTCA+ISDA prediction approach. Results . Extensive experiments on four widely used datasets indicate that: 1) ISDA-based solution performs better than other state-of-the-art methods for within-project class-imbalance problem; 2) SSTCA+ISDA proposed for cross-project class-imbalance problem significantly outperforms related methods. Conclusion . Within-project and cross-project class-imbalance problems greatly affect prediction performance, and we provide a unified and effective prediction framework for both problems.

A Multi-Objective Ensemble Method for Class Imbalance Learning

This article describes how class imbalance learning has attracted great attention in recent years as many real world domain applications suffer from this problem. Imbalanced class distribution occurs when the number of training examples for one class far surpasses the training examples of the other class often the one that is of more interest. This problem may produce an important deterioration of the classifier performance, in particular with patterns belonging to the less represented classes. Toward this end, the authors developed a hybrid model to address the class imbalance learning with focus on binary class problems. This model combines benefits of the ensemble classifiers with a multi objective feature selection technique to achieve higher classification performance. The authors' model also proposes non-dominated sets of features. Then they evaluate the performance of the proposed model by comparing its results with notable algorithms for solving imbalanced data problem. Finally, the authors utilize the proposed model in medical domain of predicting life expectancy in post-operative of thoracic surgery patients.

Ensemble based adaptive over-sampling method for imbalanced data learning in computer aided detection of microaneurysm

Diabetic retinopathy (DR) is a progressive disease, and its detection at an early stage is crucial for saving a patient's vision. An automated screening system for DR can help in reduce the chances of complete blindness due to DR along with lowering the work load on ophthalmologists. Among the earliest signs of DR are microaneurysms (MAs). However, current schemes for MA detection appear to report many false positives because detection algorithms have high sensitivity. Inevitably some non-MAs structures are labeled as MAs in the initial MAs identification step. This is a typical “class imbalance problem”. Class imbalanced data has detrimental effects on the performance of conventional classifiers. In this work, we propose an ensemble based adaptive over-sampling algorithm for overcoming the class imbalance problem in the false positive reduction, and we use Boosting, Bagging, Random subspace as the ensemble framework to improve microaneurysm detection. The ensemble based over-sampling methods we proposed combine the strength of adaptive over-sampling and ensemble. The objective of the amalgamation of ensemble and adaptive over-sampling is to reduce the induction biases introduced from imbalanced data and to enhance the generalization classification performance of extreme learning machines (ELM). Experimental results show that our ASOBoost method has higher area under the ROC curve (AUC) and G-mean values than many existing class imbalance learning methods.

ODOC-ELM: Optimal decision outputs compensation-based extreme learning machine for classifying imbalanced data

Extreme learning machine (ELM) has been one widely used learning paradigm to train single hidden layer feedforward network (SLFN). However, like many other classification algorithms, ELM may learn undesirable class boundaries from data with unbalanced classes. This paper first tries to analyze the reason of the damage caused by class imbalance for ELM, and then discusses the influence of several data distribution factors for the damage. Next, we present an optimal decision outputs compensation strategy to deal with the class imbalance problem in the context of ELM. Specifically, the outputs of the minority classes in ELM are properly compensated. For a binary-class problem, the compensation can be regarded as a single variable optimization problem, thus the golden section search algorithm is adopted to find the optimal compensation value. For a multi-class problem, the particle swarm optimization (PSO) algorithm is used to solve the multivariate optimization problem and to provide the optimal combination of compensations. Experimental results on lots of imbalanced data sets demonstrate the superiority of the proposed algorithm. Statistical results indicate that the proposed approach not only outperforms the original ELM, but also yields better or at least competitive results compared with several widely used and state-of-the-art class imbalance learning methods.

A Review of Class Imbalance Learning Methods in Bioinformatics

In recent years, research on bioinformatics has increasingly focused on the problem of class imbalance. A classification task is called class imbalance when the number of instances belonging to one class or several classes exceeds that of the other classes. Class imbalance often underestimates the performance of minority classes. This article provides a review of the most widely used class imbalance learning methods and their applications in various bioinformatic problems, including disease diagnosis based on gene expression data and protein mass spectrometry data, translation initiation site recognition based on DNA sequences, protein function classification using amino acid sequences, activities prediction of drug molecules, recognition of precursor microRNA (pre-miRNAs), etc. This article also summarizes the current challenges and future possible trends of class imbalance learning methods in Bioinformatics. Keywords: Activities prediction of drug molecules, bioinformatics, class imbalance, gene expression, protein function classification, protein mass spectrometry, recognition of precursor microRNA, translation initiation site recognition.

Class imbalance learning via a fuzzy total margin based support vector machine

The classification of imbalanced data is a major challenge for machine learning. In this paper, we presented a fuzzy total margin based support vector machine (FTM-SVM) method to handle the class imbalance learning (CIL) problem in the presence of outliers and noise. The proposed method incorporates total margin algorithm, different cost functions and the proper approach of fuzzification of the penalty into FTM-SVM and formulates them in nonlinear case. We considered an excellent type of fuzzy membership functions to assign fuzzy membership values and got six FTM-SVM settings. We evaluated the proposed FTM-SVM method on two artificial data sets and 16 real-world imbalanced data sets. Experimental results show that the proposed FTM-SVM method has higher G_Mean and F_Measure values than some existing CIL methods. Based on the overall results, we can conclude that the proposed FTM-SVM method is effective for CIL problem, especially in the presence of outliers and noise in data sets.

IRUSRT: A Novel Imbalanced Learning Technique by Combining Inverse Random Under Sampling and Random Tree

In this article, a novel technique IRUSRT (inverse random under sampling and random tree) by combining inverse random under sampling and random tree is proposed to implement imbalanced learning. The main idea is to severely under sample the majority class thus creating multiple distinct training sets. With each training set, a random tree is trained to separate the minority class from the majority class. By combining these random trees through fusion, a composite classifier is constructed. The experimental analysis on 23 real-world datasets assessed over area under the ROC curve (AUC), F-measure, and G-mean indicates that IRUSRT performs significantly better when compared with many existing class imbalance learning methods.

A Kernel Clustering-Based Possibilistic Fuzzy Extreme Learning Machine for Class Imbalance Learning

Compared with traditional computational intelligence techniques such as the support vector machine, extreme learning machine (ELM) provides better generalization performance at a much faster learning speed without tuning model parameters. Unfortunately, the training process of ELM is still sensitive to the outliers or noises in the training set. On the other hand, when it comes to imbalanced datasets, ELM produces suboptimal classification models. In this paper, a kernel possibilistic fuzzy c-means clustering-based ELM algorithm for class imbalance learning (CIL) is developed to handle the class imbalance problem in the presence of outliers and noises. A set of experiments are conducted on several artificial and real-world imbalanced datasets for testing the generalization performance of the proposed algorithm. Additionally, we compare its performance with some typical CIL methods. The results indicate that the proposed method is a very effective method for CIL, especially in the presence of outliers and noises in datasets.

Estimating harmfulness of class imbalance by scatter matrix based class separability measure

In many real world applications, class imbalance problems occur frequently, causing great underestimation for the classification performance of minority classes. In recent years, much effective solutions have been proposed to address this problem. However, the recent research found that not all skewed classification tasks are harmful and carrying out class imbalance learning methods on those unharmful ones can hardly improve and even degenerate classification performance, meanwhile increase training time to a large extent. Therefore, it is essential to design an efficient criterion to pre-estimate the harmfulness of class imbalance when we encounter skewed classification tasks. In this study, we explore the reason of harmfulness produced by class imbalance and propose a simple and ingenious strategy using scatter matrix based class separability measure to estimate the harmfulness of class imbalance with merely using training samples. The experimental results indicate that the proposed strategy can quantificationally estimate the damage for imbalanced classification tasks and provide priori information to guide us to select appropriate classification methods. Moreover, the computing complexity of the proposed strategy is quite low, thus it is practical in real-world applications.

Hierarchical Fuzzy Support Vector Machine (SVM) for Rail Data Classification

Abstract This study aims at designing a modelling architecture to deal with the imbalanced data relating to the production of rails. The modelling techniques are based on Support Vector Machines (SVMs) which are sensitive to class imbalance. An internal (Biased Fuzzy SVM) and external (data under-sampling) class imbalance learning methods were applied to the data. The performance of the techniques when implemented on the latter was better, while in both cases the inclusion of a fuzzy membership improved the performance of the SVM. Fuzzy C-Means (FCM) Clustering was analysed for reducing the number of support vectors of the Fuzzy SVM model, concluding it is effective in reducing model complexity without any significant performance deterioration.

Selecting the Optimal Combination Model of FSSVM for the Imbalance Datasets

Imbalanced data learning is one of the most active and important fields in machine learning research. The existing class imbalance learning methods can make Support Vector Machines (SVMs) less sensitive to class imbalance; they still suffer from the disturbance of outliers and noise present in the datasets. A kind of Fuzzy Smooth Support Vector Machines (FSSVMs) are proposed based on the Smooth Support Vector Machine (SSVM) of O. L. Mangasarian. SSVM can be computed by the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm or the Newton-Armijo algorithm easily. Two kinds of fuzzy memberships and three smooth functions can be chosen in the algorithms. The fuzzy memberships consider the contribution rate of each sample to the optimal separating hyperplane. The polynomial smooth functions can make the optimization problem more accurate at the inflection point. Those changes play the active effects on trials. The results of the experiments show that the FSSVMs can gain the better accuracy and the shorter time than the SSVMs and some of the other methods.

Using Class Imbalance Learning for Software Defect Prediction

To facilitate software testing, and save testing costs, a wide range of machine learning methods have been studied to predict defects in software modules. Unfortunately, the imbalanced nature of this type of data increases the learning difficulty of such a task. Class imbalance learning specializes in tackling classification problems with imbalanced distributions, which could be helpful for defect prediction, but has not been investigated in depth so far. In this paper, we study the issue of if and how class imbalance learning methods can benefit software defect prediction with the aim of finding better solutions. We investigate different types of class imbalance learning methods, including resampling techniques, threshold moving, and ensemble algorithms. Among those methods we studied, AdaBoost.NC shows the best overall performance in terms of the measures including balance, G-mean, and Area Under the Curve (AUC). To further improve the performance of the algorithm, and facilitate its use in software defect prediction, we propose a dynamic version of AdaBoost.NC, which adjusts its parameter automatically during training. Without the need to pre-define any parameters, it is shown to be more effective and efficient than the original AdaBoost.NC.