Multi-class Imbalanced Learning Research Articles

Clustering-Based Oversampling Algorithm for Multi-class Imbalance Learning

Clustering-Based Oversampling Algorithm for Multi-class Imbalance Learning

To Combat Multiclass Imbalanced Problems by Aggregating Evolutionary Hierarchical Classifiers.

Real-world datasets are often imbalanced, posing frequent challenges to canonical machine learning algorithms that assume a balanced class distribution. Moreover, the imbalance problem becomes more complicated when the dataset is multiclass. Although many approaches have been presented for imbalanced learning (IL), research on the multiclass imbalanced problem is relatively limited and deficient. To alleviate these issues, we propose a forest of evolutionary hierarchical classifiers (FEHC) method for multiclass IL (MCIL). FEHC can be seen as a classifier fusion framework with a forest structure, and it aggregates several evolutionary hierarchical multiclassifiers (EHMCs) to reduce generalization error. Specifically, a multichromosome genetic algorithm (MCGA) is designed to simultaneously select (sub)optimal features, classifiers, and hierarchical structures when generating these EHMCs. The MCGA adopts a dynamic weighting module to learn difficult classes and promote the diversity of FEHC. We also present the "stratified underbagging" (SUB) strategy to address class imbalance and the "soft tree traversal" (STT) strategy to make FEHC converge faster and better. We thoroughly evaluate the proposed algorithm using 14 multiclass imbalanced datasets with various properties. Compared with popular and state-of-the-art approaches, FEHC obtains better performance under different evaluation metrics. Codes have been made publicly available on GitHub.https://github.com/CUHKSZ-NING/FEHCClassifier.

A partition-based problem transformation algorithm for classifying imbalanced multi-label data

Multi-label learning has garnered much research interest due to its wide range of real-world applications. Many multi-label learning methods have been proposed; however, few have addressed the class imbalance problem existing in multi-label data. Even though some studies have taken this issue into account, most of them have ignored the label correlations or only considered random correlations between them. In this study, we propose a novel partition-based imbalanced multi-label learning algorithm, named Multi-label Learning based on Hierarchical Clustering (MLHC), to tackle this problem. MLHC first carries out hierarchical clustering on the original label space to divide it into several disconnected subspaces, each of which contains several labels that are strongly correlated with each other. Then, for each label subspace, we use the problem transformation strategy to convert it into a multi-class problem by binary coding. Any multi-class imbalance learning algorithm can be applied to the transformed multi-class data. Finally, the classification results will be decoded to retrieve the corresponding label subspace, and all label subspace results are combined to show the predicted label vector in the original label space. We conducted experiments not only on thirteen benchmark multi-label datasets but also carried out them on XJTU-SY which is a multi-label engineering application dataset, and the results indicated that our proposed MLHC learning algorithm outperforms several state-of-the-art class imbalance multi-label learning algorithms, demonstrating the effectiveness and necessity of discovering label correlations and transforming the original imbalanced multi-label learning problem into multiple strongly correlated multi-class imbalanced learning problems.

Novel hybrid classification model for multi-class imbalanced lithology dataset

Deep learning methods and applications assist geologists in predicting and identifying lithologies in different surveys, hence lowering operational costs and uptime. This allows accurate data analysis and completion of scientific research on data obtained at geologically different places. This study used 4 lithologies datasets with high dimensionality and multiclass imbalance problems for analysis and classification. The imbalance of data classification is one of the most important problems facing current data analysis. Data imbalance can considerably influence classification performance, especially when dealing with other difficulty variables such as the presence of overlapping class distributions. This impact is especially obvious in multi-class conditions when mutual imbalance relations across classes complicate matters even further. Moreover, the problem of high dimensionality can lead to increased computing complexity and overfitting, and thus these issues can affect classification performance. To overcome these problems, we developed a new hybrid deep learning multi-class imbalanced learning method that combines Synthetic Minority Oversampling (SMOTE) to resample the data, and Recursive Feature Elimination (RFE) to identify the most useful predictive features. Finally, we believe that our developments can help improve geology research by providing accurate classification and rapid answers about interpreting data obtained in various study areas.

Double-kernel based class-specific broad learning system for multiclass imbalance learning

Imbalance learning has gained more and more attention from researchers. Most of the efforts so far have focused on binary imbalance problems, while there are numerous unresolved multiclass imbalance problems in real-world scenarios. The diversity of data distribution and the poor performance of traditional multiclass classification algorithms present significant challenges for classifying multiclass imbalanced data. This paper proposes a double kernel-based class-specific broad learning system (DKCSBLS) for multi-class imbalance learning. Class-specific penalty coefficients are incorporated into the model to increase the focus on minority classes. Moreover, double kernel mapping mechanism is designed to extract more robust features. Extensive experiments on various real-world datasets demonstrate the superiority of our proposed approach.

One-against-all-based Hellinger distance decision tree for multiclass imbalanced learning

One-against-all-based Hellinger distance decision tree for multiclass imbalanced learning

SA-CGAN: An oversampling method based on single attribute guided conditional GAN for multi-class imbalanced learning

Imbalanced data can always be observed in our daily life and various practical tasks. A lot of well-constructed machine learning methodologies may produce ineffective performance, when conducted on this kind of data. This originates from the produced high training biases that towards the majority class instances. Among all the solutions of this problem, data generation of the minority class is always considered the most effective approach. However, in all the previous works, data are always processed sample-wisely and the distribution of each single data attribute is never noticed. So, in this paper, to estimate the mechanism of how each attribute contributes to its label, we explore the potential connection between the two items by Conditional Generative Adversarial Networks (CGAN) separately and individually. Then, the constructed new instances are purified by a designed attribute-based minimax filter and the survivors are concatenated to form the eventual generated data. In other words, different from the CGAN based data generation way, the proposed approach improves it by additionally considering all the single attribute patterns of the data that to construct new instances. In addition, we extend the binary class imbalanced learning framework to multiple class one. In the experimental part, the improved model is compared against GAN, CGAN and some other standard multiple-class oversampling algorithms on several widely used datasets. Results, in terms of four common measurements, have shown that the proposed approach can produce comparable and always superior performance when compared with the competitors.

A hybrid multi-class imbalanced learning method for predicting the quality level of diesel engines

Establishing an effective quality level prediction of diesel engines is of great significance for controlling the production quality through subsequent process improvements and reducing manufacturing costs. Data-driven methods have become the most promising methods due to their self-learning ability and independence of complex manufacturing processes. However, the dataset collected from a diesel engine assembly line is characterized by high dimensionality and multiclass imbalance. To realize the high-accuracy quality level prediction of diesel engines, this study develops a novel hybrid multi-class imbalanced learning method that combines particle swarm optimization (PSO) based wrapper method and dynamic oversampling approach with adaptive boosting (DysBoost), where PSO based wrapper method is applied to adaptively select the optimum feature subset and DySBoost is utilized to address the multi-class imbalanced classification problem. Computational experiments are conducted on three standard imbalanced datasets and one diesel engine assembly dataset, and the experimental results demonstrate the effectiveness of the proposed method in terms of accuracy and efficiency.

Classifier Selection and Ensemble Model for Multi-class Imbalance Learning in Education Grants Prediction

ABSTRACT Ensemble learning combines base classifiers to improve the performance of the models and obtains a higher classification accuracy than a single classifier. We propose a multi-classification method to predict the level of grant for each college student based on feature integration and ensemble learning. It extracted from expense, score, in/out dormitory, book loan conditions of 10885 students’ daily behavior data and constructed a 21-dimensional feature. The ensemble learning method integrated gradient boosting decision tree, random forest, AdaBoost, and Support Vector Machine classifiers for college grant classification. The proposed method is evaluated with 10885 students set and experiments show that the proposed method has an average accuracy of 0.954 5 and can be used as an effective means of assisting decision-making for college student grants.

SMOTE-Based Weighted Deep Rotation Forest for the Imbalanced Hyperspectral Data Classification

Conventional classification algorithms have shown great success in balanced hyperspectral data classification. However, the imbalanced class distribution is a fundamental problem of hyperspectral data, and it is regarded as one of the great challenges in classification tasks. To solve this problem, a non-ANN based deep learning, namely SMOTE-Based Weighted Deep Rotation Forest (SMOTE-WDRoF) is proposed in this paper. First, the neighboring pixels of instances are introduced as the spatial information and balanced datasets are created by using the SMOTE algorithm. Second, these datasets are fed into the WDRoF model that consists of the rotation forest and the multi-level cascaded random forests. Specifically, the rotation forest is used to generate rotation feature vectors, which are input into the subsequent cascade forest. Furthermore, the output probability of each level and the original data are stacked as the dataset of the next level. And the sample weights are automatically adjusted according to the dynamic weight function constructed by the classification results of each level. Compared with the traditional deep learning approaches, the proposed method consumes much less training time. The experimental results on four public hyperspectral data demonstrate that the proposed method can get better performance than support vector machine, random forest, rotation forest, SMOTE combined rotation forest, convolutional neural network, and rotation-based deep forest in multiclass imbalance learning.

A class imbalance-aware review rating prediction using hybrid sampling and ensemble learning

Imbalanced distribution of instances across the classes is a challenging issue when the underlying problem is of type classification. The reason is that classifiers will tend to favor the classes with a large number of instances i.e. instances of minority classes may be identified as instances of majority classes by the classifiers. In recent years, plenty of researches have been done to resolve the class imbalance issue in binary classification problems which resulted in many class imbalance learning techniques for binary classification problems. But, the class imbalance in multi-class classification problems did not draw much attention from the research community. Unlike binary class imbalance learning, multi-class imbalance learning techniques experience more than one majority class and more than one minority class. This paper tries to come up with a multi-class imbalanced learning technique that can overcome the effects of multi-class imbalance problem in review rating prediction tasks. The proposed model handles the multi-class imbalance issue by using the combination of hybrid sampling and ensemble learning techniques. Sampling techniques such as Random Under Sampling (RUS) and Synthetic Minority Over-sampling TEchnique(SMOTE) are jointly used in the proposed model to create balanced training sets for base learners. Also, the proposed model creates a powerful ensemble structure by amalgamating a manually created bagging ensemble and AdaBoost boosting ensembles. Experiments are done using the Amazon product dataset in order to investigate the performance of the proposed model. The experimental results show that the proposed Class Imbalance-Aware Review rating prediction(CIAR) model outperforms almost all the baseline models in-terms of G-mean, F-Score, and ROC_AUC_Score.

Boosting methods for multi-class imbalanced data classification: an experimental review

Since canonical machine learning algorithms assume that the dataset has equal number of samples in each class, binary classification became a very challenging task to discriminate the minority class samples efficiently in imbalanced datasets. For this reason, researchers have been paid attention and have proposed many methods to deal with this problem, which can be broadly categorized into data level and algorithm level. Besides, multi-class imbalanced learning is much harder than binary one and is still an open problem. Boosting algorithms are a class of ensemble learning methods in machine learning that improves the performance of separate base learners by combining them into a composite whole. This paper’s aim is to review the most significant published boosting techniques on multi-class imbalanced datasets. A thorough empirical comparison is conducted to analyze the performance of binary and multi-class boosting algorithms on various multi-class imbalanced datasets. In addition, based on the obtained results for performance evaluation metrics and a recently proposed criteria for comparing metrics, the selected metrics are compared to determine a suitable performance metric for multi-class imbalanced datasets. The experimental studies show that the CatBoost and LogitBoost algorithms are superior to other boosting algorithms on multi-class imbalanced conventional and big datasets, respectively. Furthermore, the MMCC is a better evaluation metric than the MAUC and G-mean in multi-class imbalanced data domains.

DyS-IENN: a novel multiclass imbalanced learning method for early warning of tardiness in rocket final assembly process

Establishing an effective early warning mechanism for the rocket final assembly process (RFAP) is crucial for the timely delivery of rockets and the reduction of additional production costs. To solve the unsystematic design of warning indicators and warning levels in RFAP and address the problem of low warning accuracy caused by imbalanced data distribution, this paper redesigns the warning indicators and warning levels in a systematic way, and develops a novel multiclass imbalanced learning method based on dynamic sampling algorithm (DyS) and improved ensemble neural network (IENN). The DyS algorithm dynamically determines the training set after oversampling the minority class, while the IENN can effectively suppress the oscillation in the iterative process of the DyS algorithm and improve the overall classification accuracy by removing the redundant and ineffective networks from the ensemble neural network. The experiment results indicate that the proposed method outperforms other methods in terms of accuracy and stability for early warning of tardiness in RFAP.

Data-driven rated power prediction of diesel engines using improved multi-class imbalanced learning method

Rated power is one of the most important quality indicators of diesel engines. The effective prediction of the rated power classes is of great significance in controlling the quality of diesel engines. However, the number of samples varies greatly in different classes. Creating an effective machine learning model to predict the rated power classes is difficult due to the imbalanced dataset. This paper develops an improved multi-class imbalanced learning method which integrates dynamic oversampling method with multi-class AdaBoost for producing high predictive accuracy over both the majority classes and the minority classes of the diesel engine dataset. Computational experiments have been performed on four UCI standard datasets and one diesel engine’s rated power dataset, and the experimental results indicate that the proposed algorithm is superior to the compared methods, including resampling methods and boosting-type methods.

Multiclass imbalanced learning with one-versus-one decomposition and spectral clustering

In many real-world applications, an algorithm needs to learn multiclass classification models from data with imbalanced class distributions. Multiclass imbalanced learning is currently receiving increased attention from researchers. In contrast to traditional imbalanced learning on binary datasets, multiclass imbalanced learning faces great challenges from the variety of changes in the class distributions as well as the inadequate performance of multiclass classification algorithms. In this paper, we propose a novel data preprocessing-based method to solve this problem. The proposed method combines a one-versus-one (OVO) decomposition of class pairs and a spectral clustering technique. This method first decomposes a multiclass dataset into several binary-class datasets. Then, it uses spectral clustering to divide the minority classes of binary-class subsets into subspaces and oversamples them according to the characteristics of the data. Sampling based on spectral clustering takes into account the distribution of the data and effectively avoids oversampling outliers. After the data approximately reaches the equilibrium point, multiclass classifiers can be trained from these rebalanced data. We compared the proposed method with five state-of-the-art multiclass imbalanced learning methods on seven multiclass datasets, using multiclass area under the ROC curve (MAUC), the precision of minor classes (Pmin) and the average precision of all classes (Pavg) as the performance metrics. The experimental results show that our proposed method has the best overall performance.

Imbalanced Hyperspectral Image Classification With an Adaptive Ensemble Method Based on SMOTE and Rotation Forest With Differentiated Sampling Rates

Rotation forest (RoF) is a powerful ensemble classifier and has been demonstrated the outstanding performance in hyperspectral data classification. However, the classification task suffers from the class imbalanced problem which has been considered to be one of the most important challenges. The traditional construction method of RoF biases classifying the majority classes and ignores recognizing the minority classes samples. This letter proposes a novel adaptive ensemble method based on SMOTE and RoF with differentiated sampling rates (AdaSRoF) for the multiclass imbalance problem. The proposed method adaptively generates several balanced data sets with more diversity and less noise by using SMOTE and a dynamic data sampling ratio for base classifiers. The obtained results on two publicly available hyperspectral images show that the proposed method can get more diversity and better performance than support vector machine (SVM), random forest (RF), and RoF in multiclass imbalance learning.

Dynamic Ensemble Selection and Data Preprocessing for Multi-Class Imbalance Learning

Class imbalance refers to classification problems in which many more instances are available for certain classes than for others. Such imbalanced datasets require special attention because traditional classifiers generally favor the majority class which has a large number of instances. Ensemble of classifiers has been reported to yield promising results. However, the majority of ensemble methods applied to imbalance learning are static ones. Moreover, they only deal with binary imbalanced problems. Hence, this paper presents an empirical analysis of Dynamic Selection techniques and data preprocessing methods for dealing with multi-class imbalanced problems. We considered five variations of preprocessing methods and 14 Dynamic Selection schemes. Our experiments conducted on 26 multi-class imbalanced problems show that the dynamic ensemble improves the AUC and the [Formula: see text]-mean as compared to the static ensemble. Moreover, data preprocessing plays an important role in such cases.

Dynamic Synthetic Minority Over-Sampling Technique-Based Rotation Forest for the Classification of Imbalanced Hyperspectral Data

Rotation forest (RoF) is a powerful ensemble classifier and has attracted substantial attention due to its performance in hyperspectral data classification. Multi-class imbalance learning is one of the biggest challenges in machine learning and remote sensing. The standard technique for constructing RoF ensemble tends to increase the overall accuracy; RoF has difficulty to sufficiently recognize the minority class. This paper proposes a novel dynamic SMOTE (synthetic minority oversampling technique)-based RoF algorithm for the multi-class imbalance problem. The main idea of the proposed method is to dynamically balance the class distribution before building each rotation decision tree. A resampling rate is set in each iteration (ranging from 10% in the first iteration to 100% in the last) and this ratio defines the number of minority class instances randomly resampled (with replacement) from the original dataset in each iteration. The rest of the minority class instances are generated by the SMOTE method. The reported results on three real hyperspectral datasets show that the proposed method can get better performance than random forest, RoF, and some popular data sampling methods.

Evolutionary inversion of class distribution in overlapping areas for multi-class imbalanced learning

Inductive learning from multi-class and imbalanced datasets is one of the main challenges for machine learning. Most machine learning algorithms have their predictive performance negatively affected by imbalanced data. Although several techniques have been proposed to deal with this difficulty, they are usually restricted to binary classification datasets. Thus, one of the research challenges in this area is how to deal with imbalanced multiclass classification datasets. This challenge become more difficult when classes containing fewer instances are located in overlapping regions of the data attribute space. In fact, several studies have indicated that the degree of class overlapping has a higher effect on predictive performance than the global class imbalance ratio. This paper proposes a novel evolutionary ensemble-based method for multi-class imbalanced learning called the evolutionary inversion of class distribution in overlapping areas for multi-class imbalanced learning (EVINCI). EVINCI uses a multiobjective evolutionary algorithm (MOEA) to evolve a set of samples taken from an imbalanced dataset. It selectively reduces the concentration of less representative instances of the majority classes in the overlapping areas while selecting samples that produce more accurate models. In experiments performed to evaluate its predictive accuracy, EVINCI was superior to state-of-the-art ensemble-based methods for imbalanced learning.

Generalized class-specific kernelized extreme learning machine for multiclass imbalanced learning

Class imbalanced learning is a well-known issue, which exists in real-world applications. Datasets that have skewed class distribution raise hindrance to the traditional learning algorithms. Traditional classifiers give the same importance to all the samples, which leads to the prediction biased towards the majority classes. To solve this intrinsic deficiency, numerous strategies have been proposed such as weighted extreme learning machine (WELM), weighted support vector machine (WSVM), class-specific extreme learning machine (CS-ELM) and class-specific kernelized extreme learning machine (CSKELM). This work focuses on multiclass imbalance problems, which are more difficult compared to the binary class imbalance problems. Kernelized extreme learning machine (KELM) yields better results compared to the traditional extreme learning machine (ELM), which uses random input parameters. This work presents a generalized CSKELM (GCSKELM), the extension of our recently proposed CSKELM, which addresses the multiclass imbalanced problems more effectively. The proposed GCSKELM can be applied directly to solve the multiclass imbalanced problems. GCSKELM with Gaussian kernel function avoids the non-optimal hidden node problem associated with CS-ELM and other existing variants of ELM. The proposed work also has less computational cost in contrast with kernelized WELM (KWELM) for multiclass imbalanced learning. This work employs class-specific regularization parameters, which are determined by employing class proportion. The extensive experimental analysis shows that the proposed work obtains promising generalization performance in contrast with the other state-of-the-art imbalanced learning methods.