Imbalanced Binary Classification Research Articles

Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review.

Clinical prediction models for spine surgery applications are on the rise, with an increasing reliance on machine learning (ML) and deep learning (DL). Many of the predicted outcomes are uncommon; therefore, to ensure the models' effectiveness in clinical practice it is crucial to properly evaluate them. This systematic review aims to identify and evaluate current research-based ML and DL models applied for spine surgery, specifically those predicting binary outcomes with a focus on their evaluation metrics. Overall, 60 papers were included, and the findings were reported according to the PRISMA guidelines. A total of 13 papers focused on lengths of stay (LOS), 12 on readmissions, 12 on non-home discharge, 6 on mortality, and 5 on reoperations. The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%. A total of 59 papers reported the model's area under the receiver operating characteristic (AUROC), 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed positive predictive value (PPV), 24 included the negative predictive value (NPV), 25 indicated the Brier score with 10 providing a null model Brier, and 8 detailed the F1 score. Additionally, data visualization varied among the included papers. This review discusses the use of appropriate evaluation schemes in ML and identifies several common errors and potential bias sources in the literature. Embracing these recommendations as the field advances may facilitate the integration of reliable and effective ML models in clinical settings.

Cost-Sensitive Online Adaptive Kernel Learning for Large-Scale Imbalanced Classification

Imbalanced classification is a challenging task in the fields of machine learning, data mining and pattern recognition. Cost-sensitive online algorithms are very important methods for large-scale imbalanced classification problems. At present, most of the cost-sensitive classification algorithms focus on the accuracy of the minority class and ignore the accuracy of the majority class. In order to better balance the accuracy between the minority class and the majority class, in this paper, a misclassification cost is presented to ensure that the cost-sensitive online algorithm can better deal with the imbalanced classification problems without signifificantly reducing the accuracy of the majority class. Based on the proposed misclassification cost, a novel cost-sensitive online adaptive kernel learning algorithm is proposed to boost the adaptability of kernel function when data arrives one by one. According to the essential characteristics of the imbalanced binary classification, a cost-sensitive online adaptive kernel learning algorithm is given to handle the large-scale imbalanced multi-class classification problems. Theoretical analysis of the proposed algorithms are provided. Extensive experiments demonstrate that compared with the state-of-the-art imbalanced classification algorithms, the proposed algorithms can significantly improve the classification performances on most of the large-scale imbalanced data sets.

DynaQ: online learning from imbalanced multi-class streams through dynamic sampling

AbstractOnline supervised learning from fast-evolving data streams, particularly in domains such as health, the environment, and manufacturing, is a crucial research area. However, these domains often experience class imbalance, which can skew class distributions. It is essential for online learning algorithms to analyze large datasets in real-time while accurately modeling rare or infrequent classes that may appear in bursts. While methods have been proposed to handle binary class imbalance, there is a lack of attention to multi-class imbalanced settings with varying degrees of imbalance in evolving streams. In this paper, we present the Dynamic Queues (DynaQ) algorithm for online learning in multi-class imbalanced settings to fill this knowledge gap. Our approach utilizes a batch-based resampling method that creates an instance queue for each class to balance the number of instances. We maintain a queue threshold and remove older samples during training. Additionally, we dynamically oversample minority classes based on one of four rate parameters: recall, F1-score, $$\kappa _m$$ κ m , and Euclidean distance. Our learning algorithm consists of an ensemble that uses sliding windows and a soft voting schema while incorporating a drift detection mechanism. Our experimental results demonstrate the superiority of the DynaQ approach over state-of-the-art methods.

Advanced Genetic Programming vs. State-of-the-Art AutoML in Imbalanced Binary Classification

The objective of this article is to provide a comparative analysis of two novel genetic programming (GP) techniques, differentiable Cartesian genetic programming for artificial neural networks (DCGPANN) and geometric semantic genetic programming (GSGP), with state-of-the-art automated machine learning (AutoML) tools, namely Auto-Keras, Auto-PyTorch and Auto-Sklearn. While all these techniques are compared to several baseline algorithms upon their introduction, research still lacks direct comparisons between them, especially of the GP approaches with state-of-the-art AutoML. This study intends to fill this gap in order to analyze the true potential of GP for AutoML. The performances of the different tools are assessed by applying them to 20 benchmark datasets of the imbalanced binary classification field, thus an area that is a frequent and challenging problem. The tools are compared across the four categories average performance, maximum performance, standard deviation within performance, and generalization ability, whereby the metrics F1-score, G-mean, and AUC are used for evaluation. The analysis finds that the GP techniques, while unable to completely outperform state-of-the-art AutoML, are indeed already a very competitive alternative. Therefore, these advanced GP tools prove that they are able to provide a new and promising approach for practitioners developing machine learning (ML) models. Doi: 10.28991/ESJ-2023-07-04-021 Full Text: PDF

Multimodal Classification of Anxiety Based on Physiological Signals

Multiple studies show an association between anxiety disorders and dysregulation in the Autonomic Nervous System (ANS). Thus, understanding how informative the physiological signals are would contribute to effectively detecting anxiety. This study targets the classification of anxiety as an imbalanced binary classification problem using physiological signals collected from a sample of healthy subjects under a neutral condition. For this purpose, the Electrocardiogram (ECG), Electrodermal Activity (EDA), and Electromyogram (EMG) signals from the WESAD publicly available dataset were used. The neutral condition was collected for around 20 min on 15 participants, and anxiety scores were assessed through the shortened 6-item STAI. To achieve the described goal, the subsequent steps were followed: signal pre-processing; feature extraction, analysis, and selection; and classification of anxiety. The findings of this study allowed us to classify anxiety with discriminatory class features based on physiological signals. Moreover, feature selection revealed that ECG features play a relevant role in anxiety classification. Supervised feature selection and data balancing techniques, especially Borderline SMOTE 2, increased the performance of most classifiers. In particular, the combination of feature selection and Borderline SMOTE 2 achieved the best ROC-AUC with the Random Forest classifier.

Switching synthesizing-incorporated and cluster-based synthetic oversampling for imbalanced binary classification

Oversampling is a popular yet useful method to fulfill the binary classification of imbalanced data, however many existing results of oversampling are very likely to generate redundant/unsafe/noise samples due primarily to the inadequate consideration of the data distribution. To address this issue, we propose a novel oversampling approach, namely Switching Synthesizing-Incorporated and Cluster-Based Synthetic Oversampling (SSI-CBSO). The core idea of SSI-CBSO is four-fold: (1) noise samples are removed by using K nearest neighbor strategy and Fuzzy C-Means clustering is adopted for the filtered data in the minority class; (2) the number of samples that need to be synthesized is adaptively assigned to each cluster concerning the inter-class distance and the intra-cluster similarity; (3) to better reflect the data distribution, a new method in terms of the concept of the hypersphere is put forward to measure the cluster density in a high dimensional; and (4) a new principle based on the Mahalanobis distance is provided for a better selection of the target sample. Then, a switching synthesizing strategy is established to guarantee the safety of the synthesized samples. Finally, experiments on 13 binary imbalanced data sets by using five evaluation metrics with four classifiers verify that our proposed SSI-CBSO approach can obtain desirable results.

Effective Class-Imbalance Learning Based on SMOTE and Convolutional Neural Networks

Imbalanced Data (ID) is a problem that deters Machine Learning (ML) models from achieving satisfactory results. ID is the occurrence of a situation where the quantity of the samples belonging to one class outnumbers that of the other by a wide margin, making such models’ learning process biased towards the majority class. In recent years, to address this issue, several solutions have been put forward, which opt for either synthetically generating new data for the minority class or reducing the number of majority classes to balance the data. Hence, in this paper, we investigate the effectiveness of methods based on Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) mixed with a variety of well-known imbalanced data solutions meaning oversampling and undersampling. Then, we propose a CNN-based model in combination with SMOTE to effectively handle imbalanced data. To evaluate our methods, we have used KEEL, breast cancer, and Z-Alizadeh Sani datasets. In order to achieve reliable results, we conducted our experiments 100 times with randomly shuffled data distributions. The classification results demonstrate that the mixed Synthetic Minority Oversampling Technique (SMOTE)-Normalization-CNN outperforms different methodologies achieving 99.08% accuracy on the 24 imbalanced datasets. Therefore, the proposed mixed model can be applied to imbalanced binary classification problems on other real datasets.

Global reliable data generation for imbalanced binary classification with latent codes reconstruction and feature repulsion

Global reliable data generation for imbalanced binary classification with latent codes reconstruction and feature repulsion

An imbalanced binary classification method via space mapping using normalizing flows with class discrepancy constraints

For imbalanced binary classification, most methods classify data in the original space by balancing data or modifying classification algorithms. However, for datasets with severe overlaps and complex class boundaries, the classification performance is limited by the ability of the classifier to fit the decision boundary. A practical alternative is to map the data to a latent space with lower classification difficulty. In this paper, we propose an imbalanced binary classification method via space mapping using normalizing flows with class discrepancy constraints. The flow-based model is employed to map the original data to a latent space, ensuring the accuracy of the mapping and the authenticity of the mapped distribution. To solve the problem of overlaps and within-class imbalance, class discrepancy constraints, including global and local constraints, are proposed to modify the flow-based model. The former maps different subclusters to the same cluster of the latent space, and the latter increases the separability of different classes. Classification thus can be carried out in the latent space with simpler distribution and high separability to achieve better classification performance. Experimental results on 35 KEEL and UCI public datasets indicate that the proposed method outperforms other state-of-the-art methods on F1-measure and G-mean.

An improved GEV boosting method for imbalanced data classification with application to short-term rainfall prediction

This paper considers the imbalanced binary classification problem by focusing on the application of the short-term rainfall forecasting in arid and semi-arid regions. Specifically, we present a novel boosting-type method by utilizing the generalized extreme value (GEV) distribution as the link function and applying a gradient tree boosting algorithm to capture complex interactions among covariates. The proposed method has several appealing advantages such as, it can identify rare rainfall events as well as quantifying the uncertainties; it is data-driven that without any assumption on the relationship between the covariates and the rainfall event; the fitted model is highly interpretable, making it a useful tool for studying the rainfall mechanisms in arid and semi-arid regions. Experiments on two real-world datasets show that our approach outperforms its competing methods.

Class-specific extreme learning machine based on overall distribution for addressing binary imbalance problem

Class imbalance problem occurs when the training dataset contains significantly fewer samples of one class in contrast to another class. Conventional extreme learning machine (ELM) gives the same importance to all the samples leading to the results, which favor the majority class. To solve this intrinsic deficiency, modifications of ELM have been developed such as weighted ELM (WELM) and WELM based on the overall distribution (ODW-ELM). Recently class-specific ELM (CS-ELM) was designed for class imbalance learning. It has been shown in this work that the derivation of the output weights, \(\varvec{\beta }\), is more efficient compared to class-specific cost regulation ELM (CCRELM) for handling the class imbalance problem. Motivated by CCRELM, X. Luo et al. have proposed the classifier ODW-ELM, which is also not efficient for imbalance learning. In this work, a novel class-specific ELM based on overall distribution (OD-CSELM) and the kernelized version of OD-CSELM (OD-CSKELM) is proposed to address the binary class imbalance problem more effectively. OD-CSELM and OD-CSKELM are motivated by CS-ELM. In addition, the computational complexity of OD-CSELM and OD-CSKELM is significantly lower than WELM and kernelized WELM, respectively. The proposed work is evaluated by using the benchmark real-world imbalanced datasets. The experimental results demonstrate that the proposed work gives good generalization performance in contrast to the rest of the classifiers for class imbalance learning.

Imbalanced binary classification under distribution uncertainty

Imbalanced binary classification plays an important role in many applications. Some popular classifiers, such as logistic regression (LR), usually underestimate the probability of the minority class. Therefore, in this paper, we introduce two novel methods under distribution uncertainty, the idea of which is to modify the predicted probability with an additional uncertainty estimation. We develop the mean-uncertain method and the volatility-uncertain method, respectively, by assuming that the disturbance term follows the maximal and the G-normal distributions, which are the most important distributions within a sublinear expectation framework. Experiments on the simulated dataset and 10real-life datasets are conducted to compare the newly proposed approaches to several existing ones, including two resampling methods and two regression-based methods. The results of experiments show that our methods outperform most of the others in common evaluation metrics, especially the accuracy of the minority class.

Affinity based fuzzy kernel ridge regression classifier for binary class imbalance learning

The class imbalance learning (CIL) problem indicates when one class have very low proportions of samples (minority class) compared to the other class (majority class). Even though kernel ridge regression (KRR) shows high generalization ability at a considerably quicker learning speed than conventional machine learning algorithms, it fails to achieve an excellent result for CIL problems. To address this inherent limitation of KRR, a novel affinity-based fuzzy KRR (AFKRR) is proposed for dealing with the binary CIL problem. In AFKRR, an affinity-based fuzzy membership value is linked to each training sample. The affinity of the majority class datapoint is measured using the support vector data description (SVDD) trained by the majority class datapoints. The classification ability of the proposed AFKRR is calculated using the area under the receiver optimal characteristics curve (AUC), F-measure and Geometric mean. AFKRR’s performance is compared with the support vector machine (SVM), affinity and class probability-based fuzzy SVM (ACFSVM), entropy-based fuzzy SVM (EFSVM), improved density weighted least squares SVM (IDLSSVM-CIL), KRR and intuitionistic fuzzy KRR (IFKRR) models on a few real-world as well as a few artificial imbalanced datasets. Experimental outcomes reveal the usability and efficacy of the proposed AFKRR model.

An imbalanced binary classification method based on contrastive learning using multi-label confidence comparisons within sample-neighbors pair

For imbalanced classification, data-level methods can achieve inter-class balance, but the samples generated do not contain new information and cannot avoid the problem of introducing noise. Algorithm-level methods may lead to overfitting of the model, and its classification effect is more dependent on the specific dataset and classification task, which means they lack universality. In addition, how to deeply mine the differences in the distribution of data overlap areas, and how to effectively mine the differences between categories when the absolute number of minority samples is small, are also important challenges in imbalanced classification. This paper proposes an imbalanced binary classification method using multi-label confidence comparisons based on contrastive learning. Different from the previous idea of directly learning its distribution characteristics from minority samples, combined with the idea of contrastive learning, the classification task is redefined as the multi-label matching task by mining the deep features that can represent the commonality and difference between the neighboring samples. Multiple differentiated contrastive sample groups are obtained through random sampling in its neighbor sample pool for each sample. This sample is combined with its contrastive sample groups to form multiple sample-neighbor pairs as training samples in the multi-label matching task. The original dataset is multiplied without introducing noise, laying a foundation for the effective mining of class differences when the absolute number of minority class samples is small. Based on the corresponding reconstruction error generated by Variational AutoEncoder (VAE), for sample-neighbor pairs, a multi-label matching loss between target samples and contrastive sample groups that integrates the idea of contrastive learning is designed. a robust classifier is obtained through simultaneous iterative learning of reconstruction error and multi-label matching loss, which can better mine the distribution differences of overlapping regions. In the testing phase, multiple different contrastive sample groups and the corresponding prediction results of the samples to be classified are obtained, which categories can be judged by integrating the predictions of each group for reverse reasoning. Experimental results on 38 public datasets show that the method outperforms typical imbalanced classification methods in both F1-measure and G-mean.

Predicting the likelihood of airspace user rerouting to mitigate air traffic flow management delay

At present, the most common air traffic flow management measure used by the European Network Manager to solve en-route demand and capacity balance issues is to impose air traffic flow management regulations in congested airspace, which delay flights on the ground to smooth demand. The delay imposed on a regulated flight, however, has a negative operational and economic impact on the corresponding airspace user. When the impact is severe, the airspace user may reroute the flight to tactically circumvent the regulated airspace, thereby avoiding the regulation and reducing (or even eliminating) the delay. This legitimate strategy is unquestionably effective in mitigating the delay and the associated cost, but it may also increases traffic demand uncertainty due to last-minute changes as well as the environmental impact due to sub-optimal trajectories. This paper presents a gradient-boosted decision trees model that, trained on historical data, could assist in predicting the likelihood of a regulated flight rerouting to mitigate air traffic flow management delay. Specific metrics for highly imbalanced binary classification problems reveal that, while not perfect, the model outperforms a rudimentary decision tree and a dummy classifier. Furthermore, Shapley values computed with the model uncover the most important triggers for tactical rerouting from a data-driven perspective, namely the current delay, the aircraft operator, the airspace sector where the most penalising regulation is applied and the remaining time until departure. This interpretable model could be used to assist flow managers in their decision-making, as well as to accurately mimic the behaviour of flight dispatchers when facing delays in simulators.

Equity‐weighted bootstrapping: Examples and analysis

When faced with severely imbalanced binary classification problems, we often train models on bootstrapped data in which the number of instances of each class occur in a more favorable ratio, often equal to one. We view algorithmic inequity through the lens of imbalanced classification: In order to balance the performance of a classifier across groups, we can bootstrap to achieve training sets that are balanced with respect to both labels and group identity. For an example problem with severe class imbalance—prediction of suicide death from administrative patient records—we illustrate how an equity‐directed bootstrap can bring test set sensitivities and specificities much closer to satisfying the equal odds criterion. In the context of naïve Bayes and logistic regression, we analyse the equity‐weighted bootstrap, demonstrating that it works by bringing odds ratios close to one, and linking it to methods involving intercept adjustment, thresholding, and weighting.

Prediction of Bacterial sRNAs Using Sequence-Derived Features and Machine Learning.

Small ribonucleic acid (sRNA) sequences are 50–500 nucleotide long, noncoding RNA (ncRNA) sequences that play an important role in regulating transcription and translation within a bacterial cell. As such, identifying sRNA sequences within an organism’s genome is essential to understand the impact of the RNA molecules on cellular processes. Recently, numerous machine learning models have been applied to predict sRNAs within bacterial genomes. In this study, we considered the sRNA prediction as an imbalanced binary classification problem to distinguish minor positive sRNAs from major negative ones within imbalanced data and then performed a comparative study with six learning algorithms and seven assessment metrics. First, we collected numerical feature groups extracted from known sRNAs previously identified in Salmonella typhimurium LT2 (SLT2) and Escherichia coli K12 (E. coli K12) genomes. Second, as a preliminary study, we characterized the sRNA-size distribution with the conformity test for Benford’s law. Third, we applied six traditional classification algorithms to sRNA features and assessed classification performance with seven metrics, varying positive-to-negative instance ratios, and utilizing stratified 10-fold cross-validation. We revisited important individual features and feature groups and found that classification with combined features perform better than with either an individual feature or a single feature group in terms of Area Under Precision-Recall curve (AUPR). We reconfirmed that AUPR properly measures classification performance on imbalanced data with varying imbalance ratios, which is consistent with previous studies on classification metrics for imbalanced data. Overall, eXtreme Gradient Boosting (XGBoost), even without exploiting optimal hyperparameter values, performed better than the other five algorithms with specific optimal parameter settings. As a future work, we plan to extend XGBoost further to a large amount of published sRNAs in bacterial genomes and compare its classification performance with recent machine learning models’ performance.

Density Weighted Twin Support Vector Machines for Binary Class Imbalance Learning

Density Weighted Twin Support Vector Machines for Binary Class Imbalance Learning

A dual encoder DAE neural network for imbalanced binary classification based on NSGA-III and GAN

A dual encoder DAE neural network for imbalanced binary classification based on NSGA-III and GAN

Business Anomaly Detection Method of Power Dispatching Automation System Based on Clustering Under-Sampling in the Boundary Region

Timely detecting business anomaly in the power dispatching automation system is significant for the steady operation of the power grid. Though the imbalanced binary classification method in machine learning is an effective way to achieve the business anomaly detection of the system, the overlap of boundary samples is an urgent issue affecting the classification effect. An under-sampling method by removing the clustering noises of the majority samples in the boundary region is proposed. Firstly, KNN is used to search adjacent points of the majority class, and the boundary region and the safety region are divided according to the proportion of the majority samples in adjacent points. Secondly, DBSCAN is used to cluster the majority samples in the boundary region, and noise points are removed. Finally, it’s combined with the method based on model dynamic selection driven by data partition hybrid sampling (DPHS-MDS). The purpose of reducing the overlap degree of boundary samples, balancing the dataset and improving the classification effect is achieved. Experimental results show that the proposed method is superior to the relevant mainstream methods under F-measure and G-mean.