Cost-sensitive Learning Methods Research Articles

Reject inference in credit scoring based on cost-sensitive learning and joint distribution adaptation method

As traditional credit evaluation methods generally only use accepted sample modeling, the rejected data is omitted, which means the model's prediction of new customers is biased. However, reject inference can be used to solve this credit evaluation sample selection bias. This paper proposes a new reject inference method based on joint distribution adaptation (JDA) and cost-sensitive semi-supervised support vector machines (CS4VM). First, this method uses both accepted (labeled) samples and rejected (unlabeled) samples modeling, which overcomes the deviations in traditional credit evaluation methods. Second, as the accepted sample and the rejected sample distributions are different, this method reduces the distribution differences between the accepted and rejected sample sets, which ensures that the sample data conforms to the basic assumptions in the semi-supervised model, and improves the performance of the classification model. Third, this method reduces the overall cost in the actual credit business by considering both the traditional misclassification costs when mining the default samples and the different decision weights for the accepted and rejected samples. Finally, an empirical study verifies the excellent predictive performance of the proposed method and effectively reduces the total credit costs.

MVQS: Robust multi-view instance-level cost-sensitive learning method for imbalanced data classification

Multi-view imbalanced learning is to handle the datasets with multi-view representations and imbalanced classes. Existing multi-view imbalanced learning methods can be divided into two main categories: multi-view ensemble learning and multi-view cost-sensitive learning. However, these methods suffer from the following problems: 1) neglecting either consensus or complementary information, 2) complex preprocessing and information fusion in multi-view ensemble learning and manual assignment of misclassification costs in multi-view cost-sensitive learning, and 3) limited ability to handle noisy samples. Therefore, we aim to design a concise and unified framework to grapple with the multi-view representations, imbalanced classes and noisy samples simultaneously. Inspired by the merits of support vector machine (SVM) and quadratic type squared error (QTSE) loss function, we propose a robust multi-view instance-level cost-sensitive SVM with QTSE loss (MVQS) for imbalanced data classification. The consensus regularization term and combination weight strategy are employed to fully exploit multi-view information. The QTSE loss can adaptively impose instance-level penalties to the misclassification of samples, and make MVQS be robust to noisy samples. We solve MVQS with the alternating direction method of multipliers (ADMM) and the gradient descent (GD) algorithm. Comprehensive experiments validate that MVQS is more competitive and robust than other benchmark approaches.

Universum driven cost-sensitive learning method with asymmetric loss function

Class imbalance learning (CIL) has been a critical yet challenging topic for a long time. To address the issue, this paper proposes a cost-sensitive learning method based on the linear-exponential (LINEX) loss function and the Universum (UCSSVM). On one hand, the asymmetry of LINEX loss function is leveraged to realize cost-sensitive learning, by treating each instance discriminately. On the other hand, to enhance the classification performance, a novel and effective strategy is put forward to generate Universum with the assistance of hard-to-classify samples (small margin distance). The mini-batch gradient descent (MBGD) approach is adopted to optimize UCSSVM. Meanwhile, extensive experiments demonstrate its superiority in CIL and the necessity of using Universum.

Cost-sensitive continuous ensemble kernel learning for imbalanced data streams with concept drift

In stream learning, data continuously arrives over time, often at a very high rate. For imbalanced data streams with concept drift, it becomes essential to simultaneously address classification accuracy and time efficiency. However, existing algorithms currently either demand excessive computational time or exhibit relatively lower classification performance. To address these issues, in this paper, a cost-sensitive continuous ensemble kernel learning method (CCEKL) is proposed to deal with the imbalanced data streams with concept drift. Firstly, for two-class data streams, a novel misclassification cost is proposed to adapt to the changing imbalance rate. Secondly, based on the modified loss function, a continuous kernel learning method is applied to adapt to the changing data distribution. Furthermore, the two-class method is extended to multi-class classification to deal with multi-class dynamic imbalance problems. Finally, we use an ensemble approach to address the proposed algorithm sensitivity to the initial kernel width. To further improve efficiency, a parallel approach is applied to simultaneously train multi-classifiers with different initial kernel widths. Experimental results on 30 data streams reveal that, for binary classification, CCEKL achieves superior balanced accuracy in a shorter training time. For multi-class classification, CCEKL attains better balanced accuracy without requiring excessive training time, surpassing the most of the state-of-the-art baseline algorithms.

SynDroid: An adaptive enhanced Android malware classification method based on CTGAN-SVM

Android mobile phones have the highest market share nowadays, bringing a boom of Android application programming as well as malicious software (malware) issues. Traditional machine learning and deep learning methods are widely used in Android malware detection and classification, both require plenty of application samples to train classifiers. However, the collected samples are always imbalanced, because the distribution of malware categories differs hugely in the real environment. For Android malware samples with high dimension features, and Class Imbalance Ratio coming to more than 100:1, traditional methods become weak. To fill the gap, we propose an Android malware classification model named SynDroid. The core step of SynDroid uses CTGAN-SVM to generate qualified high-dimension samples, and adaptively discards bad results. Besides, we propose KS-CIR test to help the model to determine which classes of data needed to be enhanced most. This new proposed test can measure the data in respect of both samples' quality and quantity. Lastly, Random Forest is taken as a classifier to finish the classification task. The performance of SynDroid is evaluated on CCCS-CIC-AndMal2020 on the accuracy, precision, recall and F1-score. Both longitudinal and horizontal comparison experiments have been done with traditional oversampling methods, cost-sensitive learning, and other complicated methods. The result shows that the proposed method gets 12% more accuracy than the method on the same dataset and alleviates imbalanced data problems.

Everything has its price: Foundations of cost-sensitive machine learning and its application in psychology.

Psychology has seen an increase in the use of machine learning (ML) methods. In many applications, observations are classified into one of two groups (binary classification). Off-the-shelf classification algorithms assume that the costs of a misclassification (false positive or false negative) are equal. Because this is often not reasonable (e.g., in clinical psychology), cost-sensitive machine learning (CSL) methods can take different cost ratios into account. We present the mathematical foundations and introduce a taxonomy of the most commonly used CSL methods, before demonstrating their application and usefulness on psychological data, that is, the drug consumption data set (N = 1, 885) from the University of California Irvine ML Repository. In our example, all demonstrated CSL methods noticeably reduced mean misclassification costs compared to regular ML algorithms. We discuss the necessity for researchers to perform small benchmarks of CSL methods for their own practical application. Thus, our open materials provide R code, demonstrating how CSL methods can be applied within the mlr3 framework (https://osf.io/cvks7/). (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Multi-view cost-sensitive kernel learning for imbalanced classification problem

Multi-view imbalanced learning concentrates on recognizing valuable patterns from multi-view imbalanced data. There are numerous algorithm-level multi-view imbalanced learning methods including multi-view ensemble learning methods and multi-view cost-sensitive learning methods. However, these approaches have two drawbacks. Firstly, from the multi-view representation perspective, they either ignore the agreement among different views, or fail to fully exploit the complementarity information contained in multi-view data. Secondly, from the class-imbalanced perspective, multi-view ensemble learning first needs to design a specific data pre-processing scheme to generate views and then fuses the local and global information with multi-view ensemble schemes. Such models are closely related to practical problems and lack universality. In contrast, multi-view cost-sensitive learning methods possess concise formulations. But the success of such methods depends on imposing appropriate misclassification cost to the majority and minority classes. Inspired by elegant merits of asymmetric LINEX loss function, we propose a multi-view cost-sensitive kernel learning method with LINEX loss termed as MVCSKL. It not only fully exploits the consensus and complementary information of multiple views, but also adaptively learns and adjusts the misclassification cost of different classes through the asymmetric LINEX loss function. Besides, the introduction of the kernel function enables the MVCSKL model to handle complex nonlinear problem and further guarantees its prediction performance. Furthermore, we adopt the alternating direction method of multipliers and gradient descent algorithm to solve MVCSKL. Based on the Rademacher complexity, we analyze the generalization capability of MVCSKL. Extensive experiments are conducted to verify the effectiveness of the proposed model.

Handling imbalanced class in melanoma: Kemeny–Young rule based optimal rank aggregation and Self-Adaptive Differential Evolution Optimization

Melanoma, despite its relatively low incidence compared to other types of skin cancer, accounts for a significant proportion of skin cancer-related deaths. Early detection of melanoma is crucial for improving patient survival rates. Deep learning algorithms, which heavily rely on data, are widely used in melanoma detection. However, the performance of these algorithms is greatly influenced by the distribution of the dataset, particularly class imbalance. In this manuscript, the authors present a novel method based on the Kemeny–Young rule for optimal rank aggregation to address the class imbalance problem in melanoma detection. The proposed approach aims to reduce class bias and enhance overall classification accuracy. Furthermore, a cost-sensitive learning approach is introduced to improve the classifier’s ability to handle class imbalance effectively. This novel cost-sensitive learning method utilizes Self-Adaptive Differential Evolution Optimization to determine optimal weights for each class. Our approach differs from traditional methods that assign weights based on predefined criteria. To evaluate the effectiveness of the proposed methods, extensive experiments and ablation studies are conducted on the highly imbalanced ISIC 2020 dataset, which is widely used in melanoma detection research. The Kemeny–Young rule-based majority voting achieves an overall error rate of 2.44%, while the cost-sensitive learning based on the Self-Adaptive Differential Evolution approach achieves an even lower error rate of 1.99%. Moreover, the proposed method achieves a sensitivity of 87.93% and a specificity of 98.19%. These experimental results demonstrate the competitiveness and effectiveness of the proposed methods in addressing the challenges posed by class imbalance and improving the accuracy of melanoma detection. By effectively mitigating class imbalance, these methods improve the accuracy and reliability of melanoma detection, thus offering valuable insights for developing advanced computer-aided diagnosis systems in dermatology. The relevant codes for our proposed approach are publicly available at: https://github.com/ctrl-gaurav/Handling-Imbalanced-Class-in-Melanoma.

CSAL: Cost sensitive active learning for multi-source drifting stream

Multi-source stream classification is a prominent real-world problem challenged by the limited real labels and non-stationary environment. Despite growing research achievements in this field, most existing works solved this problem by requiring all real labels of source or target streams to conduct domain adaption or transfer learning mechanisms, which brings high labeling costs. However, in real-world applications, there are usually insufficient labeled data in both source and target streams, and the annotation cost of the source and target streams are generally unequal. Thus, we propose a Cost-Sensitive Active Learning (CSAL) method for multi-source drifting streams. Specifically, a multi-source ensemble framework with an asymmetry weighting mechanism is presented to ensure beneficial knowledge transfer and avoid the negative transfer. Then, a multi-perspective similarity estimation method is proposed to evaluate the similarity of source and target streams. On this basis, a novel cost-sensitive hybrid labeling strategy that combines volatility strategy and uncertainty strategy with a cost-sensitive budget control mechanism is proposed, which adaptively selects representative samples at the appropriate time. At last, a parallel multiple hypothesis drift detection method is proposed, which can efficiently utilize real labels to detect concept drift. Experimental results on real-world and synthetic data streams show that our CSAL outperforms the state-of-the-art methods with even fewer labels.

Probability Guided Loss for Long-Tailed Multi-Label Image Classification

Long-tailed learning has attracted increasing attention in very recent years. Long-tailed multi-label image classification is one subtask and remains challenging and poorly researched. In this paper, we provide a fresh perspective from probability to tackle this problem. More specifically, we find that existing cost-sensitive learning methods for long-tailed multi-label classification will affect the predicted probability of positive and negative labels in varying degrees during training, and different processes of probability will affect the final performance in turn. We thus propose a probability guided loss which contains two components to control this process. One is the probability re-balancing which can flexibly adjust the process of training probability. And the other is the adaptive probability-aware focal which can further reduce the probability gap between positive and negative labels. We conduct extensive experiments on two long-tailed multi-label image classification datasets: VOC-LT and COCO-LT. The results demonstrate the rationality and superiority of our strategy.

Breaking Alert Fatigue: AI-Assisted SIEM Framework for Effective Incident Response

Contemporary security information and event management (SIEM) solutions struggle to identify critical security incidents effectively due to the overwhelming number of false alerts generated by disparate security products, which results in significant alert fatigue and hinders effective incident response. To overcome this challenge, we propose a next-generation SIEM framework that integrates security orchestration automation and response capabilities and utilizes a divide-and-conquer strategy to mitigate the impact of low-quality IDS alerts. The proposed framework leverages advanced machine learning and data visualization tools—including a cost-sensitive learning method and an event segmenting algorithm—to filter and correlate alerts plus an augmented visualization tool to expedite the triage process. The proposed framework was evaluated experimentally on a dataset collected from a real-world enterprise network, and we report highly convincing results. The alert screening scheme demonstrates significant potential for real-world security operations. We believe that our findings will contributing to the development of a next-generation SIEM system that effectively addresses alert fatigue and lays the foundation for future research in this field.

Seismic predictions of fluids via supervised deep learning: Incorporating various class-rebalance strategies

Seismic fluids prediction under the machine-learning framework is of great significance for the exploration and development of oil and gas resources, geothermal energy exploitation, carbon dioxide sequestration monitoring, and groundwater management. Data-driven supervised machine-learning algorithms often rely heavily on the characteristics of the data (number of labels and data distribution). The disparity in the number of different labels for the majority and minority samples can hinder the generalization ability of the machine-learning model, especially weakening the predicting power for minority groups (e.g., hydrocarbon-bearing rocks) which are often of essential interest to us. For a clastic reservoir exhibiting a typical class imbalance (the ratio of gas sandstone to other lithofluids is significantly low), under the framework of a supervised convolutional neural network, we investigate and compare various class-rebalance methods to enhance the model’s prediction ability for gas-bearing sandstones. To achieve the purpose of class rebalance, we mainly use sampling methods to obtain class-balanced data sets and cost-sensitive learning methods to modify loss functions. The crosswell blind tests indicate that the ensemble-based undersampling method of BalanceCascade is found to be most effective in enhancing the prediction performance, increasing the F1 score of gas sandstone by as much as 15%. We also propose the combination of Balance Cascade and focal-loss (FL) methods, which can further improve the F1 score of gas-bearing sandstone in several wells compared with using BalanceCascade or FL alone. By incorporating class-rebalance strategies into model building, we finally obtain more reliable seismic prediction results for gas-bearing sandstone.

Cost-Sensitive Hypergraph Learning With F-Measure Optimization

The imbalanced issue among data is common in many machine-learning applications, where samples from one or more classes are rare. To address this issue, many imbalanced machine-learning methods have been proposed. Most of these methods rely on cost-sensitive learning. However, we note that it is infeasible to determine the precise cost values even with great domain knowledge for those cost-sensitive machine-learning methods. So in this method, due to the superiority of F-measure on evaluating the performance of imbalanced data classification, we employ F-measure to calculate the cost information and propose a cost-sensitive hypergraph learning method with F-measure optimization to solve the imbalanced issue. In this method, we employ the hypergraph structure to explore the high-order relationships among the imbalanced data. Based on the constructed hypergraph structure, we optimize the cost value with F-measure and further conduct cost-sensitive hypergraph learning with the optimized cost information. The comprehensive experiments validate the effectiveness of the proposed method.

Adaptive Subspace Optimization Ensemble Method for High-Dimensional Imbalanced Data Classification.

It is hard to construct an optimal classifier for high-dimensional imbalanced data, on which the performance of classifiers is seriously affected and becomes poor. Although many approaches, such as resampling, cost-sensitive, and ensemble learning methods, have been proposed to deal with the skewed data, they are constrained by high-dimensional data with noise and redundancy. In this study, we propose an adaptive subspace optimization ensemble method (ASOEM) for high-dimensional imbalanced data classification to overcome the above limitations. To construct accurate and diverse base classifiers, a novel adaptive subspace optimization (ASO) method based on adaptive subspace generation (ASG) process and rotated subspace optimization (RSO) process is designed to generate multiple robust and discriminative subspaces. Then a resampling scheme is applied on the optimized subspace to build a class-balanced data for each base classifier. To verify the effectiveness, our ASOEM is implemented based on different resampling strategies on 24 real-world high-dimensional imbalanced datasets. Experimental results demonstrate that our proposed methods outperform other mainstream imbalance learning approaches and classifier ensemble methods.

Classification of tracheal stenosis with asymmetric misclassification errors from EMG signals using an adaptive cost-sensitive learning method

Upper airway obstruction is characterized by loss of normal airway architecture resulting from various disorders such as infections and asthma. Early detection of airway obstruction is essential to prevent medical deterioration. The objective of this study was to non-invasively identify early-stage tracheal stenosis, using electromyography (EMG) signals of inspiratory muscles. The identification of tracheal stenosis has been defined as an asymmetric misclassification cost problem. Specifically, the EMG signals are used as input to a ResNet-like architecture for tabular data with an Adaptive Cost-Sensitive Learning (AdaCSL) algorithm. The electrical activity of the external intercostal muscle of four healthy individuals was recorded while they breathed through two different tubes, one simulating a narrowed airway and the other simulating a normal airway. Two experiment settings were designed. The first setting aimed to classify tracheal stenosis in a specific subject by training the model on data from other subjects, reflecting the case of diagnosing a new subject. To overcome multi-subject variations, the second setting aimed to classify tracheal stenosis by mixing all subjects’ training and test data. The ResNet-like architecture with an AdaCSL algorithm was significantly better in the first experiment setting with costs that were 43%, 48%, and 59% lower than the cost of the second-best alternative for three different misclassification cost values. It also achieved a lower cost in the second experiment setting over other classifiers. The experiments emphasize the capability of using inspiratory muscle EMG signals to diagnose respiratory disease and demonstrate the usefulness of the AdaCSL algorithm for personalized monitoring.

Cost-Sensitive YOLOv5 for Detecting Surface Defects of Industrial Products.

Owing to the remarkable development of deep learning algorithms, defect detection techniques based on deep neural networks have been extensively applied in industrial production. Most existing surface defect detection models assign equal costs to the classification errors among different defect categories but do not strictly distinguish them. However, various errors can generate a great discrepancy in decision risk or classification costs and then produce a cost-sensitive issue that is crucial to the manufacturing process. To address this engineering challenge, we propose a novel supervised classification cost-sensitive learning method (SCCS) and apply it to improve YOLOv5 as CS-YOLOv5, where the classification loss function of object detection was reconstructed according to a new cost-sensitive learning criterion explained by a label-cost vector selection method. In this way, the classification risk information from a cost matrix is directly introduced into the detection model and fully exploited in training. As a result, the developed approach can make low-risk classification decisions for defect detection. It is applicable for direct cost-sensitive learning based on a cost matrix to implement detection tasks. Using two datasets of a painting surface and a hot-rolled steel strip surface, our CS-YOLOv5 model outperforms the original version with respect to cost under different positive classes, coefficients, and weight ratios, but also maintains effective detection performance measured by mAP and F1 scores.

Rebalance Weights AdaBoost-SVM Model for Imbalanced Data

Classification of imbalanced data is a challenging task that has captured considerable interest in numerous scientific fields by virtue of the great practical value of minority accuracy. Some methods for improving generalization performance have been developed to address this classification situation. Here, we propose a cost-sensitive ensemble learning method using a support vector machine as a base learner of AdaBoost for classifying imbalanced data. Considering that the existing methods are not well studied in terms of how to precisely control the classification accuracy of the minority class, we developed a novel way to rebalance the weights of AdaBoost, and the weights influence the base learner training. This weighting strategy increases the sample weight of the misclassified minority while decreasing the sample weight of the misclassified majority until their distributions are even in each round. Furthermore, we included P-mean as one of the assessment markers and discussed why it is necessary. Experiments were conducted to compare the proposed and comparison 10 models on 18 datasets in terms of six different metrics. Through comprehensive experimental findings, the statistical study is performed to verify the efficacy and usability of the proposed model.

Cost-sensitive learning with modified Stein loss function

Cost-sensitive learning (CSL), which has gained widespread attention in class imbalance learning (CIL), can be implemented either by tuning penalty parameters or by designing new loss functions. In this paper, we propose a cost-sensitive learning method with a modified Stein loss function (CSMS) and a robust CSMS (RCSMS). Specifically, CSMS is flexible, as it realizes CSL from above two aspects simultaneously. In contrast, RCSMS merely achieves CSL by tuning penalty parameters, but the adopted loss function makes it insensitive to noise. To our best knowledge, it is the first time for Stein loss function derived from statistics to be applied in machine learning, which not only offers two alternative class imbalance solutions but also provides a novel idea for the design of loss functions in CIL. The mini-batch stochastic sub-gradient descent (MBGD) approach is employed to optimize CSMS and RCSMS. Meanwhile, the Rademacher complexity is used to analyze their generalization error bounds. Extensive experiments profoundly confirm the superiority of both models over benchmarks.

Machine learning-enabled risk prediction of chronic obstructive pulmonary disease with unbalanced data

Background and objectiveSince the early symptoms of chronic obstructive pulmonary disease (COPD) are not obvious, patients are not easily identified, causing improper time for prevention and treatment. In present study, machine learning (ML) methods were employed to construct a risk prediction model for COPD to improve its prediction efficiency. MethodsWe collected data from a sample of 5807 cases with a complete COPD diagnosis from the 2019 COPD Surveillance Program in Shanxi Province and extracted 34 potentially relevant variables from the dataset. Firstly, we used feature selection methods (i.e., Generalized elastic net, Lasso and Adaptive lasso) to select ten variables. Afterwards, we employed supervised classifiers for class imbalanced data by combining the cost-sensitive learning and SMOTE resampling methods with the ML methods (Logistic Regression, SVM, Random Forest, XGBoost, LightGBM, NGBoost and Stacking), respectively. Last, we assessed their performance. ResultsThe cough frequently at age 14 and before and other 9 variables are significant parameters for COPD. The Stacking heterogeneous ensemble model showed relatively good performance in the unbalanced datasets. The Logistic Regression with class weighting enjoyed the best classification performance in the balancing data when these composite indicators (AUC, F1-Score and G-mean) were used as criteria for model comparison. The values of F1-Score and G-mean for the top three ML models were 0.290/0.660 for Logistic Regression with class weighting, 0.288/0.649 for Stacking with synthetic minority oversampling technique (SMOTE), and 0.285/0.648 for LightGBM with SMOTE. ConclusionsThis paper combining feature selection methods, unbalanced data processing methods and machine learning methods with data from disease surveillance questionnaires and physical measurements to identify people at risk of COPD, concluded that machine learning models based on survey questionnaires could provide an automated identification for patients at risk of COPD, and provide a simple and scientific aid for early identification of COPD.

Wasserstein Generative Adversarial Network to Address the Imbalanced Data Problem in Real-Time Crash Risk Prediction

Real-time crash risk prediction models aim to identify pre-crash conditions as part of active traffic safety management. However, traditional models which were mainly developed through matched case-control sampling have been criticised due to their biased estimations. In this study, the state-of-art class balancing method known as the Wasserstein Generative Adversarial Network (WGAN) was introduced to address the class imbalance problem in the model development. An extremely imbalanced dataset consisted of 257 crashes and over 10 million non-crash cases from M1 Motorway in United Kingdom for 2017 was then utilized to evaluate the proposed method. The real-time crash prediction model was developed by employing Deep Neural Network (DNN) and Logistic Regression (LR). Crash predictions were performed under different crash to non-crash ratios where synthetic crashes were generated by Wasserstein Generative Adversarial Network (WGAN), Synthetic Minority Over-sampling Technique (SMOTE) and Adaptive Synthetic (ADASYN) sampling respectively. Comparisons were then made with algorithmic-level class balancing methods such as cost-sensitive learning and ensemble methods. Our findings suggest that WGAN clearly outperforms other oversampling methods in terms of handling the extremely imbalanced sample and the DNN model subsequently produces a crash prediction sensitivity of about 70% with a 5% false alarm rate. Based on the findings of this study, proactive traffic management strategies including Variable Speed Limit (VSL) and Dynamic Messing Signs (DMS) could be deployed to reduce the probability of crash occurrence.