Cost-sensitive Learning Research Articles

CSE-IDS: Using cost-sensitive deep learning and ensemble algorithms to handle class imbalance in network-based intrusion detection systems

In recent times, Network-based Intrusion Detection Systems (NIDSs) have become very popular for detecting intrusions in computer networks. Existing NIDSs can easily identify those intrusions that have been frequently witnessed in the network (majority attacks), but they cannot identify new and infrequent intrusions (minority attacks) accurately. Moreover, such systems solely focus on maximizing the overall Attack Detection Rate while overlooking the number of false alarms. To address these issues, this paper proposes CSE-IDS, a three-layer NIDS, based on Cost-Sensitive Deep Learning and Ensemble algorithms. Layer 1 of the proposed CSE-IDS uses Cost-Sensitive Deep Neural Network to separate normal traffic from suspicious network traffic. These suspicious samples are then sent to Layer 2, which uses the eXtreme Gradient Boosting algorithm to classify them into normal class, different majority attack classes, and a single class representing all minority attack classes. At last, Random Forest is used at Layer 3 to classify the minority attacks identified at Layer 2 into their respective classes. The performance of the proposed CSE-IDS was evaluated on the NSL-KDD, CIDDS-001, and CICIDS2017 datasets with respect to Accuracy, Recall, Precision, F1-score, ROC curve, AUC values, and computational times. The proposed system outperforms its counterparts by achieving a high Attack Detection Rate for both majority attacks and minority attacks present in the network. Further, it minimizes the number of false alarms by correctly segregating normal traffic from attack traffic. The obtained results confirm that the proposed CSE-IDS can be deployed in the real world for performing network-based intrusion detection.

Cost-sensitive ensemble learning: a unifying framework

Over the years, a plethora of cost-sensitive methods have been proposed for learning on data when different types of misclassification errors incur different costs. Our contribution is a unifying framework that provides a comprehensive and insightful overview on cost-sensitive ensemble methods, pinpointing their differences and similarities via a fine-grained categorization. Our framework contains natural extensions and generalisations of ideas across methods, be it AdaBoost, Bagging or Random Forest, and as a result not only yields all methods known to date but also some not previously considered.

A Method of Enterprise Financial Risk Analysis and Early Warning Based on Decision Tree Model

At present, the domestic and foreign financial crisis early-warning model research will provide only prediction accuracy as the only standard of success for early-warning model, ignoring an important problem, namely, will the financial crisis early-warning model for normal business, compared with the normal enterprise, forecast the financial crisis? This paper reviews the research situation at home and abroad from the perspective of the definition of the enterprise financial crisis, the form of expression, and so on. From the theoretical level, the relationship between the cause of the financial crisis and the change of financial indicators is established by explaining the early-warning theory, early-warning theory of financial crisis, and cost-sensitive learning theory, and the framework of early warning modeling of financial crisis based on decision tree is put forward. The decision tree model is constructed on several training subsets as the base learner so that the decision tree base learner can learn the characteristics of the healthy sample and crisis sample roughly equally. Taking the bond issuing enterprises of manufacturing industry as samples, the empirical comparison shows that the financial warning model based on decision tree integration is more accurate, which indicates that the model can improve the correct identification rate of financial crisis enterprises under the premise of higher overall warning accuracy.

Cost-sensitive meta-learning framework

PurposeThis paper aims to describe the use of a meta-learning framework for recommending cost-sensitive classification methods with the aim of answering an important question that arises in machine learning, namely, “Among all the available classification algorithms, and in considering a specific type of data and cost, which is the best algorithm for my problem?”Design/methodology/approachThis paper describes the use of a meta-learning framework for recommending cost-sensitive classification methods for the aim of answering an important question that arises in machine learning, namely, “Among all the available classification algorithms, and in considering a specific type of data and cost, which is the best algorithm for my problem?” The framework is based on the idea of applying machine learning techniques to discover knowledge about the performance of different machine learning algorithms. It includes components that repeatedly apply different classification methods on data sets and measures their performance. The characteristics of the data sets, combined with the algorithms and the performance provide the training examples. A decision tree algorithm is applied to the training examples to induce the knowledge, which can then be used to recommend algorithms for new data sets. The paper makes a contribution to both meta-learning and cost-sensitive machine learning approaches. Those both fields are not new, however, building a recommender that recommends the optimal case-sensitive approach for a given data problem is the contribution. The proposed solution is implemented in WEKA and evaluated by applying it on different data sets and comparing the results with existing studies available in the literature. The results show that a developed meta-learning solution produces better results than METAL, a well-known meta-learning system. The developed solution takes the misclassification cost into consideration during the learning process, which is not available in the compared project.FindingsThe proposed solution is implemented in WEKA and evaluated by applying it to different data sets and comparing the results with existing studies available in the literature. The results show that a developed meta-learning solution produces better results than METAL, a well-known meta-learning system.Originality/valueThe paper presents a major piece of new information in writing for the first time. Meta-learning work has been done before but this paper presents a new meta-learning framework that is costs sensitive.

A random forest classifier with cost-sensitive learning to extract urban landmarks from an imbalanced dataset

ABSTRACT Urban landmarks play an important role as spatial references in spatial cognition, navigation, map design and urban planning. However, the current landmark extraction methods do not consider the imbalance between the landmark and non-landmarknon-landmark samples in a dataset, so the extraction results are biased toward the class with the majority of sample data, resulting in poor classification performance for the class with the fewest sample data. This study introduces a random forest (RF) classifier combined with cost-sensitive learning to extract urban landmarks automatically from a basic spatial database. First, the optimal feature set is determined according to the importance of features. Next, a cost-sensitive RF algorithm is applied to extract landmarks, which determines the misclassification cost according to the class distribution, and each decision tree is weighted by the classification results. The method has good performance, with a recall and area under the ROC curve (AUC) greater than 90%, and the model is also applicable to small sample sets, which can reduce the cost of manual labor.

Cost-sensitive regression learning on small dataset through intra-cluster product favoured feature selection

Massive regression and forecasting tasks are generally cost-sensitive regression learning problems with asymmetric costs between over-prediction and under-prediction. However, existing classic methods, such as clustering and feature selection, are subject to difficulties in dealing with small datasets. As one of the key challenges, it is difficult to statistically validate the importance of features using traditional algorithms (e.g. the Boruta algorithm) owing to insufficient available data. By leveraging the feature information intra-cluster (item group with similar attributes), we propose an intra-cluster product favoured (ICPF) feature selection algorithm to select the information based on the traditional filtering method (specifically the Boruta algorithm in our study). The experimental results show that the ICPF algorithm significantly reduces the number of dimensions of the selected feature set and improves the performance of cost-sensitive regression learning. The misprediction cost decreased by 33.5% (linear-linear cost function) and 32.4% (quadratic-quadratic cost function) after adopting the ICPF algorithm. In addition, the advantage of the ICPF algorithm is robust to other regression models, such as random forest and XGboost.

A novel adaptive weighted kernel extreme learning machine algorithm and its application in wind turbine blade icing fault detection

The conventional weighted kernel extreme learning machine (WKELM) is a typical cost-sensitive learning algorithm, which can provide a solution for the data imbalance problem in wind turbine blade icing fault detection. However, WKELM algorithm uses a fixed weighting strategy that only considers the number of samples to achieve imbalanced learning, which leads to its insufficient performance in data imbalance classification and fault detection. In this study, the support vector data description (SVDD) method is used to obtain sample distribution information to construct a novel adaptive weighting strategy. The adaptive weighting strategy is combined with the conventional fixed weighting strategy to construct an adaptive weighted kernel extreme learning machine (AWKELM) algorithm. The effectiveness and superiority of the AWKELM algorithm is verified through three benchmark cases, it is demonstrated that AWKELM algorithm can effectively solve the problem of data imbalance and further improve the effect of wind turbine blade icing detection.

An online weighted sequential extreme learning machine for class imbalanced data streams

When general online classification algorithms deal with imbalanced data streams, there are always some problems, such as over fitting phenomenon caused by insufficient simple learning and instability of training model. In this paper, we introduce online sequential extreme learning machine (OSELM) as the basic theory model, and combine with the cost-sensitive strategy, then propose a cost-sensitive learning based online sequential extreme learning machine algorithm (C-OSELM). Firstly, in order to solve the problem that minority classes are easily misclassified due to class imbalance, use cost-sensitive strategy, by assigning different penalty parameters to various samples, a weighting matrix is constructed to improve the misclassification cost, thereby effectively alleviating the excessive deviation of decision surface. On this basis, in order to solve the problem that the penalty parameter is too single and the algorithm is not universal, the cost adjustment function is introduced to optimize the weight parameters to select the appropriate weight. Finally, 16 class II imbalanced datasets are used for comparison and verification. The experimental results show that the classification performances of the proposed C-OSELM algorithm are better than other comparative algorithms.

Enhancement of prostate cancer diagnosis by machine learning techniques: an algorithm development and validation study.

To investigate the value of machine learning(ML) in enhancing prostate cancer(PCa) diagnosis. Consecutive systematic prostate biopsies performed from Jan 2003-June 2017 were used as the training cohort, and prospective biopsies performed from July 2017-November 2019 were used as validation cohort. Men were included if PSA was 0.4-50 ng/mL, and information of digital rectal examination (DRE), Transrectal ultrasound(TRUS) prostate volume, TRUS abnormality were known. Clinically significant PCa(csPCa) was defined as Gleason 3 + 4 or above cancers. Area-under-curve (AUC) of receiver-operating characteristics (ROC) was compared between PSA, PSA density, European Randomized Study of Screening for Prostate Cancer (ERSPC) risk calculator (ERSPC-RC), and various ML techniques using PSA, DRE and TRUS information. ML techniques used included XGBoost, LightGBM, Catboost, Support vector machine (SVM), Logistic regression (LR), and Random Forest (RF), where cost sensitive learning was applied. Training and validation cohorts included 3881 and 778 consecutive men, respectively. RF model performed better than other ML techniques and PSA, PSA density and ERSPC-RC for prediction of PCa or csPCa in the validation cohort. In csPCa prediction, AUC of PSA, PSA density, ERSPC-RC and RF was 0.71, 0.80, 0.83 and 0.88 respectively. At 90-95% sensitivity for csPCa, RF model achieved a negative predictive value (NPV) of 97.5-98.0% and avoided 38.3-52.2% unnecessary biopsies. Decision curve analyses (DCA) showed RF model provided net clinical benefit over PSA, PSA density and ERSPC-RC. By using the same clinical parameters, ML techniques performed better than ERSPC-RC or PSA density in csPCa predictions, and could avoid up to 50% unnecessary biopsies.

Risky Driver Recognition with Class Imbalance Data and Automated Machine Learning Framework.

Identifying high-risk drivers before an accident happens is necessary for traffic accident control and prevention. Due to the class-imbalance nature of driving data, high-risk samples as the minority class are usually ill-treated by standard classification algorithms. Instead of applying preset sampling or cost-sensitive learning, this paper proposes a novel automated machine learning framework that simultaneously and automatically searches for the optimal sampling, cost-sensitive loss function, and probability calibration to handle class-imbalance problem in recognition of risky drivers. The hyperparameters that control sampling ratio and class weight, along with other hyperparameters, are optimized by Bayesian optimization. To demonstrate the performance of the proposed automated learning framework, we establish a risky driver recognition model as a case study, using video-extracted vehicle trajectory data of 2427 private cars on a German highway. Based on rear-end collision risk evaluation, only 4.29% of all drivers are labeled as risky drivers. The inputs of the recognition model are the discrete Fourier transform coefficients of target vehicle’s longitudinal speed, lateral speed, and the gap between the target vehicle and its preceding vehicle. Among 12 sampling methods, 2 cost-sensitive loss functions, and 2 probability calibration methods, the result of automated machine learning is consistent with manual searching but much more computation-efficient. We find that the combination of Support Vector Machine-based Synthetic Minority Oversampling TEchnique (SVMSMOTE) sampling, cost-sensitive cross-entropy loss function, and isotonic regression can significantly improve the recognition ability and reduce the error of predicted probability.

Density-based weighting for imbalanced regression

In many real world settings, imbalanced data impedes model performance of learning algorithms, like neural networks, mostly for rare cases. This is especially problematic for tasks focusing on these rare occurrences. For example, when estimating precipitation, extreme rainfall events are scarce but important considering their potential consequences. While there are numerous well studied solutions for classification settings, most of them cannot be applied to regression easily. Of the few solutions for regression tasks, barely any have explored cost-sensitive learning which is known to have advantages compared to sampling-based methods in classification tasks. In this work, we propose a sample weighting approach for imbalanced regression datasets called DenseWeight and a cost-sensitive learning approach for neural network regression with imbalanced data called DenseLoss based on our weighting scheme. DenseWeight weights data points according to their target value rarities through kernel density estimation (KDE). DenseLoss adjusts each data point’s influence on the loss according to DenseWeight, giving rare data points more influence on model training compared to common data points. We show on multiple differently distributed datasets that DenseLoss significantly improves model performance for rare data points through its density-based weighting scheme. Additionally, we compare DenseLoss to the state-of-the-art method SMOGN, finding that our method mostly yields better performance. Our approach provides more control over model training as it enables us to actively decide on the trade-off between focusing on common or rare cases through a single hyperparameter, allowing the training of better models for rare data points.

A data science approach for early-stage prediction of Patient's susceptibility to acute side effects of advanced radiotherapy

The prediction by classification of side effects incidence in a given medical treatment is a common challenge in medical research. Machine Learning (ML) methods are widely used in the areas of risk prediction and classification. The primary objective of such algorithms is to use several features to predict dichotomous responses (e.g., disease positive/negative). Similar to statistical inference modelling, ML modelling is subject to the class imbalance problem and is affected by the majority class, increasing the false-negative rate. In this study, seventy-nine ML models were built and evaluated to classify approximately 2000 participants from 26 hospitals in eight different countries into two groups of radiotherapy (RT) side effects incidence based on recorded observations from the international study of RT related toxicity “REQUITE”. We also examined the effect of sampling techniques and cost-sensitive learning methods on the models when dealing with class imbalance. The combinations of such techniques used had a significant impact on the classification. They resulted in an improvement in incidence status prediction by shifting classifiers’ attention to the minority group. The best classification model for RT acute toxicity prediction was identified based on domain experts' success criteria. The Area Under Receiver Operator Characteristic curve of the models tested with an isolated dataset ranged from 0.50 to 0.77. The scale of improved results is promising and will guide further development of models to predict RT acute toxicities. One model was optimised and found to be beneficial to identify patients who are at risk of developing acute RT early-stage toxicities as a result of undergoing breast RT ensuring relevant treatment interventions can be appropriately targeted. The design of the approach presented in this paper resulted in producing a preclinical-valid prediction model. The study was developed by a multi-disciplinary collaboration of data scientists, medical physicists, oncologists and surgeons in the UK Radiotherapy Machine Learning Network.

An efficient method to determine sample size in oversampling based on classification complexity for imbalanced data

Resampling, one of the approaches to handle class imbalance, is widely used alone or in combination with other approaches, such as cost-sensitive learning and ensemble learning because of its simplicity and independence in learning algorithms. Oversampling methods, in particular, alleviate class imbalance by increasing the size of the minority class. However, previous studies related to oversampling generally have focused on where to add new samples, how to generate new samples, and how to prevent noise and they rarely have investigated how much sampling is sufficient. In many cases, the oversampling size is set so that the minority class has the same size as the majority class. This setting only considers the size of the classes in sample size determination, and the balanced training set can induce overfitting with the addition of too many minority samples. Moreover, the effectiveness of oversampling can be improved by adding synthetics into the appropriate locations. To address this issue, this study proposes a method to determine the oversampling size less than the sample size needed to obtain a balance between classes, while considering not only the absolute imbalance but also the difficulty of classification in a dataset on the basis of classification complexity. The effectiveness of the proposed sample size in oversampling is evaluated using several boosting algorithms with different oversampling methods for 16 imbalanced datasets. The results show that the proposed sample size achieves better classification performance than the sample size for attaining class balance.

Interactive Decision Tree Learning and Decision Rule Extraction Based on the ImbTreeEntropy and ImbTreeAUC Packages

This paper presents two new R packages ImbTreeEntropy and ImbTreeAUC for building decision trees, including their interactive construction and analysis, which is a highly regarded feature for field experts who want to be involved in the learning process. ImbTreeEntropy functionality includes the application of generalized entropy functions, such as Renyi, Tsallis, Sharma-Mittal, Sharma-Taneja and Kapur, to measure the impurity of a node. ImbTreeAUC provides non-standard measures to choose an optimal split point for an attribute (as well the optimal attribute for splitting) by employing local, semi-global and global AUC measures. The contribution of both packages is that thanks to interactive learning, the user is able to construct a new tree from scratch or, if required, the learning phase enables making a decision regarding the optimal split in ambiguous situations, taking into account each attribute and its cut-off. The main difference with existing solutions is that our packages provide mechanisms that allow for analyzing the trees’ structures (several trees simultaneously) that are built after growing and/or pruning. Both packages support cost-sensitive learning by defining a misclassification cost matrix, as well as weight-sensitive learning. Additionally, the tree structure of the model can be represented as a rule-based model, along with the various quality measures, such as support, confidence, lift, conviction, addedValue, cosine, Jaccard and Laplace.

An adaptive fault diagnosis method of power transformers based on combining oversampling and cost‐sensitive learning

Dissolved gas analysis is an important technique for the insulation condition assessment and incipient fault diagnosis of power transformers. However, the performance of the traditional ratio methods can be hardly improved due to the overreliance on absolute ratio threshold. In this paper, a novel method combining oversampling and cost-sensitive learning is proposed to improve the diagnosis accuracy of all fault types of transformers. The radial-based oversampling (RBO) method is adopted to synthesise samples for the complex fault classes. With the newly generated samples, the deep belief network (DBN) can effectively learn the features of complex fault classes and distinguish them from the other fault classes. Moreover, by integrating a cost matrix into the loss function, the parameters of DBN are adaptively updated so as to ensure the correct classification of the fault class with less samples. Based on the oversampling and cost-sensitive learning, the proposed method can form suitable classification boundaries amongst thermal, discharge and complex fault classes. The effectiveness and generalisation capability of the proposed method are verified by case studies in a real-world fault dataset of power transformers with multi-source samples. The results demonstrate that the proposed method improves the classification accuracies in all fault classes, especially in the complex fault classes. The overall accuracy can be reach over 90% by applying both RBO and cost-sensitive learning.

A Study on the Improvement of Image Classification Performance in the Defense Field through Cost-Sensitive Learning of Imbalanced Data

With the development of deep learning technology, researchers and technicians keep attempting to apply deep learning in various industrial and academic fields, including the defense. Most of these attempts assume that the data are balanced. In reality, since lots of the data are imbalanced, the classifier is not properly built and the model's performance can be low. Therefore, this study proposes cost-sensitive learning as a solution to the imbalance data problem of image classification in the defense field. In the proposed model, cost-sensitive learning is a method of giving a high weight on the cost function of a minority class. The results of cost-sensitive based model shows the test F1-score is higher when cost-sensitive learning is applied than general learning's through 160 experiments using submarine/non-submarine dataset and warship/non-warship dataset. Furthermore, statistical tests are conducted and the results are shown significantly. Keywords: ë¶ˆê· í˜•ë°ì´í„°, 비용민감학습, ì»¨ë³¼ë£¨ì ˜ ì‹ ê²½ë§ Keywords: Imbalanced Data, Cost-sensitive Learning, CNN

Efficient Utilization of Missing Data in Cost-Sensitive Learning

Different from previous imputation methods which impute missing values in the incomplete samples by using the information in the complete samples, this paper proposes a Date-drive Incremental imputation Model, DIM for short, which uses all available information in the data set to impute missing values economically, effectively, orderly, and iteratively. To this end, we propose a scoring rule to rank the missing features by taking into account both the economical criterion and the effective imputation information. The economical criterion takes both the imputation cost and the discriminative ability of the feature into account, while the effective imputation information enables to use all observed information in the data set including the imputed missing values to impute the left missing values. During the imputation process, our DIM first detects the neednot-impute samples for reducing the imputation cost and noise, and then selects the missing features with the top rank to impute first. The imputation process orderly imputes the missing features until all missing values are imputed or the imputation cost is exhausted. Experimental results on UCI data sets demonstrated the advantages of our proposed DIM, compared to the comparison methods, in terms of prediction accuracy and classification accuracy.

Multiclass Classification with Imbalanced Datasets for Car Ownership Demand Model – Cost-Sensitive Learning

In terms of the travel demand prediction from the household car ownership model, if the imbalanced data were used to support the transportation policy via a machine learning model, it would negatively affect the algorithm training process. The data on household car ownership obtained from the study project for the expressway preparation in the Khon Kaen Province (2015) was an unbalanced dataset. In other words, the number of members of the minority class is lower than the rest of the answer classes. The result is a bias in data classification. Consequently, this research suggested balancing the datasets with cost-sensitive learning methods, including decision trees, k-nearest neighbors (kNN), and naive Bayes algorithms. Before creating the 3-class model, a k-folds cross-validation method was applied to classify the datasets to define true positive rate (TPR) for the model’s performance validation. The outcome indicated that the kNN algorithm demonstrated the best performance for the minority class data prediction compared to other algorithms. It provides TPR for rural and suburban area types, which are region types with very different imbalance ratios, before balancing the data of 46.9% and 46.4%. After balancing the data (MCN1), TPR values were 84.4% and 81.4%, respectively.

Instance-dependent cost-sensitive learning for detecting transfer fraud

Card transaction fraud is a growing problem affecting card holders worldwide. Financial institutions increasingly rely upon data-driven methods for developing fraud detection systems, which are able to automatically detect and block fraudulent transactions. From a machine learning perspective, the task of detecting fraudulent transactions is a binary classification problem. Classification models are commonly trained and evaluated in terms of statistical performance measures, such as likelihood and AUC, respectively. These measures, however, do not take into account the actual business objective, which is to minimize the financial losses due to fraud. Fraud detection is to be acknowledged as an instance-dependent cost-sensitive classification problem, where the costs due to misclassification vary between instances, and requiring adapted approaches for learning a classification model. In this article, an instance-dependent threshold is derived, based on the instance-dependent cost matrix for transfer fraud detection, that allows for making the optimal cost-based decision for each transaction. Two novel classifiers are presented, based on lasso-regularized logistic regression and gradient tree boosting, which directly minimize the proposed instance-dependent cost measure when learning a classification model. The proposed methods are implemented in the R packages cslogit and csboost, and compared against state-of-the-art methods on a publicly available data set from the machine learning competition website Kaggle and a proprietary card transaction data set. The results of the experiments highlight the potential of reducing fraud losses by adopting the proposed methods.

Cost-sensitive learning for semi-supervised hit-and-run analysis

Hit-and-run crashes not only degrade the morality, but also result in delays of medical services provided to victims. However, class imbalance problem exists as the number of hit-and-run crashes is much smaller than that of non-hit-and-run crashes. The missing label problem also exists in the crash analysis due to reasons like data barrier such that the information hidden in the unlabelled samples has not been effectively utilised. In this paper, a cost-sensitive semi-supervised logistic regression (CS3LR) model is proposed for hit-and-run analysis, in order to tackle class-imbalanced data distribution and missing label problem, based on the crash dataset of Victorian, Australia (2013–2019). By performing label estimation with logistic regression jointly utilising both labelled and unlabelled data with pseudo labels in a well-designed cost-sensitive semi-supervised maximum likelihood framework, the proposed model can obtain an unbiased likelihood parameter for hit-and-run prediction and analysis. Comparing the experimental results of CS3LR model with two logistic regression models and seven machine learning methods, better performance of CS3LR model is demonstrated. The most significant contributing factors to hit-and-run crashes extracted by CS3LR with only 10% labelled data show a high degree of consistency with the true contributing factors obtained by the supervised cost-sensitive logistic regression with complete hit-and-run labels. The effects of class-weighted ratio and hyper-parameter λ on the performance of hit-and-run crash prediction model have also been analysed. The results can further provide recommendations and implications on the policies and counter-measures for preventing hit-and-run collisions and crimes. The methodology proposed in this paper can also be employed to analyse crash data with other types of missing labels, such as crash severity.