Cost-sensitive Classification Research Articles

Solving the Imbalanced and Limited Data Labeled for Automated Essay Scoring using Cost Sensitive XGBoost and Pseudo-Labeling

There are two main problems on forming the Automatic Essay Scoring Model. They are the datasets having imbalanced amount of the right and wrong answers and the minimal use of labeled data in the model training. The model forming based on these problems is divided into three main points, namely word representation, Cost-Sensitive XGBoost Classification, and adding unlabeled data with the Pseudo-Labeling Technique. The essay answer data is converted into a vector using the trained word vector fastText. Furthermore, the classification of unlabeled data was carried out using the Cost-Sensitive XGBoost Method. The data labeled by the classification model is added as training data for the new classification model form. The process is carried out iteratively. This research is about using the combination of Cost-Sensitive XGBoost Classification and Pseudo-Labeling which is expected to solve the problems. For the 0th iteration, the dataset having a ratio of the amount of "right" labeled data with the amount of "right" labeled data is close to 1, in other words a balanced dataset or a ratio that is more than 1 produces a model with better performance. Thus, the selection of training data at an early stage must pay attention to this ratio. In addition, the use of the Hybrid Method on these datasets can save labeled data 56 times compared to the AdaBoost Method. Hybrid model is able to produce F1-Measure more than 95.6%, so it can be concluded that the Hybrid Method, which combines the XGBoost and Pseudo-Labeling Cost-Sensitive Classification with Self Training, is able to overcome the problem of unbalanced datasets and data limited label.

Method of non-invasive parameters for predicting the probability of early in-hospital death of patients in intensive care unit

ObjectivesTo achieve timely and accurate prediction of the death probability of intensive care unit (ICU) patients without the invasive physiological parameters of professional medical staff and patients, a new method for predicting the death probability of ICU patients based on Extreme Gradient Boosting (XGBoost) machine learning is proposed. Methods15 non-invasive physiological parameters all from Medical Information Mart for Intensive Care III (MIMIC III) database were used in this study, and a total of 151 characteristic variables were extracted. And based on the XGBoost machine learning method, the ten-fold cross-validation method and the cost-sensitive classification threshold adjustment method are used to optimize the model parameters to establish the optimal prediction model of patient death probability. ResultIn the case of using the entire feature set, the optimal prediction results of the XGBoost machine learning model are 0.858 for accuracy (ACC) and 0.941 for area under receiver operating characteristic curve (AUC). Better than traditional scoring methods (the ACC and AUC of the optimal prediction results of the traditional scoring method are 0.736 and 0.816, respectively). ConclusionsCompared with the traditional scoring system, the machine learning algorithm proposed in this study uses the patient's non-invasive physiological parameters, and can provide a more accurate prediction of the death probability of ICU patients in a timely manner without medical professionals and invasive physiological parameters. This makes it possible for the death probability of ICU patients’ prediction to be widely used and timely and appropriate clinical decision-making possible.

Apportioned margin approach for cost sensitive large margin classifiers

We consider the problem of cost sensitive multiclass classification, where we would like to increase the sensitivity of an important class at the expense of a less important one. We adopt an apportioned margin framework to address this problem, which enables an efficient margin shift between classes that share the same boundary. The decision boundary between all pairs of classes divides the margin between them in accordance with a given prioritization vector, which yields a tighter error bound for the important classes while also reducing the overall out-of-sample error. In addition to demonstrating an efficient implementation of our framework, we derive generalization bounds, demonstrate Fisher consistency, adapt the framework to Mercer’s kernel and to neural networks, and report promising empirical results on all accounts.

Cost-sensitive meta-learning framework

PurposeThis paper aims to describe the use of a meta-learning framework for recommending cost-sensitive classification methods with the aim of answering an important question that arises in machine learning, namely, “Among all the available classification algorithms, and in considering a specific type of data and cost, which is the best algorithm for my problem?”Design/methodology/approachThis paper describes the use of a meta-learning framework for recommending cost-sensitive classification methods for the aim of answering an important question that arises in machine learning, namely, “Among all the available classification algorithms, and in considering a specific type of data and cost, which is the best algorithm for my problem?” The framework is based on the idea of applying machine learning techniques to discover knowledge about the performance of different machine learning algorithms. It includes components that repeatedly apply different classification methods on data sets and measures their performance. The characteristics of the data sets, combined with the algorithms and the performance provide the training examples. A decision tree algorithm is applied to the training examples to induce the knowledge, which can then be used to recommend algorithms for new data sets. The paper makes a contribution to both meta-learning and cost-sensitive machine learning approaches. Those both fields are not new, however, building a recommender that recommends the optimal case-sensitive approach for a given data problem is the contribution. The proposed solution is implemented in WEKA and evaluated by applying it on different data sets and comparing the results with existing studies available in the literature. The results show that a developed meta-learning solution produces better results than METAL, a well-known meta-learning system. The developed solution takes the misclassification cost into consideration during the learning process, which is not available in the compared project.FindingsThe proposed solution is implemented in WEKA and evaluated by applying it to different data sets and comparing the results with existing studies available in the literature. The results show that a developed meta-learning solution produces better results than METAL, a well-known meta-learning system.Originality/valueThe paper presents a major piece of new information in writing for the first time. Meta-learning work has been done before but this paper presents a new meta-learning framework that is costs sensitive.

OCEAn: Ordinal classification with an ensemble approach

Generally, classification problems catalog instances according to their target variable without considering the relation among the different labels. However, there are real problems in which the different values of the class are related to each other. Because of interest in this type of problem, several solutions have been proposed, such as cost-sensitive classifiers. Ensembles have proven to be very effective for classification tasks; however, as far as we know, there are no proposals that use a genetic-based methodology as the metaheuristic to create the models. In this paper, we present OCEAn, an ordinal classification algorithm based on an ensemble approach, which makes a final prediction according to a weighted vote system. This weighted voting takes into account weights obtained by a genetic algorithm that tries to minimize the cost of classification. To test the performance of this approach, we compared our proposal with ordinal classification algorithms in the literature and demonstrated that, indeed, our approach improves on previous results.

Three-way decision and conformal prediction: Isomorphisms, differences and theoretical properties of cautious learning approaches

The aim of this article is to study the relationship between two popular Cautious Learning approaches, namely: Three-way decision (TWD) and conformal prediction (CP). Based on the novel proposal of a technique to transform three-way decision classifiers into conformal predictors, and vice versa, we provide conditions for the equivalence between TWD and CP. These theoretical results provide error-bound guarantees for TWD, together with a formal construction to define cost-sensitive cautious classifiers based on CP. The proposed techniques are then applied and evaluated on a collection of benchmark and real-world datasets. The results of the experiments show that the proposed techniques can be used to obtain cautious learning classifiers that are competitive with, and often out-perform, state-of-the-art approaches. Further, through a qualitative medical case study we discuss the usefulness of cautious learning in the development of robust Machine Learning.

A Novel Example-Dependent Cost-Sensitive Stacking Classifier to Identify Tax Return Defaulters

Tax evasion refers to an entity indulging in illegal activities to avoid paying their actual tax liability. A tax return statement is a periodic report comprising information about income, expenditure, etc. One of the most basic tax evasion methods is failing to file tax returns or delay filing tax return statements. The taxpayers who do not file their returns, or fail to do so within the stipulated period are called tax return defaulters. As a result, the Government has to bear the financial losses due to a taxpayer defaulting, which varies for each taxpayer. Therefore, while designing any statistical model to predict potential return defaulters, we have to consider the real financial loss associated with the misclassification of each individual. This paper proposes a framework for an example-dependent cost-sensitive stacking classifier that uses cost-insensitive classifiers as base generalizers to make predictions on the input space. These predictions are used to train an example-dependent cost-sensitive meta generalizer. Based on the meta-generalizer choice, we propose four variant models used to predict potential return defaulters for the upcoming tax-filing period. These models have been developed for the Commercial Taxes Department, Government of Telangana, India. Applying our proposed variant models to GST data, we observe a significant increase in savings compared to conventional classifiers. Additionally, we develop an empirical study showing that our approach is more adept at identifying potential tax return defaulters than existing example-dependent cost-sensitive classification algorithms.

Intrusion detection in computer networks using a cost sensitive ensemble classifier

The growing use of Internet technology and the attack on computer networks have made intrusion detection systems an essential part of computer security. Conventional intrusion control methods such as firewalls or access control systems are no longer alone able to withstand attacks. Therefore, the need to detect new attacks and anomalies is inevitable. The dataset used in this paper is called NSL-KDD which includes 5 classes: one of them is normal and the other four classes are attacks. In the presented work, an ensemble classifier based on the mean probability of attacks is adopted. The true detection rate of the proposed system is $99.89%$ which is more than other competing methods. Moreover, the ensemble classifier achieved an F1-measure of $92.48%$. To improve the F1 measure, we used a meta-classifier called meta-cost which incorporates a cost matrix to transform the original classifier into a cost-sensitive classifier. By this idea, we achieved an F1-measure of $94.1%$ which outperforms than non-cost sensitive ensemble classifier. These results show that the proposed system can be used as a suitable defence tool to detect intrusion against cyber-attacks.

Fusion Algorithm for Imbalanced EEG Data Processing in Seizure Detection

Purpose: Seizure detection algorithms (SDAs) based on electroencephalography (EEG) have been described in previous studies, but the imbalanced data distribution of ictal and interictal states continue to pose a technical challenge. This study proposes a novel algorithm to address the imbalanced classification problem and improve seizure detection performance. Method: The proposed algorithm is designed based on hybrid sampling and a cost-sensitive (CS) classifier. Hybrid sampling resamples the imbalanced EEG data at data-level, and the CS classifier, which is used as an algorithm-level tool, reduces the overall misclassification cost of seizure detection. The synthetic minority oversampling technique and undersampling TomekLink technique are combined to reduce the imbalanced ratio between ictal and interictal states while retaining the generalization ability. Finally, CS support vector machine classifies the resampled EEG feature vectors, assigning different cost sensitive parameters to moderate the poor performance resulting from the imbalanced distribution problem. Result: The proposed algorithm improved the average sensitivity and AUC by 46.67% and 0.0482, respectively, compared with the original results without using the imbalanced EEG data processing (IEDP) technique. Experimental results showed that an average sensitivity and AUC of 86.34% and 0.9837, respectively, could be obtained across all cases. Finally, a performance evaluation showed that the proposed algorithm outperformed published methods in terms seizure detection. Conclusion: A fusion algorithm combining data- and algorithm-level methods can achieve high sensitivity and AUC compared with existing IEDP methods. Thus, SDA performance can be improved, enabling their clinical use with EEG-based SMSs.

A cost-sensitive ensemble deep forest approach for extremely imbalanced credit fraud detection

Credit fraud detection modeling helps prevent default risks and reduce economic losses, and increasingly sophisticated methods have been designed for predicting the default probability of clients. In such problems, the fact that the class of fraud clients is much smaller than the class of good clients makes it a challenge to detect the fraud class. To minimize the financial losses in extremely imbalanced datasets, this paper delivers a novel cost-sensitive ensemble model under the framework of deep forest. The model first introduces a cost-sensitive strategy to assign a higher cost to the fraud class, thereby improving the attention of the model to the fraud samples. As everyone knows, for the basic classifiers of ensemble learning, the greater their differences, the better the performance after ensemble. So the model adds superior cost-sensitive base classifiers into the cascade structure to improve the overall performance. The model also introduces Type II error as the convergence index to automatically adjust the depth of the cascade structure. The experiments conducted on the European credit dataset and a private electronic transaction dataset are presented to demonstrate the performance of the proposed method. The results indicate that the proposed model outperforms most benchmarks in detecting fraud samples.

Instance-dependent cost-sensitive learning for detecting transfer fraud

Card transaction fraud is a growing problem affecting card holders worldwide. Financial institutions increasingly rely upon data-driven methods for developing fraud detection systems, which are able to automatically detect and block fraudulent transactions. From a machine learning perspective, the task of detecting fraudulent transactions is a binary classification problem. Classification models are commonly trained and evaluated in terms of statistical performance measures, such as likelihood and AUC, respectively. These measures, however, do not take into account the actual business objective, which is to minimize the financial losses due to fraud. Fraud detection is to be acknowledged as an instance-dependent cost-sensitive classification problem, where the costs due to misclassification vary between instances, and requiring adapted approaches for learning a classification model. In this article, an instance-dependent threshold is derived, based on the instance-dependent cost matrix for transfer fraud detection, that allows for making the optimal cost-based decision for each transaction. Two novel classifiers are presented, based on lasso-regularized logistic regression and gradient tree boosting, which directly minimize the proposed instance-dependent cost measure when learning a classification model. The proposed methods are implemented in the R packages cslogit and csboost, and compared against state-of-the-art methods on a publicly available data set from the machine learning competition website Kaggle and a proprietary card transaction data set. The results of the experiments highlight the potential of reducing fraud losses by adopting the proposed methods.

PULNS: Positive-Unlabeled Learning with Effective Negative Sample Selector

Positive-unlabeled learning (PU learning) is an important case of binary classification where the training data only contains positive and unlabeled samples. The current state-of-the-art approach for PU learning is the cost-sensitive approach, which casts PU learning as a cost-sensitive classification problem and relies on unbiased risk estimator for correcting the bias introduced by the unlabeled samples. However, this approach requires the knowledge of class prior and is subject to the potential label noise. In this paper, we propose a novel PU learning approach dubbed PULNS, equipped with an effective negative sample selector, which is optimized by reinforcement learning. Our PULNS approach employs an effective negative sample selector as the agent responsible for selecting negative samples from the unlabeled data. While the selected, likely negative samples can be used to improve the classifier, the performance of classifier is also used as the reward to improve the selector through the REINFORCE algorithm. By alternating the updates of the selector and the classifier, the performance of both is improved. Extensive experimental studies on 7 real-world application benchmarks demonstrate that PULNS consistently outperforms the current state-of-the-art methods in PU learning, and our experimental results also confirm the effectiveness of the negative sample selector underlying PULNS.

Privacy-Preserving Cost-Sensitive Learning.

Cost-sensitive learning methods guaranteeing privacy are becoming crucial nowadays in many applications where increasing use of sensitive personal information is observed. However, there has no optimal learning scheme developed in the literature to learn cost-sensitive classifiers under constraint of enforcing differential privacy. Our approach is to first develop a unified framework for existing cost-sensitive learning methods by incorporating the weight constant and weight functions into the classical regularized empirical risk minimization framework. Then, we propose two privacy-preserving algorithms with output perturbation and objective perturbation methods, respectively, to be integrated with the cost-sensitive learning framework. We showcase how this general framework can be used analytically by deriving the privacy-preserving cost-sensitive extensions of logistic regression and support vector machine. Experimental evidence on both synthetic and real data sets verifies that the proposed algorithms can reduce the misclassification cost effectively while satisfying the privacy requirement. A theoretical investigation is also conducted, revealing a very interesting analytic relation, i.e., that the choice of the weight constant and weight functions does not only influence the Fisher-consistent property (population minimizer of expected risk with a specific loss function leads to the Bayes optimal decision rule) but also interacts with privacy-preserving levels to affect the performance of classifiers significantly.

Robust SVM for Cost-Sensitive Learning

Although the performance of cost-sensitive support vector machine (CS-SVM) has been demonstrated to approximate to the cost-sensitive Bayes risk, previous CS-SVM methods still suffer from the influence of outlier samples and redundant features. Recently, a few studies have focused on separately solving these two issues by the sparse theory. In this paper, we propose a new robust cost-sensitive support vector machine to simultaneously solve them in a unified framework. To do this, we employ robust statistics and sparse theory, respectively, to take the sample importance and the feature importance into account, for avoiding the influence of outliers and redundant features. Furthermore, we propose a new optimization method to solve the primal problem of our proposed objective function. Experimental results on synthetic and real data sets show that our proposed method outperforms all the comparison methods in terms of cost-sensitive classification.

Mapping of tank silt application using Sentinel-2 images over the Berambadi catchment (India)

Mapping soil properties is becoming more and more challenging due to the increase in anthropogenic modification of the landscape, calling for new methods to identify these changes. A striking example of anthropogenic modifications of soil properties is the widespread practice in South India of applying large quantities of silt from dry river dams (or “tanks”) to agricultural fields. Whereas several studies have demonstrated the interest of tank silt for soil fertility, no assessment of the actual extent of this age-old traditional practice exists. Over South-Indian pedological context, this practice is characterized by an application of black-colored tank silt to red-colored soils such as Ferralsols. The objective of this work was to evaluate the usefulness of Sentinel-2 images for mapping tank silt applications, hypothesizing that observed changes in soil surface color can be a proxy for tank silt application. We used data collected in a cultivated watershed in South India including 217 soil surface samples characterized in terms of Munsell color. We used two Sentinel-2 images acquired on February and April 2017. The surface soil color over each Sentinel-2 image was classified into two soil types (“Black” and “Red” soils). A change of soil color from “Red” in February 2017 to “Black” in April 2017 was attributed to tank silt application. Soil color changes were analyzed accounting for possible surface soil moisture changes. The proposed methodology was based on a well-balanced Calibration data created from the initial imbalanced Calibration dataset thanks to the Synthetic Minority Over-sampling Technique (SMOTE) methodology, coupled to the Cost-Sensitive Classification And Regression Trees (Cost-Sensitive CART) algorithm. To estimate the uncertainties of i) the two-class classification at each date and ii) the change of soil color from “Red” to “Black”, a bootstrap procedure was used providing fifty two-class classifications for each Sentinel-2 image. The results showed that 1) the CART method allowed to classify the “Red” and “Black” soil with correct overall accuracy from both Sentinel-2 images, 2) the tank silt application was identified over 202 fields and 3) the soil color changes were not related to a surface soil moisture change between both dates. With the actual availability of the Sentinel-2 and the past availability of the LANDSAT satellite imageries, this study may open a way toward a simple and accurate method for delivering tank silt application mapping and so to study and possibly quantify retroactively this farmer practice.

Developing Interval-Based Cost-Sensitive Classifiers by Genetic Programming for Binary High-Dimensional Unbalanced Classification [Research Frontier

Cost-sensitive learning is a popular approach to addressing the problem of class imbalance for many classification algorithms in machine learning. However, most cost-sensitive algorithms are dependent on manually designed cost matrices. Unfortunately, in many cases, it is often not easy for humans, even experts, to accurately specify misclassification costs for different mistakes due to the lack of domain knowledge related to actual situations in some complex unbalanced problems. As a result, these cost-sensitive algorithms cannot be directly applied. This paper proposes a new genetic programmingbased approach to developing cost-sensitive classifiers that are independent of manually designed cost matrices. The proposed method is able to construct classifiers and learn cost intervals automatically and simultaneously. In the proposed method, a tree representation, terminal sets and function sets are designed and developed. We examine the effectiveness of the proposed method on ten high-dimensional unbalanced datasets. The experimental results show that the proposed method often outperforms compared methods for highdimensional unbalanced classification. Furthermore, according to the analysis of evolved trees, the constructed classifiers often only need a small number of features to achieve a good classification performance.

Cardiotocography signal abnormality classification using time-frequency features and Ensemble Cost-sensitive SVM classifier

BackgroundCardiotocography (CTG) signal abnormality classification plays an important role in the diagnosis of abnormal fetuses. This classification problem is made difficult by the non-stationary nature of CTG and the dataset imbalance. This paper introduces a novel application of Time-frequency (TF) features and Ensemble Cost-sensitive Support Vector Machine (ECSVM) classifier to tackle these problems. MethodsFirstly, CTG signals are converted into TF-domain representations by Continuous Wavelet Transform (CWT), Wavelet Coherence (WTC), and Cross-wavelet Transform (XWT). From these representations, a novel image descriptor is used to extract the TF features. Then, the linear feature is derived from the time-domain representation of the CTG signal. The linear and TF features are fed to the ECSVM classifier for prediction and classification of fetal outcome. ResultsThe TF features show the significant difference (p-value<0.05) in distinguishing abnormal CTG signals, but not for traditional nonlinear features. In ECSVM abnormality classification, using only linear features, the sensitivity, specificity, and quality index are 59.3%, 78.3%, and 68.1%, respectively, whereas more effective results (sensitivity: 85.2%, specificity: 66.1%, and quality index: 75.0%) are obtained using a combination of linear and TF features, with a performance improvement index of 10.1%. Especially, the area under the receiver operating characteristic curve (0.77 vs. 0.64) is significantly increased with the ECSVM vs. SVM. ConclusionOur method can greatly improve the classification results, especially for sensitivity. It improves the true positive rate of CTG abnormality classification and reduces the false positive rate, which may help detect and treat abnormal fetuses during labor.

Predicting defects in object-oriented software using cost-sensitive classification

In this software era, it is vital to produce reliable and good quality software. Early detection of defects aids in building accurate software with reduced cost and other resources. Researchers have a keen interest in producing machine learning models for effective and accurate software defect prediction in the early stages of software development. Object-oriented metrics of the software are used in developing these models. These models may result in biased predictions owing to the class imbalance problem existing in most of the software datasets. This paper provides an effective defect prediction framework for imbalanced data by employing cost-sensitive classifiers and stable performance measures like GMean, Balance, and AUC. Four decision tree-based classifiers with different cost ratios are investigated to predict defects in three Apache projects. The empirical results are statistically validated using the nonparametric Friedman test and Wilcoxon signed-rank test. The results state with 99% confidence that the predictive capability of J48, AdaBoostM1, Bagging, and RandomSubSpace improved after employing cost-sensitive learning for the four classifiers used in this study.

Cost-Sensitive Siamese Network for PCB Defect Classification.

After the production of printed circuit boards (PCB), PCB manufacturers need to remove defected boards by conducting rigorous testing, while manual inspection is time-consuming and laborious. Many PCB factories employ automatic optical inspection (AOI), but this pixel-based comparison method has a high false alarm rate, thus requiring intensive human inspection to determine whether alarms raised from it resemble true or pseudo defects. In this paper, we propose a new cost-sensitive deep learning model: cost-sensitive siamese network (CSS-Net) based on siamese network, transfer learning and threshold moving methods to distinguish between true and pseudo PCB defects as a cost-sensitive classification problem. We use optimization algorithms such as NSGA-II to determine the optimal cost-sensitive threshold. Results show that our model improves true defects prediction accuracy to 97.60%, and it maintains relatively high pseudo defect prediction accuracy, 61.24% in real-production scenario. Furthermore, our model also outperforms its state-of-the-art competitor models in other comprehensive cost-sensitive metrics, with an average of 33.32% shorter training time.

Cost-Sensitive Classification for Evolving Data Streams with Concept Drift and Class Imbalance.

Class imbalance and concept drift are two primary principles that exist concurrently in data stream classification. Although the two issues have drawn enough attention separately, the joint treatment largely remains unexplored. Moreover, the class imbalance issue is further complicated if data streams with concept drift. A novel Cost-Sensitive based Data Stream (CSDS) classification is introduced to overcome the two issues simultaneously. The CSDS considers cost information during the procedures of data preprocessing and classification. During the data preprocessing, a cost-sensitive learning strategy is introduced into the ReliefF algorithm for alleviating the class imbalance at the data level. In the classification process, a cost-sensitive weighting schema is devised to enhance the overall performance of the ensemble. Besides, a change detection mechanism is embedded in our algorithm, which guarantees that an ensemble can capture and react to drift promptly. Experimental results validate that our method can obtain better classification results under different imbalanced concept drifting data stream scenarios.