Cost-sensitive Decision Tree Research Articles

A survey of cost-sensitive decision tree induction algorithms

The past decade has seen a significant interest on the problem of inducing decision trees that take account of costs of misclassification and costs of acquiring the features used for decision making. This survey identifies over 50 algorithms including approaches that are direct adaptations of accuracy-based methods, use genetic algorithms, use anytime methods and utilize boosting and bagging. The survey brings together these different studies and novel approaches to cost-sensitive decision tree learning, provides a useful taxonomy, a historical timeline of how the field has developed and should provide a useful reference point for future research in this field.

Cost-Sensitive Decision Tree Induction for Feature Selection and Sequential Minimal Optimisation for Classification: CSattribSelectorC4.5()

Feature selection is the important step in overall data mining process. There exist several methods for feature selection. This paper discusses numerous methods of feature selection and their impact on the credit approval dataset. Cost-sensitive feature selection using a decision-tree-based algorithm is proposed (CSattribSelectorC4.5). Sequential minimal optimisation (SMO) is used for classification. The classification results show that the CSattribSelectorC4.5 helps SMO to reduce the overall misclassification cost on the selected dataset.

Research of an Algorithm for Generating Cost-Sensitive Decision Tree Based on Attribute Significance

Research of an Algorithm for Generating Cost-Sensitive Decision Tree Based on Attribute Significance

Decision tree classifiers sensitive to heterogeneous costs

There are nine major types of cost involved in cost-sensitive learning that are with heterogeneous units in general, referred to heterogeneous costs. Extant cost-sensitive learning (CSL) algorithms are based on the assumption that all involved costs can be transformed into a unified unit, called as homogeneous assumption of costs. While it is a challenge to construct many suitable transformation functions for the costs with diverse units, this paper designs a heterogeneous-cost sensitive learning (HCSL) algorithm to make split attribute selection more effective. This paper first proposes an efficient method of reducing the heterogeneity caused by both cost mechanisms and attribute information. And then, all heterogeneous costs with attribution information together are incorporated into the process of split attribute selection, called as HCAI-based split attribute selection. Third, the over-fitting is tackled by designing a simple and effective smoothing strategy, so as to build cost-sensitive decision tree classifiers with the HCSL algorithm. Experiments are conducted to evaluate the proposed HCSL algorithm on six UCI datasets. Experimental results show that the proposed approach outperforms existing methods for handling the heterogeneity caused by cost mechanisms and attribute information.

Building cost-sensitive decision trees for medical applications

This thesis presents strategies for cost-sensitive learning. We have developed an algorithm for decision tree induction that considers various types of costs. The main ones were attribute costs and misclassification costs. Other costs included, for instance, the “risk”, that is a measure of how invasive the test is. We applied our strategy to train and to evaluate cost-sensitive decision trees on medical data. The resulting trees provided a better cost-effective solution for a given problem.

Handling over-fitting in test cost-sensitive decision tree learning by feature selection, smoothing and pruning

Cost-sensitive learning algorithms are typically designed for minimizing the total cost when multiple costs are taken into account. Like other learning algorithms, cost-sensitive learning algorithms must face a significant challenge, over-fitting, in an applied context of cost-sensitive learning. Specifically speaking, they can generate good results on training data but normally do not produce an optimal model when applied to unseen data in real world applications. It is called data over-fitting. This paper deals with the issue of data over-fitting by designing three simple and efficient strategies, feature selection, smoothing and threshold pruning, against the TCSDT (test cost-sensitive decision tree) method. The feature selection approach is used to pre-process the data set before applying the TCSDT algorithm. The smoothing and threshold pruning are used in a TCSDT algorithm before calculating the class probability estimate for each decision tree leaf. To evaluate our approaches, we conduct extensive experiments on the selected UCI data sets across different cost ratios, and on a real world data set, KDD-98 with real misclassification cost. The experimental results show that our algorithms outperform both the original TCSDT and other competing algorithms on reducing data over-fitting.

Splitting attribute selection method based on cost performance

Cost-sensitive decision trees usually concern the discussion of the test cost and misclassification cost.During the classification process,splitting attribute selection is the most important.The paper analyzed the disadvantages and the advantages of the existing methods and proposed a novel method that combined the information ratio in information theory with the cost including the test cost and the misclassification cost to select the split attributes.The experimental results show that this method outperforms significantly the existing methods.

Building a cost-constrained decision tree with multiple condition attributes

Costs are often an important part of the classification process. Cost factors have been taken into consideration in many previous studies regarding decision tree models. In this study, we also consider a cost-sensitive decision tree construction problem. We assume that there are test costs that must be paid to obtain the values of the decision attribute and that a record must be classified without exceeding the spending cost threshold. Unlike previous studies, however, in which records were classified with only a single condition attribute, in this study, we are able to simultaneously classify records with multiple condition attributes. An algorithm is developed to build a cost-constrained decision tree, which allows us to simultaneously classify multiple condition attributes. The experimental results show that our algorithm satisfactorily handles data with multiple condition attributes under different cost constraints.

Test strategies for cost-sensitive decision trees

In medical diagnosis, doctors must often determine what medical tests (e.g., X-ray and blood tests) should be ordered for a patient to minimize the total cost of medical tests and misdiagnosis. In this paper, we design cost-sensitive machine learning algorithms to model this learning and diagnosis process. Medical tests are like attributes in machine learning whose values may be obtained at a cost (attribute cost), and misdiagnoses are like misclassifications which may also incur a cost (misclassification cost). We first propose a lazy decision tree learning algorithm that minimizes the sum of attribute costs and misclassification costs. Then, we design several novel "test strategies" that can request to obtain values of unknown attributes at a cost (similar to doctors' ordering of medical tests at a cost) in order to minimize the total cost for test examples (new patients). These test strategies correspond to different situations in real-world diagnoses. We empirically evaluate these test strategies, and show that they are effective and outperform previous methods. Our results can be readily applied to real-world diagnosis tasks. A case study on heart disease is given throughout the paper

"Missing is useful": missing values in cost-sensitive decision trees

Many real-world data sets for machine learning and data mining contain missing values and much previous research regards it as a problem and attempts to impute missing values before training and testing. In this paper, we study this issue in cost-sensitive learning that considers both test costs and misclassification costs. If some attributes (tests) are too expensive in obtaining their values, it would be more cost-effective to miss out their values, similar to skipping expensive and risky tests (missing values) in patient diagnosis (classification). That is, "missing is useful" as missing values actually reduces the total cost of tests and misclassifications and, therefore, it is not meaningful to impute their values. We discuss and compare several strategies that utilize only known values and that "missing is useful" for cost reduction in cost-sensitive decision tree learning.

Inducing safer oblique trees without costs

Abstract: Decision tree induction has been widely studied and applied. In safety applications, such as determining whether a chemical process is safe or whether a person has a medical condition, the cost of misclassification in one of the classes is significantly higher than in the other class. Several authors have tackled this problem by developing cost-sensitive decision tree learning algorithms or have suggested ways of changing the distribution of training examples to bias the decision tree learning process so as to take account of costs. A prerequisite for applying such algorithms is the availability of costs of misclassification. Although this may be possible for some applications, obtaining reasonable estimates of costs of misclassification is not easy in the area of safety. This paper presents a new algorithm for applications where the cost of misclassifications cannot be quantified, although the cost of misclassification in one class is known to be significantly higher than in another class. The algorithm utilizes linear discriminant analysis to identify oblique relationships between continuous attributes and then carries out an appropriate modification to ensure that the resulting tree errs on the side of safety. The algorithm is evaluated with respect to one of the best known cost-sensitive algorithms (ICET), a well-known oblique decision tree algorithm (OC1) and an algorithm that utilizes robust linear programming.

CC4.5: Cost-sensitive Decision Tree Pruning

There are many methods to prune decision trees, but the idea of cost-sensitive pruning has received much less investigation even though additional flexibility and increased performance can be obtained from this method. In this paper, we introduce a cost-sensitive decision tree pruning algorithm called CC4.5 based on the C4.5 algorithm. This algorithm uses the same method as C4.5 to construct the original decision tree, but the pruning methods in CC4.5 are different from that in C4.5. CC4.5 includes three cost-sensitive pruning methods to deal with misclassification cost in the decision tree. Unlike many other pruning algorithms, CC4.5 uses intelligent inexact classification to consider both error and cost when pruning. Moreover, experiments show that CC4.5 results in improved decision trees with respect to the cost and its comprehensibility and accuracy are also satisfactory.

Automated detection of immunofluorescently labeled cytomegalovirus-infected cells in isolated peripheral blood leukocytes using decision tree analysis.

Cytomegalovirus (CMV) infection continues to be a major problem for immunocompromised patients. Detection of viral antigens in leukocytes (antigenemia assay) is widely used for the diagnosis of CMV infection and for guiding antiviral therapy. The antigenemia technique, contingent upon the manual microscopic analysis of rare cells, is a laborious task that is subject to human error. In this study, we combine automated microscopy with artificial intelligence for reliable detection of fluorescently labeled CMV-infected cells. Cytospin preparations of peripheral blood leukocytes were immunofluorescently labeled for the CMV lower matrix phosphoprotein (pp65) and scanned in the Rare Event Imaging System (REIS), a fully automated image cytometer. The REIS detected potential positive objects and digitally recorded 49 measured cellular features for each identified case. The measurement data of these objects were analyzed by the See5 decision tree (DT) algorithm to ascertain whether they were true-positive detections. The DT was built from the measurement data of 2,047 true- and 2,028 false-positive detections, collected from 32 patient samples. By designating misclassifications of false-negatives three times more costly, the 10-fold cross-validation sensitivity, specificity, and misclassification error of the assay was 94.3%, 56.2%, and 25%, respectively. The method was also validated using an independent test set of 21 patient samples, in which similar results were obtained. To our knowledge, this study represents the first attempt to improve the accuracy of rare event image cytometry through the implementation of artificial intelligence methodology. Results suggest that cost-sensitive decision tree analysis of digitally measured cellular features vastly improves the performance of rare event image cytometry.