Decision Tree Induction Algorithm Research Articles

Improving the accuracy of decision tree induction by feature preselection

Selecting the right set of features for classification is one of the most important problems in designing a good classifier. Decision tree induction algorithms such as C4.5 have incorporated in their learning phase an automatic feature selection strategy, while some other statistical classification algorithms require the feature subset to be selected in a preprocessing phase. It is well known that correlated and irrelevant features may degrade the performance of the C4.5 algorithm. In our study, we evaluated the influence of feature preselection on the prediction accuracy of C4.5 using a real-world data set. We observed that accuracy of the C4.5 classifier could be improved with an appropriate feature preselection phase for the learning algorithm. Beyond that, the number of features used for classification can be reduced, which is important for image interpretation tasks since feature calculation is a time-consuming process.

Ensembles of Classifiers for Morphological Galaxy Classification

We compare the use of three algorithms for performing automated morphological galaxy classification using a sample of 800 galaxies. Classifiers are created using a single training set as well as bootstrap replicates of the training set, producing an ensemble of classifiers. We use a Naive Bayes classifier, a neural network trained with backpropagation, and a decision-tree induction algorithm with pruning. Previous work in the field has emphasized backpropagation networks and decision trees. The Naive Bayes classifier is easy to understand and implement and often works remarkably well on real-world data. For each of these algorithms, we examine the classification accuracy of individual classifiers using 10-fold cross validation and of ensembles of classifiers trained using 25 bootstrap data sets and tested on the same cross-validation test sets. Our results show that (1) the neural network produced the best individual classifiers (lowest classification error) for the majority of cases, (2) the ensemble approach significantly reduced the classification error for the neural network and the decision-tree classifiers but not for the Naive Bayes classifier, (3) the ensemble approach worked better for decision trees (typical error reduction of 12%-23%) than for the neural network (typical error reduction of 7%-12%), and (4) the relative improvement when using ensembles decreases as the number of output classes increases. While more extensive comparisons are needed (e.g., a variety of data and classifiers), our work is the first demonstration that the ensemble approach can significantly increase the performance of certain automated classification methods when applied to the domain of morphological galaxy classification.

Using Decision Tree Induction Systems for Modeling Space‐Time Behavior

Discrete choice models are commonly used to predict individuals' activity and travel choices either separately or simultaneously in activity‐scheduling models. This paper investigates the possibilities of decision tree induction systems as an alternative approach. The ability of decision trees to represent heuristic decision rules is evaluated and a method of capturing interactions across decisions in a sequential decision model is outlined. Decision tree induction algorithms, such as C4.5, CART, and CHAID, are suited to derive the decision rules from empirical data. A case study to illustrate the approach considers decisions of individuals when they are faced with the choice to combine different out‐of‐home activities into a multipurpose, multistop trip or make a trip for each activity separately. Data from a large‐scale activity diary survey are used to induce the decision rules. Possible directions of future research are identified.

Induction of decision trees in numeric domains using set-valued attributes

Conventional algorithms for decision tree induction use an attribute-val ue representatio n scheme for instances. This paper explores the empirical consequences of using set-valued attributes. This simple representational extension, when used as a pre-processor for numeric data, is shown to yield significant gains in accuracy combined with attractive build times. It is also shown to improve the accuracy for the second best classification option, which has valuable ramifications for post-processing. To do so an intuitive and practical version of pre-pruning is employed. Moreover, the implementation of a simple pruning scheme serves as an example of pruning applicability over the resulted trees and also as an indication that the proposed discretization absorbs much of pruning potential. Finally, we construct several versions of the basic algorithm to examine the value of every component that comprises it.

Tree Induction of Spatial Choice Behavior

Discussed is the merit of inductive learning as a tool with which to discover geographic knowledge in data-rich environments, and particularly as an analysis tool in spatial decision-making theory. The capability and applicability of Quinlan’s C4.5 decision tree induction algorithm to the class of problems involving the choice among discrete travel destinations within an urban area are analyzed. The C4.5 algorithm and its relation to other decision tree induction algorithms and to spatial behavioral modeling are described, and its implementation on spatial behavior data from the Minneapolis-St. Paul metropolitan area is illustrated.

Learning Algorithms for Keyphrase Extraction

Many academic journals ask their authors to provide a list of about five to fifteen keywords, to appear on the first page of each article. Since these key words are often phrases of two or more words, we prefer to call them keyphrases. There is a wide variety of tasks for which keyphrases are useful, as we discuss in this paper. We approach the problem of automatically extracting keyphrases from text as a supervised learning task. We treat a document as a set of phrases, which the learning algorithm must learn to classify as positive or negative examples of keyphrases. Our first set of experiments applies the C4.5 decision tree induction algorithm to this learning task. We evaluate the performance of nine different configurations of C4.5. The second set of experiments applies the GenEx algorithm to the task. We developed the GenEx algorithm specifically for automatically extracting keyphrases from text. The experimental results support the claim that a custom-designed algorithm (GenEx), incorporating specialized procedural domain knowledge, can generate better keyphrases than a general-purpose algorithm (C4.5). Subjective human evaluation of the keyphrases generated by GenEx suggests that about 80% of the keyphrases are acceptable to human readers. This level of performance should be satisfactory for a wide variety of applications.

Strategic induction of decision trees

An algorithm for decision-tree induction is presented in which attribute selection is based on the evidence-gathering strategies used by doctors in sequential diagnosis. Since the attribute selected by the algorithm at a given node is often the best attribute according to the Quinlan's information gain criterion, the decision tree it induces is often identical to the ID3 tree when the number of attributes is small. In problem-solving applications of the induced decision tree, an advantage of the approach is that the relevance of a selected attribute or test can be explained in strategic terms. An implementation of the algorithm in an environment providing integrated support for incremental learning, problem solving and explanation is presented.

A mass assignment based ID3 algorithm for decision tree induction

A mass assignment based ID3 algorithm for learning probabilistic fuzzy decision trees is introduced. Fuzzy partitions are used to discretize continuous feature universes and to reduce complexity when universes are discrete but with large cardinalities. Furthermore, the fuzzy partitioning of classification universes facilitates the use of these decision trees in function approximation problems. Generally the incorporation of fuzzy sets into this paradigm overcomes many of the problems associated with the application of decision trees to real-world problems. The probabilities required for the trees are calculated according to mass assignment theory applied to fuzzy labels. The latter concept is introduced to overcome computational complexity problems associated with higher dimensional mass assignment evaluations on databases. ©1997 John Wiley & Sons, Inc.

A mass assignment based ID3 algorithm for decision tree induction

A mass assignment based ID3 algorithm for learning probabilistic fuzzy decision trees is introduced. Fuzzy partitions are used to discretize continuous feature universes and to reduce complexity when universes are discrete but with large cardinalities. Furthermore, the fuzzy partitioning of classification universes facilitates the use of these decision trees in function approximation problems. Generally the incorporation of fuzzy sets into this paradigm overcomes many of the problems associated with the application of decision trees to real-world problems. The probabilities required for the trees are calculated according to mass assignment theory applied to fuzzy labels. The latter concept is introduced to overcome computational complexity problems associated with higher dimensional mass assignment evaluations on databases. ©1997 John Wiley & Sons, Inc.

Cost-sensitive classification

This paper introduces ICET, a new algorithm for cost-sensitive classification. ICET uses a genetic algorithm to evolve a population of biases for a decision tree induction algorithm. The fitness fu...

Cost-Sensitive Classification: Empirical Evaluation of a Hybrid Genetic Decision Tree Induction Algorithm

This paper introduces ICET, a new algorithm for cost-sensitive classification. ICET uses a genetic algorithm to evolve a population of biases for a decision tree induction algorithm. The fitness function of the genetic algorithm is the average cost of classification when using the decision tree, including both the costs of tests (features, measurements) and the costs of classification errors. ICET is compared here with three other algorithms for cost-sensitive classification - EG2, CS-ID3, and IDX - and also with C4.5, which classifies without regard to cost. The five algorithms are evaluated empirically on five real-world medical datasets. Three sets of experiments are performed. The first set examines the baseline performance of the five algorithms on the five datasets and establishes that ICET performs significantly better than its competitors. The second set tests the robustness of ICET under a variety of conditions and shows that ICET maintains its advantage. The third set looks at ICET's search in bias space and discovers a way to improve the search.

Debugging Finite Element Programs Input Data with Machine Learning

Abstract: Debugging of the input data of a structural analysis program is a troublesome task which is heavily dependent on empirical knowledge. The paper describes an effort that applies machine learning to build an expert system for debugging faults in structural analysis program input data. ID3 decision tree induction algorithm is employed to build the automatic learning mechanism. This method Oflkrs an attractive potential for knowledge acquisition in the civil engineering domain in which expertise plays a dominant role. This paper is novel in at least three aspects: (1) machine learning is employed to build expert systems automatically; (2) a novel impurity function is proposed for splitting the decision tree; (3) a formulated comparison is proposed for evaluating learning results.