Multi-pruning and Restricted Multi-pruning of Decision Trees

Abstract

In this chapter, we consider two questions related to decision trees: (i) how to construct decision trees with reasonable number of nodes and reasonable number of misclassifications when they are used for knowledge representation, and (ii) how to improve the prediction accuracy of decision trees when they are used as classifiers. We created so-called multi-pruning approach based on dynamic programming algorithms for bi-criteria optimization of CART-like decision trees relative to the number of nodes and the number of misclassifications. This approach allows us to construct the set of all Pareto optimal points and to derive, for each such point, decision trees with parameters corresponding to that point. Experiments with decision tables from the UCI ML Repository show that, very often, we can find a suitable Pareto optimal point and derive a decision tree with small number of nodes at the expense of small increment in the number of misclassifications. Multi-pruning approach includes a procedure which constructs decision trees that, as classifiers, often outperform decision trees constructed by CART. We considered a modification of multi-pruning approach (restricted multi-pruning) that requires less memory and time but usually keeps the quality of constructed trees as classifiers or as a way for knowledge representation. Based on the uncertainty measure abs which is applicable both to decision tables with single- and many-valued decisions, we extended the considered approaches to the case of decision tables with many-valued decisions.