Improving decision trees by Tsallis Entropy Information Metric method

Abstract

The construction of efficient and effective decision trees remains a key topic in machine learning because of their simplicity and flexibility. A lot of heuristic algorithms have been proposed to construct near-optimal decision trees. Most of them, however, are greedy algorithms that have the drawback of obtaining only local optimums. Besides, conventional split criteria they used, e.g. Shannon entropy, Gain Ratio and Gini index, cannot select informative attributes efficiently. To address the above issues, we propose a novel Tsallis Entropy Information Metric (TEIM) algorithm with a new split criterion and a new construction method of decision trees. Firstly, the new split criterion is based on two terms of Tsallis conditional entropy, which is better than conventional split criteria. Secondly, the new construction method is based on a two-stage approach that avoids local optimum to a certain extent. The TEIM algorithm takes advantages of the generalization ability of Tsallis entropy and the low greediness property of two-stage approach. Experimental results on UCI datasets indicate that, compared with the state-of-the-art decision trees algorithms, the TEIM algorithm yields statistically significantly better decision trees in classification accuracy as well as tree complexity.