Unifying attribute splitting criteria of decision trees by Tsallis entropy

Abstract

Owing to its simplicity and flexibility, the decision tree remains an important analysis tool in many real-world learning tasks. A lot of decision tree algorithms have been proposed, such as ID3, C4.5 and CART, which represent three most prevalent criteria of attribute splitting, i.e., Shannon entropy, Gain Ratio and Gini index respectively. These splitting criteria seem to be independent and to work in isolation. However, in this paper, we find that these three attribute splitting criteria can be unified in a Tsallis entropy framework. More importantly, theoretically, we reveal the relations between Tsallis entropy and the above three prevalent attribute splitting criteria. In addition, we generalize the splitting criterion of the decision tree, and provide a new simple but efficient approach, Unified Tsallis Criterion Decision Tree algorithm (UTCDT), to enhance the performance of the decision tree. Experimental evidences demonstrate that UTCDT achieves statistically significant improvement over the classical decision tree algorithms, even yields comparable performance to state-of-the-art classification algorithm.