Comparison of Shannon, Renyi and Tsallis Entropy Used in Decision Trees

Abstract

Shannon entropy used in standard top-down decision trees does not guarantee the best generalization. Split criteria based on generalized entropies offer different compromise between purity of nodes and overall information gain. Modified C4.5 decision trees based on Tsallis and Renyi entropies have been tested on several high-dimensional microarray datasets with interesting results. This approach may be used in any decision tree and information selection algorithm.KeywordsDecision rulesentropyinformation theoryinformation selectiondecision trees