Rule-Extraction from Soft Decision Trees

Abstract

Soft Decision Trees have been proven to have better performance in terms of accuracy and complexity compared to conventional Decision Trees. Nevertheless, they do not possess inherent explanation capability to interpret the learning process of which the conventional Decision Trees are capable because a Soft Decision Tree redirects instances to all its children nodes with probabilities given by a “soft” function. In this paper, we apply a common rule-extraction methodology to Soft Decision Trees and propose a new method to extract rules from Soft Decision Trees in order to better understand the rationale of the learning procedural. This first method can be referred to as the Learning Based Method, which utilizes the labels predicted by a trained Soft Decision Tree model to train a conventional Decision Tree model so that univariate rules can be extracted. We also propose the Sparse Maximum Likelihood (SML) method, which provides sparse multivariate rules based on the architectures of Soft Decision Trees. Since Soft Decision Trees are multivariate (or oblique) Decision Trees,L 1 regularization is utilized in order to make the weight vectors at root node and each internal node sparse. Moreover, due to the fact that Soft Decision Trees redirect instances by probabilities, Maximum Likelihood Estimation (MLE) could provide a way of extracting sparse multivariate rules from Soft Decision Trees based on the trees’ structure. The performances of the rules extracted using these two methods are measured in terms of accuracy and fidelity (the extent to which extracted representations accurately model the learning algorithm) on 16 datasets and comparisons with the rules extracted from two commonly used Decision Trees are presented. In general, the accuracy of rules extracted using Learning Based Method and SML are better than the rules extracted from conventional Decision Trees on the 16 datasets. Both methods have high fidelity.