Distributed Decision Tree

Abstract

Decision Tree is a tree-structured plan of a set of attributes to test in order to predict the output. MapReduce and Spark is a programming model used for processing data on a distributed file system. In this paper, MapReduce and Spark implementation of Decision Tree is named as Distributed Decision Tree (DDT) and Spark Tree (ST) respectively. Decision Tree (DT), Ensemble of Trees (BT), DDT and ST are compared over accuracy, size of tree and number of leaves of tree(s) generated. DDT and ST is empirically evaluated over 10 selected datasets. Using DDT, size of tree is reduced by 71% and 82% as compared to DT and BT respectively. In case of ST size of tree is reduced by 48% and 67% as compared to DT and BT. Number of leaves is reduced by 70% and 81% with respect to DT and BT, respectively using DDT. Whereas, it is reduced by 45% and 65% with respect to DT and BT in case of ST. We evaluated DDT and ST using Yahoo! Webscope dataset. Our evaluation shows improvement in accuracy as well as reduction in size of tree and number of leaves. Hence, DDT and ST outperformed DT and BT with respect to size of tree and number of leaves with adequate classification accuracy.