D2TS: a dual diversity tree selection approach to pruning of random forests

Abstract

Random Forest is one of the most effective classification techniques. It is an ensemble technique with typically decision trees as its classifiers. Each tree votes for an outcome when a new instance is being classified, and a majority vote is taken to decide the final output. Two main factors play an essential role in Random Forests performance, namely diversity among trees in the forest and their number. Higher diversity increases prediction accuracy, whereas lower numbers of trees result in faster predictions. This paper aims at optimizing these two factors by using clustering analysis of trees in order to prune correlated trees while keeping outlier trees to maintain diversity. We group the trees into clusters and only take a number of representatives from each cluster while also keeping some or all of the outliers to preserve diversity. The resulting subset of trees will constitute a random forest of a reduced size. We will use the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm for clustering. DBSCAN is one of the most commonly used clustering techniques and is immune to outliers. We will use DBSCAN to (i) group trees in clusters based on their prediction behaviour and (ii) identify outliers. Each of the clustered and outlier trees bring an element of diversity into the pruned random forest, thus giving our approach its dual diversity aspect. Our approach achieved up to a 99% pruning level while resulting in similar, or even better, accuracy compared to the original forests for 19 public datasets with varying properties. Our source code is publicly available on GitHub.