A forest of trees with principal direction specified oblique split on random subspace

Abstract

No matter whether they are univariate or multivariate decision forests, most of previous decision forests determine their partition hyperplanes at split nodes by exhaustive search from candidates or by random generation, which makes some dent in either efficiency or accuracy. In this paper, we propose a new oblique/multivariate decision forest, a forest of trees with principal direction specified oblique split on random subspace (FPDS), where each split of trees is uniquely deterministic once the random feature subspace is determined, the largest principal direction of Principal Component Analysis (PCA) on the sample data at the corresponding split node and the median value of all the current sample points’ projections on the largest principal direction directly specified as the normal direction and the cut-point of the partition hyperplane. This method avoids either tediously searching for the optimal split or casually randomly generating the split. The heuristic method to obtain the hyperplanes guarantees accuracy of trees, and the random feature subspace selection adequately ensures the diversity among individual trees in the forest. In addition, each tree of the FPDS uses the whole training set instead of the sampling subset. Therefore, the only randomness factor in the FPDS derives from the random feature subspace selection, which to some extent enhances the robustness. It proves that the proposed forest FPDS is an alternative classifier which can match or even outperform the existing ensemble classifiers or other classifiers.