Curvature-Oriented Splitting for Multivariate Model Trees

Abstract

Model trees typically partition the input space by axis-orthogonal splits into local regions for local regression. Due to these splits, properties of the function to be approximated can only be taken into account to a limited extent, which increases the bias of model trees. To minimize this bias, multivariate model trees with axis-oblique splits can be used. However, existing methods to build axis-oblique splits are either not applicable, ineffective, or limited to certain data distributions and properties of the approximated function. In this work, we present a novel method for axis-oblique splitting that overcomes these drawbacks by using the average direction of non-linearity of the function. This direction is estimated by extending the dimensionality reduction method refined Outer Product of Gradients. Moreover, we present a tree construction algorithm in which our method is integrated, and evaluate the resulting multivariate model tree COMT on synthetic data in an experimental study. In this study, we compare COMT and our split method in terms of prediction accuracy with common regression models and a proven method to identify axis-oblique split directions called Principal Hessian Directions. We also analyze the improvements in bias and variance by our axis-oblique splitting over conventional (univariate) model trees and multivariate regression trees. On the used data, COMT outperforms the competitors in terms of prediction accuracy, and our split method significantly reduces bias and variance.