Learning with continuous piecewise linear decision trees

Abstract

In this paper, we propose a piecewise linear decision tree and its generalized form, namely the (G)PWL-DT, which introduces piecewise linearity and overcomes the discontinuity of the existing piecewise constant decision trees (PWC-DT). The proposed (G)PWL-DT inherits the basic topology and interpretability of decision trees by recursively partitioning the domain into subregions, which are represented by leaf nodes. Rather than the indicator function, the (G)PWL-DT employs rectifier linear units (ReLU) to interpret domain partitions, where the nested ReLUs are combined to formulate the corresponding PWL decision rules. Due to the piecewise linearity of each leaf node, additional boundaries among linear areas are obtained to approach greater flexibility than the existing PWC-DT under the same tree structure, where the continuity can also be guaranteed. Then, an optimization algorithm is constructed analogously based on the second-order approximation. The proposed (G)PWL-DT can be flexibly applied as a novel decision tree in different tree learning methods and it can also be regarded as a simple extension of ReLUs to the framework of tree learning. Numerical experiments verify the effectiveness of the proposed (G)PWL-DT and its potential as an alternative of the existing PWC-DT to approach better performance even with more concise structures.