A Theoretical Study of the Representational Power of Weighted Randomised Univariate Regression Tree Ensembles

Abstract

Univariate regression trees have representation problems for non-orthogonal regression functions. Ensembles of univariate regression trees have better representational power. In some cases, weighted ensembles have shown better performance than unweighted ensembles. In this paper, we study the properties of ensembles of regression trees by using regression classification models. We propose a theoretical framework to study the representational power of infinite-sized weighted ensembles, consisting of randomised finite-sized regression trees. We show for some datasets that the weighted ensembles may have better representational power than unweighted ensembles, but the performance is highly dependent on the weighting scheme and the properties of datasets. Our model cannot be used for all the datasets. However, for some datasets, we can accurately predict the experimental results of ensembles of regression trees.