Estimating ensemble weights for bagging regressors based on the mean–variance portfolio framework

Abstract

This paper presents a novel ensemble learning framework inspired by modern portfolio optimization to address regression problems. This formulation in the ensemble learning field allows determining the ensemble’s weights considering the error predictions of the base learners, the variability in their predictions, and the covariance among their predictions’ errors, which is completely aligned with the bias–variance–covariance theory. Under the framework proposed, four potential instantiations have also been provided. The first two ensemble models impose the non-negativity constraints on the ensemble’s weights (along with the equality constraint that the ensemble’s weights sum up to one) and are solved with the active set algorithm. The second two ensemble models do not include the non-negativity constraints on the ensemble’s weights (which in the financial literature are called the non-shorting constraints), giving rise to a convex quadratic programming (QP) problem (as the matrix included in the quadratic term is symmetric and positive definite) that is solved by the Lagrangian procedure. Extensive experiments with regression datasets evaluate the proposed ensemble framework. Comparisons with other state-of-the-art ensemble methods confirm that the ensemble framework yields the best overall performance.