Designing ensemble learning algorithms using kernel methods

Abstract

Ensemble methods such as boosting combine multiple learners to obtain better prediction than could be obtained from any individual learner. Here we propose a principled framework for directly constructing ensemble learning methods from kernel methods. Unlike previous studies showing the equivalence between boosting and support vector machines (SVMs) which need a translation procedure, we show that it is possible to design boosting-like procedure to solve the SVM optimisation problems. In other words, it is possible to design ensemble methods directly from SVM without any middle procedure. This finding not only enables us to design new ensemble learning methods directly from kernel methods, but also makes it possible to take advantage of those highly-optimised fast linear SVM solvers for ensemble learning. The resulted model is as effective as kernel methods while being as efficient as ensemble methods. We exemplify this framework for designing new binary and multi-class classification ensemble learning as well as a new quantile regression ensemble learning method. Experimental results demonstrate the flexibility and usefulness of the proposed framework.