A fast and robust model selection algorithm for multi-input multi-output support vector machine

Abstract

Multi-Input Multi-Output (MIMO) regression estimation problems widely exist in engineering fields. As an efficient approach for MIMO modeling, multi-dimensional support vector regression, named M-SVR, is generally capable of obtaining better predictions than many traditional methods. However, M-SVR is sensitive to the perturbation of hyper-parameters when facing small-scale sample problems, and most of currently used model selection methods for conventional SVR cannot be applied to M-SVR directly due to its special structure. In this paper, a fast and robust model selection algorithm for M-SVR is proposed. Firstly, a new training algorithm for M-SVR is proposed to reduce efficiently the numerical errors in training procedure. Based on this algorithm, a new leave-one-out (LOO) error estimate for M-SVR is derived through a virtual LOO cross-validation procedure. This LOO error estimate can be straightway calculated once a training process ended with less computational complexity than traditional LOO method. Furthermore, a robust implementation of this LOO estimate via Cholesky factorization is also proposed. Finally, the gradients of the LOO estimate are calculated, and the hyper-parameters with lowest LOO error can be found by means of gradient decent method. Experiments on toy data and real-life dynamical load identification problems are both conducted, demonstrating comparable results of the proposed algorithm in terms of generalization performance, numerical stability and computational cost.