Speeding up L 2-loss support vector regression by random Fourier features

Abstract

To avoid the expensive quadratic programming in the L 2-loss support vector regression (SVR) model, smooth approximation and iteratively reweighted least square (IRLS) techniques were introduced in literature, resulting in smoothed SVR (SSVR) and IRLS-SVR. However, for nonlinear models, SSVR and IRLS-SVR both involve operations with matrices and vectors of the same size as the training set. Thus, as the training set becomes large, nonlinear SSVR and IRLS-SVR both need long training time and large memory. To further alleviate the training cost, this paper projects the original data into a low dimensional space via random Fourier feature. The inner product of the random Fourier features of two data points is approximately the same as the kernel function evaluated at these two data points. Hence, it is possible to use a linear model in the new low dimensional space to approximate the original nonlinear model, and consequently the time/memory efficient linear training algorithms could be applied. This paper applies the idea of random Fourier features to nonlinear SSVR and IRLS-SVR, and our testing results on real-world datasets show that, the introduction of random Fourier features makes SSVR and IRLS-SVR achieve similar prediction accuracy as the original nonlinear version with substantially higher time efficiency.