Efficient large scale kernel ridge regression via ensemble SPSD approximation

Abstract

Kernel Ridge Regression (KRR) is a powerful method for non-parametric regression and has been studied extensively in fields of machine learning and data mining. KRR suffers from cubic computational complexity and quadratic storage complexity w.r.t. the number of training samples. Thus, KRR is difficult to handle large scale data set. The general way of avoiding high complexity is to use single sketching methods, such as Nystrom method and random features, to generate a sketch instead of the full kernel matrix. However, the complexity reduction caused by single sketching method is usually at the expense of approximation accuracy which hurts the regression performance of KRR. In this paper, we propose an efficient large scale KRR based on ensemble symmetric positive semi-definite (SPSD) approximation method (shortened as EKRR). On one hand, the proposed EKRR improves the approximation accuracy for kernel matrix by taking advantage of ensemble sketching method; on the other hand, the proposed EKRR achieves low computational and storage complexity by using an iterative inverse matrix computation process. Empirical experiments show that EKRR outperforms single sketching based regression methods in terms of regression performance with comparable low computational and storage complexity.