Abstract
This paper proposes a novel scheme for reduced-rank Gaussian process regression. The method is based on an approximate series expansion of the covariance function in terms of an eigenfunction expansion of the Laplace operator in a compact subset of mathbb {R}^d. On this approximate eigenbasis, the eigenvalues of the covariance function can be expressed as simple functions of the spectral density of the Gaussian process, which allows the GP inference to be solved under a computational cost scaling as mathcal {O}(nm^2) (initial) and mathcal {O}(m^3) (hyperparameter learning) with m basis functions and n data points. Furthermore, the basis functions are independent of the parameters of the covariance function, which allows for very fast hyperparameter learning. The approach also allows for rigorous error analysis with Hilbert space theory, and we show that the approximation becomes exact when the size of the compact subset and the number of eigenfunctions go to infinity. We also show that the convergence rate of the truncation error is independent of the input dimensionality provided that the differentiability order of the covariance function increases appropriately, and for the squared exponential covariance function it is always bounded by {sim }1/m regardless of the input dimensionality. The expansion generalizes to Hilbert spaces with an inner product which is defined as an integral over a specified input density. The method is compared to previously proposed methods theoretically and through empirical tests with simulated and real data.
Highlights
Gaussian processes (GPs, Rasmussen and Williams 2006) are powerful tools for nonparametric Bayesian inference and learning
As can be noted from Eq (20), the basis functions in the reduced-rank approximation do not depend on the hyperparameters of the covariance function
We have proposed a novel approximation scheme for forming approximate eigendecompositions of covariance functions in terms of the Laplace operator eigenbasis and the spectral density of the covariance function
Summary
Gaussian processes (GPs, Rasmussen and Williams 2006) are powerful tools for nonparametric Bayesian inference and learning. This model is often written in the form f ∼ GP(0, k(x, x )), (1). Yi = f (xi ) + εi , To overcome this problem, over the years, several schemes have been proposed. They typically reduce the storage requirements to O(nm) and complexity to O(nm2), where m < n. From a spectral point of view, several of these methods (e.g., SOR, DTC, VAR, FIC) can be interpreted as modifications to the so-called Nyström method (see Baker 1977; Williams and Seeger 2001), a scheme for approximating the eigenspectrum
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
