Abstract
Consider the classical supervised learning problem: we are given data (yi,xi), i≤n, with yi a response and xi∈X a covariates vector, and try to learn a model fˆ:X→R to predict future responses. Random feature methods map the covariates vector xi to a point ϕ(xi) in a higher dimensional space RN, via a random featurization map ϕ. We study the use of random feature methods in conjunction with ridge regression in the feature space RN. This can be viewed as a finite-dimensional approximation of kernel ridge regression (KRR), or as a stylized model for neural networks in the so called lazy training regime. We define a class of problems satisfying certain spectral conditions on the underlying kernels, and a hypercontractivity assumption on the associated eigenfunctions. These conditions are verified by classical high-dimensional examples. Under these conditions, we prove a sharp characterization of the error of random feature ridge regression. In particular, we address two fundamental questions: (1) What is the generalization error of KRR? (2) How big N should be for the random feature approximation to achieve the same error as KRR? In this setting, we prove that KRR is well approximated by a projection onto the top ℓ eigenfunctions of the kernel, where ℓ depends on the sample size n. We show that the test error of random feature ridge regression is dominated by its approximation error and is larger than the error of KRR as long as N≤n1−δ for some δ>0. We characterize this gap. For N≥n1+δ, random features achieve the same error as the corresponding KRR, and further increasing N does not lead to a significant change in test error.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.