Kolmogorov width decay and poor approximators in machine learning: shallow neural networks, random feature models and neural tangent kernels

Abstract

We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor \(L^{2}\)-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the \(L^{2}\)-topology.