Approximation Properties of Ridge Functions and Extreme Learning Machines

Abstract

For a compact set $D\subset\mathbb{R}^{m}$ we consider the problem of approximating a function $f$ over $D$ by sums of ridge functions ${x}\mapsto\varphi({w}^{T}{x})$ with ${w}$ in a given set $\mathcal{W}$. While such sums are dense in $C(D)$ if $\mathcal{W}=\mathbb{R}^{m}$, to understand the effectiveness of these approximations we consider finite $\mathcal{W}$ and estimate the infimum of $\left\Vert f-g\right\Vert _{\infty}$ over $g\in V_{\mathcal{W}}:={span}\left\{ \,{x}\mapsto\varphi({w}^{T}{x})\mid\varphi\in C(\mathbb{R}),\,{w}\in\mathcal{W}\,\right\} $ in terms of the Lipschitz constant of $f$. In particular, we show lower bounds for the worst case approximation of functions of Lipschitz constant one by considering unapproximable functions $f$ ($\left\Vert f-g\right\Vert _{\infty}\geq\left\Vert f\right\Vert _{\infty}$ for any $g\in V_{\mathcal{W}}$) of Lipschitz constant one. Accurate approximations appear to require $\left|\mathcal{W}\right|=\Omega(m^{2})$ for $D$ a unit hypercube in $\mathbb{R}^{m}$: if $\left|\mathcal{W}\right|\leq\frac{1}{2}m(m-1)$ and $m\geq2$, then $\sup_{f}\inf_{g\in V_{\mathcal{W}}}\left\Vert f-g\right\Vert _{\infty}\geq1/\sqrt{m-1}$, where $f$ ranges over Lipschitz functions of Lipschitz constant one. Similar results hold for sums of products of pairs of ridge functions with weight vectors in $\mathcal{W}$.