Lower bounds for approximation by MLP neural networks

Abstract

The degree of approximation by a single hidden layer MLP model with n units in the hidden layer is bounded below by the degree of approximation by a linear combination of n ridge functions. We prove that there exists an analytic, strictly monotone, sigmoidal activation function for which this lower bound is essentially attained. We also prove, using this same activation function, that one can approximate arbitrarily well any continuous function on any compact domain by a two hidden layer MLP using a fixed finite number of units in each layer.