Optimal function approximation with ReLU neural networks

Abstract

In this paper, we consider the optimal approximations of univariate functions with feed-forward ReLU neural networks. We attempt to answer the following questions. For given function and network, what is the minimal possible approximation error? How fast does the optimal approximation error decrease with network size? Is optimal approximation attainable by current network training techniques? Theoretically, we introduce necessary and sufficient conditions for optimal approximations of convex functions. We give lower and upper bounds of optimal approximation errors, and approximation rate that measures how fast approximation error decreases with network size. ReLU network architectures are presented to generate optimal approximations. We then propose an algorithm to compute optimal approximations and prove its convergence. We conduct experiments to validate its effectiveness and compare with other approaches. We also demonstrate that the theoretical limit of approximation errors is not attained by ReLU networks trained with stochastic gradient descent optimization, which indicates that the expressive power of ReLU networks has not been exploited to its full potential.