Abstract
In this paper, it is shown that C_beta -smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters {0,pm frac{1}{2}, pm 1, 2}. The l_0 and l_1 parameter norms of considered networks are thus equivalent. The depth, the width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in [-1,1]. In particular, this implies that the nonparametric regression estimation with constructed networks achieves, up to logarithmic factors, the same minimax convergence rates as with sparse networks with parameters in [-1,1].
Highlights
The problem of function approximation with neural networks has been of big interest in mathematical research for the last several decades
There are at most 1/ε2 parameters outside of the interval (−ε2, ε2); an entropy bound of order O((2/L)2L−1/ε2) has been obtained by taking in the covering networks the remaining parameters to be 0
One of the ways to approximate functions by neural networks is based on the neural network approximation of local Taylor polynomials of those functions
Summary
The problem of function approximation with neural networks has been of big interest in mathematical research for the last several decades. Various results have been obtained that describe the approximation rates in terms of the structures of the networks and the properties of the approximated functions. One of the most remarkable results in this direction is the universal approximation theorem, which shows that even shallow (but sufficiently wide) networks can approximate continuous functions arbitrarily well (see [9] for the overview and possible proofs of the theorem). In [6] it was shown that integrable functions can be approximated by networks with fixed width. Those networks, may need to be very deep to attain small approximation errors
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
