Abstract
In this paper, it is shown that C_beta -smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters {0,pm frac{1}{2}, pm 1, 2}. The l_0 and l_1 parameter norms of considered networks are thus equivalent. The depth, the width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in [-1,1]. In particular, this implies that the nonparametric regression estimation with constructed networks achieves, up to logarithmic factors, the same minimax convergence rates as with sparse networks with parameters in [-1,1].
Highlights
The problem of function approximation with neural networks has been of big interest in mathematical research for the last several decades
There are at most 1/ε2 parameters outside of the interval (−ε2, ε2); an entropy bound of order O((2/L)2L−1/ε2) has been obtained by taking in the covering networks the remaining parameters to be 0
One of the ways to approximate functions by neural networks is based on the neural network approximation of local Taylor polynomials of those functions
Summary
The problem of function approximation with neural networks has been of big interest in mathematical research for the last several decades. Various results have been obtained that describe the approximation rates in terms of the structures of the networks and the properties of the approximated functions. One of the most remarkable results in this direction is the universal approximation theorem, which shows that even shallow (but sufficiently wide) networks can approximate continuous functions arbitrarily well (see [9] for the overview and possible proofs of the theorem). In [6] it was shown that integrable functions can be approximated by networks with fixed width. Those networks, may need to be very deep to attain small approximation errors
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
