Neural Networks with Superexpressive Activations and Integer Weights

Abstract

AbstractAn example of an activation function \(\sigma \) is given such that networks with activations \(\{\sigma , \lfloor \cdot \rfloor \}\), integer weights and a fixed architecture depending only on the input dimension d approximate continuous functions on \([0,1]^d\). The range of integer weights required for \(\varepsilon \)-approximation of Hölder continuous functions is derived, which, together with our discrete choice of weights, allows to obtain the number of networks needed to attain a given approximation rate. Combining this number with the obtained speed of approximation and applying an oracle inequality we get a prediction rate \(n^{\frac{-2\beta }{2\beta +d}}\log _2n\) for neural network regression estimation of an unknown \(\beta \)-Hölder continuous function with given n samples. Thus, up to a logarithmic factor \(\log _2n\), the attained rate coincides with the minimax estimation rate for the prediction error of \(\beta \)-smooth functions. As the network sizes are fixed and their weights are integers, the constructed networks are not only easily encodable but they also reduce the problem of finding the best predictor to a simple procedure of minimization over the finite set of candidates.KeywordsNeural networksFunction approximationEntropyNonparametric regression