Neural networks, approximation theory, and finite precision computation

Abstract

The concepts of universal approximation and best approximation are discussed in relation to artificial neural networks explicitly addressing the fact that networks are typically simulated on computers. In relation to the property of universal approximation, it is shown that networks can be considered as producing a polynomial approximation to the training data, and only a finite number of coefficients of this polynomial can be manipulated. Thus, they are not capable of universal approximation. In relation to the property of best approximation, a short discussion shows that possession, or not, of this property is irrelevant when comparing the approximation abilities of different networks. In summary, existence proofs derived from approximation theory prove to be irrelevant when the numerical limitations imposed by computer simulation are taken into account.