Universal Approximation Power of Deep Residual Neural Networks Through the Lens of Control

Abstract

In this paper we show that deep residual neural networks have the power of universal approximation by using, in an essential manner, the observation that these networks can be modeled as nonlinear control systems. We first study the problem of using a deep residual neural network to exactly memorize training data by formulating it as a controllability problem for an ensemble control system. Using techniques from geometric control theory, we identify a class of activation functions that allow us to ensure controllability on an open and dense submanifold of sample points. Using this result, and resorting to the notion of monotonicity, we establish that any continuous function can be approximated on a compact set to arbitrary accuracy, with respect to the uniform norm, by this class of neural networks. Moreover, we provide optimal bounds on the number of required neurons.