Universal approximation properties for an ODENet and a ResNet: Mathematical analysis and numerical experiments

Abstract

We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let $ n $ and $ m $ be the dimension of input and output data, and assume $ m\leq n $. Then we show that ODENet of width $ n+m $ with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on $ \mathbb{R}^n $. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.