Exponential ReLU DNN Expression of Holomorphic Maps in High Dimension

J A A Opschoor,J Zech,Ch Schwab

doi:10.1007/s00365-021-09542-5

Abstract

For a parameter dimension din {mathbb {N}}, we consider the approximation of many-parametric maps u: [-,1,1]^drightarrow {mathbb R} by deep ReLU neural networks. The input dimension d may possibly be large, and we assume quantitative control of the domain of holomorphy of u: i.e., u admits a holomorphic extension to a Bernstein polyellipse {{mathcal {E}}}_{rho _1}times cdots times {{mathcal {E}}}_{rho _d} subset {mathbb {C}}^d of semiaxis sums rho _i>1 containing [-,1,1]^{d}. We establish the exponential rate O(exp (-,bN^{1/(d+1)})) of expressive power in terms of the total NN size N and of the input dimension d of the ReLU NN in W^{1,infty }([-,1,1]^d). The constant b>0 depends on (rho _j)_{j=1}^d which characterizes the coordinate-wise sizes of the Bernstein-ellipses for u. We also prove exponential convergence in stronger norms for the approximation by DNNs with more regular, so-called “rectified power unit” activations. Finally, we extend DNN expression rate bounds also to two classes of non-holomorphic functions, in particular to d-variate, Gevrey-regular functions, and, by composition, to certain multivariate probability distribution functions with Lipschitz marginals.

Highlights

In recent years, so-called deep artificial neural networks (“DNNs” for short) have seen dramatic development in applications from data science and machine learning.after early results in the 1990s on genericity and universality of DNNs, in recent years the refined mathematical analysis of their approximation properties, viz. “expressive power,” has received increasing attention
The polynomials which appear in such expansions can, in turn, be represented by DNNs, either exactly for certain activation functions, or approximately for example for the so-called rectified linear unit (“ReLU”) activation with exponentially small representation error [18,37]
We focus on ReLU DNNs, but comment in passing on versions of our results for other DNN activation functions

Summary

Introduction

-called deep artificial neural networks (“DNNs” for short) have seen dramatic development in applications from data science and machine learning.after early results in the 1990s on genericity and universality of DNNs (see [27] for a survey and references), in recent years the refined mathematical analysis of their approximation properties, viz. “expressive power,” has received increasing attention. A particular class of many-parametric maps whose DNN approximation needs to be considered in many applications are real-analytic and holomorphic maps. The question of DNN expression rate bounds for such maps has received some attention in the approximation theory literature [21,22,36]. It is well known that multi-variate, holomorphic maps admit exponential expression rates by multivariate polynomials. Countably parametric maps u : [− 1, 1]∞ → R can be represented under certain conditions by so-called generalized polynomial chaos expansions with quantified sparsity in coefficient sequences. The polynomials which appear in such expansions can, in turn, be represented by DNNs, either exactly for certain activation functions, or approximately for example for the so-called rectified linear unit (“ReLU”) activation with exponentially small representation error [18,37]

Objectives

Results

Conclusion