Integral representations of shallow neural network with rectified power unit activation function

Abstract

In this paper we characterize the set of functions that can be represented by infinite width neural networks with RePU activation function max(0,x)p, when the network coefficients are regularized by an ℓ2/p (quasi)norm. Compared to the more well-known ReLU activation function (which corresponds to p=1), the RePU activation functions exhibit a greater degree of smoothness which makes them preferable in several applications. Our main result shows that such representations are possible for a given function if and only if the function is κ-order Lipschitz and its R-norm is finite. This extends earlier work on this topic that has been restricted to the case of the ReLU activation function and coefficient bounds with respect to the ℓ2 norm. Since for q<2, ℓq regularizations are known to promote sparsity, our results also shed light on the ability to obtain sparse neural network representations.