Learning Polynomial Neural Networks via Low Rank Tensor Recovery

Abstract

In this paper, we study connections between training neural networks and low rank tensor recovery. We focus on two-layer neural networks with polynomial activation functions, where the number of hidden units is smaller than the dimension of the input. We leverage the fact that the relationship between network input and output can be described in terms of the inner product between two tensors, one depending on input and another solely on the weights. Based on this relationship, we propose an iterative algorithm to learn the neural network. The algorithm consists of two main steps: bias correction and low rank tensor projection. The proposed method is faster than those based on convex relaxation and generalizes previous approaches for training polynomial neural network with quadratic non-linearity.1