Active Neuron Least Squares: A Training Method for Multivariate Rectified Neural Networks

Abstract

We propose a novel training method for multivariate two-layer rectified neural networks. The core mechanism is the augmentation of standard gradient descent direction by the inclusion of search vectors which are chosen to explicitly adjust the activation patterns of the neurons. Active neuron least squares (ANLS) proceeds iteratively with each iteration consisting of three steps: (a) generation of search vectors (including ones designed to change activation patterns), (b) identification of the candidate weights, and (c) a decision on which candidate weights are selected for update. We develop stable and efficient procedures for implementing the method. Numerical examples are provided that demonstrate the effectiveness of ANLS compared with existing popular approaches on various learning tasks ranging from function approximation to solving PDEs.