Convergence analysis of BP neural networks via sparse response regularization

Abstract

Backpropagation (BP) algorithm is the typical strategy to train the feedforward neural networks (FNNs). Gradient descent approach is the popular numerical optimal method which is employed to implement the BP algorithm. However, this technique frequently leads to poor generalization and slow convergence. Inspired by the sparse response character of human neuron system, several sparse-response BP algorithms were developed which effectively improve the generalization performance. The essential idea is to impose the responses of hidden layer as a specific L1 penalty term on the standard error function of FNNs. In this paper, we mainly focus on the two remaining challenging tasks: one is to solve the non-differential problem of the L1 penalty term by introducing smooth approximation functions. The other aspect is to provide a rigorous convergence analysis for this novel sparse response BP algorithm. In addition, an illustrative numerical simulation has been done to support the theoretical statement.