Online gradient method with smoothing ℓ0 regularization for feedforward neural networks

Abstract

ℓp regularization has been a popular pruning method for neural networks. The parameter p was usually set as 0<p≤2 in the literature, and practical training algorithms with ℓ0 regularization are lacking due to the NP-hard nature of the ℓ0 regularization problem; however, the ℓ0 regularization tends to produce the sparsest solution, corresponding to the most parsimonious network structure which is desirable in view of the generalization ability. To this end, this paper considers an online gradient training algorithm with smoothing ℓ0 regularization (OGTSL0) for feedforward neural networks, where the ℓ0 regularizer is approximated by a series of smoothing functions. The underlying principle for the sparsity of OGTSL0 is provided, and the convergence of the algorithm is also theoretically analyzed. Simulation examples support the theoretical analysis and illustrate the superiority of the proposed algorithm.