用于神经网络权值稀疏化的<italic>L</italic><sub>1/2</sub>正则化方法

Abstract

On the premise of appropriate learning accuracy, the number of the neurons of a neural network should be as less as possible (constructional sparsification), so as to reduce the cost, and to improve the robustness and the generalization accuracy. We study the constructional sparsification of feedforward neural networks by using regularization methods. Apart from the traditional L 1 regularization for sparsification, we mainly use the L 1/2 regularization. To remove the oscillation in the iteration process due to the nonsmoothness of the L 1/2 regularizer, we propose to smooth it in a neighborhood of the nonsmooth point to get a smoothing L 1/2 regularizer. By doing so, we expect to improve the efficiency of the L 1/2 regularizer so as to surpass the L 1 regularizer. Some of our recent works in this respect are summarized in this paper, including the works on BP feedforward neural networks, higher order neural networks, double parallel neural networks and Takagi-Sugeno fuzzy models.