Improvement of learning for CNN with ReLU activation by sparse regularization

Abstract

This paper introduces the sparse regularization for the convolutional neural network (CNN) with the rectified linear units (ReLU) in the hidden layers. By introducing the sparseness for the inputs of the ReLU, there is effect to push the inputs of the ReLU to zero in the learning process. Thus it is expected that the unnecessary increase of the outputs of the ReLU can be prevented. This is the similar effect with the Batch Normalization. Also the unnecessary negative values of the inputs of the ReLU can be reduced by introducing the sparseness. This can improve the generalization of the trained network. The relations between the proposed approach and the Batch Normalization or the modifications of the activation function such as Exponential Linear Unit (ELU) are also discussed. The effectiveness of the proposed method was confirmed through the detail experiments.