Gradient rectified parameter unit of the fully connected layer in convolutional neural networks

Abstract

Existing visualization approaches of the convolutional neural network (CNN) present the importance of the positive gradient in backpropagation, and remove the irrelevant negative gradients for improvement. However, the training procedure in CNN pays the same attention to positive and negative gradients’ optimizations. In this work, we present a gradient rectified parameter unit of the fully connected layer (GRU-FC) approach, which rectifies the corresponding parameters generating the negative gradient in the fully connected layer by zero clearing and retrains the networks with the rectified parameters. Besides, a simplified version of GRU-FC is provided to accelerate the training of the network that has a single fully connected layer for classification. Theoretical analysis of the rationalization of GRU-FC presents that GRU-FC-2L is an appropriate approach for networks with more than one fully connected layer. Experiments on the convergence analysis of the network by GRU-FC-2L is conducted. The GRU-FC approach is verified on several datasets (i.e., SV HN, STL10, CIFAR10 and ImageNet) with the recognition accuracy increased effectively. Furthermore, the GRU-FC approach shows a way to dropout unimportant weights regularly instead of randomness.