Versatile kernel reactivation for deep convolutional neural networks

Abstract

Several networks exhibiting excellent performance in the field of computer vision still suffer from the challenge of over-parameterization. Over-parameterized networks have several parameters that cannot be trained, owing to the poor backpropagation process caused by a small L1 norm, and these parameters suppress the potential for network performance improvement. This study proposes a kernel reactivation method to improve network performance by reusing invalid kernels that cannot be utilized for training because of the small L1 norm. The results indicate that the accuracy of Cifar-10 in ResNet-110 is improved by 0.94% compared to the baseline, and the top-1 and top-5 accuracies of Tiny-ImageNet in ResNet-50 are improved by 1.87% and 1.03% compared to the baseline, respectively.