Binary Convolutional Neural Network with High Accuracy and Compression Rate

Abstract

In this paper, we propose an efficient scheme to train a binary convolutional neural network that has high compression rate and classification accuracy. In binary neural networks, weights and activations are binarized to +1 or -1. This brings two benefits: 1)The model size is greatly reduced; 2)Arithmetic operations can be replaced by more efficient bitwise operations based on binary values, resulting in much faster inference speed and lower power consumption. However, binarizing neural networks will result in severe prediction accuracy degradation compared to their counterpart full-precision networks. To solve this problem, we apply three strategies: 1) By summarizing the previous work, we conclude that the large performance loss of binary networks is mainly caused by the binarization of activations, so we propose to apply multiple banalizations to activations. Compared to apply multiple banalizations to weights, it effectively maintains compression rate while improving the accuracy; 2) In order to alleviate gradient mismatches between the forward and backward propagation, We adopt a more precise differentiable approximation when calculating the gradients in backward propagation; 3) In order to further improve the compression rate, we also binarize the last layer with the help of a scale layer.