Memristive KDG-BNN: Memristive binary neural networks trained via knowledge distillation and generative adversarial networks

Abstract

With the increasing requirements for the combination of software and hardware, network compression and hardware deployment have become hot research topics. In network compression, binary neural networks (BNNs) are widely applied in artificial intelligence chips because of memory saving, high computational efficiency, and hardware friendliness. However, there is a performance gap between BNNs and full-precision neural networks (FNNs). This paper proposes a BNN training framework called KDG-BNN, consisting of three modules: a full-precision network, a 1-bit binary network, and a discriminator. The full-precision network guides the 1-bit binary network to train through distillation loss in this framework. Meanwhile, the 1-bit binary network acts as a generator and conducts adversarial training with the discriminator. By simultaneously optimizing the adversarial loss and distillation loss, the 1-bit binary network can learn the feature distribution of the full-precision network more accurately. Then, the generative adversarial network (GAN) is replaced by Wasserstein GAN with gradient penalty (WGAN-GP) to deal with gradient disappearance, and KDG-BNN is developed into KDWG-BNN. Experiments show that AdamBNN trained with KDWG-BNN can achieve 85.89% and 70.7% accuracy on CIFAR-10 and ImageNet, respectively, exceeding 0.76% on CIFAR-10 and 0.2% on ImageNet. The memristor has many features for hardware deployment, such as memory functions, continuous input and output, nanoscale size, etc., making it an ideal device for deploying neural networks. Therefore, this paper further proposes a memristor-based KDG-BNN implementation scheme by levering the merits of memristors and the lightweight BNNs in the hope of realizing and promoting end-side intelligent applications.