Data-free Knowledge Distillation via Adversarial

Abstract

Network Compression is a challenging task, but it is crucial for using the deeper network in the low-performance device. If the original training datasets could be obtained, the traditional network compression approaches are useful for training a compact deep model. This paper proposes a novel framework for knowledge distillation without original training datasets via Generative Adversarial Network(GAN). We arrange the fixed pre-trained deeper network and the compact network as the discriminators to generate the training dataset. We also use the deeper network and the compact network as the generators, then introduce one simple full connection network as the discriminator to compress the complex network. We propose (i) a series of new images generation loss functions. (ii) a knowledge distillation method via generating adversarial networks. Finally, we show the superiority of our approach by contrasting with SOTA by benchmark datasets.