BookKD: A novel knowledge distillation for reducing distillation costs by decoupling knowledge generation and learning

Abstract

Knowledge distillation guides student networks’ training and enhances their performance through excellent teacher networks. However, along with the performance advantages, knowledge distillation also entails a huge computational burden, sometimes tens or even hundreds of times that of traditional training methods. So, this paper designs a book-based knowledge distillation (BookKD) to minimize the costs of knowledge distillation while improving performance. First, a decoupling-based knowledge distillation framework is designed. By decoupling the traditional knowledge distillation process into two independent sub-processes, book-making and book-learning, knowledge distillation can be completed with little resource consumption. Second, a book-making method based on knowledge ensemble and knowledge regularization is developed, which makes books by organizing and processing the knowledge generated by teachers. These books can replace these teachers to provide sufficient knowledge with little distillation costs. Finally, a book-learning method based on entropy dynamic adjustment and label smoothing is designed. The entropy dynamic adjustment optimizes the training loss and mitigates student networks’ difficulty in learning books. Label smoothing alleviates the student network’s over-confidence in ground truth labels, which increases its attention to the class similarity knowledge in books. BookKD is tested on three image classification datasets, CIFAR100, ImageNet and ImageNet100, and an object detection dataset PASCAL VOC 2007. The experiment results indicate the advantages of BookKD in reducing distillation costs and improving distillation performance.