Adaptive Teacher Finetune: Towards high-performance knowledge distillation through adaptive fine-tuning

Abstract

Knowledge distillation is a widely used method to transfer knowledge from a large model to a small model. Traditional methods use pre-trained large models to supervise the training of small models, called Offline Knowledge Distillation, However, the structural gap between teachers and students limits its performance. After that, Oneline Knowledge Distillation retrained the teacher-student network from the beginning and the method of echo teaching greatly improved the performance. But there is very little work to explore the difference between the two. In this paper, we first point out that the essential difference between Offline and Oneline Knowledge Distillation is actually whether the weight of the teacher-student network has a process of mutual adaptation. If they adopt the teacher network and the student network jointly train to implement Offline Knowledge Distillation, there is no obvious difference in the final performance, no matter whether it is a joint distillation training. This shows that teacher-student network adaptation is important for Knowledge Distillation. Then, we propose an Adaptive Teacher Finetune (ATF) to adapt the teacher model to the student network. It will use student model information for Tinetune during the Offline Knowledge Distillation process. With normalized logical distribution and alpha-divergence, the performance improvement of ATF clearly exceeds the existing Offline and Oneline Knowledge Distillation method. Extensive experiments conducted on cifar and ImageNet support our aforementioned analysis and conclusions. With the newly introduced ATF, we obtained state-of-the-art performance on ResNet 18 on ImageNet.