Generative Denoise Distillation: Simple stochastic noises induce efficient knowledge transfer for dense prediction

Abstract

Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly, which tend to learn each spatial location’s features indiscriminately. Real-world datasets frequently exhibit noise, motivating models to acquire compact and representative features instead of memorizing every details in the training data. To derive a more compact representation (concept feature) from the teacher, we propose an innovative method, termed Generative Denoise Distillation (GDD), where stochastic noises are added to the concept feature of the student to embed them into the generated instance feature from a shallow network. Then, the generated instance feature is aligned with the knowledge of the instance from the teacher. This process diminishes model redundancy while concurrently augmenting its capacity for generalization. We extensively experiment with object detection, instance segmentation, and semantic segmentation to demonstrate the versatility and effectiveness of our method. Notably, GDD achieves new state-of-the-art performance in the tasks mentioned above. We have achieved substantial improvements in semantic segmentation by enhancing PspNet and DeepLabV3, both of which are based on ResNet-18, resulting in mIoU scores of 74.67 and 77.69, respectively, surpassing their previous scores of 69.85 and 73.20 on the Cityscapes dataset of 20 categories. The code is available at https://github.com/ZhgLiu/GDD.