Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation

Abstract

Deep neural networksDeep Neural Network have achieved remarkable performance for artificial intelligence tasks. The success behind intelligent systems often relies on large-scale models with high computational complexity and storage costs. The over-parameterized networks are often easy to optimize and can achieve better performance. However, it is challenging to deploy them over resource-limited edge-devices. Knowledge DistillationKnowledge distillation (KD) aims to optimize a lightweight network from the perspective of over-parameterized training. The traditional offline KD transfers knowledge from a cumbersome teacher to a small and fast student network. When a sizeable pre-trained teacher network is unavailable, online KDKnowledge distillation can improve a group of models by collaborative or mutual learning. Without needing extra models, Self-KD boosts the network itself using attached auxiliary architectures. KDKnowledge distillation mainly involves knowledge extraction and distillation strategies these two aspects. Beyond KDKnowledge distillation schemes, various KDKnowledge distillation algorithms are widely used in practical applications, such as multi-teacher KD, cross-modal KD, attention-based KD, data-free KD and adversarial KD. This paper provides a comprehensive KDKnowledge distillation survey, including knowledge categoriesKnowledge category, distillation schemes and algorithms, as well as some empirical studies on performance comparison. Finally, we discuss the open challenges of existing KD works and prospect the future directions.