SemCKD: Semantic Calibration for Cross-Layer Knowledge Distillation

Abstract

Knowledge distillation is a technique to enhance the generalization ability of a student model by exploiting outputs from a teacher model. Recently, feature-map based variants explore knowledge transfer between manually assigned teacher-student pairs in intermediate layers for further improvement. However, layer semantics may vary in different neural networks, resulting in performance degeneration due to negative regularization from semantic mismatch in manual layer associations. To address this issue, we propose semantic calibration for cross-layer knowledge distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple teacher layers rather than a specific intermediate layer for appropriate cross-layer supervision. We further provide theoretical analysis of the association weights and conduct extensive experiments to demonstrate the effectiveness of our approach. On average, SemCKD improves the student Top-1 classification accuracy by 4.27\% across twelve different teacher-student model combinations on CIFAR-100. Code is available at \url{https://github.com/DefangChen/SemCKD}.