Generalized Knowledge Distillation via Relationship Matching.

Abstract

The knowledge of a well-trained deep neural network (a.k.a. the teacher) is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model (a.k.a. the student), which expands the student's knowledge and improves its learning efficacy. Instead of restricting the teacher from working on the same task as the student, we borrow the knowledge of a teacher trained from a general label space --- in this Generalized Knowledge Distillation (GKD), the classes of the teacher and the student may be the same, completely different, or partially overlapped. We claim that the comparison ability between instances acts as an essential factor threading knowledge across tasks, and propose the Relationship Facilitated Local Classifier Distillation (ReFilled) approach, which decouples the GKD flow of the embedding and the top-layer classifier. In particular, different from reconciling the instance-label confidence between models, ReFilled requires the teacher to reweight the hard tuples push forwarded by the student adaptively and then matches the similarity comparison levels between instances. ReFilled demonstrates strong discriminative ability when the classes of the teacher vary from the same to a fully non-overlapped set w.r.t. the student.