Relation-Based Knowledge Distillation for Anomaly Detection

Abstract

Anomaly detection is a binary classification task, which is to judge whether the input image contains an anomaly or not and the difficulty is that only normal samples are available at training. Due to this unsupervised nature, the classic supervised classification methods will fail. Knowledge distillation-based methods for unsupervised anomaly detection have recently drawn attention as it has shown the outstanding performance. In this paper, we present a novel knowledge distillation-based approach for anomaly detection (RKDAD). We propose to use the “distillation” of the “FSP matrix” from adjacent layers of a teacher network, pre-trained on ImageNet, into a student network which has the same structure as the teacher network to solve the anomaly detection problem, we show that the “FSP matrix” are more discriminative features for normal and abnormal samples than other traditional features like the latent vectors in autoencoders. The “FSP matrix” is defined as the inner product between features from two layers and we detect anomalies using the discrepancy between teacher’s and student’s corresponding “FSP matrix”. To the best of our knowledge, it is the first work to use the relation-based knowledge distillation framework to solve the unsupervised anomaly detection task. We show that our method can achieve competitive results compared to the state-of-the-art methods on MNIST, F-MNIST and surpass the state-of-the-art results on the object images in MVTecAD.