Target Category Agnostic Knowledge Distillation With Frequency-Domain Supervision

Abstract

Existing knowledge distillation approaches require task-related data to train portable student networks for satisfactory performance. Nevertheless, in real-world applications, a majority of the data is unavailable over concerns about personal privacy, commercial confidentiality, etc. To mitigate the difficulty of acquiring the target dataset, the mainstream knowledge distillation methods generate training samples from the teacher network. However, the generated images still differ from the authentic ones, which limits the student network performance. To solve this issue, we propose a convenient and cost-negligible method to build datasets for training student networks. Specifically, we crawl the data from the web without knowing any category of the target dataset, named Irrelevant Category Crawler Data (ICCD). To prevent the performance collapse due to the data distribution gap between our ICCD and the target dataset, we propose a Pseudo-Classification strategy and Frequency-domain Supervision (PCFS) for target-category agnostic knowledge distillation with ICCD. The Pseudo-Classification strategy classifies ICCD into different pseudo-categories by the teacher network, and uniformly but randomly samples images from each pseudo-category to construct the pseudo-target dataset. Furthermore, we transform the feature maps from the spatial-domain to the frequency-domain, and utilize the high-frequency and low-frequency signals of the teacher network to impose strong constraints on the student network. Extensive experiments conducted on various test sets demonstrate the effectiveness of our proposed PCFS, which outperforms existing data-free methods and achieves comparable performance to those using the target training set. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/SCUT-BIP-Lab/PCFS-DFKD</uri> .