Distilling and transferring knowledge via cGAN-generated samples for image classification and regression

Abstract

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a student model based on the knowledge from a teacher model. However, applying KD in image regression with a scalar response variable is also important (e.g., age estimation) yet has been rarely studied. Besides, existing KD methods often require a practitioner to carefully select or adjust the teacher and student architectures, making these methods less flexible in practice. To address the above problems in a unified way, we propose a comprehensive KD framework based on conditional generative adversarial networks (cGANs), termed cGAN-KD. Fundamentally different from existing KD methods, cGAN-KD distills and transfers knowledge from a teacher model to a student model via specifically processed cGAN-generated samples. This novel mechanism makes cGAN-KD suitable for both classification and regression tasks, compatible with other KD methods, and insensitive to the teacher and student architectures. An error bound for a student model trained in the cGAN-KD framework is derived in this work, providing a theory for why cGAN-KD is effective as well as guiding the practical implementation of cGAN-KD. Extensive experiments on CIFAR-100 and ImageNet-100 (a subset of ImageNet with only 100 classes) datasets show that the cGAN-KD framework can leverage state-of-the-art KD methods to yield a new state of the art. Moreover, experiments on Steering Angle and UTKFace datasets demonstrate the effectiveness of cGAN-KD in image regression tasks. Notably, in classification, incorporating cGAN-KD into training improves the state-of-the-art SSKD by an average of 1.32% in test accuracy on ImageNet-100 across five different teacher–student pairs. In regression, cGAN-KD decreases the test mean absolute error of a WRN16 × 1 student model from 5.74 to 1.79 degrees (i.e., 68.82% drop) on Steering Angle.