ADCL: Adversarial Distilled Contrastive Learning on lightweight models for self-supervised image classification

Abstract

With the development of modern sensors, numerous images are collected in edge application scenarios; however, their utilization is quite expensive because a massive effort is required to label them for further usage. Self-supervised learning, with no need for labeled data, shows great potential in this context; however, notable performance degradations can be observed when training lightweight networks which are essential in edge implementation. We propose an effective distillation method called Adversarial Distilled Contrastive Learning (ADCL) to mitigate this issue. Specifically, we introduced knowledge distillation into self-supervised learning to transfer underlying feature clustering relations from teacher models to shallow models. We adopted an online-updated rather than a pretrained teacher model to realize convenient implementation to specific data domains. An adversarial loss item was introduced to alleviate unstable optimization caused by an online trained teacher by forcing the teacher model to find feature relations beyond the recognition of the student model. Compared with other self-supervised knowledge distillation methods that maintain data queues consisting of positive and negative examples, the asymmetric contrastive learning method was employed to further relieve the memory bottleneck during training. The experimental results prove the effectiveness of our method. When ResNet-50 is used as a teacher to teach ResNet-18 on ImageNet, ADCL achieves top-1 accuracies of 60.3% , which surpasses other knowledge distillation methods with online teachers and is comparable to approaches using pretrained teachers and data queues.