Continual Learning Based on Knowledge Distillation and Representation Learning

Abstract

AbstractIn recent years, continual learning that is more in line with real-world scenarios has received more attention. In order to solve the catastrophic forgetting problem in continual learning, researchers have put forward various solutions, which are simply summarized into three types: network structure-based methods, rehearsal-based methods and regularization-based methods. Inspired by pseudo-rehearsal and regularization methods, we propose a novel Continual Learning Based on Knowledge Distillation and Representation Learning (KRCL) model, which employs Beta-VAE as a representation learning module to extract a shared representation of learned tasks. In addition, Beta-VAE is also used as a generative model to generate pseudo samples of historical task, and KRCL trains the pseudo samples of the previous tasks together with the data of the current task, and then combines the knowledge distillation process to extract the dark knowledge from the old task model to alleviate the catastrophic forgetting. We compare KRCL with the Finetune, LWF, IRCL and KRCL_real baseline methods on four benchmark datasets. The result shows that the KRCL model achieves state-of-the-art performance in standard continual learning tasks.KeywordsContinual learningClass incremental learningRepresentation learningKnowledge distillationCatastrophic forgetting