Target layer regularization for continual learning using Cramer-Wold distance

Abstract

We propose an effective regularization strategy (CW-TaLaR) for solving continual learning problems. It uses a penalizing term that is based on measuring the dissimilarity between two probability distributions defined on a target layer of an underlying neural network shared by all tasks. We consider two complementary ways of modeling a target layer distribution that our strategy preserves while learning a new task, both of them trained and applied with the use of the Cramer-Wold distance. Simultaneously, we do not require remembering previous tasks datasets. We perform experiments involving several common supervised frameworks, which prove the competitiveness of the CW-TaLaR method in comparison to a few existing state-of-the-art continual learning models.