An incremental learning approach for sustainable regional isolation and integration

Abstract

Humans are capable of acquiring new knowledge on a constant basis, while integrating and optimizing old knowledge without forgetting them. This is mainly attributed to the human brain's ability of partitioned learning and memory replay. In this paper, we simulate this ability and propose an incremental learning network of Sustainable Regional Isolation and Integration (SRII). SRII consists of two phases, regional isolation and regional integration, which are iterated to achieve continuous incremental class learning. Regional isolation isolates new learning processes to avoid interfering with existing knowledge, while regional integration uses knowledge distillation and a margin loss regularization term. The knowledge distillation transfers replay knowledge for alleviating catastrophic forgetting, while the margin loss regularization term is to clarify the boundaries of new and old knowledge for alleviating recency bias. Experimental results on the CIFAR100 and miniImageNet datasets demonstrate that SRII outperforms the state-of-the-arts on avoiding catastrophic forgetting. In all 5-stage and 10-stage incremental settings, SRII outperforms the baseline and achieves at least 5.09%+ average accuracy improvement. Our source code is available at https://github.com/Wuziyi123/SRII.