Continual Learning With Structured Inheritance for Semantic Segmentation in Aerial Imagery

Abstract

With the rapid update and iteration of current aerial image data, the continual learning scenarios and catastrophic forgetting problem attracted increased attention, especially in the semantic segmentation task. However, the existing methods mainly focus on the class continual learning in a single task and are not satisfactory when extended to multiple tasks. In this article, we consider more realistic and complicated settings, namely task continual learning. We revisit the characteristics of semantic segmentation and knowledge distillation (KD) strategy, then propose a general and effective framework, named structured inheritance, to learn new tasks while retaining high performance on old tasks. Specifically, we present two structure-preserving penalties: pixel affinity structure loss and representation consistency structure loss. The former breaks the isolation of pixels and retains the pixel interactive information learned by the old tasks. At the same time, the latter protects high-frequency stationary information between sequence semantic segmentation tasks. Our approach does not need to add extra parameters nor does it need to access the data stream of the old tasks. Therefore, it can be applied in practical applications with strict computational burden, memory cost, and storage budget. Extensive continual learning experiments on four semantic segmentation datasets of Vaihingen, Potsdam, DeepGlobe, and Gaofen challenge semantic segmentation dataset (GCSS) prove the effectiveness of our proposed framework, which outperforms the current state-of-the-art methods and even exceeds the theoretical upper-bound performance of multitask learning. The code and models will be made publicly available.