A semi-parallel CNN-transformer fusion network for semantic change detection

Abstract

Semantic change detection (SCD) can recognize the region and the type of changes in remote sensing images. Existing methods are either based on transformer or convolutional neural network (CNN), but due to the size of various ground objects is different, it is necessary to have global modeling ability and local information extraction ability at the same time. Therefore, in this paper we propose a fusion semantic change detection network (FSCD) with both global modeling ability and local information extraction ability by fusing transformer and CNN. A semi-parallel fusion block has also been proposed to construct FSCD. It can not only have global and local features in parallel, but also fuse them as deeply as serial. To better adaptively decide which mechanism is applied to which pixel, we design a self-attention and convolution selection module (ACSM). ACSM is a self-attention mechanism used to selectively combine transformer and CNN. Specifically, the importance of each mechanism is automatically obtained by learning. According to the importance, the mechanism suitable for a pixel is selected, which is better than using either mechanism alone. We evaluate the proposed FSCD on two datasets, and the proposed network has a significant improvement compared with the state-of-the-art network.