Temporal-agnostic change region proposal for semantic change detection

Abstract

Remote sensing imagery allows temporal and large-scale observation of the Earth, and advanced techniques such as deep learning have been developed to deal with the massive data. As a result, by combining remote sensing with deep learning, it is possible to realize change detection, which can reveal the change in land cover and land use. Recent work has focused on semantic change detection (SCD), which simultaneously locates where the changes took place while identifying what objects changed between the different periods. Deep learning based SCD methods are typically based on a multi-task structure, which combines the classification and binary change detection (BCD). In this structure, the change region proposals are derived from the transformation (e.g., concatenation, differencing) of multi-temporal images, but after being processed by a convolutional neural network, there can be an incomplete response in the spatio-temporal features. Because of the high similarity of the color, texture, and geometric information of the ground objects in temporal high-resolution remote sensing imagery, this reduces the distinguishability of features and makes it difficult to identify changed areas. Meanwhile, there is less interaction between the BCD and classification, resulting in semantic inconsistency in changed areas. In this paper, the temporal-agnostic change region proposal network (TCRPN) is proposed, which is based on exploring the saliency of the changed regions in single-temporal images to highlight the changed areas in the spatio-temporal features and improve the response to change. As the changed areas can be the salient regions in the single-temporal images, a single-image salient area extractor (SSAE) is proposed to extract the salient areas from the different temporal images, obtaining salient change region candidates. As the change-related salient regions are temporally agnostic, temporal saliency fusion is used to integrate the candidates extracted from the single-temporal images to obtain a more reliable salient area map, where the salient area map is used to enhance the spatio-temporal features. Meanwhile, a shortcut is proposed to further make the change representation more distinguishable and suppress incorrectly highlighted regions. Directly utilizing multi-temporal images for change region proposal can reduce the change information loss and increase the interaction between the classification and BCD tasks to obtain better spatio-temporal features. The TCRPN is simple and flexible, and can be plugged into any multi-task SCD network. We selected four advanced multi-task SCD methods as the baseline, and added the TCRPN in their encoder or decoder part to conduct comprehensive experiments on three large-scale SCD datasets. The experimental results confirmed the effectiveness of the proposed method.