Remote Sensing Image Fusion With Task-Inspired Multiscale Nonlocal-Attention Network

Abstract

Recently, convolutional neural networks (CNNs) have been developed for remote sensing image fusion (RSIF). To obtain competitive fusion performance, network design becomes more complicated by stacking convolutional layers deeper and wider. However, problems still remain when applying existing networks in practical applications. On the one hand, researchers focus on improving spatial resolution but ignore that the fused images will be used in subsequent interpretation applications, e.g., objection detection. On the other hand, RSIF involves different tasks with different image sources e.g., pansharpening of the panchromatic and multispectral image, hypersharpening of the panchromatic and hyperspectral image, etc. However, existing networks only solve one of them, failing to be compatible with other tasks. To address the above problems, a convenient task-inspired multiscale nonlocal-attention network (MNAN) is proposed for RSIF. The proposed MNAN focuses more on enhancing the multi-scale targets in the scene when improving the resolution of the fused image. In addition, the proposed network can be applied to both pansharpening and hypersharpening tasks without any modification.