Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network

Abstract

Real-time monitoring and surveillance play an important role in the field of remote sensing, where multi-spectral (MS) images with high spatial resolution are widely desired for better analysis. However, high-resolution MS images cannot be directly obtained due to the limitations of sensors and bandwidth. As an essential way to alleviate this problem, pan-sharpening aims at fusing the complementary information of a low-resolution MS image and a high-resolution panchromatic (PAN) image to reconstruct a high-resolution MS image. Most previous deep-learning based methods can meet the real-time requirements with the help of graphics processing unit (GPU). However, they don’t fully exploit the favorable hierarchical information, sparing huge room for performance improvement. In this paper, to meet the requirement of real-time implementation and achieve more effective performance simultaneously, we propose a multi-scale fusion network (MSFN) to make full use of hierarchical complementary features of PAN and MS images. Specifically, we introduce an encoder–decoder structure and coarse-to-fine strategy to effectively extract multi-scale features of PAN and MS images, separately. Meanwhile, an information pool is adopted to preserve primitive information. Then a multi-scale feature fusion module is applied to fuse multi-scale features from the decoder and information pool. Finally, the fused features are utilized to reconstruct the high-resolution MS image. Extensive experiments demonstrate that our proposed method achieves favorable performance against other methods in terms of quantitative metrics and visual quality. Besides, the results on running time indicate that our method can achieve real-time performance.