MCnet: Multiscale visible image and infrared image fusion network

Abstract

In both civil and military fields, remote sensing image fusion is a popular method for improving images. Multimodal image fusion is typically employed in remote sensing image fusion. Multimodal image fusion acquires synthetic images containing rich image information by fusing image information from different wavebands. Current fusion networks concentrate on fusing features at a single image scale, and the resulting image lacks either spatial or texture features. To address these problems and achieve high-quality fusion, we propose MCnet (multiscale network). MCnet is a new method for multimodal image fusion of visible remote sensing images and infrared remote sensing images in a multiscale framework. Specifically, MCnet first deals with the fusion of image features at different scales in a coarse-to-fine manner. Second, MCnet adaptively provides the amount of information in each image at different scales and images for fine fusion feature supplementation in the coarse fusion stage. In the step of fine fusion, the outcomes of the previous stage will be supplied with the missing characteristics. Finally, we design an objective function with three components: structure loss, region loss, and image quality loss. Structure loss and region loss maintain convergence on overall image similarity and region feature similarity. The image quality constraint partially mitigates the effect of low-quality results on model convergence. MCnet emphasizes the texture features and edge contours of the results, which not only boost the quality of the fusion results but also cause the images to show better discriminative properties. We conduct sufficient experiments based on VIS (visible) and IR (infrared) datasets. The results demonstrate that our proposed model achieves state-of-the-art performance. We also conduct generalizability studies on the proposed method, which likewise yield positive results, demonstrating that MCnet is successful and applicable in a variety of situations.