Multi-level adaptive perception guidance based infrared and visible image fusion

Abstract

The current constraint rules of end-to-end infrared and visible image fusion (IVIF) networks based on deep learning solely focus on the pixel level, disregarding the consideration of deep features and global information. To address this limitation, this paper proposes an multi-level adaptive perception-guided image fusion approach, named MAPFusion. The main idea is to design a specific network structure and use features to guide the training of the network. Specifically, on the one hand, a new loss function strategy is designed, which combines pixel-level, structure-level, and feature-level strategies to comprehensively enhance the information of the fused image. In particular, the multi-level adaptive perceptual loss is introduced as a feature-level strategy, which preserves both low-level positional data and high-level semantic data by constraining the original image and fused image features. Moreover, adaptive weights are constructed using the information measure between feature maps, which measure the degree of information preservation of different source images. On the other hand, a new fusion network is proposed, which uses the improved U-Net structure to extract the multi-scale features of the image, and adds the Atrous Spatial Pyramid Pooling (ASPP) to the specific network layer, which can increase the receptive field and effectively take advantage of the global information of the image. Comparative experiments demonstrate that the proposed method effectively utilizes the information existing in infrared and visible images, resulting in superior visual quality and objective evaluation compared to SOTA algorithms. Ablation and supplementary experiments further validate the rationality of the proposed method and its potential for advanced vision tasks.