CMFA_Net: A cross-modal feature aggregation network for infrared-visible image fusion

Abstract

Infrared and visible image fusion is a typical cross-modal information enhancement technology, which aims to fetch the complementary cues from different sensors to reconstruct an informative image or video. Many related works focus on designing the hand-crafted fusion rules and ignore the inner complementarity potentials of modalities, resulting in failing to mine the ability of the deep model thoroughly. In this work, an unsupervised cross-modal feature aggregation network (CMFA_Net) is developed, which explores the latent correlations between the internal characteristics effectively and processes these information to fuse a satisfactory image. Firstly, a densely integrated structure and an attention module are proposed to form a feature extractor. Subsequently, the l1-norm and the attention mechanism are combined to fuse the affinity features of the cross-modal images. Finally, the fused image is reconstructed by the deconvolution block. To guarantee the clarity and rich information of the fused image, a specific loss function is put forward by utilizing the average pixel decision for structural similarity (SSIM-p) and content-gram variation (CGV) for training the model on the KAIST dataset. Extensive and solid experiments verify the effectiveness and robustness of the proposed model and demonstrate that the proposed method outperforms the state-of-the-arts and achieves advanced performance as well as less computational consumption both in subjective and objective evaluations.