SDFuse: Semantic-injected dual-flow learning for infrared and visible image fusion

Abstract

Infrared and visible image fusion (IVIF) strives to render fused results that preserve the strengths of the source images (e.g., texture details and thermal highlights) while boosting downstream tasks. However, existing semantic-driven methods typically combine segmentation and fusion tasks via cascade architecture, which fails to sufficiently exploit semantic knowledge. To address this limitation, an innovative semantic-injected network with dual flows is proposed for IVIF, called SDFuse. Concretely, we first apply two pseudo-Siamese feature extraction modules (FEM) to obtain shallow structure features, which involve skip connections to prevent the disappearance of information. Subsequently, dual flows are constructed to support the joint learning paradigm of two tasks. The segmentation flow excavates abundant semantic knowledge from shallow features through fulfilling high-level segmentation task. The fusion flow is introduced to achieve the aggregation of highlights and textures. Note that SGM in fusion flow effectuates the injection of deep semantic knowledge into shallow structural features. Eventually, the region consistency module (RCM) with group normalization leads the network to focus on foreground objects and generate context-consistent fusion images. Our fusion results not only effectively fuse the information of the source image but also have advantages in downstream tasks. Extensive experiments over multiple representative benchmarks prove that SDFuse outperforms current state-of-the-art approaches.