Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity

Abstract

Image fusion aims to integrate complementary characteristics of source images into a single fused image that better serves human visual observation and machine vision perception. However, most existing image fusion algorithms primarily focus on improving the visual appeal of fused images. Although there are some semantic-driven methods that consider semantic requirements of downstream applications, none of them have demonstrated the potential of image-level fusion compared to feature-level fusion, which fulfills high-level vision tasks directly on multi-modal features rather than on a fused image. To overcome these limitations, this paper presents a practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity constraints, termed PSFusion. First of all, the sparse semantic perception branch extracts sufficient semantic features, which are then progressively integrated into the fusion network using the semantic injection module to fulfill the semantic requirements of high-level vision tasks. The scene fidelity path within the scene restoration branch is devised to ensure that the fusion features contain complete information for reconstructing the source images. Additionally, the contrast mask and salient target mask are employed to construct the fusion loss to maintain impressive visual effects of fusion results. In particular, we provide quantitative and qualitative analyses to demonstrate the potential of image-level fusion compared to feature-level fusion for high-level vision tasks. With the rapid advancement of large-scale models, image-level fusion can expeditiously leverage the advantages of multi-modal data and state-of-the-art (SOTA) unimodal segmentation to achieve superior performance. Furthermore, extensive comparative experiments demonstrate the superiority of our PSFusion over SOTA image-level fusion alternatives in terms of visual appeal and high-level semantics. Even under harsh circumstances, our method offers satisfactory fusion results to serve subsequent high-level vision applications. The source code is available at https://github.com/Linfeng-Tang/PSFusion.