SPFusion: A multi-task semantic perception infrared and visible light fusion method with quality assessment

Abstract

The fusion of infrared and visible light aims to combine the advantages of infrared highlighting thermal targets and the rich textures of visible light, to produce a fused image that is both texture-rich and has prominent targets. However, under extreme conditions such as heavy fog and overexposure, directly fusing low-quality visible light images with infrared images may retain excessive redundant information in these areas. Therefore, this paper proposes a method for achieving adaptive fusion under extreme conditions, called SPFusion. Specifically, the network is guided by a quality assessment module and advanced visual tasks such as segmentation, classification, and reconstruction, to learn an adaptive balance point for rationally distributing infrared and visible light information. Additionally, a progressive fusion strategy and a modality complementarity module are employed to fully integrate information from each modality. Consequently, the network can autonomously allocate visible and infrared information based on image quality and semantics. Extensive experiments have demonstrated that SPFusion achieves state-of-the-art (SOTA) performance on the TNO, MSRS, LLVIP, and RoadScene datasets, and enhances the performance of advanced visual tasks such as object detection. The code is available as open source at linshenj/SPFusion (github.com).