Image fusion via hierarchical extractor and maximum feature distribution learning

Abstract

Most existing learning-based image fusion solutions are prone to discard important high-frequency information such as edges and detail textures of the image due to they tend to build deeper network structures. In this work, we report a performance-efficient model for fusing infrared and visible images that combines the idea of GAN architecture with the prior knowledge of edge and intensity. We innovatively introduce the maximum edge and maximum intensity information of the paired dual-source images directly into the hierarchical feature extractor, which can diversify the inputs and mitigate the loss of intermediate information during feature extraction process. Considering that features extracted from different encoding branches contain lots of redundant information, fused images are reconstructed via the decoder after preliminarily merging of locally encoded features using the attention mechanism. The two discriminators are employed to determine the authenticity of the fusion results compared with maximum feature map rather than images themselves, thus the generated images will both contain more intensity information and edge details. A prior information-based hybrid loss function is designed to guide the proposed model to learn the maximum feature distribution of dual-source image, rather than being biased towards learning the distribution of one source image or the mean distribution of two source images, thereby improving the reconstruction performance. Extensive evaluations demonstrate that our method performs better than the leading algorithms on the public datasets in both qualitative and quantitative terms. Our fusion results not only present abundant textures, brighter thermal targets, and high-contrast appearance, but also eliminate unpleasant halo artifacts along edges.