A full-scale hierarchical encoder-decoder network with cascading edge-prior for infrared and visible image fusion

Abstract

Most existing deep learning-based infrared and visible image fusion methods always fail to consider the full-scale long-range correlation and the prior knowledge, resulting in the fused images with low-contrast salient objects and blurred edge details. To overcome these drawbacks, a full-scale hierarchical encoder-decoder network with cascading edge-prior for infrared and visible image fusion is proposed. First, a top-down encoder extracts the hierarchical representations from source image. Then, to inject edge priors into the network and capture the progressive semantic correlations, a triple fusion mechanism is proposed including edge image fusion based on maximum fusion strategy, single-scale shallow layer fusion and full-scale semantic layer fusion based on dual-attention fusion (DAF) strategy. The fused full-scale semantic features (F2SF) are obtained by capturing the long-range affinities of the full-scale. At the same time, a cascading edge-prior branch (CEPB) is designed to embed the fused edge knowledge into fused single-scale shallow features, jointly guiding the decoder to focus on abundant details layer-by-layer on the basis of F2SF, thus recovering the edge and texture details of the fused image well. Finally, a novel loss function consisting of SSIM, intensity and edge loss is constructed to further maintain the network with better edge representation and reconstruction capability. Compared with existing state-of-the-art fusion methods, the proposed method has better performance in terms of both visual evaluation and objective evaluation on public datasets. The source code is available at https://github.com/lxq-jnu/FSFusion.