Laplacian Pyramid Generative Adversarial Network for Infrared and Visible Image Fusion

Abstract

Generative adversarial network (GAN) has recently demonstrated a powerful tool for infrared and visible image fusion. However, existing methods extract the features incompletely, miss some textures, and lack the stability of training. To cope with these issues, this article proposes a novel image fusion Laplacian pyramid GAN (IF-LapGAN). Firstly, a generator is constructed which consists of shallow features extraction module, Laplacian pyramid module, and reconstruction module. Specifically, the Laplacian pyramid module is a pyramid-style encoder-decoder architecture, which progressively extracts the multi-scale features. Moreover, the attention module is equipped in the decoder to effectively decode the salient features. Then, two discriminators are adopted to discriminate the fused image and two different modalities respectively. To improve the stability of adversarial learning, we propose to develop another side supervised loss based on the side pre-trained fusion network. Extensive experiments show that IF-LapGAN achieves 3.27%, 27.28%, 6.32%, 1.39%, 3.14%, 1.15% and 1.07% improvement gains in terms of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q_{NMI}$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q_{M}$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q_{Yang}$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q^{AB/F}$</tex-math></inline-formula> , MI, VIF, and FMI, respectively, compared with the second best values.