MFHOD: Multi-modal image fusion method based on the higher-order degradation model

Abstract

The task of multimodal image fusion aims to preserve the respective advantages of each modality, such as the detailed texture information from visible light images and the salient target information from infrared images. However, images in real environments are often not perfect high-quality images but are affected by various degradation factors such as noise, blur, and glare. Existing unsupervised multimodal image fusion algorithms use undegraded source images as input to fusion networks and generate high-quality fusion images through complex fusion strategies. This single linear learning approach from high-quality to high-quality cannot be applied to the fusion task of degraded multimodal images in real environments. To address this issue, this paper proposes an end-to-end multimodal image fusion network based on a high-order local degradation model (MFHOD). Firstly, inspired by the idea of probabilistic degradation, we propose a high-order local random degradation model (HODM), which inputs the source multimodal images into the degradation model to obtain degraded images before feeding them into the network. Secondly, we design a simple and efficient dual-branch feature extraction encoder to extract deep features from images. Then, from the perspectives of image pixels, brightness, and gradients, we propose an improved composite loss composed of multiple loss functions to constrain network training. Finally, we propose a L2-norm fusion strategy for preserving brightness information in low-light nighttime images. Our MFHOD demonstrates good performance on infrared and visible light image datasets as well as medical image datasets. Experimental results show that MFHOD can effectively suppress the effects of glare, noise, and smoke in adverse environments, and also improve the quality of fusion images in low-light and nighttime environments.