Fusionnet: Multispectral Fusion of RGB and NIR Images Using Two Stage Convolutional Neural Networks

Abstract

In low light condition, color (RGB) images captured by imaging systems suffer from severe noise causing loss of colors and textures. Near infrared (NIR) images, which tend to ignore interference from external lights, have advantage of capturing invisible information that can not be obtained by regular RGB cameras. In this paper, we propose multispectral fusion of RGB and NIR images using two stage convolutional neural networks (CNNs), called FusionNet. Lack of training data is a huge obstacle to the learning-based fusion. We synthesize noisy RGB images for training by adding multiscale Gaussian noise. We adopt two stage CNNs for RGB-NIR fusion that consists of denoising and fusion. First, we use a compact denoising subnetwork to remove severe noise from the input RGB image. Then, we utilize a fusion subnetwork to recover textures of the denoised RGB image with the help of its corresponding NIR image. We provide a perceptually motivated loss function to ensure color/texture consistency between the input RGB image and the output fusion result. Experimental results show that the proposed method produces natural looking fusion results by successfully recovering colors and textures. Moreover, the proposed method outperforms state-of-the-art fusion methods in terms of visual quality and quantitative measurements.