Infrared and visible image fusion based on tight frame learning via VGG19 network

Abstract

A composite image of infrared and visible images should contain salient features of two source images which come from different sensors in the same scene. In this paper, we propose a fusion method based on a tight frame combined with VGG 19 to integrate the infrared and visible images. After training the tight frame, the pre-registered source images are decomposed into the base and detailed images by classifying all the learned frame filters into two sets with a total-variation-like measure. Then, the max-ℓ1 rule is adopted to fuse the base images, while the VGG19-based method is adopted to fuse the detailed images. For the detailed images, the pre-trained VGG19 is used to extract the multi-layer feature maps, multiple fused detailed images are reconstructed by the ℓ1-norm combined with the weighted evaluation rules, and the final fused detailed image is selected by the max rule. Finally, the final fused image is obtained by directly combining the base image and the detailed image. The experimental results demonstrate that the proposed method achieves the state-of-the-art fusion performance in both quantitative evaluation and subjective quality.