Attention-based hierarchical fusion of visible and infrared images

Abstract

This paper presents a two-level hierarchical multi-resolution image fusion method based on attention, which focus on combining multi-sensor images from visible and infrared spectrums into an image while maintaining the salient information necessary for human visual. Attention-based one-level fusion employs adaptive weighting fusion rules to obtain the initial fused image which may being the boundary of saliency regions, then two-level fusion based on nonsubsampled contourlet transform (NSCT) is used to hide possible imprints of the initial fused image. Firstly, for the given visible and infrared images from the same scene, salient regions are detected by visual attention model. Secondly, salient sizes are acquired by calculating maximum entropy. Thirdly, one-level fusion is achieved by the adoptive weighing fusion rules. Various weight values are adopted according to the visual saliency order from high to low. The initial fused image contains not only the salient information of visible image, but also the salient thermal target information of infrared image. Finally, two-level fusion based on NSCT is employed to fuse the visible source image, the infrared source image and the initial fused image, and the final fused image is obtained. Two different data sets from TNO, Human Factors are employed and the experimental results show that the proposed approach is efficient and also performed better than some existing methods.