An unsupervised multi‐focus image fusion method based on Transformer and U‐Net

Abstract

This work presents a multi-focus image fusion method based on Transformer and U-Net with an unsupervised training fashion. In this work, the authors introduce Transformer into image fusion because it has great ability to capture the global dependencies and low-frequency features. In image processing, convolutional neural network (CNN) has good performance of detailed feature extraction but a weakness for global feature extraction, and Transformer has limited power in local or detailed information extraction but a strong capacity for global feature extraction. Thus, this work combines the advantages of CNN and Transformer to propose an unsupervised decision map making model for image fusion joint U-Net. The authors construct a model including feature extraction and feature reconstruction modules which correspond to the encoder and decoder network of U-Net, respectively. In addition, perceptual loss is introduced on the basis of structural similarity loss because the combination of these two loss functions can achieve better performance with lower training cost. Experiments show that the proposed image fusion method performs better fusion performance compared with the existing methods.