Image fusion meets deep learning: A survey and perspective

Abstract

Image fusion, which refers to extracting and then combining the most meaningful information from different source images, aims to generate a single image that is more informative and beneficial for subsequent applications. The development of deep learning has promoted tremendous progress in image fusion, and the powerful feature extraction and reconstruction capabilities of neural networks make the fused results promising. Recently, several latest deep learning technologies have made image fusion explode, e.g., generative adversarial networks, autoencoder, etc. However, a comprehensive review and analysis of latest deep-learning methods in different fusion scenarios is lacking. To this end and in this survey, we first introduce the concept of image fusion, and classify the methods from the perspectives of the deep architectures adopted and fusion scenarios. Then, we review the state-of-the-art on the use of deep learning in various types of image fusion scenarios, including the digital photography image fusion, the multi-modal image fusion and the sharpening fusion. Subsequently, the evaluation for some representative methods in specific fusion tasks are performed qualitatively and quantitatively. Moreover, we briefly introduce several typical applications of image fusion, including photography visualization, RGBT object tracking, medical diagnosis, and remote sensing monitoring. Finally, we provide the conclusion, highlight the challenges in image fusion, and look forward to potential future research directions.