FATFusion: A functional–anatomical transformer for medical image fusion

Abstract

Medical image fusion (MIF) strives to obtain an informative fused image by integrating complementary information, e.g., functional metabolic properties and anatomical soft tissues, to facilitate disease diagnosis and treatment planning. To solve the problems of existing MIF algorithms, including inconvenient and manually designed transformation methods, complicated fusion strategies, global complementary information loss, and insufficient consideration of the modality-specific intrinsic characteristics, in this paper, we innovatively propose a Functional–Anatomical Transformer for MIF named FATFusion. In the proposed method, a functional multiscale branch (FMB) and an anatomical multiscale branch (AMB) are designed to extract interscale characteristics belonging to each modality. In addition, functional-guided transformer modules (FGTMs) and anatomical-guided transformer modules (AGTMs) are elaborated to communicate and gather the perceived features for further global complementary information interaction and aggregation. Moreover, the pixel loss and total variation loss are designed to train FATFusion in an end-to-end and unsupervised manner, guiding the deep fusion model to leverage more significant properties. Promising quantitative and qualitative assessments illustrate that FATFusion surpasses alternative state-of-the-art MIF methodologies, excelling in both objective evaluation and perceptible observation. Furthermore, successful generalization results reveal that FATFusion has impressive generalization ability. The source code of the proposed method is available at https://github.com/tthinking/FATFusion.