Multimodal Medical Image Fusion Research Articles

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19.

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

MMCL: Meta-mutual contrastive learning for multi-modal medical image fusion

The number of datasets and computational efficiency are always hindrances in the multi-modal medical image fusion (MMIF) research. To address these challenges, we propose a contrastive learning framework inspired meta-mutual, which divides the medical image fusion task into subtasks and pre-trains an optimal meta-representation suitable for all subtasks. We then fine-tune our proposed network using this optimal meta-representation as initialization, achieving the best model with only a few short datasets. Additionally, extracting source image features in pairs can lead to redundant information due to the invariant and unique features of multi-modal images. Therefore, we introduce novelty mutual contrastive coupled pairs to extract both invariant and unique features from source images. Experimental results demonstrate that our method outperforms other state-of-the-art fusion methods.

Medical Image Fusion for Multiple Diseases Features Enhancement

ABSTRACTThroughout the past 20 years, medical imaging has found extensive application in clinical diagnosis. Doctors may find it difficult to diagnose diseases using only one imaging modality. The main objective of multimodal medical image fusion (MMIF) is to improve both the accuracy and quality of clinical assessments by extracting structural and spectral information from source images. This study proposes a novel MMIF method to assist doctors and postoperations such as image segmentation, classification, and further surgical procedures. Initially, the intensity‐hue‐saturation (IHS) model is utilized to decompose the positron emission tomography (PET)/single photon emission computed tomography (SPECT) image, followed by a hue‐angle mapping method to discriminate high‐ and low‐activity regions in the PET images. Then, a proposed structure feature adjustment (SFA) mechanism is used as a fusion strategy for high‐ and low‐activity regions to obtain structural and anatomical details with minimum color distortion. In the second step, a new multi‐discriminator generative adversarial network (MDcGAN) approach is proposed for obtaining the final fused image. The qualitative and quantitative results demonstrate that the proposed method is superior to existing MMIF methods in preserving the structural, anatomical, and functional details of the PET/SPECT images. Through our assessment, involving visual analysis and subsequent verification using statistical metrics, it becomes evident that color changes contribute substantial visual information to the fusion of PET and MR images. The quantitative outcomes demonstrate that, in the majority of cases, the proposed algorithm consistently outperformed other methods. Yet, in a few instances, it achieved the second‐highest results. The validity of the proposed method was confirmed using diverse modalities, encompassing a total of 1012 image pairs.

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Many image fusion methods have been proposed to leverage the advantages of functional and anatomical images while compensating for their shortcomings. These methods integrate functional and anatomical images while presenting physiological and metabolic organ information, making their diagnostic efficiency far greater than that of single-modal images. Currently, most existing multimodal medical imaging fusion methods are based on multiscale transformation, which involves obtaining pyramid features through multiscale transformation. Low-resolution images are used to analyse approximate image features, and high-resolution images are used to analyse detailed image features. Different fusion rules are applied to achieve feature fusion at different scales. Although these fusion methods based on multiscale transformation can effectively achieve multimodal medical image fusion, much detailed information is lost during multiscale and inverse transformation, resulting in blurred edges and a loss of detail in the fusion images. A multimodal medical image fusion method based on interval gradients and convolutional neural networks is proposed to overcome this problem. First, this method uses interval gradients for image decomposition to obtain structure and texture images. Second, deep neural networks are used to extract perception images. Three methods are used to fuse structure, texture, and perception images. Last, the images are combined to obtain the final fusion image after colour transformation. Compared with the reference algorithms, the proposed method performs better in multiple objective indicators of QEN\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{EN}$$\\end{document}, QNIQE\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{NIQE}$$\\end{document}, QSD\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{SD}$$\\end{document}, QSSEQ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{SSEQ}$$\\end{document} and QTMQI\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{TMQI}$$\\end{document}.

Towards accurate diagnosis: exploring knowledge distillation and self-attention in multimodal medical image fusion

ABSTRACT Multimodal medical image fusion aims to aggregate significant information based on the characteristics of medical images from different modalities. Existing research in image fusion faces several major limitations, including a scarcity of paired data, noisy and inconsistent modalities, a lack of contextual relationships, and suboptimal feature extraction and fusion techniques. In response to these challenges, this research proposes a novel adaptive fusion approach. Our knowledge distillation (KD) model extracts informative features from multimodal medical images using various key components. A teacher network is employed to emphasise the suitability and complexity of capturing high-level abstract features. The soft labels are utilised to transfer the knowledge between the teacher network as well as the student network. During student network training, we minimise the divergence between these soft labels. To enhance the adaptive fusion of extracted features from different modalities, we apply a self-attention mechanism. Training this self-attention mechanism minimises the loss function, encouraging attention scores to capture relevant contextual relationships between features. Additionally, a cross-modal consistency module aligns the extracted features to ensure spatial consistency and meaningful fusion. Our adaptive fusion strategy effectively combines features to enhance the diagnostic value and quality of fused images. We employ generator and discriminator architectures for synthesising fused images and distinguishing between real and generated fused images. Comprehensive analysis is conducted on the basis of diverse evaluation measures. Experimental results demonstrate improved fusion outcomes with values of 0.92, 41.58, 7.25, 0.958, 0.759, 0.947, 0.90, 7.05, 0.0726, and 76 s for SSIM, PSNR, FF, VIF, UIQI, FMI, EITF, entropy, RMSE, and execution time, respectively.

CT and MRI Image Fusion via Coupled Feature-Learning GAN

The fusion of multimodal medical images, particularly CT and MRI, is driven by the need to enhance the diagnostic process by providing clinicians with a single, comprehensive image that encapsulates all necessary details. Existing fusion methods often exhibit a bias towards features from one of the source images, making it challenging to simultaneously preserve both structural information and textural details. Designing an effective fusion method that can preserve more discriminative information is therefore crucial. In this work, we propose a Coupled Feature-Learning GAN (CFGAN) to fuse the multimodal medical images into a single informative image. The proposed method establishes an adversarial game between the discriminators and a couple of generators. First, the coupled generators are trained to generate two real-like fused images, which are then used to deceive the two coupled discriminators. Subsequently, the two discriminators are devised to minimize the structural distance to ensure the abundant information in the original source images is well-maintained in the fused image. We further empower the generators to be robust under various scales by constructing a discriminative feature extraction (DFE) block with different dilation rates. Moreover, we introduce a cross-dimension interaction attention (CIA) block to refine the feature representations. The qualitative and quantitative experiments on common benchmarks demonstrate the competitive performance of the CFGAN compared to other state-of-the-art methods.

Multi-Dimensional Medical Image Fusion With Complex Sparse Representation.

In the field of medical imaging, the fusion of data from diverse modalities plays a pivotal role in advancing our understanding of pathological conditions. Sparse representation (SR), a robust signal modeling technique, has demonstrated noteworthy success in multi-dimensional (MD) medical image fusion. However, a fundamental limitation appearing in existing SR models is their lack of directionality, restricting their efficacy in extracting anatomical details from different imaging modalities. To tackle this issue, we propose a novel directional SR model, termed complex sparse representation (ComSR), specifically designed for medical image fusion. ComSR independently represents MD signals over directional dictionaries along specific directions, allowing precise analysis of intricate details of MD signals. Besides, current studies in medical image fusion mostly concentrate on addressing either 2D or 3D fusion problems. This work bridges this gap by proposing a MD medical image fusion method based on ComSR, presenting a unified framework for both 2D and 3D fusion tasks. Experimental results across six multi-modal medical image fusion tasks, involving 93 pairs of 2D source images and 20 pairs of 3D source images, substantiate the superiority of our proposed method over 11 state-of-the-art 2D fusion methods and 4 representative 3D fusion methods, in terms of both visual quality and objective evaluation. The source code of our fusion method is available at https://github.com/Imagefusions/imagefusions/tree/main.

Multimodal Medical Image Fusion Enhancement Based on Wavelet Transform

Multimodal Medical Image Fusion Enhancement Based on Wavelet Transform

Enhanced multimodal medical image fusion via modified DWT with arithmetic optimization algorithm

Medical image fusion (MIF) techniques are proficient in combining medical images in distinct morphologies to obtain a reliable medical analysis. A single modality image could not offer adequate data for an accurate analysis. Therefore, a novel multimodal MIF-based artificial intelligence (AI) method has been presented. MIF approaches fuse multimodal medical images for exact and reliable medical recognition. Multimodal MIF improves diagnostic accuracy and clinical decision-making by combining complementary data in different imaging modalities. This article presents a new multimodal medical image fusion model utilizing Modified DWT with an Arithmetic Optimization Algorithm (MMIF-MDWTAOA) approach. The MMIF-MDWTAOA approach aims to generate a fused image with the significant details and features from each modality, leading to an elaborated depiction for precise interpretation by medical experts. The bilateral filtering (BF) approach is primarily employed for noise elimination. Next, the image decomposition process uses a modified discrete wavelet transform (MDWT) approach. However, the approximation coefficient of modality_1 and the detailed coefficient of modality_2 can be fused interchangeably. Furthermore, a fusion rule is derived from combining the multimodality data, and the AOA model is enforced to ensure the optimum selection of the fusion rule parameters. A sequence of simulations is accomplished to validate the enhanced output of the MMIF-MDWTAOA technique. The investigational validation of the MMIF-MDWTAOA technique showed the highest entropy values of 7.568 and 7.741 bits/pixel over other approaches.

Interactive Multi-scale Fusion: Advancing Brain Tumor Detection Through Trans-IMSM Model.

Multi-modal medical image (MI) fusion assists in generating collaboration images collecting complement features through the distinct images of several conditions. The images help physicians to diagnose disease accurately. Hence, this research proposes a novel multi-modal MI fusion modal named guided filter-based interactive multi-scale and multi-modal transformer (Trans-IMSM) fusion approach to develop high-quality computed tomography-magnetic resonance imaging (CT-MRI) fused images for brain tumor detection. This research utilizes the CT and MRI brain scan dataset to gather the input CT and MRI images. At first, the data preprocessing is carried out to preprocess these input images to improve the image quality and generalization ability for further analysis. Then, these preprocessed CT and MRI are decomposed into detail and base components utilizing the guided filter-based MI decomposition approach. This approach involves two phases: such as acquiring the image guidance and decomposing the images utilizing the guided filter. A canny operator is employed to acquire the image guidance comprising robust edge for CT and MRI images, and the guided filter is applied to decompose the guidance and preprocessed images. Then, by applying the Trans-IMSM model, fuse the detail components, while a weighting approach is used for the base components. The fused detail and base components are subsequently processed through a gated fusion and reconstruction network, and the final fused images for brain tumor detection are generated. Extensive tests are carried out to compute the Trans-IMSM method's efficacy. The evaluation results demonstrated the robustness and effectiveness, achieving an accuracy of 98.64% and an SSIM of 0.94.

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Convolutional laplacian gaussian pyramid approach multimodal medical image fusion

Convolutional laplacian gaussian pyramid approach multimodal medical image fusion

Performance analysis of multimodal medical image fusion using AMT-DWT-based pre-processing and customized CNN for denoising

Performance analysis of multimodal medical image fusion using AMT-DWT-based pre-processing and customized CNN for denoising

Adaptive decomposition with guided filtering and Laplacian pyramid-based image fusion method for medical applications

Medical image fusion enhances diagnostic precision and facilitates clinical decision-making by integrating information from multiple medical imaging modalities. However, this field is still challenging as the output integrated image, whether from spatial or transform domain algorithms, may suffer from drawbacks such as low contrast, blurring effect, noise, over smoothness, etc. Also, some existing novel works are restricted to specific image datasets. So, to address such issues, a new multi-modal medical image fusion approach based on the advantageous effects of multiple transforms has been introduced in the present work. For this, we use an adaptive image decomposition tool known as Hilbert vibration decomposition (HVD). HVD decomposes an image into different energy components, and after a proper decomposition of the source images, the desirable features of the decomposed components are then passed through a guided filter (GF) for edge preservation. Then, the Laplacian pyramid integrates these filtered parts using the choose max rule. Since HVD offers better spatial resolution and is independent of fixed cut-off frequencies like other transforms, the subjective outputs from this method for different publicly available medical image datasets are clear and better than the previously 20 state-of-the-art published results. Moreover, the obtained values of different objective evaluation metrics such as information entropy (IE): 7.6943, 5.9737, mean: 110.6453, 54.6346, standard deviation (SD): 85.5376, 61.8129, average gradient (AG): 109.2818, 64.6451, spatial frequency (SF): 0.1475, 0.1100, and edge metric (QHK/S): 0.5400, 0.6511 demonstrate its comparability to others. The algorithm's running period of just 0.161244 s also indicates high computational efficiency.

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

Medical imaging is indispensable for accurate diagnosis and effective treatment, with modalities like MRI and CT providing diverse yet complementary information. Traditional image fusion methods, while essential in consolidating information from multiple modalities, often suffer from poor image quality and loss of crucial details due to inadequate handling of semantic information and limited feature extraction capabilities. This paper introduces a novel medical image fusion technique leveraging unsupervised image segmentation to enhance the semantic understanding of the fusion process. The proposed method, named DUSMIF, employs a multi-branch, multi-scale deep learning architecture that integrates advanced attention mechanisms to refine the feature extraction and fusion processes. An innovative approach that utilizes unsupervised image segmentation to extract semantic information is introduced, which is then integrated into the fusion process. This not only enhances the semantic relevance of the fused images but also improves the overall fusion quality. The paper proposes a sophisticated network structure that extracts and fuses features at multiple scales and across multiple branches. This structure is designed to capture a comprehensive range of image details and contextual information, significantly improving the fusion outcomes. Multiple attention mechanisms are incorporated to selectively emphasize important features and integrate them effectively across different modalities and scales. This approach ensures that the fused images maintain high quality and detail fidelity. A joint loss function combining content loss, structural similarity loss, and semantic loss is formulated. This function not only guides the network in preserving image brightness and texture but also ensures that the fused image closely resembles the source images in both content and structure. The proposed method demonstrates superior performance over existing fusion techniques in objective assessments and subjective evaluations, confirming its effectiveness in enhancing the diagnostic utility of fused medical images.

Retraction Note: MAMIF: multimodal adaptive medical image fusion based on B-spline registration and non-subsampled shearlet transform

Retraction Note: MAMIF: multimodal adaptive medical image fusion based on B-spline registration and non-subsampled shearlet transform

SIMFusion: A semantic information-guided modality-specific fusion network for MR Images

Multi-modal medical image fusion aims to integrate distinct imaging modalities to yield more comprehensive and precise medical images, which can benefit the subsequent image analysis tasks. However, prevailing state-of-the-art image fusion methods, despite substantial advancements, do not explicitly address the efficient handling of complementary information between modalities. Moreover, most current multi-modal medical image fusion methods encounter challenges when integrating with practical tasks, which lack the guidance of semantic information, thus hindering the generation of high-quality images for accurate identification of lesion areas. To deal with these challenges, this paper introduces a novel semantic information-guided modality-specific fusion network for multi-modal magnetic resonance (MR) images, named SIMFusion. Specifically, we propose the decomposition branch to capture common and specific features from MR images of different modalities for fusion to reduce information redundancy through correlated mutual information loss. Subsequently, we obtain their semantic characteristics via the semantic branch utilizing a pre-train segmentation network and, finally, achieve an adaptive balance between the two sets of features through a specialized fusion strategy. Extensive experiments demonstrate the superiority of SIMFusion over existing competing techniques on both BraTS2019 and ISLES2022 datasets, with 8.1% improvement in MI and 38.7% in VIFF in T2-T1ce image pairs, highlighting the potential of the proposed method as a promising solution for MR image fusion in practical applications. Our code will be released at https://github.com/Zhangxw-ustc/SIMFusion.

Multimodal Medical Image Fusion based on the VGG19 Model in the NSCT Domain

Aim: To deal with the drawbacks of the traditional medical image fusion methods, such as the low preservation ability of the details, the loss of edge information, and the image distortion, as well as the huge need for the training data for deep learning, a new multi-modal medical image fusion method based on the VGG19 model and the non-subsampled contourlet transform (NSCT) is proposed, whose overall objective is to simultaneously make the full use of the advantages of the NSCT and the VGG19 model. Methodology: Firstly, the source images are decomposed into the high-pass and low-pass subbands by NSCT, respectively. Then, the weighted average fusion rule is implemented to produce the fused low-pass sub-band coefficients, while an extractor based on the pre-trained VGG19 model is constructed to obtain the fused high-pass subband coefficients. Result and Discussion: Finally, the fusion results are reconstructed by the inversion transform of the NSCT on the fused coefficients. To prove the effectiveness and the accuracy, experiments on three types of medical datasets are implemented. Conclusion: By comparing seven famous fusion methods, both of the subjective and objective evaluations demonstrate that the proposed method can effectively avoid the loss of detailed feature information, capture more medical information from the source images, and integrate them into the fused images.

CFIFusion: Dual‐Branch Complementary Feature Injection Network for Medical Image Fusion

ABSTRACTThe goal of fusing medical images is to integrate the diverse information that multimodal medical images hold. However, the challenges lie in the limitations of imaging sensors and the issue of incomplete modal information retention, which make it difficult to produce images encompassing both functional and anatomical information. To overcome these obstacles, several medical image fusion techniques based on CNN or transformer architectures have been presented. Nevertheless, CNN technique struggles to establish extensive dependencies between the fused and source images, and transformer architecture often overlooks shallow complementary features. To augment both the feature extraction capacity and the stability of the model, we introduce a framework, called dual‐branch complementary feature injection fusion (CFIFusion) technique, a for multimodal medical image fusion framework that combines unsupervised models of CNN model and transformer techniques. Specifically, in our framework, the entire source image and segmented source image are input into an adaptive backbone network to learn global and local features, respectively. To further retain the source images' complementary information, we design a multi‐scale complementary feature extraction framework as an auxiliary module, focusing on calculating feature differences at each level to capture the shallow complementary information. Then, we design a shallow information preservation module tailored for sliced image characteristics. Experimental results on the Harvard whole brain atlas dataset demonstrate that CFIFusion shows greater benefits than recent state‐of‐the‐art algorithms in terms of both subjective and objective evaluations.

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer.

The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.