Multimodal Medical Image Fusion Research Articles

Shearlet Transform-Based Novel Method for Multimodality Medical Image Fusion Using Deep Learning

Multi-modality medical image fusion (MMIF) methods were widely used in a variety of clinical settings. For specialists, MMIF could provide an image containing anatomical and physiological information that can help develop diagnostic procedures. Different models linked to MMIF were proposed previously. However, there would be a need to enhance the functionality of prior methodologies. In this proposed model, a unique fusion model depending upon optimal thresholding and deep learning approaches are presented. An enhanced monarch butterfly optimization (EMBO) determines an optimal threshold with fusion rules as in shearlet transform. The efficiency of the fusion process mainly depends on the fusion rule and the optimization of the fusion rule can improve the efficiency of the fusion. The extraction element of the deep learning approach was then utilized to fuse high- and low-frequency sub-bands. The fusion technique was carried out using a convolutional neural network (CNN). The studies were carried out for MRI and CT images. The fusion results were attained and the proposed model was proved to offer effective performance with reduced values of error and improved values of correlation.

DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network

Multi-modal medical image fusion is a long-standing important research topic that can obtain informative medical images and assist doctors diagnose and treat diseases more efficiently. However, most fusion methods extract and fuse features by subjectively defining constraints, which easily distorts the unique information of source images. In this work, we present a novel end-to-end unsupervised network to fuse multi-modal medical images. It is composed of a generator and two symmetrical discriminators. The former aims to generate a ”real-like” fused image based on a specifically designed content and structure loss, while the latter are devoted to distinguishing the differences between the fused image and the source ones. They are trained alternately until discriminators cannot distinguish the fused image from the source ones. In addition, the symmetrical discriminator scheme is conducive to maintaining the feature consistency among different modalities. More importantly, to enhance the retention degree of texture details, U-Net is adopted as the generator heuristically, where the up-sampling method is modified to bilinear interpolation for avoiding checkerboard artifacts. As for the optimization, we define the content loss function, which preserves the gradient information and pixel activity of source images. Both visual analysis and quantitative evaluation of experimental results show the superiority of our method as compared to the cutting-edge baselines.

A Meta-Heuristic Multi-Objective Optimization Method for Alzheimer’s Disease Detection Based on Multi-Modal Data

Alzheimer’s disease (AD) is a neurodegenerative disease that affects a large number of people across the globe. Even though AD is one of the most commonly seen brain disorders, it is difficult to detect and it requires a categorical representation of features to differentiate similar patterns. Research into more complex problems, such as AD detection, frequently employs neural networks. Those approaches are regarded as well-understood and even sufficient by researchers and scientists without formal training in artificial intelligence. Thus, it is imperative to identify a method of detection that is fully automated and user-friendly to non-AI experts. The method should find efficient values for models’ design parameters promptly to simplify the neural network design process and subsequently democratize artificial intelligence. Further, multi-modal medical image fusion has richer modal features and a superior ability to represent information. A fusion image is formed by integrating relevant and complementary information from multiple input images to facilitate more accurate diagnosis and better treatment. This study presents a MultiAz-Net as a novel optimized ensemble-based deep neural network learning model that incorporate heterogeneous information from PET and MRI images to diagnose Alzheimer’s disease. Based on features extracted from the fused data, we propose an automated procedure for predicting the onset of AD at an early stage. Three steps are involved in the proposed architecture: image fusion, feature extraction, and classification. Additionally, the Multi-Objective Grasshopper Optimization Algorithm (MOGOA) is presented as a multi-objective optimization algorithm to optimize the layers of the MultiAz-Net. The desired objective functions are imposed to achieve this, and the design parameters are searched for corresponding values. The proposed deep ensemble model has been tested to perform four Alzheimer’s disease categorization tasks, three binary categorizations, and one multi-class categorization task by utilizing the publicly available Alzheimer neuroimaging dataset. The proposed method achieved (92.3 ± 5.45)% accuracy for the multi-class-classification task, significantly better than other network models that have been reported.

Water wave optimized nonsubsampled shearlet transformation technique for multimodal medical image fusion

SummaryMedical image fusion has advanced to the point that it is now possible to combine multiple medical images for accurate disease diagnosis and treatment. The state‐of‐art techniques based on spatial and transform domains suffer from different limitations such as low fused image quality, spectral degradation, contrast reduction, low edge information preserving, lack of shift‐invariance, high computational complexity, classification accuracy, and sensitivity to noise. The main motivation of this work is to generate a single image with excellent visual clarity that retains the features of the source images. This article proposes a water wave optimized nonsubsampled shearlet transformation technique (NSST) for multimodal medical image fusion, in which the water wave optimization (WWO) algorithm is used to allocate the weights of the NSST approach's high‐frequency subbands. The NSST approach is primarily used in this work due to its ability to withstand shift‐invariance and its potential to improve the visual clarity of the fused multimodal image by preserving the essential features present in the image's various directions and edges. We combined the NSST technique with the WWO algorithm, which processes the edges, details, and contourlets of medical images using a max selection strategy based on the fitness function, to improve image quality and computational costs. The WWO algorithm is mainly applied to the NSST to minimize the L1 distance between the fused and the source images. Hence to overcome this problem a condition CNN optimized with a hybrid tunicate swarm memetic (TSM) algorithm is used to incorporate both the benefits offered by the condition CNN–TSM algorithm and NSST. The TSM optimized condition CNN architecture is used to preserve the coefficients of the image and improve the perceiving capability of the high‐frequency sub‐bands. An inverse NSST is used for fused frequency sub‐band integration. Finally, the efficiency of the proposed methodology is evaluated in terms of enhanced visual feature quality, edge detection, contour detection, and computational performance.

A multimodal fusion method for Alzheimer's disease based on DCT convolutional sparse representation.

The medical information contained in magnetic resonance imaging (MRI) and positron emission tomography (PET) has driven the development of intelligent diagnosis of Alzheimer's disease (AD) and multimodal medical imaging. To solve the problems of severe energy loss, low contrast of fused images and spatial inconsistency in the traditional multimodal medical image fusion methods based on sparse representation. A multimodal fusion algorithm for Alzheimer' s disease based on the discrete cosine transform (DCT) convolutional sparse representation is proposed. The algorithm first performs a multi-scale DCT decomposition of the source medical images and uses the sub-images of different scales as training images, respectively. Different sparse coefficients are obtained by optimally solving the sub-dictionaries at different scales using alternating directional multiplication method (ADMM). Secondly, the coefficients of high-frequency and low-frequency subimages are inverse DCTed using an improved L1 parametric rule combined with improved spatial frequency novel sum-modified SF (NMSF) to obtain the final fused images. Through extensive experimental results, we show that our proposed method has good performance in contrast enhancement, texture and contour information retention.

Multimodal Medical Image Fusion Using Two- Stage Decomposition Technique to Combine the Significant Features of Spatial Fuzzy Plane and Transformed Frequency Plane

Multimodal medical image fusion is becoming a powerful technique in various clinical applications such as disease diagnosis and treatment planning. In this work, a two-stage decomposition technique is proposed to capture the significant features from the spatial and spectral domains as well. The first stage decomposition method employs a multicluster fuzzy c-means (FCM) algorithm to produce gross- and fine-structural components. The fine-structural components containing the edge topology are again decomposed into amplitude and phase spectra in terms of the discrete cosine transformation-based discrete Fourier transform (DCT-DFT). Finally, all parts of decomposed images are integrated with three different fusion rules. A new window-based feature quality measuring (WFQM) filter is proposed for fusing the information of gross-structural components, while the singular value decomposition (SVD) method and gray-wolf optimization-based pulse-coupled neural network (GWO-PCNN) model are applied for fusing the amplitude and phase spectrum distributions of input images. As the WFQM filter and GWO-PCNN model are designed to fuse the significant information from the spatial fuzzy plane and transformed frequency plane respectively, relative contrast degradation and spectral distortion are less in the proposed fusion method. Experimental results are reported on Harvard University datasets and some qualitative as well as quantitative measures are presented for comparing the proposed method with other state-of-art-techniques. The quantitative evaluation used peak signal-to-noise ratio (PSNR), mutual information (MI), structural similarity index measurement (SSIM), visual information fidelity (VIF), and entropy (ENT) to study its superiority.

Fidelity-driven Optimization Reconstruction and Details Preserving Guided Fusion for Multi-Modality Medical Image

By integrating effective features of multi-modality medical images to provide richer information, multi-modality medical image fusion has been substantially used in computer-aided diagnosis applications. However, many existing fusion schemes do not consider how to eliminate the effects of the noise in source medical images and cannot provide enough details and textures for disease diagnosis. To address the problems above, we propose a new fidelity-driven optimization (FDO) reconstruction and details preserving guided-based fusion method for multi-modality medical images. To overcome the influence of noise in multi-modality medical images, a rank coefficient optimization method of low-rank approximation based on weighted mean curvature is proposed to reconstruct multi-modality medical image. Moreover, we propose an iterative detail preserving guided fusion (DPGF) method to integrate more textures and detail information of source multi-modality medical images, while ensuring high signal-to noise ratios. The experimental results show that the proposed method outperforms some of the state-of-the-art fusion methods. Specifically, the extensive experiments prove that our method has high robustness for noisy medical images, which also indicates the application prospects in diagnosis applications. Supplemental material and codes of this work are publicly available at <uri>https://github.com/VCMHE/FDO_DPGF</uri>.

Using Generative Adversarial Networks and Ensemble Learning for Multi-Modal Medical Image Fusion to Improve the Diagnosis of Rare Neurological Disorders

Using Generative Adversarial Networks and Ensemble Learning for Multi-Modal Medical Image Fusion to Improve the Diagnosis of Rare Neurological Disorders

MRSCFusion: Joint Residual Swin Transformer and Multiscale CNN for Unsupervised Multimodal Medical Image Fusion

MRSCFusion: Joint Residual Swin Transformer and Multiscale CNN for Unsupervised Multimodal Medical Image Fusion

A Brief Analysis of Multimodal Medical Image Fusion Techniques

Recently, image fusion has become one of the most promising fields in image processing since it plays an essential role in different applications, such as medical diagnosis and clarification of medical images. Multimodal Medical Image Fusion (MMIF) enhances the quality of medical images by combining two or more medical images from different modalities to obtain an improved fused image that is clearer than the original ones. Choosing the best MMIF technique which produces the best quality is one of the important problems in the assessment of image fusion techniques. In this paper, a complete survey on MMIF techniques is presented, along with medical imaging modalities, medical image fusion steps and levels, and the assessment methodology of MMIF. There are several image modalities, such as Computed Tomography (CT), Positron Emission Tomography (PET), Magnetic Resonance Imaging (MRI), and Single Photon Emission Computed Tomography (SPECT). Medical image fusion techniques are categorized into six main categories: spatial domain, transform fusion, fuzzy logic, morphological methods, and sparse representation methods. The MMIF levels are pixel-level, feature-level, and decision-level. The fusion quality evaluation metrics can be categorized as subjective/qualitative and objective/quantitative assessment methods. Furthermore, a detailed comparison between obtained results for significant MMIF techniques is also presented to highlight the pros and cons of each fusion technique.

FDGNet: A pair feature difference guided network for multimodal medical image fusion

Most multimodal medical image fusion (MMIF) methods suffer from insufficient complementary feature extraction and luminance degradation, such that the fused results cannot effectively assist in clinical diagnosis. Therefore, a novel end-to-end unsupervised learning fusion network is proposed to address these defects, termed a pair feature difference guided network (FDGNet). To adequately extract complementary features from source images, the MMIF task is modeled as feature-weighted guided learning, where the feature extraction framework is dedicated to calculating the difference among features at various levels, such that the feature reconstruction framework can generate a pair of interactive weights via the guidance of the feature differences to directly produce the fused result. Simultaneously, a hybrid loss composed of weighted fidelity loss and feature difference loss is introduced to effectively train the proposed network. Particularly, a weight estimation is designed in the weighted fidelity loss, resorting to a joint of enhanced saliency and pixel intensity of source images to prevent luminance degradation of the fused image. Extensive experiments on six category multimodal medical images demonstrate that FDGNet not only preserves rich luminance (CT), tissue texture (MRI), and functional (PET/SPECT) details from source images but also improves the quantitative metrics of the NMI, QABF, QY, and VIFP with about 68.68%, 6.73%, 12.52%, and 18.33%, respectively over the second-best algorithm.

Hybrid Multimodal Medical Image Fusion Method Based on LatLRR and ED-D2GAN

In order to better preserve the anatomical structure information of Computed Tomography (CT) source images and highlight the metabolic information of lesion regions in Positron Emission Tomography (PET) source images, a hybrid multimodal medical image fusion method (LatLRR-GAN) based on Latent low-rank representation (LatLRR) and the dual discriminators Generative Adversarial Network (ED-D2GAN) is proposed. Firstly, considering the denoising capability of LatLRR, source images were decomposed by LatLRR. Secondly, the ED-D2GAN model was put forward as the low-rank region fusion method, which can fully extract the information contained by the low-rank region images. Among them, encoder and decoder networks were used in the generator; convolutional neural networks were also used in dual discriminators. Thirdly, a threshold adaptive weighting algorithm based on the region energy ratio is proposed as the salient region fusion rule, which can improve the overall sharpness of the fused image. The experimental results show that compared with the best methods of the other six methods, this paper is effective in multiple objective evaluation metrics, including the average gradient, edge intensity, information entropy, spatial frequency and standard deviation. The results of the two experiments are improved by 35.03%, 42.42%, 4.66%, 8.59% and 11.49% on average.

Multimodal medical volumetric image fusion using 3-D Shearlet transform and T-S fuzzy reasoning

Multimodal medical volumetric image fusion using 3-D Shearlet transform and T-S fuzzy reasoning

DFENet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion

In recent times, several medical image fusion techniques based on the convolutional neural network (CNN) have been proposed for various medical imaging fusion tasks. However, these methods cannot model the long-range dependencies between the fused image and the source images. To address this limitation, we propose DFENet, a multimodal medical image fusion framework that integrates CNN feature learning and vision transformer feature learning using self-supervised learning. DFENet is based on an encoder-decoder network, which can be trained on large-scale natural image dataset without the need for carefully collated ground truth fusion images. The proposed network consists of an encoder, a feature fuser, and a decoder. The encoder is composed of a CNN module and a transformer module, which is used to extract local and global features of images. In order to avoid the use of simple up-sampling and concatenate processing, a new global semantic information aggregation module is proposed to efficiently aggregate the multi-scale features obtained by the transformer module, which enhances the quality of the reconstructed images. The decoder is composed of six convolution layers with two skip connections, which are used for the reconstruction from fused features. We also propose a fusion strategy combining local energy and gradient information for the feature fusion process of magnetic resonance imaging and functional medical images. Compared to conventional fusion rules, our fusion strategy is more robust to noisy images. And compared with the existing competitive methods, our method retains more texture details of the original images and outputs a more natural and realistic fused image.

Multimodal medical image fusion using convolutional neural network and extreme learning machine.

The emergence of multimodal medical imaging technology greatly increases the accuracy of clinical diagnosis and etiological analysis. Nevertheless, each medical imaging modal unavoidably has its own limitations, so the fusion of multimodal medical images may become an effective solution. In this paper, a novel fusion method on the multimodal medical images exploiting convolutional neural network (CNN) and extreme learning machine (ELM) is proposed. As a typical representative in deep learning, CNN has been gaining more and more popularity in the field of image processing. However, CNN often suffers from several drawbacks, such as high computational costs and intensive human interventions. To this end, the model of convolutional extreme learning machine (CELM) is constructed by incorporating ELM into the traditional CNN model. CELM serves as an important tool to extract and capture the features of the source images from a variety of different angles. The final fused image can be obtained by integrating the significant features together. Experimental results indicate that, the proposed method is not only helpful to enhance the accuracy of the lesion detection and localization, but also superior to the current state-of-the-art ones in terms of both subjective visual performance and objective criteria.

Multi-modal medical image super-resolution fusion based on detail enhancement and weighted local energy deviation

Multi-modal medical image fusion (MMIF) integrates medical images of different modalities into an image with rich information to boost the accuracy and efficiency of clinical diagnosis and treatment. There are two main problems in medical image fusion: 1) It is difficult to balance the computational efficiency and fusion quality; 2) in the clinic, it is necessary to observe medical images in high resolution. To overcome these problems, a multi-modal medical image super-resolution (SR) fusion (MMISRF) method is proposed based on detail enhancement and weighted local energy deviation (WLED). The method has two major novelties. For the first problem, to improve the efficiency, the proposed method decomposes the SR image into one base layer and one detail layer through a two-scale decomposition method. Then, for the detail layer, a fusion rule is proposed based on detail enhancement and information refinement to enhance the detail information and retain the salient feature information. For the base layer, a WLED-based rule is designed to better preserve the energy information from the source images to the fused image. The final fused image is obtained by combining the fused detail and base layers. For the second problem, this paper introduces the bicubic interpolation-based SR into the field of MMIF for the first time. The experimental results indicate that the proposed MMISRF outperforms the state-of-the-art approaches in terms of subjective visual effect and objective evaluation. Furthermore, the proposed method is more effective for MMIF task at different resolutions.

Multimodal medical image fusion algorithm based on pulse coupled neural networks and nonsubsampled contourlet transform

Combining two medical images from different modalities is more helpful for using the resulting image in the healthcare field. Medical image fusion means combining two or more images coming from multiple sensors. This technology obtains an output image that presents more effective and useful information from two images. This paper proposes a multi-modal medical image fusion algorithm based on the nonsubsampled contourlet transform (NSCT) and pulse coupled neural networks (PCNN) methods. The input images are decomposed using the NSCT method into low- and high-frequency subbands. The PCNN is a fusion rule for integrating both low- and high-frequency subbands. The inverse of the NSCT method is to reconstruct the fused image. The results of medical image fusion help doctors with disease diagnosis and patient treatment. The proposed algorithm is tested on six groups of multi-modal medical images using 100 pairs of input images. The proposed algorithm is compared with eight fusion methods. We evaluate the performance of the proposed algorithm using the fusion metrics: peak signal to noise ratio (PSNR), mutual information (MI), entropy (EN), weighted edge information (Q^{AB/F}), nonlinear correlation information entropy (Q_{ncie}), standard deviation (SD), and average gradient (AG). Experimental results show that the proposed algorithm can perform better than other medical image fusion methods and achieve promising results.Graphical abstract

Tri-modal medical image fusion based on adaptive energy choosing scheme and sparse representation

Multimodal medical image fusion integrates useful information from multiple single-modal medical images, generating a more comprehensive and objective fused image that better assist clinical applications. In this paper, a novel tri-modal medical image fusion method based on cartoon-texture decomposition is proposed and performed using a rolling guidance filter, and sparse representation, to fuse the texture components. Furthermore, a novel adaptive energy choosing scheme is proposed to fuse the cartoon components; through this approach, the brightness of cartoon components can be effectively detected. Finally, the fused image is reconstructed by combining the fused texture and cartoon components. Experimental results demonstrate that the proposed method yields better performance than some state-of-the-art methods in subjective and objective assessments. Meanwhile, the average level of the proposed method are 28.44%, 8.94%, 0.07%, 16.09%, 58.66%, and 0.34% higher than the compared methods evaluated by the metrics including QMI, QTE, QNCIE, QG, QP and EN, respectively.

Enhanced JAYA optimization based medical image fusion in adaptive non subsampled shearlet transform domain

Multi-modal image fusion has gained popularity in the medical field as it assists doctors to view the diverse medical image modalities in a single image. The treatment is effectively planned by looking into the fused image that helps doctors diagnose diseases. The medical image fusion aims to merge the texture features from multiple images in a single image. The proposed method includes the application of Adaptive window-based Non-Subsampled Shearlet Transform (ANSST) on source images to separate the low and high-frequency directional sub-bands. Further, an enhanced JAYA (EJAYA) optimization framework is utilized to obtain the adaptive weights for combining high-frequency sub-bands for a multi-modal medical image fusion. The low-frequency bands are fused using the max rule based on the average energy of low-frequency sub-bands. The entire process focuses on preserving the low-frequency band’s energy while improving the texture details in the combined image. In the end, inverse ANSST is applied on merged low-frequency and high-frequency components to get the fused image. Extensive experiments are conducted on data sets obtained from the Brain Atlas website comprising more than 100 images. The significance of the current approach is validated by qualitative and quantitative assessments. The proposed method exhibits good performance in terms of subjective analysis compared to the recent well-known image fusion techniques.

An Adaptive Image Fusion Algorithm in the NSST Based on CDF 9/7 for Neurodegenerative Diseases

The neurodegenerative disease such as: Parkinson's disease (PD), mild Alzheimer’s affects many people and has a serious influence on their life, With the quick advancement of computer-aided diagnostic (CAD) methods, early detection is crucial since effective treatment halts the spread of the disease. Image fusion is useful for medical diagnostics. In this paper we propose a multi-modality medical image fusion algorithm in NSST domain. Shearlets (NSST) are decomposed similarly to contourlets (NSCT), except that instead of applying the Laplacian pyramid followed by directional filtering, shearlets use a shear matrix. In this article the Biorthogonal CDF9/7 filter is applied in the shift-invariant shearlet filter banks, then the coefficients of low frequency bands are selected using maximum rule, and using the gradient in each subband high frequency image to motivate the modified pulse coupled neural networks (Modified PCNN). Finally reverse IHS to get the fused color image, all this to optimize the calculation performance and improve the characteristics of the fused image for medical diagnosis. Our approach was validated with several brain diseases modalities: Alzheimer’s…etc. The findings reveal that the suggested image fusion technique has a higher quality than those fused by previous algorithms existing.