Multimodal Medical Image Fusion Research Articles

Discrete Wavelet Transformation based Multimodal Medical Image Fusion for Disease Identification

In medical science, image processing techniques play a significant function. Computational automation of the treatment is the most authentic and prominent method. The disease of the brain is identified using Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). Many more scan variations of MRI and PET have been executed for the medical diagnosis. The medical expert needs a solid strain of the computational scan and it’s related for diagnosis. The current era of computer research is turning towards clinical diagnosis and etiological analysis based on multimodal image processing. Based on individual medical modalities the accuracy of diseases diagnosis is decreases. Medical community requires a high accuracy in the disease’s identification based on multimodal scan image data. For the diagnosis and treatment of disorders requires precise information that is attained through various modalities of medical images such as Computed Tomography (CT), Positron Emission Tomography (PET), and Magnetic Resonance Imaging (MRI). In image processing the image fusion is the method of merging two images into a single picture. The obtained single fused image using various multi-modality medical images is enhanced anatomical, highly desirable spectral information compared to the raw single scanned image. This multi – modal fused image is useful for clinical diagnosis of medical experts. In this research work, the system is prepared for the preprocessing of the MRI and PET scan images. The pre-processing techniques enhance the quality of the input images which are degraded and non-readable. For the pre-processing approach we have applied the Gaussian filters of spatial filtering techniques. The enhanced images passed to the fusion of different region of brain images using Discrete Wavelet Transform (DWT). The system achieved around 90- 95% more accurate outcomes by diluting the color change. The outcome fused image is achieved without losing the spectral and anatomical data. The experiment has been tested on Alzheimer’s, normal axis, and normal coronal brain disease images dataset. The quantitative and graphical analysis indicates that’s the Discrete Wavelet Transform (DWT) significantly improves the quality of fused images.

Multi-modal medical image fusion using improved dual-channel PCNN.

This paper proposes a medical image fusion method in the non-subsampled shearlet transform (NSST) domain to combine a gray-scale image with the respective pseudo-color image obtained through different imaging modalities. The proposed method applies a novel improved dual-channel pulse-coupled neural network (IDPCNN) model to fuse the high-pass sub-images, whereas the Prewitt operator is combined with maximum regional energy (MRE) to construct the fused low-pass sub-image. First, the gray-scale image and luminance of the pseudo-color image are decomposed using NSST to find the respective sub-images. Second, the low-pass sub-images are fused by the Prewitt operator and MRE-based rule. Third, the proposed IDPCNN is utilized to get the fused high-pass sub-images from the respective high-pass sub-images. Fourth, the luminance of the fused image is obtained by applying inverse NSST on the fused sub-images, which is combined with the chrominance components of the pseudo-color image to construct the fused image. A total of 28 diverse medical image pairs, 11 existing methods, and nine objective metrics are used in the experiment. Qualitative and quantitative fusion results show that the proposed method is competitive with and even outpaces some of the existing medical fusion approaches. It is also shown that the proposed method efficiently combines two gray-scale images.

MACTFusion: Lightweight Cross Transformer for Adaptive Multimodal Medical Image Fusion.

Multimodal medical image fusion aims to integrate complementary information from different modalities of medical images. Deep learning methods, especially recent vision Transformers, have effectively improved image fusion performance. However, there are limitations for Transformers in image fusion, such as lacks of local feature extraction and cross-modal feature interaction, resulting in insufficient multimodal feature extraction and integration. In addition, the computational cost of Transformers is higher. To address these challenges, in this work, we develop an adaptive cross-modal fusion strategy for unsupervised multimodal medical image fusion. Specifically, we propose a novel lightweight cross Transformer based on cross multi-axis attention mechanism. It includes cross-window attention and cross-grid attention to mine and integrate both local and global interactions of multimodal features. The cross Transformer is further guided by a spatial adaptation fusion module, which allows the model to focus on the most relevant information. Moreover, we design a special feature extraction module that combines multiple gradient residual dense convolutional and Transformer layers to obtain local features from coarse to fine and capture global features. The proposed strategy significantly boosts the fusion performance while minimizing computational costs. Extensive experiments, including clinical brain tumor image fusion, have shown that our model can achieve clearer texture details and better visual quality than other state-of-the-art fusion methods.

Medical Image Fusion Based on Local Saliency Energy and Multi-scale Fractal Dimension.

At present, there are some problems in multimodal medical image fusion, such as texture detail loss, leading to edge contour blurring and image energy loss, leading to contrast reduction. To solve these problems and obtain higher-quality fusion images, this study proposes an image fusion method based on local saliency energy and multi-scale fractal dimension. First, by using a non-subsampled contourlet transform, the medical image was divided into 4 layers of high-pass subbands and 1 layer of low-pass subband. Second, in order to fuse the high-pass subbands of layers 2 to 4, the fusion rules based on a multi-scale morphological gradient and an activity measure were used as external stimuli in pulse coupled neural network. Third, a fusion rule based on the improved multi-scale fractal dimension and new local saliency energy was proposed, respectively, for the low-pass subband and the 1st closest to the low-pass subband. Layerhigh pass sub-bands were fused. Lastly, the fused image was created by performing the inverse non-subsampled contourlet transform on the fused sub-bands. On three multimodal medical image datasets, the proposed method was compared with 7 other fusion methods using 5 common objective evaluation metrics. Experiments showed that this method can protect the contrast and edge of fusion image well and has strong competitiveness in both subjective and objective evaluation.

Multimodal medical image fusion and classification using deep learning techniques

Multimodal medical image fusion and classification using deep learning techniques

End-to-end dynamic residual focal transformer network for multimodal medical image fusion

End-to-end dynamic residual focal transformer network for multimodal medical image fusion

A multi-directional fractional-order variation with luminance term for infrared and visible image fusion

Infrared and visible image fusion strives to effectively combine the advantageous information from input images to achieve a fused result. In this paper, we propose a novel multi-directional fractional-order variation to fuse infrared-visible images. First, each fidelity term of each source image is modeled straightforwardly by Euclidean norm, ensuring robust optimization. Then, the detail regularization term is formulated based on fractional variation in two, four, and eight directions instead of integral variation, which enables the capture of more comprehensive detail while avoiding the undesirable staircase effect. Furthermore, the fused image is enhanced by transferring the highest level of brightness from the source images in the luminance regularization. Finally, based on the extensive experiments, the proposed method exhibits superior performance in both subjective and objective evaluations compared to existing others. Moreover, our method is further expanded to the multi-modal medical image fusion, achieving promising performance preliminarily.

Advancements In Multi-Modality Medical Image Fusion: A Comprehensive Review

Multimodality medical image fusion involves the amalgamation of multiple images acquired through single or multiple imaging modalities. The purpose of medical image fusion methods is to enhance the quality of medical images by effectively capturing the key features within the fused output. This consequently enhances the practicality of medical images for diagnostic assessment and issue identification. Medical imaging modalities like magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET) have been developed and are extensively utilized for clinical diagnoses. These multimodality images harbor vital information crucial for precise and efficient diagnosis of brain ailments. As a result, amalgamating diverse image modalities in the medical domain to create a distinct image brimming with intricate anatomical details and heightened spectral information has become exceedingly valuable in clinical diagnosis. This manuscript offers an exhaustive examination of existing literature on image fusion, elucidating the fundamental concepts and materials essential for a comprehensive grasp of diverse medical fusion techniques.

Advancing multimodal medical image fusion: an adaptive image decomposition approach based on multilevel Guided filtering.

With the rapid development of medical imaging methods, multimodal medical image fusion techniques have caught the interest of researchers. The aim is to preserve information from diverse sensors using various models to generate a single informative image. The main challenge is to derive a trade-off between the spatial and spectral qualities of the resulting fused image and the computing efficiency. This article proposes a fast and reliable method for medical image fusion depending on multilevel Guided edge-preserving filtering (MLGEPF) decomposition rule. First, each multimodal medical image was divided into three sublayer categories using an MLGEPF decomposition scheme: small-scale component, large-scale component and background component. Secondly, two fusion strategies-pulse-coupled neural network based on the structure tensor and maximum based-are applied to combine the three types of layers, based on the layers' various properties. The three different types of fused sublayers are combined to create the fused image at the end. A total of 40 pairs of brain images from four separate categories of medical conditions were tested in experiments. The pair of images includes various case studies including magnetic resonance imaging (MRI) , TITc, single-photon emission computed tomography (SPECT) and positron emission tomography (PET). We included qualitative analysis to demonstrate that the visual contrast between the structure and the surrounding tissue is increased in our proposed method. To further enhance the visual comparison, we asked a group of observers to compare our method's outputs with other methods and score them. Overall, our proposed fusion scheme increased the visual contrast and received positive subjective review. Moreover, objective assessment indicators for each category of medical conditions are also included. Our method achieves a high evaluation outcome on feature mutual information (FMI), the sum of correlation of differences (SCD), Qabf and Qy indexes. This implies that our fusion algorithm has better performance in information preservation and efficient structural and visual transferring.

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

LRFNet: A real-time medical image fusion method guided by detail information

Multimodal medical image fusion (MMIF) technology plays a crucial role in medical diagnosis and treatment by integrating different images to obtain fusion images with comprehensive information. Deep learning-based fusion methods have demonstrated superior performance, but some of them still encounter challenges such as imbalanced retention of color and texture information and low fusion efficiency. To alleviate the above issues, this paper presents a real-time MMIF method, called a lightweight residual fusion network. First, a feature extraction framework with three branches is designed. Two independent branches are used to fully extract brightness and texture information. The fusion branch enables different modal information to be interactively fused at a shallow level, thereby better retaining brightness and texture information. Furthermore, a lightweight residual unit is designed to replace the conventional residual convolution in the model, thereby improving the fusion efficiency and reducing the overall model size by approximately 5 times. Finally, considering that the high-frequency image decomposed by the wavelet transform contains abundant edge and texture information, an adaptive strategy is proposed for assigning weights to the loss function based on the information content in the high-frequency image. This strategy effectively guides the model toward preserving intricate details. The experimental results on MRI and functional images demonstrate that the proposed method exhibits superior fusion performance and efficiency compared to alternative approaches. The code of LRFNet is available at https://github.com/HeDan-11/LRFNet.

Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics

Advancements in artificial intelligence (AI) have driven extensive research into developing diverse multimodal data analysis approaches for smart healthcare. There is a scarcity of large-scale analysis of literature in this field based on quantitative approaches. This study performed a bibliometric and topic modeling examination on 683 articles from 2002 to 2022, focusing on research topics and trends, journals, countries/regions, institutions, authors, and scientific collaborations. Results showed that, firstly, the number of articles has grown from 1 in 2002 to 220 in 2022, with a majority being published in interdisciplinary journals that link healthcare and medical research and information technology and AI. Secondly, the significant rise in the quantity of research articles can be attributed to the increasing contribution of scholars from non-English speaking countries/regions and the noteworthy contributions made by authors in the USA and India. Thirdly, researchers show a high interest in diverse research issues, especially, cross-modality magnetic resonance imaging (MRI) for brain tumor analysis, cancer prognosis through multi-dimensional data analysis, and AI-assisted diagnostics and personalization in healthcare, with each topic experiencing a significant increase in research interest. There is an emerging trend towards issues such as applying generative adversarial networks and contrastive learning for multimodal medical image fusion and synthesis and utilizing the combined spatiotemporal resolution of functional MRI and electroencephalogram in a data-centric manner. This study is valuable in enhancing researchers’ and practitioners’ understanding of the present focal points and upcoming trajectories in AI-powered smart healthcare based on multimodal data analysis.

A Novel Hybrid Multimodal Medical Image Fusion Scheme Based on Non-subsampled Shearlet Transform

A Novel Hybrid Multimodal Medical Image Fusion Scheme Based on Non-subsampled Shearlet Transform

Adaptive convolutional sparsity with sub-band correlation in the NSCT domain for MRI image fusion

Objective. Multimodal medical image fusion (MMIF) technologies merges diverse medical images with rich information, boosting diagnostic efficiency and accuracy. Due to global optimization and single-valued nature, convolutional sparse representation (CSR) outshines the standard sparse representation (SR) in significance. By addressing the challenges of sensitivity to highly redundant dictionaries and robustness to misregistration, an adaptive convolutional sparsity scheme with measurement of the sub-band correlation in the non-subsampled contourlet transform (NSCT) domain is proposed for MMIF. Approach. The fusion scheme incorporates four main components: image decomposition into two scales, fusion of detail layers, fusion of base layers, and reconstruction of the two scales. We solved a Tikhonov regularization optimization problem with source images to obtain the base and detail layers. Then, after CSR processing, detail layers were sparsely decomposed using pre-trained dictionary filters for initial coefficient maps. NSCT domain’s sub-band correlation was used to refine fusion coefficient maps, and sparse reconstruction produced the fused detail layer. Meanwhile, base layers were fused using averaging. The final fused image was obtained via two-scale reconstruction. Main results. Experimental validation of clinical image sets revealed that the proposed fusion scheme can not only effectively eliminate the interference of partial misregistration, but also outperform the representative state-of-the-art fusion schemes in the preservation of structural and textural details according to subjective visual evaluations and objective quality evaluations. Significance. The proposed fusion scheme is competitive due to its low-redundancy dictionary, robustness to misregistration, and better fusion performance. This is achieved by training the dictionary with minimal samples through CSR to adaptively preserve overcompleteness for detail layers, and constructing fusion activity level with sub-band correlation in the NSCT domain to maintain CSR attributes. Additionally, ordering the NSCT for reverse sparse representation further enhances sub-band correlation to promote the preservation of structural and textural details.

Medical image fusion based on machine learning for health diagnosis and monitoring of colorectal cancer

With the rapid development of medical imaging technology and computer technology, the medical imaging artificial intelligence of computer-aided diagnosis based on machine learning has become an important part of modern medical diagnosis. With the application of medical image security technology, people realize that the difficulty of its development is the inherent defect of advanced image processing technology. This paper introduces the background of colorectal cancer diagnosis and monitoring, and then carries out academic research on the medical imaging artificial intelligence of colorectal cancer diagnosis and monitoring and machine learning, and finally summarizes it with the advanced computational intelligence system for the application of safe medical imaging.In the experimental part, this paper wants to carry out the staging preparation stage. It was concluded that the staging preparation stage of group Y was higher than that of group X and the difference was statistically significant. Then the overall accuracy rate of multimodal medical image fusion was 69.5% through pathological staging comparison. Finally, the diagnostic rate, the number of patients with effective treatment and satisfaction were analyzed. Finally, the average diagnostic rate of the new diagnosis method was 8.75% higher than that of the traditional diagnosis method. With the development of computer science and technology, the application field was expanding constantly. Computer aided diagnosis technology combining computer and medical images has become a research hotspot.

GeSeNet: A General Semantic-Guided Network With Couple Mask Ensemble for Medical Image Fusion.

At present, multimodal medical image fusion technology has become an essential means for researchers and doctors to predict diseases and study pathology. Nevertheless, how to reserve more unique features from different modal source images on the premise of ensuring time efficiency is a tricky problem. To handle this issue, we propose a flexible semantic-guided architecture with a mask-optimized framework in an end-to-end manner, termed as GeSeNet. Specifically, a region mask module is devised to deepen the learning of important information while pruning redundant computation for reducing the runtime. An edge enhancement module and a global refinement module are presented to modify the extracted features for boosting the edge textures and adjusting overall visual performance. In addition, we introduce a semantic module that is cascaded with the proposed fusion network to deliver semantic information into our generated results. Sufficient qualitative and quantitative comparative experiments (i.e., MRI-CT, MRI-PET, and MRI-SPECT) are deployed between our proposed method and ten state-of-the-art methods, which shows our generated images lead the way. Moreover, we also conduct operational efficiency comparisons and ablation experiments to prove that our proposed method can perform excellently in the field of multimodal medical image fusion. The code is available at https://github.com/lok-18/GeSeNet.

Multimodal medical image fusion method based on structural functional cross neural network

Multimodal medical image fusion method based on structural functional cross neural network

Multi-Dimensional Medical Image Fusion With Complex Sparse Representation.

In the field of medical imaging, the fusion of data from diverse modalities plays a pivotal role in advancing our understanding of pathological conditions. Sparse representation (SR), a robust signal modeling technique, has demonstrated noteworthy success in multi-dimensional (MD) medical image fusion. However, a fundamental limitation appearing in existing SR models is their lack of directionality, restricting their efficacy in extracting anatomical details from different imaging modalities. To tackle this issue, we propose a novel directional SR model, termed complex sparse representation (ComSR), specifically designed for medical image fusion. ComSR independently represents MD signals over directional dictionaries along specific directions, allowing precise analysis of intricate details of MD signals. Besides, current studies in medical image fusion mostly concentrate on addressing either 2D or 3D fusion problems. This work bridges this gap by proposing a MD medical image fusion method based on ComSR, presenting a unified framework for both 2D and 3D fusion tasks. Experimental results across six multi-modal medical image fusion tasks, involving 93 pairs of 2D source images and 20 pairs of 3D source images, substantiate the superiority of our proposed method over 11 state-of-the-art 2D fusion methods and 4 representative 3D fusion methods, in terms of both visual quality and objective evaluation. The source code of our fusion method is available at https://github.com/Imagefusions/imagefusions/tree/main.

Multimodality Medical Image Fusion Based on Pixel Significance with Edge-Preserving Processing for Clinical Applications

Multimodality Medical Image Fusion Based on Pixel Significance with Edge-Preserving Processing for Clinical Applications

Multimodal Medical Image Fusion Network Based on Target Information Enhancement

Multimodal Medical Image Fusion Network Based on Target Information Enhancement