Multimodal Medical Image Fusion Research Articles

Multimodal Medical Image Fusion Network Based on Target Information Enhancement

Multimodal Medical Image Fusion Network Based on Target Information Enhancement

Multi Modalities Medical Image Fusion Using Deep Learning and Metaverse Technology: Healthcare 4.0 A Futuristic Approach

Multimodal medical image fusion is the efficient integration of various imaging modalities to improve the ability to assess, direct therapy, treat patients, or predict outcomes. As image fusion offers additional essential information, the correctness of the image generated from different medical imaging modalities has a substantial impact on the success of a disease's diagnosis. A single medical imaging modality cannot provide complete and precise information. In the modern research area, multimodality medical image fusion approach is one of the meaningful research in the area of medical imaging and radiation medicine. The fusion of medical images is the process of enrolling and combining multiple images from one or more imaging modalities, enhancing the image quality and to achieve randomness and redundancy, heighten the clinical utility of medical images in the diagnosis and evaluation of medical problems. The thought is to enhance the image occurrence Magnetic resonance imaging (MRI) is achieved by combining images like computerized tomography (CT) as well as magnetic resonance imaging (MRI) gives fine soft tissue information as long as CT gives fine facts over denser tissue. In this research paper, we have an account that features for future development with demanding performance requirements and processing speed.

Retracted: Multimodal Medical Image Fusion of Positron Emission Tomography and Magnetic Resonance Imaging Using Generative Adversarial Networks.

[This retracts the article DOI: 10.1155/2022/6878783.].

SegCoFusion: An Integrative Multimodal Volumetric Segmentation Cooperating With Fusion Pipeline to Enhance Lesion Awareness.

Multimodal volumetric segmentation and fusion are two valuable techniques for surgical treatment planning, image-guided interventions, tumor growth detection, radiotherapy map generation, etc. In recent years, deep learning has demonstrated its excellent capability in both of the above tasks, while these methods inevitably face bottlenecks. On the one hand, recent segmentation studies, especially the U-Net-style series, have reached the performance ceiling in segmentation tasks. On the other hand, it is almost impossible to capture the ground truth of the fusion in multimodal imaging, due to differences in physical principles among imaging modalities. Hence, most of the existing studies in the field of multimodal medical image fusion, which fuse only two modalities at a time with hand-crafted proportions, are subjective and task-specific. To address the above concerns, this work proposes an integration of multimodal segmentation and fusion, namely SegCoFusion, which consists of a novel feature frequency dividing network named FDNet and a segmentation part using a dual-single path feature supplementing strategy to optimize the segmentation inputs and suture with the fusion part. Furthermore, focusing on multimodal brain tumor volumetric fusion and segmentation, the qualitative and quantitative results demonstrate that SegCoFusion can break the ceiling both of segmentation and fusion methods. Moreover, the effectiveness of the proposed framework is also revealed by comparing it with state-of-the-art fusion methods on 2D two-modality fusion tasks, our method achieves better fusion performance than others. Therefore, the proposed SegCoFusion develops a novel perspective that improves the performance in volumetric fusion by cooperating with segmentation and enhances lesion awareness.

Survey on Multi-Modal Medical Image Fusion

Abstract: Multi-modality medical or clinical image fusion is a field of study aimed at enhancing diagnostic accuracy and aid in decisions to be taken by medical professional. Various fusion techniques such as pixel-based, region-based, and transformbased approaches are applied in image fusion to provide accurate fusion. Different devices which take scans of body such as MRI, CT, PET, SPECT, Ultrasound hold and carry different features, and different medical sensors obtain different information of the particular part of the body. Each of these imaging modalities offer only specific information that is used for the detection and analysis of specific problem. The idea behind fusion is to achieve and get better contrast and better fused image. The algorithm is making use of the common pyramid type and similarity type fusion algorithm with the neural networks model to achieve a better and more flexible fusion method. The advantages of image fusion medically are widespread. It plays a pivotal role in tumour localization, surgical planning and in treatment assessment.

Multimodal Medical Image Fusion Utilizing Two-scale Image Decomposition via Saliency Detection.

Modern medical imaging modalities used by clinicians have many applications in the diagnosis of complicated diseases. These imaging technologies reveal the internal anatomy and physiology of the body. The fundamental idea behind medical image fusion is to increase the image's global and local contrast, enhance the visual impact, and change its format so that it is better suited for computer processing or human viewing while preventing noise magnification and accomplishing excellent real-time performance. The top goal is to combine data from various modal images (CT/MRI and MR-T1/MR-T2) into a solitary image that, to the greatest degree possible, retains the key characteristics (prominent features) of the source images. The clinical accuracy of medical issues is compromised because innumerable classical fusion methods struggle to conserve all the prominent features of the original images. Furthermore, complex implementation, high computation time, and more memory requirements are key problems of transform domain methods. With the purpose of solving these problems, this research suggests a fusion framework for multimodal medical images that makes use of a multi-scale edge-preserving filter and visual saliency detection. The source images are decomposed using a two-scale edge-preserving filter into base and detail layers. Base layers are combined using the addition fusion rule, while detail layers are fused using weight maps constructed using the maximum symmetric surround saliency detection algorithm. The resultant image constructed by the presumed method has improved objective evaluation metrics than other classical methods, as well as unhindered edge contour, more global contrast, and no ringing effect or artifacts. The methodology offers a dominant and symbiotic arsenal of clinical symptomatic, therapeutic, and biomedical research competencies that have the prospective to considerably strengthen medical practice and biological understanding.

New Approach for Multimodal Medical Image Fusion using PDE-based Technique

Accurate diagnosis of diseases is essential for effective medical treatment, and medical image fusion plays a crucial role in achieving this goal. This article proposes a novel technique for multimodal medical image fusion based on fourth order PDE-based techniques. The proposed technique uses inputted CT and brain MRI images, and applies a PDE-based technique for noise reduction and edge preservation. Subsequently, a computed fusion rule using principal component analysis is applied to generate a fused image of the brain that presents clear and essential components for medical diagnosis and analysis. The article provides a comparative study of the proposed technique with other existing techniques, demonstrating its optimal performance. The proposed technique can significantly enhance medical image diagnosis and analysis for accurate disease detection and treatment.

Multi-modal medical image fusion via multi-dictionary and truncated Huber filtering

Multi-modal medical image fusion provides comprehensive and objective descriptions of lesions for clinical medical assistance. However, retaining useful information while achieving noise robustness remains challenging for existing techniques. In this paper, we propose a novel medical image fusion algorithm based on multi-dictionary convolutional sparse representation. Especially, truncated Huber filtering is first introduced to achieve detail-base layer decomposition of source images. Subsequently, multiple-dictionary decisions and nuclear energy-based rules are proposed to fuse the details and base layers, respectively. The fused image is reconstructed by synthesizing the fused detail and base components. The proposed model effectively fuses the source global structure and texture information and exhibits strong robustness against noise. Experiments involving extensive noise-free and noisy anatomical and functional medical image fusion on a public dataset covering five fusion categories demonstrate that the proposed method outperforms other state-of-the-art methods in subjective and objective evaluations. The source code of this study is publicly available at https://github.com/JEI981214/MDHU-fusion.

F-DARTS: Foveated Differentiable Architecture Search Based Multimodal Medical Image Fusion.

Multimodal medical image fusion (MMIF) is highly significant in such fields as disease diagnosis and treatment. The traditional MMIF methods are difficult to provide satisfactory fusion accuracy and robustness due to the influence of such possible human-crafted components as image transform and fusion strategies. Existing deep learning based fusion methods are generally difficult to ensure image fusion effect due to the adoption of a human-designed network structure and a relatively simple loss function and the ignorance of human visual characteristics during weight learning. To address these issues, we have presented the foveated differentiable architecture search (F-DARTS) based unsupervised MMIF method. In this method, the foveation operator is introduced into the weight learning process to fully explore human visual characteristics for the effective image fusion. Meanwhile, a distinctive unsupervised loss function is designed for network training by integrating mutual information, sum of the correlations of differences, structural similarity and edge preservation value. Based on the presented foveation operator and loss function, an end-to-end encoder-decoder network architecture will be searched using the F-DARTS to produce the fused image. Experimental results on three multimodal medical image datasets demonstrate that the F-DARTS performs better than several traditional and deep learning based fusion methods by providing visually superior fused results and better objective evaluation metrics.

AQWT: adaptive quaternion wavelet transform and hybrid heuristic improvement for multi-modal medical image fusion model

AQWT: adaptive quaternion wavelet transform and hybrid heuristic improvement for multi-modal medical image fusion model

Review on multi-modality medical image fusion using deep learning

Multi-Modality medical image fusion is a method in which multiple images are merged having either single or multiple imaging modalities. This process is carried out to improve the quality of imaging while preserving all the essential and distinct features. Many areas such as Machine Learning, Artificial Intelligence, Image Processing, and Computer Vision are covered by Medical Image Fusion. This method has been adopted on a large scale by physicians to apprehend any damage or injury caused in organ tissues in clinical trials by performing a fusion of images with different modalities. In this review, Deep Learning methods carried out in the medical image fusion field have been discussed along with a comparison between their accuracies. The main objective of this paper is to list some of the most effective techniques in this domain and discuss their performance. At last, the paper concludes with the fact that although the development and growth in this area have increased over the years, many challenges have also come along the way.

Enhanced multimodal medical image fusion based on Pythagorean fuzzy set: an innovative approach

Medical image fusion is the process of combining a multi-modality image into a single output image for superior information and a better visual appearance without any vagueness or uncertainties. It is suitable for better diagnosis. Pythagorean fuzzy set (PFS)-based medical image fusion was proposed in this manuscript. In the first phase, a two-scale gaussian filter was used to decompose the source images into base and detail layers. In the second phase, a spatial frequency (SF)-based fusion rule was employed for detail layers to preserve the more edge-oriented details. However, the base layer images were converted into pythagorean fuzzy images (PFIs) using the optimum value obtained by pythagorean fuzzy entropy (PFE). The blackness and whiteness count fusion rule were performed for image blocks decomposed from two PFIs in the third phase. Finally, the enhanced fused image was obtained by reconstructions of fused PFI blocks, which performed the defuzzification process. The proposed method was evaluated on different datasets for disease diagnosis and achieved better mean (M), standard deviation (SD), average gradient (AG), SF, modified spatial frequency (MSF), mutual information (MI), and fusion symmetry (FS) values than compared to state-of-art methods. This advancement is important in the field of healthcare and medical imaging, including enhanced diagnostics and treatment planning.

Multimodality medical image fusion analysis with multi-plane features of PET and MRI images using ONSCT

ABSTRACT The Multimodal Medical Image Fusion (MMIF) is affected by poor image quality, which leads to the extraction of inefficient features. The main intent of this work is to fuse various planes in the PET and MRI medical images efficiently using the MMIF approach. Initially, the sample images containing the axial plane of PET and MRI images are aggregated from standard datasets. Then, the collected images are employed for the decomposition process, which is accomplished via Optimal Non-Subsampled Contourlet Transform (ONSCT). The parameters in the NSCT are optimized using the Modified Water Strider Algorithm (MWSA. Once the images are decomposed, it is segmented into two sub-bands as high frequency and low-frequency sub-bands. Consequently, the high-frequency sub-bands of both PET and MRI images are fused by using the optimal weighted average fusion, in which the weight factor is obtained optimally by the MWSA algorithm. Similarly, the low-frequency sub-bands of both medical images are combined by sparse fusion technique. Finally, both the resultant fused images are subjected to Inverse Non-Subsampled Contourlet Transform (INSCT) to get desired fused images. The experimental findings suggest that the proposed model has effectively fused the images, and it also enhances the similarity score with axial planes.

Fusion of Magnetic Resonance Elastography Images With Computed Tomography and Magnetic Resonance Imaging Using the Human Visual System.

Magnetic resonance elastography (MRE) is used to assess the stiffness of the liver to rule out cirrhosis or fibrosis. The image, nevertheless, is regarded as shear-wave imaging and does not depict any anatomical features. Multimodality medical image fusion (MMIF), such as the fusion of MRE with computed tomography (CT) scan or magnetic resonance imaging (MRI), can help doctors optimize the advantages of each imaging technique. As a result, perceptions serve as valid and valuable assessment criteria. The contrast sensitivity function (CSF), which describes the rates of visual contrast sensitivity through the changing of spatial frequencies, is used mathematically to characterize the human visual system (HVS). As a result, we suggest novel methods for fusing images that use discrete wavelets transform (DWT) based on HVS and CSF models. Images from MRI or CT scan were combined with MRE images, and the outcomes were assessed both subjectively and objectively. Visual inspection of merging images was done throughout the qualitative analysis. The CT-MRE fused images in all four datasets were shown to be superior at maintaining bones and spatial resolution, despite the MRI-MRE being better at exhibiting soft tissues and contrast resolution. It is clear from all four datasets that the liver soft tissue in MRI and CT images mixed successfully with the red-colored stiffness distribution seen in MRE images. The proposed approach outperformed DWT, which produced visual artifacts such as signal loss. Quantitative evaluation using mean, standard deviation, and entropy showed that the generated images from the proposed technique performed better than the source images and DWT. Additionally, peak signal-to-noise ratio, mean square error, correlation coefficient, and structural similarity index measure were employed to compare the two fusion approaches, namely, MRI-MRE and CT-MRE. The comparison did not show the superiority of one approach over the other. In conclusion, both subjective and objective evaluation approaches revealed that the combined images contained more information and characteristics. Hence, the proposed method might be a useful procedure to diagnose and localize the stiffness regions on the liver soft tissue by fusion of MRE with MRI or CT.

Medical Diagnosis with Multimodal Image Fusion Techniques

Image Fusion is an effective approach utilized to draw out all the significant information from the source images, which supports experts in evaluation and quick decision making. Multi modal medical image fusion produces a composite fused image utilizing various sources to improve quality and extract complementary information. It is extremely challenging to gather every piece of information needed using just one imaging method. Therefore, images obtained from different modalities are fused Additional clinical information can be gleaned through the fusion of several types of medical image pairings. This study's main aim is to present a thorough review of medical image fusion techniques which also covers steps in fusion process, levels of fusion, various imaging modalities with their pros and cons, and the major scientific difficulties encountered in the area of medical image fusion. This paper also summarizes the quality assessments fusion metrics. The various approaches used by image fusion algorithms that are presently available in the literature are classified into four broad categories i) Spatial fusion methods ii) Multiscale Decomposition based methods iii) Neural Network based methods and iv) Fuzzy Logic based methods. the benefits and pitfalls of the existing literature are explored and Future insights are suggested. Moreover, this study is anticipated to create a solid platform for the development of better fusion techniques in medical applications.

A novel framework of multimodal medical image fusion using adaptive NSST and optimized deep learning approach

ABSTRACT Multimodal medical image fusion plays a pivotal role in the medical and imaging industry. Existing works of deep learning method suffers from blurred texture characteristics and computing efficiency. Thus, a novel deep learning model is proposed for multimodal medical image fusion. Initially, an Adaptive Non-Subsampled Shearlet Transform (ANSST) approach is developed for decomposing the images, where the filter coefficient is optimized by Hybrid Water Strider-Dingo Optimization (HWS-DOX). The fusion of the high sub-bands of source image 1 and high sub-bands of source image 2 is done by an Optimized Deep Neural Network (ODNN). Then, the lower sub-bands of source image 1 and lower sub-bands of source image 2 will be fused through a weighted averaging scheme. Finally, the final fused images are attained by applying the inverse ANSST. Thus, the performance of recommended module is compared over classical heuristics and various transform methods.

IE-CFRN: Information exchange-based collaborative feature representation network for multi-modal medical image fusion

Multi-modal medical image fusion (MMIF) is a valuable approach to integrating functional metabolic information and tissue structural details from different modalities, which facilitates clinical diagnosis and surgical navigation efficiently. In this paper, we propose a collaborative feature representation network based on information exchange called IE-CFRN, which distributes the contribution of each encoder feature at the channel level to achieve more accurate fusion. Furthermore, due to the different emphasis on shallow-level and high-level features, we construct a hierarchical feature enhancement network (HFEN) to integrate significant information on multi-level features. Additionally, to adaptively estimate the contribution coefficient in the fidelity loss for guiding the training process, we introduce a learnable weight estimation network (LWEN). Extensive experimentation on eleven state-of-the-art algorithms shows that our IE-CFRN method yields superior consequences with respect to visual effects, and qualitative and quantitative evaluation.

Gradient modulated contrastive distillation of low-rank multi-modal knowledge for disease diagnosis.

The fusion of multi-modal data, e.g., medical images and genomic profiles, can provide complementary information and further benefit disease diagnosis. However, multi-modal disease diagnosis confronts two challenges: (1) how to produce discriminative multi-modal representations by exploiting complementary information while avoiding noisy features from different modalities. (2) how to obtain an accurate diagnosis when only a single modality is available in real clinical scenarios. To tackle these two issues, we present a two-stage disease diagnostic framework. In the first multi-modal learning stage, we propose a novel Momentum-enriched Multi-Modal Low-Rank (M3LR) constraint to explore the high-order correlations and complementary information among different modalities, thus yielding more accurate multi-modal diagnosis. In the second stage, the privileged knowledge of the multi-modal teacher is transferred to the unimodal student via our proposed Discrepancy Supervised Contrastive Distillation (DSCD) and Gradient-guided Knowledge Modulation (GKM) modules, which benefit the unimodal-based diagnosis. We have validated our approach on two tasks: (i) glioma grading based on pathology slides and genomic data, and (ii) skin lesion classification based on dermoscopy and clinical images. Experimental results on both tasks demonstrate that our proposed method consistently outperforms existing approaches in both multi-modal and unimodal diagnoses.

Hierarchical Progressive Network for Multimodal Medical Image Fusion in Healthcare Systems

Deep learning (DL)-based multisource information processing plays an essential role in the Internet of Medical Things (IoMT). In this field, medical image fusion integrates scan results from different devices, supporting healthcare systems to make a more informed diagnosis. This study proposes a DL-based network to fuse multimodal medical images. In our method, the lattice unit (LU) is designed to improve the representation capability of the fusion network. Moreover, to acquire hierarchical features from images, the progressive module (PM) treats the network's shallow and deep layers differently. The shallow layers represent the structure of the source image; the deep layers correspond to details. Different loss functions are utilized for these two kinds of information to retain fused images' salient structures and functional information. Experiments show that the proposed algorithm performs well on visual quality and objective evaluation, which provides a reliable reference for medical diagnosis. In addition, this method is lightweight and speedy compared to existing algorithms, facilitating further placement in the IoMT's specific devices.

A Systematic Literature Review on Multimodal Medical Image Fusion

A Systematic Literature Review on Multimodal Medical Image Fusion