Multimodal Medical Image Fusion Method Research Articles

Multimodal medical image fusion method based on structural functional cross neural network

Multimodal medical image fusion method based on structural functional cross neural network

Shearlet Transform-Based Novel Method for Multimodality Medical Image Fusion Using Deep Learning

Multi-modality medical image fusion (MMIF) methods were widely used in a variety of clinical settings. For specialists, MMIF could provide an image containing anatomical and physiological information that can help develop diagnostic procedures. Different models linked to MMIF were proposed previously. However, there would be a need to enhance the functionality of prior methodologies. In this proposed model, a unique fusion model depending upon optimal thresholding and deep learning approaches are presented. An enhanced monarch butterfly optimization (EMBO) determines an optimal threshold with fusion rules as in shearlet transform. The efficiency of the fusion process mainly depends on the fusion rule and the optimization of the fusion rule can improve the efficiency of the fusion. The extraction element of the deep learning approach was then utilized to fuse high- and low-frequency sub-bands. The fusion technique was carried out using a convolutional neural network (CNN). The studies were carried out for MRI and CT images. The fusion results were attained and the proposed model was proved to offer effective performance with reduced values of error and improved values of correlation.

Hybrid Multimodal Medical Image Fusion Method Based on LatLRR and ED-D2GAN

In order to better preserve the anatomical structure information of Computed Tomography (CT) source images and highlight the metabolic information of lesion regions in Positron Emission Tomography (PET) source images, a hybrid multimodal medical image fusion method (LatLRR-GAN) based on Latent low-rank representation (LatLRR) and the dual discriminators Generative Adversarial Network (ED-D2GAN) is proposed. Firstly, considering the denoising capability of LatLRR, source images were decomposed by LatLRR. Secondly, the ED-D2GAN model was put forward as the low-rank region fusion method, which can fully extract the information contained by the low-rank region images. Among them, encoder and decoder networks were used in the generator; convolutional neural networks were also used in dual discriminators. Thirdly, a threshold adaptive weighting algorithm based on the region energy ratio is proposed as the salient region fusion rule, which can improve the overall sharpness of the fused image. The experimental results show that compared with the best methods of the other six methods, this paper is effective in multiple objective evaluation metrics, including the average gradient, edge intensity, information entropy, spatial frequency and standard deviation. The results of the two experiments are improved by 35.03%, 42.42%, 4.66%, 8.59% and 11.49% on average.

A Disentangled Representation Based Brain Image Fusion via Group Lasso Penalty.

Complementary and redundant relationships inherently exist between multi-modal medical images captured from the same brain. Fusion processes conducted on intermingled representations can cause information distortion and the loss of discriminative modality information. To fully exploit the interdependency between source images for better feature representation and improve the fusion accuracy, we present the multi-modal brain medical image fusion method in a disentangled pipeline under the deep learning framework. A three-branch auto-encoder with two complementary branches and a redundant branch is designed to extract the exclusive modality features and common structure features from input images. Especially, to promote the disentanglement of complement and redundancy, a complementary group lasso penalty is proposed to constrain the extracted feature maps. Then, based on the disentangled representations, different fusion strategies are adopted for complementary features and redundant features, respectively. The experiments demonstrate the superior performance of the proposed fusion method in terms of structure preservation, visual quality, and running efficiency.

Medical image fusion based on improved multi-scale morphology gradient-weighted local energy and visual saliency map

Medical image fusion refers to the process of fusing images of different modes of the same object using image processing technology to maximize the mining of image information and improve image quality. At present, many methods are available to study image fusion, but they usually have shortcomings such as low image contrast and a weak ability to retain image details and edges. To solve these problems, we propose a new multimodal medical image fusion method. In this algorithm, the original image is decomposed into high-frequency and low-frequency information by non-subsampled shearlet transform (NSST). Low-frequency information is fused by visual saliency maps, which avoids edge loss caused by direct use of the coefficient maximum fusion rule. High-frequency information is fused by a method jointly guided by the improved multi-scale morphology gradient and weighted sum of eight-neighborhood-based modified Laplacian, which retains the texture details and the edge of the image. Finally, the fused image is generated by the NSST inverse transform. This strategy solves the problem of insufficient detail extraction in traditional algorithms, improves the overall appearance of the fused image, and enhances the contrast. To verify the effectiveness of the algorithm, we applied this technique to four different medical image modality combinations, compared the results with nine image fusion methods published in recent years, and evaluated the fused images using image quality evaluation indexes. Our algorithm achieved better results in terms of subjective vision and objective image quality evaluations and therefore should be competitive with existing technologies.

Fusion of medical images based on salient features extraction by PSO optimized fuzzy logic in NSST domain

The aim of multimodal medical image fusion is to extract information from different modal images into a single one so that the single fusion image contains the salient features of the source images to the maximum extent. Different medical imaging modalities such as CT and MRI are presented as different visual morphology, which often show different complementary salient features in clinical diagnosis. According to this characteristic, in order to fuse different salient features of multimodal medical images into a single image and provide a new dimension of information for clinical diagnosis to improve the diagnosis accuracy, an improved image fusion algorithm based on visual salience detection is proposed in this paper. The visual salience of two registered source images is calculated by the GBVS algorithm, and the low-frequency and high-frequency sub-bands are obtained by decomposing the source images in NSST domain. For the low-frequency sub-bands, fuzzy logic system is used to obtain the respective weights of the fused low-frequency sub-band with the local energy and GBVS graph as input. For the high-frequency sub-bands, the NSML values of each sub-band are calculated and compared to obtain the fused high-frequency sub-band. The final fused image is obtained by the inverse NSST transformation. In addition, PSO algorithm is used to optimize the membership function of fuzzy logic system for better adaption to medical images and feature extraction. By applying this multimodal medical image fusion method, the visual quality of the image is effectively improved and the salient features of tissues are preserved well. Experiments on fusion of grayscale, color and ultrasound multimodal medical images show that the proposed method has advantages on retention of image salient features. Compared with other models, the fused image obtained by the proposed method has higher objective indexes with clearer edge contour, higher overall contrast, and no ringing effect and artifacts.

Measurement and analysis of multi-modal image fusion metrics based on structure awareness using domain transform filtering

Due to varrying imaging principles and interwined complexity of human organ structures, different types of medical images must be combined, as single-modality medical images may only provide limited information. In this paper, a multimodal medical image fusion method that integrates multimodal medical images having low resolution with reduced computational complexity to improve the accuracy of target recognition and for providing a basis for clinical diagnosis. Initially salient structure extraction (SSE) approach, which employ a rolling guidance filter (RGF) over the source images for removing small scale structures while preserving the image textures and thereby recovering the salient edges has been implemented. Subsequently image gradient operator is employed to restores large-scale structures from the filtered images. A DTF (Domain Transfer Filtering) is further used to recover the small-scale details in the neighborhood of large-scale structures of the images. The output of DTF is used as a weighted map that is combined with the source images to recover fusion result by a weighted-sum rule. Image fusion measurement for quality assessment and objective analysis is carried out using various fusion metrics. Experimental result shows that the proposed method can obtain high quantitative and qualitative performance as compared to other state-of-the-art methods and can eventually provide effective reference for doctors to assess patient condition.

Multi-modal medical image fusion framework using co-occurrence filter and local extrema in NSST domain

The fusion of vital imaging information has turned into a primary issue for biomedical applications. In this paper, a new multi-modality medical image fusion method is proposed in the shearlet domain. In the proposed algorithm, input images are decomposed using Non-subsampled shearlet transform (NSST) to get low and high frequencies components. A novel procedure to decompose and combine the base layers and detail layers using the local extrema (LE) approach is used and fused using a Co-occurrence filter (CoF) in low-frequency components. In high-frequency components, an edge-preserving image fusion strategy is performed using sum modified Laplacian (SML) to combine the high-frequency coefficients. The experimental outcomes and comparative analysis are performed over the Multi-modal medical image dataset using proposed and existing methods. It is exhibited through test outcomes and assessments that the proposed technique beats the cutting-edge fusion strategies concerning edge preservation in subjective and objective assessment criteria.

Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain.

Multimodal medical image fusion aims to fuse images with complementary multisource information. In this paper, we propose a novel multimodal medical image fusion method using pulse coupled neural network (PCNN) and a weighted sum of eight-neighborhood-based modified Laplacian (WSEML) integrating guided image filtering (GIF) in non-subsampled contourlet transform (NSCT) domain. Firstly, the source images are decomposed by NSCT, several low- and high-frequency sub-bands are generated. Secondly, the PCNN-based fusion rule is used to process the low-frequency components, and the GIF-WSEML fusion model is used to process the high-frequency components. Finally, the fused image is obtained by integrating the fused low- and high-frequency sub-bands. The experimental results demonstrate that the proposed method can achieve better performance in terms of multimodal medical image fusion. The proposed algorithm also has obvious advantages in objective evaluation indexes VIFF, QW, API, SD, EN and time consumption.

Multimodal medical image fusion based on joint bilateral filter and local gradient energy

• An effective, fast and noise robust multimodal medical image fusion method is proposed. An image is modeled as a superposition of the structure layer and the energy layer decomposed by joint filter, then a novel local gradient energy operator is proposed to fuse the structure layer and the abs-max rule to fuse the energy layer. • 118 co-registered pairs of medical images cover five different categories of medical image fusion problems (MR-T1/MR-T2, CT/MR, MR-Gad/MR-T2, MR/PET, and MR/SPECT) are tested. To the best of our knowledge, this is the most comprehensive experiment both in quantity and classes for medical image fusion. • The merits of noniterative, local, and without multiscale decomposition or reconstruction make the proposed method easy to implementation and achieve high computational efficiency. • Our method can be effectively extended to other image fusion problems, such as multifocus image fusion, infrared and visible image fusion, millimeter-wave and visible image fusion, panchromatic and multispectral image fusion. As a powerful assistance technique for biomedical diagnosis, multimodal medical image fusion has emerged as a hot topic in recent years. Unfortunately, the trade-off among fusion performance, time consumption and noise robustness for many medical image fusion algorithms remains an enormous challenge. In this paper, an effective, fast and robust medical image fusion method is proposed. A two-layer decomposition scheme is introduced by the joint bilateral filter, the energy layer containing rich intensity information, and the structure layer capturing ample details. Then a novel local gradient energy operator based on the structure tensor and neighbor energy is proposed to fuse the structure layer and the l 1 -max rule is introduced to fuse the energy layer. A total of 118 co-registered pairs of medical images covering five different categories of medical image fusion problems are tested in experiments. Seven latest representative medical image fusion methods are compared, and six representative quality evaluation metrics with complementary characteristics are fully employed to objectively evaluate the fused results. Extensive experimental results demonstrate that the proposed method yields better performance than some state-of-the-art methods in both visual quality and quantitative evaluation, and achieves nearly real-time computational efficiency and robustness to noise.

OPTIMIZED DUAL-TREE COMPLEX WAVELET TRANSFORM AND FUZZY ENTROPY FOR MULTI-MODAL MEDICAL IMAGE FUSION: A HYBRID META-HEURISTIC CONCEPT

In recent times, multi-modal medical image fusion has emerged as an important medical application tool. An important goal is to fuse the multi-modal medical images from diverse imaging modalities into a single fused image. The physicians broadly utilize this for precise identification and treatment of diseases. This medical image fusion approach will help the physician perform the combined diagnosis, interventional treatment, pre-operative planning, and intra-operative guidance in various medical applications by developing the corresponding information from clinical images through different modalities. In this paper, a novel multi-modal medical image fusion method is adopted using the intelligent method. Initially, the images from two different modalities are applied with optimized Dual-Tree Complex Wavelet Transform (DT-CWT) for splitting the images into high-frequency subbands and low-frequency subbands. As an improvement to the conventional DT-CWT, the filter coefficients are optimized by the hybrid meta-heuristic algorithm named as Hybrid Beetle and Salp Swarm Optimization (HBSSO) by merging the Salp Swarm Algorithm (SSA), and Beetle Swarm Optimization (BSO). Moreover, the fusion of the source images’ high-frequency subbands was done by the optimized type-2 Fuzzy Entropy. The upper and lower membership limits are optimized by the same hybrid HBSSO. The optimized type-2 fuzzy Entropy automatically selects high-frequency coefficients. Also, the fusion of the low-frequency sub-images is performed by the Averaging approach. Further, the inverse optimized DT-CWT on the fused image sets helps to obtain the final fused medical image. The main objective of the optimized DT-CWT and optimized type-2 fuzzy Entropy is to maximize the SSIM. The experimental results confirm that the developed approach outperforms the existing fusion algorithms in diverse performance measures.

Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform

Although many powerful convolutional neural networks (CNN) have been applied to various image processing fields, due to the lack of datasets for network training and the significant different intensities of diverse multi-modal source images at the same location, CNN cannot be directly used for the field of medical image fusion (MIF), which is a major problem and limits the development of this field. In this article, a novel multimodal medical image fusion method based on non-subsampled contourlet transform (NSCT) and CNN is presented. The proposed algorithm not only solves this problem, but also exploits the advantages of both NSCT and CNN to obtain better fusion results. In the proposed algorithm, source multi-modality images are decomposed into low and high frequency subbands. For high frequency subbands, a new perceptual high frequency CNN (PHF-CNN), which is trained in the frequency domain, is designed as an adaptive fusion rule. In the matter of the low frequency subband, two result maps are adopted to generate the decision map. Finally, fused frequency subbands are integrated by the inverse NSCT. To verify the effectiveness of the proposed algorithm, ten state-of-the-art MIF algorithms are selected as comparative algorithms. Subjective evaluations by five doctors as well as objective evaluations by seven image quality metrics, demonstrate that the proposed algorithm is superior to the other comparative algorithms in terms of fusing multimodal medical images.

An Optimal Registration on Shearlet domain with Novel Weighted Energy fusion for Multi-Modal Medical Images

Medical images usually display various attributes of data on human viscera and abnormal tissue in different modalities. The fusion of images, ensures effective deployment of all relevant information from several modalities into a single image. Finally, the paper leads to a novel multi-modal medical image fusion method based on Non-Subsampled Shearlet Transform (NSST) called Denoised Optimum B-Spline Shearlet Image Fusion(DOBSIF) that is based on real-time and standard radiological datasets. The proposed, novel registration based fusion technique DOBSIF, is compared with existing state-of-art techniques like PCA, DWT, SWT etc., and in comparison the proposed DOBSIF works effectively for both the color and grayscale images. To improve the fusion function, a novel pre-fusion method is carried out with the help of the Whale Optimization Algorithm(WOA), using the ideal B-spline-based registration method and in addition a novel Weighted Energy fusion rule is applied to derive relevant information from the source images. The proposed work initially follows normal pre-processing measures such as Gaussian filtering, edge sharpening, and resizing followed by optimal registration, fusion with novel fusion rule and segmentation of tumor part from the fused image. The visual quality of the fused image was evaluated by expert radiologists and the fused resultant image is robust, subject to the modified Active Contour(ACM) model for lesion identification.

Survey study of multimodality medical image fusion methods

Multimodality medical image fusion is the process of combining multiple images from single or multiple modalities of imaging. Medical image fusion methods are adopted to increase the quality of medical images by attaining the salient features in the fusion results. Hence, they raise the clinical applicability of medical images for appraisal and diagnosis problems. This purpose is achieved by capturing the complementary information presented in two or more images of different modalities in the fusion result. Medical image fusion is generally concerned with Magnetic Resonance Imaging (MRI), Magnetic Resonance Angiogram (MRA), Positron Emission Tomography (PET), Structural Positron Emission Tomography (SPET), Computerized Tomography (CT), and Single-Photon Emission Computed Tomography (SPECT) modalities. Each modality has its merits and drawbacks. This induces new fusion methods for merging information from multiple imaging modalities. Researchers have presented several methods for medical image fusion, and these methods achieved good results. However, medical image fusion is a resurgent field that needs to be enhanced to conquer the increasing challenges. This paper presents a comprehensive survey of some existing medical image fusion methods. It is expected that this study will be useful for the researchers scrutinizing medical image fusion. Furthermore, it is expected to establish a concrete foundation for developing more powerful fusion methods for medical applications.

Improving medical image fusion method using fuzzy entropy and nonsubsampling contourlet transform

AbstractMany types of medical images must be fused, as single‐modality medical images can only provide limited information due to the imaging principles and the complexity of human organ structures. In this paper, a multimodal medical image fusion method that combines the advantages of nonsubsampling contourlet transform (NSCT) and fuzzy entropy is proposed to provide a basis for clinical diagnosis and improve the accuracy of target recognition and the quality of fused images. An image is initially decomposed into low‐ and high‐frequency subbands through NSCT. The corresponding fusion rules are adopted in accordance with the different characteristics of the low‐ and high‐frequency components. The membership degree of low‐frequency coefficients is calculated. The fuzzy entropy is also computed and subsequently used to guide the fusion of coefficients to preserve image details. High‐frequency components are fused by maximizing the regional energy. The final fused image is obtained by inverse transformation. Experimental results show that the proposed method achieves good fusion effect based on the subjective visual effect and objective evaluation criteria. This method can also obtain high average gradient, SD, and edge preservation and effectively retain the details of the fused image. The results of the proposed algorithm can provide effective reference for doctors to assess patient condition.

Multimodal Medical Image Fusion using Rolling Guidance Filter with CNN and Nuclear Norm Minimization.

Medical image fusion is very important for the diagnosis and treatment of diseases. In recent years, there have been a number of different multi-modal medical image fusion algorithms that can provide delicate contexts for disease diagnosis more clearly and more conveniently. Recently, nuclear norm minimization and deep learning have been used effectively in image processing. A multi-modality medical image fusion method using a rolling guidance filter (RGF) with a convolutional neural network (CNN) based feature mapping and nuclear norm minimization (NNM) is proposed. At first, we decompose medical images to base layer components and detail layer components by using RGF. In the next step, we get the basic fused image through the pretrained CNN model. The CNN model with pre-training is used to obtain the significant characteristics of the base layer components. And we can compute the activity level measurement from the regional energy of CNN-based fusion maps. Then, a detail fused image is gained by NNM. That is, we use NNM to fuse the detail layer components. At last, the basic and detail fused images are integrated into the fused result. From the comparison with the most advanced fusion algorithms, the results of experiments indicate that this fusion algorithm has the best effect in visual evaluation and objective standard. The fusion algorithm using RGF and CNN-based feature mapping, combined with NNM, can improve fusion effects and suppress artifacts and blocking effects in the fused results.

Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation

Multi-modal medical image fusion refers to the fusion of two or more medical images obtained by different imaging methods into one image. Multi-modal medical images contain a lot of useful information that helps doctors to make a diagnosis. In this study, a multi-modal medical image fusion method is proposed based on Laplacian pyramid (LP) decomposition and adaptive sparse representation (ASR). ASR was used to reduce the noise of high-frequency information in the image fusion process and it did not need a high redundancy dictionary as traditional sparse representation (SR) methods. The proposed fusion method first used the LP decomposition to split medical images into four images of different sizes. Then ASR was performed to fuse the decomposed four layers, respectively. Finally, the fused image was obtained by the inverse Laplace pyramid transform. Experimental results showed that the proposed method could effectively fuse the medical images with the detailed information perfectly integrated, and could also reduce the influences of artifacts, noise and block effect. The research results are of great significance in the field of medical image fusion.

Research of Multimodal Medical Image Fusion Based on Parameter-Adaptive Pulse-Coupled Neural Network and Convolutional Sparse Representation.

Visual effects of medical image have a great impact on clinical assistant diagnosis. At present, medical image fusion has become a powerful means of clinical application. The traditional medical image fusion methods have the problem of poor fusion results due to the loss of detailed feature information during fusion. To deal with it, this paper proposes a new multimodal medical image fusion method based on the imaging characteristics of medical images. In the proposed method, the non-subsampled shearlet transform (NSST) decomposition is first performed on the source images to obtain high-frequency and low-frequency coefficients. The high-frequency coefficients are fused by a parameter‐adaptive pulse-coupled neural network (PAPCNN) model. The method is based on parameter adaptive and optimized connection strength β adopted to promote the performance. The low-frequency coefficients are merged by the convolutional sparse representation (CSR) model. The experimental results show that the proposed method solves the problems of difficult parameter setting and poor detail preservation of sparse representation during image fusion in traditional PCNN algorithms, and it has significant advantages in visual effect and objective indices compared with the existing mainstream fusion algorithms.

MRI and CT Medical Image Fusion Based on Synchronized-Anisotropic Diffusion Model

Medical image fusion can combine information from multi-modality images and express them through a single image. How to design a fusion method to preserve more information becomes a hot topic. In this paper, we propose a novel multi-modality medical image fusion method based on Synchronized-Anisotropic Diffusion Equation (S-ADE). First, the modified S-ADE model which is more suitable for Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) images is employed to decompose two source images. We get the base layers and texture layers. Next, the “Maximum Absolute Value” rule is used for base layers fusion. On texture layers, the fusion decision map is calculated by New Sum of Modified Anisotropic Laplacian (NSMAL) algorithm which is designed using common decomposition coefficients by anisotropic diffusion. Furthermore, the consistency check is constructed on the decision map to mitigate the staircase effect. After that, the fused image is obtained by a simple linear combination of layers. Finally, the fused MR/CT image is obtained after image correction. Its aim is to eliminate redundant texture information which is from MRI images in the contour part. The extensive experimental results demonstrate that the proposed method preserves much information as well as guarantees image quality and visual effects. It outperforms other state-of-the-art methods in terms of subjective and objective evaluations.

Fine grained medical image fusion using type-2 fuzzy logic

<p>In recent years, many fast-growing technologies coupled with wide volume of medical data for the digitalization of that data. Thus, researchers have shown their immense interest on Multi-sensor image fusion technologies which convey image information based on data from various sensor modalities into a single image. The image fusion technique is a widespread technique for the diagnosis of medical instrumentation and measurement. Therefore, in this paper we have introduced a novel multimodal sensor medical image fusion method based on type-2 fuzzy logic is proposed using Sugeno model. Moreover, a Gaussian smoothen filter is introduced to extract the detailed information of an image using sharp feature points.Type-2 fuzzy algorithm is used to achieve highly efficient feature points from both the b images to provide high visually classified resultant image. The experimental results demonstrate that the proposed method can achieve better performance than the state-of-the- art methods in terms of visual quality and objective evaluation.</p>