Multimodal Medical Research Articles

Multimodal Medical Image Fusion based on Gray Wolf Optimization and Hilbert Transform

Image fusion combines complimentary information from multiple images acquired through different sensors, into a single image. In this paper, an image fusion technique based on Gray wolf optimization is proposed for fusion of medical images, Magnetic Resonance Image (MRI) and Positron Emission Tomography (PET). Firstly, Hilbert transformation is applied to extort the informative data from input images. The relevant part of the input images based on information present, is selected for image fusion. The selected portion of the input images is fused by using Gray wolf optimization based method. The simulation results show improved performance of proposed framework as compared to conventional techniques.

Multimodal Medical Image Fusion Using Hybrid Layer Decomposition With CNN-Based Feature Mapping and Structural Clustering

Multimodal medical image fusion aims to integrate multisensory medical data into a single image for better prediction of the diagnostic details. This article presents a feature-level multimodal medical image fusion with the use of a two-scale ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> - ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> hybrid layer decomposition scheme to maximize the structural details with the suppression of significant noise and artifacts. The proposed method uses a convolutional neural network (CNN) with consistency verification and structural patch clustering (fuzzy c-means-based) for the decomposed base and detail layer fusion, respectively. At first, a color space transform is used to separate the luminance and chrominance components extracted from both the source images and the luminance part of each image is decomposed. In the second step, a pretrained CNN model is used to extract the prominent features from each decomposed base layer component. For the output feature map, a regional energy-based activity measure is computed and generated a fusion score, which is refined in the consistency verification step to optimize the weight map for fusing the base layers. The two-scale detail layers are merged by the clustering-based prelearned dictionary for efficiently mapping of structural details of the layers. The color components associated with both the images are also combined using a color saliency measure. Finally, the fused base layer, detail layers, and color components are merged to get the resultant fused image. The experimental results justify the superiority of the proposed approach in terms of both subjective and objective assessments. The performance of the proposed scheme is tested on a large multimodal medical data set of MR-SPECT, MR-PET, and CT-MR neurological images. Furthermore, the proposed approach shows better results than the other state-of-the-art approaches with improved fusion quality and computational performances.

Deep Learning-based Image Segmentation on Multimodal Medical Imaging.

Multi-modality medical imaging techniques have been increasingly applied in clinical practice and research studies. Corresponding multi-modal image analysis and ensemble learning schemes have seen rapid growth and bring unique value to medical applications. Motivated by the recent success of applying deep learning methods to medical image processing, we first propose an algorithmic architecture for supervised multi-modal image analysis with cross-modality fusion at the feature learning level, classifier level, and decision-making level. We then design and implement an image segmentation system based on deep Convolutional Neural Networks (CNN) to contour the lesions of soft tissue sarcomas using multi-modal images, including those from Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Positron Emission Tomography (PET). The network trained with multi-modal images shows superior performance compared to networks trained with single-modal images. For the task of tumor segmentation, performing image fusion within the network (i.e. fusing at convolutional or fully connected layers) is generally better than fusing images at the network output (i.e. voting). This study provides empirical guidance for the design and application of multi-modal image analysis.

Exploiting Quality-Guided Adaptive Optimization for Fusing Multimodal Medical Images

Multimodal medical image fusion (MMIF) plays critical roles in image-guided clinical diagnostics and treatment. Pulse coupled neural network (PCNN) has been applied in image fusion for several years. In the schemes of image fusion based on PCNN, the authors have adjusted variables manually, so that it is difficult to get satisfying effects which limit in dealing with medical images with different modalities. This paper presents a quality-guided adaptive optimization method for MMIF, which is based on PCNN optimized by multi-swarm fruit fly optimization algorithm (MFOA). To reduce the implementation cost and improve the performance of the MFOA, quality assessment for multimodal medical image fusion was chosen to be the hybrid fitness function. Guided by such quality measurement, the adaptive PCNN using the MFOA (PCNN-MFOA) is proposed, which could automatically fit the optimal variables to the source images and enhance the fusion effect. The experimental results visually and quantitatively show that the proposed fusion strategy is more effective than the state-of-the-art methods and it is more effective in processing medical images with different modalities.

Multimodal Medical Image Fusion using Guided Filter in NSCT Domain

Multimodal medical image fusion aims at minimizing the redundancy and collecting the relevant information using the input images acquired from different medical sensors. The main goal is to produce a single fused image having more information and has higher efficiency for medical applications. In this paper modified fusion method has been proposed in which NSCT decomposition is used to decompose the wavelet coefficients obtained after wavelet decomposition. NSCT being multidirectional,shift invariant transform provide better results.Guided filter has been used for the fusion of high frequency coefficients on account of its edge preserving property. Phase congruency is used for the fusion of low frequency coefficients due to its insensitivity to illumination contrast hence making it suitable for medical images. The simulated results show that the proposed technique shows better performance in terms of entropy, structural similarity index, Piella metric. The fusion response of the proposed technique is also compared with other fusion approaches; proving the effectiveness of the obtained fusion results.

Multimodal Medical Image Fusion Based on Fuzzy Discrimination With Structural Patch Decomposition.

Multimodal medical image fusion, emerging as a hot topic, aims to fuse images with complementary multi-source information. In this paper, we propose a novel multimodal medical image fusion method based on structural patch decomposition (SPD) and fuzzy logic technology. First, the SPD method is employed to extract two salient features for fusion discrimination. Next, two novel fusion decision maps called an incomplete fusion map and supplemental fusion map are constructed from salient features. In this step, the supplemental map is constructed by our defined two different fuzzy logic systems. The supplemental and incomplete maps are then combined to construct an initial fusion map. The final fusion map is obtained by processing the initial fusion map with a Gaussian filter. Finally, a weighted average approach is adopted to create the final fused image. Additionally, an effective color medical image fusion scheme that can effectively prevent color distortion and obtain superior diagnostic effects is also proposed to enhance fused images. Experimental results clearly demonstrate that the proposed method outperforms state-of-the-art methods in terms of subjective visual and quantitative evaluations.

Enhancement of Hybrid Multimodal Medical Image Fusion Techniques for Clinical Disease Analysis

Multimodal medical image fusion is one the most significant and useful disease analytic techniques. This research article proposes the hybrid multimodality medical image fusion methods and discusses the most essential advantages and disadvantages of these methods. The hybrid multimodal medical image fusion algorithms are used to improve the quality of fused multimodality medical image. Magnetic resonance imaging, positron emission tomography, and single photon emission computed tomography are the input multimodal therapeutic images used for fusion process. An experimental result of proposed hybrid fusion techniques provides the fused multimodal medical images of highest quality, shortest processing time, and best visualization. Both traditional and hybrid multimodal medical image fusion algorithms are evaluated using several quality metrics. Compared with existing techniques the proposed result gives the better processing performance in both qualitative and quantitative evaluation criteria. This is favorable, especially for helping in accurate clinical disease analysis.

A Hybrid Multimodal Medical Image Fusion Technique for CT and MRI Brain Images

Estimating the type, size, location and spread of a brain tumor is vital in the diagnosis and treatment of brain cancer. Fused CT and MRI brain images assist in faster detection and diagnosis of brain tumors. They provide superior results in comparison to individual CT or MRI images. Multiscale transforms (MSTs) are widely used in fusing multimodal images like CT and MRI. However, they have a few drawbacks like reduced contrast, poor edge detection, redundancy and high computation time. This article describes how MSTs coupled with sparse representation (SR) aims to overcome the drawbacks. Non-Subsampled Contourlet Transform (NSCT) is widely used on MSTs for fusing multifocal images. Therefore, a novel technique using NSCT and SR is proposed for better quality fused CT and MRI images. The experimental results show superior performance.

PCANet-Based Structural Representation for Nonrigid Multimodal Medical Image Registration.

Nonrigid multimodal image registration remains a challenging task in medical image processing and analysis. The structural representation (SR)-based registration methods have attracted much attention recently. However, the existing SR methods cannot provide satisfactory registration accuracy due to the utilization of hand-designed features for structural representation. To address this problem, the structural representation method based on the improved version of the simple deep learning network named PCANet is proposed for medical image registration. In the proposed method, PCANet is firstly trained on numerous medical images to learn convolution kernels for this network. Then, a pair of input medical images to be registered is processed by the learned PCANet. The features extracted by various layers in the PCANet are fused to produce multilevel features. The structural representation images are constructed for two input images based on nonlinear transformation of these multilevel features. The Euclidean distance between structural representation images is calculated and used as the similarity metrics. The objective function defined by the similarity metrics is optimized by L-BFGS method to obtain parameters of the free-form deformation (FFD) model. Extensive experiments on simulated and real multimodal image datasets show that compared with the state-of-the-art registration methods, such as modality-independent neighborhood descriptor (MIND), normalized mutual information (NMI), Weber local descriptor (WLD), and the sum of squared differences on entropy images (ESSD), the proposed method provides better registration performance in terms of target registration error (TRE) and subjective human vision.

Correction to: Adopting Quaternion Wavelet Transform to Fuse Multi-Modal Medical Images

The article “Adopting Quaternion Wavelet Transform to Fuse Multi-Modal Medical Images”, written by Peng Geng, Xiuming Sun, Jianhua Liu was originally published Online First without open access. After publication in volume [37], issue [2], page [230–239] the author decided to opt for Open Choice and to make the article an open access publication. Therefore, the copyright of the article has been changed to © The Author(s) [2018] and the article is forthwith distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Adaptive Parameter estimation based Multimodal Medical Image Fusion Frame work in SWT domain

Adaptive Parameter estimation based Multimodal Medical Image Fusion Frame work in SWT domain

Multimodal Medical Image Registration Based on an Information-Theory Measure with Histogram Estimation of Continuous Image Representation

This work presents a novel method for multimodal medical registration based on histogram estimation of continuous image representation. The proposed method, regarded as “fast continuous histogram estimation,” employs continuous image representation to estimate the joint histogram of two images to be registered. The Jensen–Arimoto (JA) divergence is a similarity measure to measure the statistical dependence between medical images from different modalities. The estimated joint histogram is exploited to calculate the JA divergence in multimodal medical image registration. In addition, to reduce the grid effect caused by the grid-aligning transformations between two images and improve the implementation speed of the registration method, random samples instead of all pixels are extracted from the images to be registered. The goal of the registration is to optimize the JA divergence, which would be maximal when two misregistered images are perfectly aligned using the downhill simplex method, and thus to get the optimal geometric transformation. Experiments are conducted on an affine registration of 2D and 3D medical images. Results demonstrate the superior performance of the proposed method compared to standard histogram, Parzen window estimations, particle filter, and histogram estimation based on continuous image representation without random sampling.

Multimodal Medical Image Registration Based on Feature Spheres in Geometric Algebra

Multi-modal image registration in medical image analysis is very challenging as the appearance of body structures in images from different imaging devices can be very different. In this paper, we propose a new method [GA-speeded up robust features (SURF)], which incorporates the geometric algebra (GA) into SURF framework, to detect features from images. We model the volumetric data and register the multi-modal medical images using feature spheres formulated in conformal geometric algebra (CGA). Specifically, we first extract features from medical images using GA-SURF. Second, we construct the feature spheres using the feature points and find the correspondence of feature spheres in the two images using CGA. With that, we can register the images based on the correspondence of the feature spheres. The experimental results evaluated by RIRE have shown that our method can register the multi-modal images with high accuracy. The maximum registration error is less than $4mm$ .

Multimodal Medical Image Fusion based on Deep Learning Neural Network for Clinical Treatment Analysis

Multimodal Medical Image Fusion based on Deep Learning Neural Network for Clinical Treatment Analysis

LPPCO: A Novel Multimodal Medical Image Registration Using New Feature Descriptor Based on the Local Phase and Phase Congruency of Different Orientations

At present, many feature descriptors-based registration methods have been proven to be robust in case of complex intensity distortions. However, most of these feature descriptors are only related to intensity information in a patch of neighboring pixels and ignoring the neighbor orientation information, which make the registration performance for medical images with low-resolution appear to be weak robustness and low accuracy. To improve the robustness and accuracy, a novel feature descriptor, named Local-Phase mean and Phase-Congruency values of different Orientations (LPPCO), is developed using filter-bank of Log-Gabor filters at different orientations and frequencies. Next, a similarity measure named LPPCOncc is developed using the normalized cross correlation (NCC) of the LPPCO descriptors, followed by a fast template matching techniques for detecting correspondences between the different images. Additionally, a more sensitivity of phase deviation function is presented for the calculation of phase congruency. The main steps of constructing the similarity measure LPPCOncc include: firstly, we extract the local phase mean and phase congruency values for each pixel in each orientation; secondly, local phase mean orientation histograms and phase congruency values orientation histograms over all the pixels are computed respectively, where the maximum response values are chosen to vote for the corresponding bin; thirdly, combining the two resulting histograms obtains the feature descriptor LPPCO; finally, the LPPCO descriptor is compared across images using NCC. Experimental results show that LPPCOncc is robust to complex intensity distortions between multimodal medical images and outperforms other similarity metrics such as NCC and DLSC. Furthermore, LPPCOncc-based registration algorithm preformed on various types of multimodal medical image pairs shows that it outperforms the NMI-based and DLSC-based registration methods in the registration robustness and accuracy.

Cascaded Lifting Wavelet and Contourlet Framework Based Dual Stage Fusion Scheme for Multimodal Medical Images

Cascaded Lifting Wavelet and Contourlet Framework Based Dual Stage Fusion Scheme for Multimodal Medical Images

An Efficient Algorithm for Multimodal Medical Image Fusion based On Feature Selection and PCA Using DTCWT

Background: In the two past decades, medical image fusion has become an essential part of modern medicine due to the availability of numerous imaging modalities (MRI, CT, SPECT. etc). This paper presents a new medical image fusion algorithm based on DTCWT and uses different fusion rules in order to obtain a new image which contains more information than any of the input images. Methods: In order to improve the visual quality of the fused image, we propose a new image fusion algorithm based on Dual Tree Complex Wavelet Transform (DTCWT). Using different fusion rules in a single algorithm leads to a perfect reconstruction of the output (fused image).This combination will create a new method which exploits the advantages of each method separately. DTCWT present good directionality since it considers the edge information in six directions and provides approximate shift invariant. The goal of Principal Component Analysis (PCA) is to extract the most significant features (wavelet coefficients in our case) in order to improve the spatial resolution. The proposed algorithm fuses the detailed wavelet coefficients of input images using features selection rule. Results: We have conducted several experiments over different sets of multimodal medical images such as CT/MRI, MRA/T1-MRI; however, only results of two sets have been presented (due to pages-limit). The proposed fusion algorithm is compared to recent fusion methods presented in the literature (eight methods) in terms of visual quality and quantitatively using well known fusion performance metrics (five metrics). Results showed that the proposed algorithm outperforms the existing ones in terms of visual and quantitative evaluations. Conclusion: This paper focuses on image fusion of medical images obtained from different modalities. We have proposed a novel algorithm based on DTCWT in order to merge multimodal medical images. Experiments have been performed over two different sets of multimodal medical images. The results show that the proposed method significantly outperforms other techniques reported in the literature.

Multimodal Medical Imaging Fusion for Patient Specific Musculoskeletal Modeling of the Lumbar Spine System in Functional Posture

Current musculoskeletal modeling does not take the functionally initial standing posture of the lumbar spine system into consideration. This may lead to inaccurate simulation outcomes for clinical decision support. This present study aimed to develop a fusion process from multimodal medical images for developing, simulating, and assessing patient specific musculoskeletal model of the lumbar spine system in functional posture. Computed tomography (CT) and X-rays were acquired on a patient. A 3D/2D matching procedure was developed using feature-based approach to transform CT-based 3D geometries into standing posture. Then, a musculoskeletal model was developed including 7 segments, 18-DOF (degrees of freedom) and 126 muscle fascicles. Model evaluation was performed using X-ray data and curvature metric. Muscle force deviations were quantified for the comparison between image-based model and generic-scaled model as well as for the sensitivity of psoas major attachment point definition. Curvatures of the simulation outcomes fall within the range of X-ray data for extension posture. Difference was noted for flexed posture. The use of generic model leads to a deviation of 66% of the muscle forces according to the image-based model. A maximal relative deviation of 4% of the estimated psoas major fascicle force was found for the analysis of the sensitivity of muscle attachment point. This study proposed a useful data fusion process from multimodal medical images to create patient specific lumbar spine model in correct initial posture leading to perform accurate dynamic simulations. Moreover, this study suggested that image-based model needs to be developed for clinical decision making related to spinal disorders.

An Efficient DWT and Intuitionistic Fuzzy Based Multimodality Medical Image Fusion

AbstractImage fusion aims to integrate complementary information from multiple modalities into a single image with none distortion and loss of data. Image fusion is important in medical imaging, specifically for the purpose of detecting the tumor and identification of diseases. In this article, completely unique discrete wavelet transform (DWT) and intuitionistic fuzzy sets (IFSs) based fusion method (DWT‐IFS) is proposed. For fusion, initially, all source images are fused using DWT with the average, maximum, and entropy fusion rules. Besides, on the fused image IFS is applied. In the IFS process images are converted into intuitionistic fuzzy images (IFIs) by selecting an optimum value for the parameter in membership, non‐membership, and hesitation degree function using entropy. Then, the resulting IFIs are decomposed into the blocks, and the corresponding blocks of the images are fused using the intersection and union operations of IFS. The efficiency of the proposed DWT‐IFS fusion method is recognized by examining it with other existing methods, such as Averaging (AVG), Principal Component Analysis (PCA), Laplacian Pyramid Approach (LPA), Contrast Pyramid Approach (CPA), Discrete Wavelet Transform (DWT), Morphological Pyramid Approach (MPA), Redundancy Discrete Wavelet Transform (RDWT), Contourlet Transform (CONTRA), and Intuitionistic Fuzzy Set (IFS) using subjective and objective performance evaluation measures. The experimental results reveal that the proposed DWT‐IFS fusion method provides higher quality of information in terms of physical properties and contrast as compared to the existing methods.

Multimodal Medical Image Fusion Based on Sugeno's Intuitionistic Fuzzy Sets

Multimodal medical image fusion is the process of retrieving valuable information from medical images. The primary goal of medical image fusion is to...