Non-rigid Multimodal Image Registration Research Articles

Interpretable Multi-modal Image Registration Network Based on Disentangled Convolutional Sparse Coding.

Multi-modal image registration aims to spatially align two images from different modalities to make their feature points match with each other. Captured by different sensors, the images from different modalities often contain many distinct features, which makes it challenging to find their accurate correspondences. With the success of deep learning, many deep networks have been proposed to align multi-modal images, however, they are mostly lack of interpretability. In this paper, we first model the multi-modal image registration problem as a disentangled convolutional sparse coding (DCSC) model. In this model, the multi-modal features that are responsible for alignment (RA features) are well separated from the features that are not responsible for alignment (nRA features). By only allowing the RA features to participate in the deformation field prediction, we can eliminate the interference of the nRA features to improve the registration accuracy and efficiency. The optimization process of the DCSC model to separate the RA and nRA features is then turned into a deep network, namely Interpretable Multi-modal Image Registration Network (InMIR-Net). To ensure the accurate separation of RA and nRA features, we further design an accompanying guidance network (AG-Net) to supervise the extraction of RA features in InMIR-Net. The advantage of InMIR-Net is that it provides a universal framework to tackle both rigid and non-rigid multi-modal image registration tasks. Extensive experimental results verify the effectiveness of our method on both rigid and non-rigid registrations on various multi-modal image datasets, including RGB/depth images, RGB/near-infrared (NIR) images, RGB/multi-spectral images, T1/T2 weighted magnetic resonance (MR) images and computed tomography (CT)/MR images. The codes are available at https://github.com/lep990816/Interpretable-Multi-modal-Image-Registration.

Non-rigid multi-modal brain image registration based on two-stage generative adversarial nets

This paper has proposed the two-stage generative adversarial nets (GAN) based multi-modal brain image registration method. The first stage uses the GAN to indirectly recover image deformation and the second stage employs the GAN to directly estimate the registered image. At the first stage, the GAN uses structural representations of reference and float images generated by the improved modality independent neighborhood descriptor (MIND) as inputs, and outputs the predicted structural representation of the registered image. At the second stage, the GAN estimates the intensities of the registered image according to the generated structural representations at the first stage. The two GANs designed based on the U-Net are trained using the different loss functions related to mean square error, mean structural similarity (MSSIM) and the least squares loss. Experiments on the 3D brain images from the Atlas, BrainWeb and RIRE datasets show that compared with state-of-the-art registration methods based on sum of squared differences on entropy images (ESSD), MIND, self-similarity context (SSC), voxelmorph (Morph), volume tweening network (VTN) and adversarial similarity network (ASN), the proposed method has competitive or better registration performance in terms of dice value, MSSIM and computational efficiency.

Non-rigid Multi-Modal Medical Image Registration Based on Improved Maximum Mutual Information PV Image Interpolation Method.

With the continuous improvement of medical imaging equipment, CT, MRI and PET images can obtain accurate anatomical information of the same patient site. However, due to the fuzziness of medical image physiological evaluation and the unhealthy understanding of objects, the registration effect of many methods is not ideal. Therefore, based on the medical image registration model of Partial Volume (PV) image interpolation method and rigid medical image registration method, this paper established the non-rigid registration model of maximum mutual information Novel Partial Volume (NPV) image interpolation method. The proposed NPV interpolation method uses the Davidon-Fletcher-Powell algorithm (DFP) algorithm optimization method to solve the transformation parameter matrix and realize the accurate transformation of the floating image. In addition, the cubic B-spline is used as the kernel function to improve the image interpolation, which effectively improves the accuracy of the registration image. Finally, the proposed NPV method is compared with the PV interpolation method through the human brain CT-MRI-PET image to obtain a clear CT-MRI-PET image. The results show that the proposed NPV method has higher accuracy, better robustness, and easier realization. The model should also have guiding significance in face recognition and fingerprint recognition.

Effect of optimization framework on rigid and non-rigid multimodal image registration

Effect of optimization framework on rigid and non-rigid multimodal image registration

Deep learning based data-adaptive descriptor for non-rigid multi-modal medical image registration

Non-rigid multi-modal medical image registration is a challenging task in the field of medical image processing and analysis because of the unpredictable complicated deformations and the nonfunctional intensity relation between images. The structural representation based registration (SRR) method can address these factors to some extent by transforming this challenging problem into the mono-modal image registration. However, the existing SSR algorithms generally rely on handcrafted features for structural representation, which tend to lead to inaccurate registered results. To address this problem, this paper has proposed a novel Laplacian Eigenmaps based deep learning network for 2D medical image registration. The proposed network is firstly used to extract the intrinsic features of different modal medical images. Then the self-similarity of extracted features is explored to construct the learning based data-adaptive descriptor (LDAD) for structural representations. The sum of squared differences (SSD) between the structural representations of the reference and floating images is used as the similarity metric to construct the objective function for image registration. The experimental results show that the LDAD based registration method can produce visually better registered results and provide higher registration accuracy for different test images than several state-of-the-art registration methods in terms of target registration error.

Non-Rigid Multi-Modal 3D Medical Image Registration Based on Foveated Modality Independent Neighborhood Descriptor.

The non-rigid multi-modal three-dimensional (3D) medical image registration is highly challenging due to the difficulty in the construction of similarity measure and the solution of non-rigid transformation parameters. A novel structural representation based registration method is proposed to address these problems. Firstly, an improved modality independent neighborhood descriptor (MIND) that is based on the foveated nonlocal self-similarity is designed for the effective structural representations of 3D medical images to transform multi-modal image registration into mono-modal one. The sum of absolute differences between structural representations is computed as the similarity measure. Subsequently, the foveated MIND based spatial constraint is introduced into the Markov random field (MRF) optimization to reduce the number of transformation parameters and restrict the calculation of the energy function in the image region involving non-rigid deformation. Finally, the accurate and efficient 3D medical image registration is realized by minimizing the similarity measure based MRF energy function. Extensive experiments on 3D positron emission tomography (PET), computed tomography (CT), T1, T2, and (proton density) PD weighted magnetic resonance (MR) images with synthetic deformation demonstrate that the proposed method has higher computational efficiency and registration accuracy in terms of target registration error (TRE) than the registration methods that are based on the hybrid L-BFGS-B and cat swarm optimization (HLCSO), the sum of squared differences on entropy images, the MIND, and the self-similarity context (SSC) descriptor, except that it provides slightly bigger TRE than the HLCSO for CT-PET image registration. Experiments on real MR and ultrasound images with unknown deformation have also be done to demonstrate the practicality and superiority of the proposed method.

Co-Sparse Analysis Model Based Image Registration to Compensate Brain Shift by Using Intra-Operative Ultrasound Imaging.

Notwithstanding the widespread use of image guided neurosurgery systems in recent years, the accuracy of these systems is strongly limited by the intra-operative deformation of the brain tissue, the so-called brain shift. Intra-operative ultrasound (iUS) imaging as an effective solution to compensate complex brain shift phenomena update patients coordinate during surgery by registration of the intra-operative ultrasound and the pre-operative MRI data that is a challenging problem.In this work a non-rigid multimodal image registration technique based on co-sparse analysis model is proposed. This model captures the interdependency of two image modalities; MRI as an intensity image and iUS as a depth image. Based on this model, the transformation between the two modalities is minimized by using a bimodal pair of analysis operators which are learned by optimizing a joint co-sparsity function using a conjugate gradient.Experimental validation of our algorithm confirms that our registration approach outperforms several of other state-of-the-art registration methods quantitatively. The evaluation was performed using seven patient dataset with the mean registration error of only 1.83 mm. Our intensity-based co-sparse analysis model has improved the accuracy of non-rigid multimodal medical image registration by 15.37% compared to the curvelet based residual complexity as a powerful registration method, in a computational time compatible with clinical use.

PCANet-Based Structural Representation for Nonrigid Multimodal Medical Image Registration.

Nonrigid multimodal image registration remains a challenging task in medical image processing and analysis. The structural representation (SR)-based registration methods have attracted much attention recently. However, the existing SR methods cannot provide satisfactory registration accuracy due to the utilization of hand-designed features for structural representation. To address this problem, the structural representation method based on the improved version of the simple deep learning network named PCANet is proposed for medical image registration. In the proposed method, PCANet is firstly trained on numerous medical images to learn convolution kernels for this network. Then, a pair of input medical images to be registered is processed by the learned PCANet. The features extracted by various layers in the PCANet are fused to produce multilevel features. The structural representation images are constructed for two input images based on nonlinear transformation of these multilevel features. The Euclidean distance between structural representation images is calculated and used as the similarity metrics. The objective function defined by the similarity metrics is optimized by L-BFGS method to obtain parameters of the free-form deformation (FFD) model. Extensive experiments on simulated and real multimodal image datasets show that compared with the state-of-the-art registration methods, such as modality-independent neighborhood descriptor (MIND), normalized mutual information (NMI), Weber local descriptor (WLD), and the sum of squared differences on entropy images (ESSD), the proposed method provides better registration performance in terms of target registration error (TRE) and subjective human vision.

Nonrigid registration of multimodal medical images based on hybrid model

ABSTRACT Background and Objectives: Multimodal image registration is a crucial step in prostate cancer radiation therapy scheme. However, it can be challenging due to the obvious appearance difference between computed tomography (CT) and magnetic resonance imaging (MRI) and unavoidable organ motion. Accordingly, a nonrigid registration framework for precisely registering multimodal prostate images is proposed in this paper. Materials and Methods: In this work, multimodal prostate image registration between CT and MRI is achieved using a hybrid model that integrates multiresolution strategy and Demons algorithm. Furthermore, to precisely describe the deformation of prostate, B-spline-based registration is utilized to refine the initial registration result of multiresolution Demons algorithm. Results: To evaluate our method, experiments on clinical prostate data sets of nine participants and comparison with the conventional Demons algorithm are conducted. Experimental results demonstrate that the proposed registration method outperforms the Demons algorithm by a large margin in terms of mutual information and correlation coefficient. Conclusions: These results show that our method outperforms the Demons algorithm and can achieve excellent performance on multimodal prostate images even the appearances of prostate change significantly. In addition, the results demonstrate that the proposed method can help to localize the prostate accurately, which is feasible in clinical.

The Spiking Cortical Model Based Structural Representations for Non-Rigid Multi-Modal Medical Image Registration

The Spiking Cortical Model Based Structural Representations for Non-Rigid Multi-Modal Medical Image Registration

EVolution: an edge-based variational method for non-rigid multi-modal image registration

Image registration is part of a large variety of medical applications including diagnosis, monitoring disease progression and/or treatment effectiveness and, more recently, therapy guidance. Such applications usually involve several imaging modalities such as ultrasound, computed tomography, positron emission tomography, x-ray or magnetic resonance imaging, either separately or combined.In the current work, we propose a non-rigid multi-modal registration method (namely EVolution: an edge-based variational method for non-rigid multi-modal image registration) that aims at maximizing edge alignment between the images being registered. The proposed algorithm requires only contrasts between physiological tissues, preferably present in both image modalities, and assumes deformable/elastic tissues. Given both is shown to be well suitable for non-rigid co-registration across different image types/contrasts (T1/T2) as well as different modalities (CT/MRI). This is achieved using a variational scheme that provides a fast algorithm with a low number of control parameters. Results obtained on an annotated CT data set were comparable to the ones provided by state-of-the-art multi-modal image registration algorithms, for all tested experimental conditions (image pre-filtering, image intensity variation, noise perturbation). Moreover, we demonstrate that, compared to existing approaches, our method possesses increased robustness to transient structures (i.e. that are only present in some of the images).

Self-similarity inspired local descriptor for non-rigid multi-modal image registration

Non-rigid multi-modal image registration plays an important role in medical image analysis. It remains a challenging problem due to the significant intensity distortion and the non-rigid transformation between images. Existing registration methods based on the information theoretic measures and image representations cannot address this problem effectively. In this paper, we have proposed a novel self-similarity inspired local descriptor to determine the similarity metrics, a key component in image registration. The self-similarity is determined based on the Zernike moments of image patches in a local neighborhood, and it is utilized to generate the local descriptor. The Euclidean distance between the local descriptors computed in the reference and moving images is used as the similarity metrics. Distinctively, the proposed local descriptor can provide an effective representation of complicated image features due to its robustness to noise and rotational invariance, which provides the proposed method with good registration performance. Extensive experiments on both simulated and real multi-modal image datasets demonstrate that the proposed method has higher registration accuracy appreciated by the target registration error (TRE) than the state-of-the-art registration methods based on the normalized mutual information (NMI), the sum of squared differences on entropy images (ESSD), the Weber local descriptor (WLD) and the modality independent neighborhood descriptor (MIND).

Curvelet based residual complexity objective function for non-rigid registration of pre-operative MRI with intra-operative ultrasound images.

Intra-operative ultrasound as an imaging based method has been recognized as an effective solution to compensate non rigid brain shift problem in recent years. Measuring brain shift requires registration of the pre-operative MRI images with the intra-operative ultrasound images which is a challenging task. In this study a novel hybrid method based on the matching echogenic structures such as sulci and tumor boundary in MRI with ultrasound images is proposed. The matching echogenic structures are achieved by optimizing the Residual Complexity (RC) in the curvelet domain. At the first step, the probabilistic map of the MR image is achieved and the residual image as the difference between this probabilistic map and intra-operative ultrasound is obtained. Then curvelet transform as a sparse function is used to minimize the complexity of residual image. The proposed method is a compromise between feature-based and intensity-based approaches. Evaluation was performed using 14 patients data set and the mean of registration error reached to 1.87 mm. This hybrid method based on RC improves accuracy of nonrigid multimodal image registration by 12.5% in a computational time compatible with clinical use.

Non-rigid multi-modal medical image registration by combining L-BFGS-B with cat swarm optimization

Non-rigid multi-modal image registration plays an important role in medical image processing and analysis. Optimization is a key component of image registration. Mapped as a large-scale optimization problem, non-rigid image registration often requires global optimization methods because the functions defined by similarity metrics are generally non-convex and irregular. In this paper, a novel optimization method is proposed by combining the limited memory Broyden–Fletcher–Goldfarb–Shanno with boundaries (L-BFGS-B) with cat swarm optimization (CSO) for non-rigid multi-modal image registration using the normalized mutual information (NMI) measure and the free-form deformations (FFD) model. The proposed hybrid L-BFGS-B and CSO (HLCSO) method uses cooperative coevolving to tackle non-rigid image registration, and employs block grouping as the grouping strategy to capture the interdependency among variables. Moreover, to achieve faster convergence and higher accuracy of the final solution, the local optimization method L-BFGS-B and the roulette wheel method are introduced into the seeking mode and the tracing mode of the HLCSO, respectively. Extensive experiments on 3D CT, PET, T1, T2 and PD weighted MR images demonstrate that the proposed method outperforms the L-BFGS-B method and the CSO method in terms of registration accuracy, and it is provided with reasonable computational efficiency.

A hybrid method for non-rigid registration of intra-operative ultrasound images with pre-operative MR images.

In recent years intra-operative ultrasound images have been used for many procedures in neurosurgery. The registration of intra-operative ultrasound images with preoperative magnetic resonance images is still a challenging problem. In this study a new hybrid method based on residual complexity is proposed for this problem. A new two stages method based on the matching echogenic structures such as sulci is achieved by optimizing the residual complexity (RC) value with quantized coefficients between the ultrasound image and the probabilistic map of MR image. The proposed method is a compromise between feature-based and intensity-based approaches. The evaluation is performed on both a brain phantom and patient data set. The results of the phantom data set confirmed that the proposed method outperforms the accuracy of conventional RC by 39%. Also the mean of fiducial registration errors reached to 1.45, 1.94 mm when the method was applied on phantom and clinical data set, respectively. This hybrid method based on RC enables non-rigid multimodal image registration in a computational time compatible with clinical use as well as being accurate.

Brain-shift compensation by non-rigid registration of intra-operative ultrasound images with preoperative MR images based on residual complexity

Compensation for brain shift is often necessary for image-guided neurosurgery, requiring registration of intra-operative ultrasound (US) images with preoperative magnetic resonance images (MRI). A new image similarity measure based on residual complexity (RC) to overcome challenges of registration of intra-operative US and preoperative MR images was developed and tested. A new two-stage method based on the matching echogenic structures such as sulci is achieved by optimizing the residual complexity value in the wavelet domain between the ultrasound image and the probabilistic map of the MR image. The proposed method is a compromise between feature-based and intensity-based approaches. Evaluation was performed using a specially designed brain phantom and an in vivo patient data set. The results of the phantom data set registration confirmed that the proposed objective function outperforms the accuracy of adapted RC for multi-modal cases by 48 %. The mean fiducial registration error reached 1.17 and 2.14 mm when the method was applied on phantom and clinical data sets, respectively. This improved objective function based on RC in the wavelet domain enables accurate non-rigid multi-modal (US and MRI) image registration which is robust to noise. This technology is promising for compensation of intra-operative brain shift in neurosurgery.

Efficient Acceleration of Mutual Information Computation for Nonrigid Registration Using CUDA

In this paper, we propose an efficient acceleration method for the nonrigid registration of multimodal images that uses a graphics processing unit. The key contribution of our method is efficient utilization of on-chip memory for both normalized mutual information (NMI) computation and hierarchical B-spline deformation, which compose a well-known registration algorithm. We implement this registration algorithm as a compute unified device architecture program with an efficient parallel scheme and several optimization techniques such as hierarchical data organization, data reuse, and multiresolution representation. We experimentally evaluate our method with four clinical datasets consisting of up to 512 × 512 × 296 voxels. We find that exploitation of on-chip memory achieves a 12-fold increase in speed over an off-chip memory version and, therefore, it increases the efficiency of parallel execution from 4% to 46%. We also find that our method running on a GeForce GTX 580 card is approximately 14 times faster than a fully optimized CPU-based implementation running on four cores. Some multimodal registration results are also provided to understand the limitation of our method. We believe that our highly efficient method, which completes an alignment task within a few tens of seconds, will be useful to realize rapid nonrigid registration.

Two Phase Non-Rigid Multi-Modal Image Registration Using Weber Local Descriptor-Based Similarity Metrics and Normalized Mutual Information

Non-rigid multi-modal image registration plays an important role in medical image processing and analysis. Existing image registration methods based on similarity metrics such as mutual information (MI) and sum of squared differences (SSD) cannot achieve either high registration accuracy or high registration efficiency. To address this problem, we propose a novel two phase non-rigid multi-modal image registration method by combining Weber local descriptor (WLD) based similarity metrics with the normalized mutual information (NMI) using the diffeomorphic free-form deformation (FFD) model. The first phase aims at recovering the large deformation component using the WLD based non-local SSD (wldNSSD) or weighted structural similarity (wldWSSIM). Based on the output of the former phase, the second phase is focused on getting accurate transformation parameters related to the small deformation using the NMI. Extensive experiments on T1, T2 and PD weighted MR images demonstrate that the proposed wldNSSD-NMI or wldWSSIM-NMI method outperforms the registration methods based on the NMI, the conditional mutual information (CMI), the SSD on entropy images (ESSD) and the ESSD-NMI in terms of registration accuracy and computation efficiency.

Non-rigid registration of multi-modal images using both mutual information and cross-correlation

The hierarchical subdivision strategy which decomposes a non-rigid matching problem into numerous local rigid transformations is a very common approach in image registration. While mutual information (MI) has proven to be a very robust and reliable similarity measure for intensity-based matching of multi-modal images, numerous problems have to be faced if it is applied to small-sized images, compromising its usefulness for such subdivision schemes. We examine and explain the loss of MI's statistical consistency along the hierarchical subdivision. Information theoretical measures are proposed to identify the problematic regions in order to overcome the MI drawbacks. This does not only improve the accuracy and robustness of the registration, but also can be used as a very efficient stopping criterion for the further subdivision of nodes in the hierarchy, which drastically reduces the computational cost of the entire registration procedure. Moreover, we present a new intensity mapping technique allowing to replace MI by more reliable measures for small patches. Integrated into the hierarchical framework, this mapping can locally transform the multi-modal images into an intermediate pseudo-modality. This intensity mapping uses the local joint intensity histograms of the coarsely registered sub-images and allows the use of the more robust and computationally more efficient cross-correlation coefficient (CC) for the matching at lower levels of the hierarchy.

Non-Rigid Multi-Modal Image Registration Using Cross-Cumulative Residual Entropy.

In this paper we present a new approach for the non-rigid registration of multi-modality images. Our approach is based on an information theoretic measure called the cumulative residual entropy (CRE), which is a measure of entropy defined using cumulative distributions. Cross-CRE between two images to be registered is defined and maximized over the space of smooth and unknown non-rigid transformations. For efficient and robust computation of the non-rigid deformations, a tri-cubic B-spline based representation of the deformation function is used. The key strengths of combining CCRE with the tri-cubic B-spline representation in addressing the non-rigid registration problem are that, not only do we achieve the robustness due to the nature of the CCRE measure, we also achieve computational efficiency in estimating the non-rigid registration. The salient features of our algorithm are: (i) it accommodates images to be registered of varying contrast+brightness, (ii) faster convergence speed compared to other information theory-based measures used for non-rigid registration in literature, (iii) analytic computation of the gradient of CCRE with respect to the non-rigid registration parameters to achieve efficient and accurate registration, (iv) it is well suited for situations where the source and the target images have field of views with large non-overlapping regions. We demonstrate these strengths via experiments on synthesized and real image data.