3D Medical Image Registration Research Articles

Particle Swarm Optimization in 3D Medical Image Registration: A Systematic Review

Particle Swarm Optimization in 3D Medical Image Registration: A Systematic Review

An efficient similarity metric for 3D medical image registration

An efficient similarity metric for 3D medical image registration

Deep learning-based 3D brain multimodal medical image registration.

Medical image registration is a critical preprocessing step in medical image analysis. While traditional medical image registration techniques have matured, their registration speed and accuracy still fall short of clinical requirements. In this paper, we propose an improved VoxelMorph network incorporating ResNet modules and CBAM (RCV-Net), for 3D multimodal unsupervised registration. Unlike popular convolution-based U-shaped registration networks like VoxelMorph, RCV-Net incorporates the convolutional block attention module (CBAM) during the convolution process. This inclusion enhances the feature map information extraction capabilities during training and effectively prevents information loss. Additionally, we introduce a lightweight and residual network module at the network's base, which enhances learning ability without significantly increasing training parameters. To evaluate the superiority of our registration model, we utilize evaluation metrics such as structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and mean square error (MSE). Experimental results demonstrate that our proposed network structure outperforms current state-of-the-art methods, yielding better performance in multimodal registration tasks. Furthermore, generalization testing on databases outside of the training set has confirmed the optimal registration effectiveness of our model.

Topology-preserving image registration with novel multi-dimensional Beltrami regularization

Beltrami coefficient provides a proper measure to quantify local distortions for a topology-preserving mapping, a.k.a general quasi-conformal mapping, in 2D. Recent studies managed to extend Beltrami coefficient from 2D to nD, and introduced its corresponding Beltrami-like measures to encode local deformation information in challenging nD topology-preserving image registration tasks. However, all these nD measures are not consistent with the classical 2D Beltrami coefficient. We propose a novel nD measure for evaluating local distortions of an nD diffeomorphic mapping in this work. We show its key mathematical properties, and prove that the new measure coincides with the classical 2D Beltrami coefficient. We further introduce a new nD topology-preserving image registration model by means of regularizing such proposed Beltrami-consistent coefficient, and show the existence of its solution. Meanwhile, a generalized Gauss-Newton scheme is applied to establish a fast multi-level numerical solver. Extensive experiments over different 3D medical image registration tasks, including the challenging multi-modal medical image registration, illustrate that the proposed approach by regularizing the introduced Beltrami-consistent coefficient outperforms the other state-of-the-art topology-preserving methods in both efficiency and accuracy.

3D Biological/Biomedical Image Registration with enhanced Feature Extraction and Outlier Detection.

In various applications, such as computer vision, medical imaging and robotics, three-dimensional (3D) image registration is a significant step. It enables the alignment of various datasets into a single coordinate system, consequently providing a consistent perspective that allows further analysis. By precisely aligning images we can compare, analyze, and combine data collected in different situations. This paper presents a novel approach for 3D or z-stack microscopy and medical image registration, utilizing a combination of conventional and deep learning techniques for feature extraction and adaptive likelihood-based methods for outlier detection. The proposed method uses the Scale-invariant Feature Transform (SIFT) and the Residual Network (ResNet50) deep neural learning network to extract effective features and obtain precise and exhaustive representations of image contents. The registration approach also employs the adaptive Maximum Likelihood Estimation SAmple Consensus (MLESAC) method that optimizes outlier detection and increases noise and distortion resistance to improve the efficacy of these combined extracted features. This integrated approach demonstrates robustness, flexibility, and adaptability across a variety of imaging modalities, enabling the registration of complex images with higher precision. Experimental results show that the proposed algorithm outperforms state-of-the-art image registration methods, including conventional SIFT, SIFT with Random Sample Consensus (RANSAC), and Oriented FAST and Rotated BRIEF (ORB) methods, as well as registration software packages such as bUnwrapJ and TurboReg, in terms of Mutual Information (MI), Phase Congruency-Based (PCB) metrics, and Gradiant-based metrics (GBM), using 3D MRI and 3D serial sections of multiplex microscopy images.

A deformable patch-based transformer for 3D medical image registration.

Medical image registration is of great importance in clinical medicine. However, medical image registration algorithms are still in the development stage due to the challenges posed by the related complex physiological structures. The objective of this study was to design a 3D medical image registration algorithm that satisfies the need for high accuracy and speed of complex physiological structures. We present a new unsupervised learning algorithm, "DIT-IVNet," for 3D medical image registration. Unlike the more popular convolution-based U-shaped registration network architectures like VoxelMorph, DIT-IVNet uses a combined convolution and transformer network architecture. To better extract image information features and reduce the heavy training parameters, we improved the 2D_Depatch module to a 3D_Depatch module, thus replacing the patch embedding in the original Vision Transformer which adaptively performs patch embedding based on 3D image structure information. We also designed inception blocks in the down-sampling part of the network to help coordinate feature learning from images to different scales. Dice score, Negative Jacobian determinant, Hausdorff distance, and Structural Similarity evaluation metrics were used to evaluate the registration effects. The results showed that our proposed network had the best metric results compared with some state-of-the-art methods. Moreover, our network obtained the highest Dice score in the generalization experiments which indicated better generalizability of our model. We proposed an unsupervised registration network and evaluated its performance in deformable medical image registration. The results of the evaluation metrics showed that the network structure outperformed state-of-the-art methods for the registration of brain datasets.

Deformable 3D medical image registration with convolutional neural network and transformer

Deformable registration of medical images based on deep learning has been the research focus this year. Convolutional Neural Network (CNN) and the transformer are the most common backbone and have been shown to enhance registration accuracy. However, CNN lacks the ability to contact long-distance information, and the transformer lacks the ability to capture local information. Whichever subtle feature loss may lead to disastrous consequences in the analysis of clinical medicine. This paper presented a novel registration network named Information Complementation Network (ICN). We aim to improve the registration accuracy by complementing the lost information. Pure transformers can establish long-distance spatial information about the image. Proposed meshing patch embedding can minimize the loss of local information and expand the receptive field to extract long-distance information. The dual-path decoder in ICN is designed to restore information furthest. We experimented on 3D brain MRI data and quantitatively compared several excellent registration models. Compared with conventional methods, the dice coefficient increased by 3%. Compared with the advanced methods, the dice coefficient increased by 1%. The number of foldings was reduced by about 50% without any loss of registration accuracy. Each evaluation metric of the trained models on liver CT images was higher than other methods. By fully complementing the lost or invalid information, ICN achieved higher registration accuracy and smoother deformation field. The innovation and contribution of this paper have the potential to be applied to clinical research or medical image processing.

Registration of 3D medical images based on unsupervised cooperative cascade of deep networks

In this paper, a deformable registration network (DR-Net) and a multi-scale cascading strategy are designed for the registration of largely deformed 3D medical images. Our DR-Net appears as a U-shaped convolutional neural network with a pyramidal input module (PIM), a light weighted sequential Inception module and an SCAM convolutional attention module. Our multi-scale cooperative cascading strategy integrates the deformation field information within and between sub-networks at different scales to synthesize the cascaded deformation fields. To cooperatively train the cascaded network, not only the output of the final network layer but also the multi-scale outputs from different layers of the decoder in the last cascaded sub-network are used to calculate loss function. As compared with the VoxelMorph and IVTN, the average dice similarity coefficients (Dice) achieved with our DR-Net are 2.4% and 2.5% higher on the Sliver dataset and are 2.5% and 2.4% higher on the LiTS dataset. The average Dice coefficients achieved with our multi-scale cascading strategy of three DR-Nets are 1.6% and 1.9% higher than those of the VM-CR3 and are 1.5% and 1.7% higher than those of the IVTN-CR3 on these two datasets, respectively. These results show that not only our proposed DR-Net itself but also the cascade of them outperform the state-of-the-art methods and their cascades in registration accuracy.

Attention Guided Policy Optimization for 3D Medical Image Registration

Learning-based image registration approaches typically learn to map from input images to a transformation matrix. Regarding the current deep-learning-based image rigid registration approaches learn a transformation matrix in a one-shot way. Our purpose is to present a deep reinforcement learning (DRL) based method for image registration to explicitly model the step-wise nature of the human registration process. We cast an image registration process as a Markov Decision Process (MDP) where actions are defined as global image adjustment operations. Then we train our proxy to learn the optimal action sequences to achieve a good registration. More specifically, we propose a DRL proxy incorporating an attention mechanism to address the challenge of large differences in appearance between images from different modalities. Registration experiments on 3D CT-MR image pairs of patients with nasopharyngeal carcinoma and on publicly available 3D PET-MR image pairs show that our approach significantly outperforms other methods, and achieves state-of-the-art performance in multi-m-modal medical image registration.

Similarity attention-based CNN for robust 3D medical image registration

In recent years, deep learning (DL)-based registration technology has significantly improved the calculation speed of medical image registration. Existing DL-based registration methods generally use raw data features to predict the deformation field. However, this strategy may not be very effective for difficult registration tasks. Hence, in this study, we propose a similarity attention-based convolutional neural network (CNN) for accurate and robust three-dimensional medical image registration. We first introduce a similarity-based local attention model as an auxiliary module for building a displacement searching space, instead of a direct displacement prediction based on raw data. The proposed model can help the network focus on spatial correspondences with high similarities and ignore those with low similarities. A multi-scale CNN is then integrated with the similarity-based local attention for providing non-local attention, lightweight network, and coarse-to-fine registration. We evaluated the proposed method for various applications, such as the registration of large-scope abdominal computerized tomography (CT) images and chest CT images acquired at different respiratory phases, and atlas registration in magnetic resonance imaging. The experimental results demonstrate that the proposed method can provide a more accurate and robust registration performance than state-of-the-art registration methods.

Joint few-shot registration and segmentation self-training of 3D medical images

Medical image segmentation and registration are very important related steps in clinical medical diagnosis. In the past few years, deep learning techniques for joint segmentation and registration have achieved good results in both segmentation and registration tasks through one-way assisted learning or mutual utilization. However, they often rely on large labeled datasets for supervised training or directly use pseudo-labels without quality estimation. We propose a joint registration and segmentation self-training framework (JRSS), which aims to use segmentation pseudo-labels to promote shared learning between segmentation and registration in scenarios with few manually labeled samples while improving the performance of dual tasks. JRSS combines weakly supervised registration and semi-supervised segmentation learning in a self-training framework. Segmentation self-training generates high-quality pseudo-labels for unlabeled data by injecting noise, pseudo-labels screening, and uncertainty correction. Registration utilizes pseudo-labels to facilitate weakly supervised learning, and as input noise as well as data augmentation to facilitate segmentation self-training. Experiments on two public 3D medical image datasets, abdominal CT and brain MRI, demonstrate that our proposed method achieves simultaneous improvements in segmentation and registration accuracy under few-shot scenarios. Outperforms the single-task fully-supervised training state-of-the-art model in the metrics of Dice similarity coefficient and standard deviation of the Jacobian determinant.

Cross-Resolution Distillation for Efficient 3D Medical Image Registration

Images captured in clinic such as MRI scans are usually in 3D formats with high spatial resolutions. Existing learning-based models for medical image registration consume large GPU memories and long inference time, which is difficult to be deployed in resource-limited diagnosis scenarios. To address this problem, instead of shrinking the model size as in previous works, we turn to reducing the input resolution of existing registration models and boosting their performance through knowledge distillation. Specifically, we propose a cross-resolution distillation (CRD) scheme, which is designed to train low-resolution models under the guidance of corresponding high-resolution models. Nevertheless, due to the resolution gap between features in high/low-resolution models, straightforward distillation is difficult to apply. To overcome this challenge, we first introduce a feature-shifted teacher (FST) to shift and fuse features of high/low-resolution models. Then, we exploit this teacher model to guide the learning of the low-resolution student model with distillation losses on both features and deformation fields. Finally, we only need to use the distilled student model during inference. Experimental results on four 3D medical image datasets demonstrate that the low-resolution models trained through our CRD scheme use fewer than 20% GPU memories and less than 20% inference time while achieving competitive performance compared with corresponding high-resolution models.

LDVoxelMorph: A precise loss function and cascaded architecture for unsupervised diffeomorphic large displacement registration.

The traditional learning-based non-rigid registration methods for medical images are trained by an invariant smoothness regularization parameter, which cannot satisfy the registration accuracy and diffeomorphic property simultaneously. The diffeomorphic property reflects the credibility of the registration results. To improve the diffeomorphic property in 3D medical image registration, we propose a diffeomorphic cascaded network based on the compressed loss (CL), named LDVoxelMorph. The proposed network has several constituent U-Nets and is trained with deep supervision, which uses a different spatial smoothness regularization parameter in each constituent U-Nets for training. This cascade-variant smoothness regularization parameter can maintain the diffeomorphic property in top cascades with large displacement and achieve precise registration in bottom cascades. Besides, we develop the CL as a penalty for the velocity field, which can accurately limit the velocity field that causes the deformation field overlap after the velocity field integration. In our registration experiments, the dice scores of our method were 0.892 ± 0.040 on liver CT datasets SLIVER37 , 0.848 ± 0.044 on liver CT datasets LiTS38 , 0.689 ± 0.014 on brain MRI datasets LPBA38 , and the number of overlapping voxels of deformation field were 325, 159, and 0, respectively. Ablation study shows that the CL improves the diffeomorphic property more effectively than increases. Experiment results show that our method can achieve higher registration accuracy assessed by dice scores and overlapping voxels while maintaining the diffeomorphic property for large deformation.

A review of deep learning-based three-dimensional medical image registration methods.

Medical image registration is a vital component of many medical procedures, such as image-guided radiotherapy (IGRT), as it allows for more accurate dose-delivery and better management of side effects. Recently, the successful implementation of deep learning (DL) in various fields has prompted many research groups to apply DL to three-dimensional (3D) medical image registration. Several of these efforts have led to promising results. This review summarized the progress made in DL-based 3D image registration over the past 5 years and identify existing challenges and potential avenues for further research. The collected studies were statistically analyzed based on the region of interest (ROI), image modality, supervision method, and registration evaluation metrics. The studies were classified into three categories: deep iterative registration, supervised registration, and unsupervised registration. The studies are thoroughly reviewed and their unique contributions are highlighted. A summary is presented following a review of each category of study, discussing its advantages, challenges, and trends. Finally, the common challenges for all categories are discussed, and potential future research topics are identified.

A 3D medical image registration method based on multi-scale feature fusion

Deformable medical image registration refers to finding a certain transformation so that the corresponding points of two medical images can be aligned in space. This has important clinical applications. In this article, we propose an unsupervised end-to-end medical image registration method. In this method, the fixed image and the moving image are concatenated in series and input into the convolution neural network to obtain the feature images of different scales. In order to improve the ability of neural network to capture global and local information, we fuse feature maps of different scales. The spatial transformation network uses the deformation field to deform the moving image, so as to realize the registration of the two pairs. We validate our method in the ABIDE data set and compare it with some classic state-of-the-art methods. The experimental results show that our method improves the registration accuracy of image pairs.

Joint affine and deformable three-dimensional networks for brain MRI registration.

Volumetric medical image registration has important clinical significance. Traditional registration methods may be time-consuming when processing large volumetric data due to their iterative optimizations. In contrast, existing deep learning-based networks can obtain the registration quickly. However, most of them require independent rigid alignment before deformable registration; these two steps are often performed separately and cannot be end-to-end. We propose an end-to-end joint affine and deformable network for three-dimensional (3D) medical image registration. The proposed network combines two deformation methods; the first one is for obtaining affine alignment and the second one is a deformable subnetwork for achieving the nonrigid registration. The parameters of the two subnetworks are shared. The global and local similarity measures are used as loss functions for the two subnetworks, respectively. Moreover, an anatomical similarity loss is devised to weakly supervise the training of the whole registration network. Finally, the trained network can perform deformable registration in one forward pass. The efficacy of our network was extensively evaluated on three public brain MRI datasets including Mindboggle101, LPBA40, and IXI. Experimental results demonstrate our network consistently outperformed several state-of-the-art methods with respect to the metrics of Dice index (DSC), Hausdorff distance (HD), and average symmetric surface distance (ASSD). The proposed network provides accurate and robust volumetric registration without any pre-alignment requirement, which facilitates the end-to-end deformable registration.

Multimodal 3D medical image registration guided by shape encoder-decoder networks.

Nonlinear multimodal image registration, for example, the fusion of computed tomography (CT) and magnetic resonance imaging (MRI), fundamentally depends on a definition of image similarity. Previous methods that derived modality-invariant representations focused on either global statistical grayscale relations or local structural similarity, both of which are prone to local optima. In contrast to most learning-based methods that rely on strong supervision of aligned multimodal image pairs, we aim to overcome this limitation for further practical use cases. We propose a new concept that exploits anatomical shape information and requires only segmentation labels for both modalities individually. First, a shape-constrained encoder-decoder segmentation network without skip connections is jointly trained on labeled CT and MRI inputs. Second, an iterative energy-based minimization scheme is introduced that relies on the capability of the network to generate intermediate nonlinear shape representations. This further eases the multimodal alignment in the case of large deformations. Our novel approach robustly and accurately aligns 3D scans from the multimodal whole-heart segmentation dataset, outperforming classical unsupervised frameworks. Since both parts of our method rely on (stochastic) gradient optimization, it can be easily integrated in deep learning frameworks and executed on GPUs. We present an integrated approach for weakly supervised multimodal image registration. Achieving promising results due to the exploration of intermediate shape features as registration guidance encourages further research in this direction.

Unsupervised 3D End-to-End Medical Image Registration With Volume Tweening Network.

3D medical image registration is of great clinical importance. However, supervised learning methods require a large amount of accurately annotated corresponding control points (or morphing), which are very difficult to obtain. Unsupervised learning methods ease the burden of manual annotation by exploiting unlabeled data without supervision. In this article, we propose a new unsupervised learning method using convolutional neural networks under an end-to-end framework, Volume Tweening Network (VTN), for 3D medical image registration. We propose three innovative technical components: (1)An end-to-end cascading scheme that resolves large displacement; (2)An efficient integration of affine registration network; and (3)An additional invertibility loss that encourages backward consistency. Experiments demonstrate that our algorithm is 880x faster (or 3.3x faster without GPU acceleration) than traditional optimization-based methods and achieves state-of-the-art performance in medical image registration.

Non-Rigid Multi-Modal 3D Medical Image Registration Based on Foveated Modality Independent Neighborhood Descriptor.

The non-rigid multi-modal three-dimensional (3D) medical image registration is highly challenging due to the difficulty in the construction of similarity measure and the solution of non-rigid transformation parameters. A novel structural representation based registration method is proposed to address these problems. Firstly, an improved modality independent neighborhood descriptor (MIND) that is based on the foveated nonlocal self-similarity is designed for the effective structural representations of 3D medical images to transform multi-modal image registration into mono-modal one. The sum of absolute differences between structural representations is computed as the similarity measure. Subsequently, the foveated MIND based spatial constraint is introduced into the Markov random field (MRF) optimization to reduce the number of transformation parameters and restrict the calculation of the energy function in the image region involving non-rigid deformation. Finally, the accurate and efficient 3D medical image registration is realized by minimizing the similarity measure based MRF energy function. Extensive experiments on 3D positron emission tomography (PET), computed tomography (CT), T1, T2, and (proton density) PD weighted magnetic resonance (MR) images with synthetic deformation demonstrate that the proposed method has higher computational efficiency and registration accuracy in terms of target registration error (TRE) than the registration methods that are based on the hybrid L-BFGS-B and cat swarm optimization (HLCSO), the sum of squared differences on entropy images, the MIND, and the self-similarity context (SSC) descriptor, except that it provides slightly bigger TRE than the HLCSO for CT-PET image registration. Experiments on real MR and ultrasound images with unknown deformation have also be done to demonstrate the practicality and superiority of the proposed method.

Integrating computer vision and non-linear optimization for automated deformable registration of 3D medical images

Deformable image registration (DIR), the process of estimating and applying non-linear transforms to spatially align sets of two or more images, is a challenging task with many important clinical applications including kinetic analysis, cancer treatment targeting, and evaluation of treatment response. Current techniques use non-linear optimizations only to reach a local minimum and not a globally optimal solution, limiting application to cases with small spatial displacements. Various semi-dense feature-based methods drawing inspiration from mammalian systems as the basis for 2D visual processing have been implemented for automated wide baseline registration and object detection applications with great success. Extension to the 3D case, which has been shown to enable highly efficient coarse global image search, in theory could be adapted also to allow precise semi-dense global optimization. The algorithm described in this paper, dubbed constrained robust affine feature transform (CRAFT), incorporates paradigms from various computer-vision techniques to combine aspects of the human visual pathway with proven non-linear optimization methods to automate general deformable registration with unprecedented robustness. This hybrid technique is able to estimate registration confidence and can serve as the basis of machine perception of medical images for machine learning.