Multimodal Deformable Registration Research Articles

ContraReg: Contrastive Learning of Multi-modality Unsupervised Deformable Image Registration.

Establishing voxelwise semantic correspondence across distinct imaging modalities is a foundational yet formidable computer vision task. Current multi-modality registration techniques maximize hand-crafted inter-domain similarity functions, are limited in modeling nonlinear intensity-relationships and deformations, and may require significant re-engineering or underperform on new tasks, datasets, and domain pairs. This work presents ContraReg, an unsupervised contrastive representation learning approach to multi-modality deformable registration. By projecting learned multi-scale local patch features onto a jointly learned inter-domain embedding space, ContraReg obtains representations useful for non-rigid multi-modality alignment. Experimentally, ContraReg achieves accurate and robust results with smooth and invertible deformations across a series of baselines and ablations on a neonatal T1-T2 brain MRI registration task with all methods validated over a wide range of deformation regularization strengths.

Multimodal deformable registration based on unsupervised learning

Multimodal deformable registration is designed to solve dense spatial transformations and is used to align images of two different modalities It is a key issue in many medical image analysis applications Multimodal image registration based on traditional methods aims to solve the optimization problem of each pair of images, and usually achieves excellent registration performance, but the calculation cost is high and the running time is long The deep learning method greatly reduces the running time by learning the network used to perform registration These learning-based methods are very effective for single-modality registration However, the intensity distribution of different modal images is unknown and complex Most existing methods rely heavily on label data Faced with these challenges, this paper proposes a deep multimodal registration framework based on unsupervised learning Specifically, the framework consists of feature learning based on matching amount and deformation field learning based on maximum posterior probability, and realizes unsupervised training by means of spatial conversion function and differentiable mutual information loss function In the 3D image registration tasks of MRI T1, MRI T2 and CT, the proposed method is compared with the existing advanced multi-modal registration methods In addition, the registration performance of the proposed method is demonstrated on the latest COVID-19 CT data A large number of results show that the proposed method has a competitive advantage in registration accuracy compared with other methods, and greatly reduces the calculation time © 2021, Editorial Board of JBUAA All right reserved

Reducing motion artifact in sequential-scan dual-energy CT imaging by incorporating deformable registration within joint statistical image reconstruction

Dual-energy computed tomography (DECT) has been widely used to reconstruct basis components. In previous studies, ou DECT algorithm has shown high accuracy in stopping power ratio (SPR) estimation of fixed objects for proton radiotherapy planning. However, patient movement between sequential data acquisitions may lead to severe motion artifacts in the component images. In order to reduce or eliminate the motion artifacts in clinical applications, we combine a deformable registration method with an accurate joint statistical iterative reconstruction algorithm, dual-energy alternating minimization (DEAM). Image registration is a process of geometrically aligning two or more images. We implement a multi-modality symmetric deformable registration method based on Advanced Normalization Tools (ANTs) to automatically align the scans we acquire for the same patient. The precalculated registration mapping and its inverse are then embedded into each iteration of the DEAM algorithm. The performance of warped DEAM is quantitatively assessed. Theoretically, the performance of warped DEAM on moved patients should be comparable to the performance of the original DEAM algorithm on fixed objects. The warped DEAM algorithm reduces motion artifacts while preserving the accuracy of the iterative joint statistical CT reconstruction algorithm, which enables us to reconstruct accurate results from sequentially scanned dual-energy patient data.

Memory-efficient 2.5D convolutional transformer networks for multi-modal deformable registration with weak label supervision applied to whole-heart CT and MRI scans.

PURPOSE: Despite its potential for improvements through supervision, deep learning-based registration approaches are difficult to train for large deformations in 3D scans due to excessive memory requirements. METHODS: We propose a new 2.5D convolutional transformer architecture that enables us to learn a memory-efficient weakly supervised deep learning model for multi-modal image registration. Furthermore, we firstly integrate a volume change control term into the loss function of a deep learning-based registration method to penalize occurring foldings inside the deformation field. RESULTS: Our approach succeeds at learning large deformations across multi-modal images. We evaluate our approach on 100 pair-wise registrations of CT and MRI whole-heart scans and demonstrate considerably higher Dice Scores (of 0.74) compared to a state-of-the-art unsupervised discrete registration framework (deeds with Dice of 0.71). CONCLUSION: Our proposed memory-efficient registration method performs better than state-of-the-art conventional registration methods. By using a volume change control term in the loss function, the number of occurring foldings can be considerably reduced on new registration cases.

Performance evaluation of MIND demons deformable registration of MR and CT images in spinal interventions

Accurate intraoperative localization of target anatomy and adjacent nervous and vascular tissue is essential to safe, effective surgery, and multimodality deformable registration can be used to identify such anatomy by fusing preoperative CT or MR images with intraoperative images. A deformable image registration method has been developed to estimate viscoelastic diffeomorphisms between preoperative MR and intraoperative CT using modality-independent neighborhood descriptors (MIND) and a Huber metric for robust registration. The method, called MIND Demons, optimizes a constrained symmetric energy functional incorporating priors on smoothness, geodesics, and invertibility by alternating between Gauss–Newton optimization and Tikhonov regularization in a multiresolution scheme. Registration performance was evaluated for the MIND Demons method with a symmetric energy formulation in comparison to an asymmetric form, and sensitivity to anisotropic MR voxel-size was analyzed in phantom experiments emulating image-guided spine-surgery in comparison to a free-form deformation (FFD) method using local mutual information (LMI). Performance was validated in a clinical study involving 15 patients undergoing intervention of the cervical, thoracic, and lumbar spine. The target registration error (TRE) for the symmetric MIND Demons formulation (1.3 ± 0.8 mm (median ± interquartile)) outperformed the asymmetric form (3.6 ± 4.4 mm). The method demonstrated fairly minor sensitivity to anisotropic MR voxel size, with median TRE ranging 1.3–2.9 mm for MR slice thickness ranging 0.9–9.9 mm, compared to TRE = 3.2–4.1 mm for LMI FFD over the same range. Evaluation in clinical data demonstrated sub-voxel TRE (<2 mm) in all fifteen cases with realistic deformations that preserved topology with sub-voxel invertibility (0.001 mm) and positive-determinant spatial Jacobians. The approach therefore appears robust against realistic anisotropic resolution characteristics in MR and yields registration accuracy suitable to application in image-guided spine-surgery.

Cross contrast multi-channel image registration using image synthesis for MR brain images.

Multi-modal deformable registration is important for many medical image analysis tasks such as atlas alignment, image fusion, and distortion correction. Whereas a conventional method would register images with different modalities using modality independent features or information theoretic metrics such as mutual information, this paper presents a new framework that addresses the problem using a two-channel registration algorithm capable of using mono-modal similarity measures such as sum of squared differences or cross-correlation. To make it possible to use these same-modality measures, image synthesis is used to create proxy images for the opposite modality as well as intensity-normalized images from each of the two available images. The new deformable registration framework was evaluated by performing intra-subject deformation recovery, intra-subject boundary alignment, and inter-subject label transfer experiments using multi-contrast magnetic resonance brain imaging data. Three different multi-channel registration algorithms were evaluated, revealing that the framework is robust to the multi-channel deformable registration algorithm that is used. With a single exception, all results demonstrated improvements when compared against single channel registrations using the same algorithm with mutual information.

MIND Demons: Symmetric Diffeomorphic Deformable Registration of MR and CT for Image-Guided Spine Surgery.

Intraoperative localization of target anatomy and critical structures defined in preoperative MR/CT images can be achieved through the use of multimodality deformable registration. We propose a symmetric diffeomorphic deformable registration algorithm incorporating a modality-independent neighborhood descriptor (MIND) and a robust Huber metric for MR-to-CT registration. The method, called MIND Demons, finds a deformation field between two images by optimizing an energy functional that incorporates both the forward and inverse deformations, smoothness on the integrated velocity fields, a modality-insensitive similarity function suitable to multimodality images, and smoothness on the diffeomorphisms themselves. Direct optimization without relying on the exponential map and stationary velocity field approximation used in conventional diffeomorphic Demons is carried out using a Gauss-Newton method for fast convergence. Registration performance and sensitivity to registration parameters were analyzed in simulation, phantom experiments, and clinical studies emulating application in image-guided spine surgery, and results were compared to mutual information (MI) free-form deformation (FFD), local MI (LMI) FFD, normalized MI (NMI) Demons, and MIND with a diffusion-based registration method (MIND-elastic). The method yielded sub-voxel invertibility (0.008 mm) and nonzero-positive Jacobian determinants. It also showed improved registration accuracy in comparison to the reference methods, with mean target registration error (TRE) of 1.7 mm compared to 11.3, 3.1, 5.6, and 2.4 mm for MI FFD, LMI FFD, NMI Demons, and MIND-elastic methods, respectively. Validation in clinical studies demonstrated realistic deformations with sub-voxel TRE in cases of cervical, thoracic, and lumbar spine.

MiLBP: a robust and fast modality-independent 3D LBP for multimodal deformable registration.

PurposeComputer-assisted intervention often depends on multimodal deformable registration to provide complementary information. However, multimodal deformable registration remains a challenging task.MethodsThis paper introduces a novel robust and fast modality-independent 3D binary descriptor, called miLBP, which integrates the principle of local self-similarity with a form of local binary pattern and can robustly extract the similar geometry features from 3D volumes across different modalities. miLBP is a bit string that can be computed by simply thresholding the voxel distance. Furthermore, the descriptor similarity can be evaluated efficiently using the Hamming distance.ResultsmiLBP was compared to vector-valued self-similarity context (SSC) in artificial image and clinical settings. The results show that miLBP is more robust than SSC in extracting local geometry features across modalities and achieved higher registration accuracy in different registration scenarios. Furthermore, in the most challenging registration between preoperative magnetic resonance imaging and intra-operative ultrasound images, our approach significantly outperforms the state-of-the-art methods in terms of both accuracy (2.15pm 1.1 hbox { mm}) and speed (29.2 s for one case).ConclusionsRegistration performance and speed indicate that miLBP has the potential of being applied to the time-sensitive intra-operative computer-assisted intervention.

TH‐CD‐206‐10: Clinical Application of the MIND Demons Algorithm for Symmetric Diffeomorphic Deformable MR‐To‐CT Image Registration in Spinal Interventions

Purpose.Accurate intraoperative localization of target anatomy (e.g., tumors and vertebral levels) and critical structures (e.g., nerves and vessels) is essential to safe, effective surgery. Preoperative CT/MR images can be used to identify such vital anatomy in intraoperative images through multimodality deformable registration. We proposed a deformable registration method to align preoperative MR to intraoperative CT using modality‐independent neighborhood descriptors (MIND) and a Huber metric for robust registration.Methods.The method, called MIND Demons, estimates time‐dependent diffeomorphisms between two images by optimizing a constrained symmetric energy functional incorporating priors on smoothness, conservation of momentum, geodesic length, invertibility, and closure under composition. Alternating optimization is performed using a regularized Gauss‐Newton (GN) method in a multiresolution scheme. Performance was measured in phantom experiments emulating image‐guided spine‐surgery and in a retrospective clinical study (N=15 patients) evaluated in terms of target registration error (TRE) and diffeomorphic properties of the resulting deformations.Results.The MIND Demons method outperformed free‐form deformation (FFD) methods and conventional Demons in phantom experiments (median TRE = 1.5 mm, compared to 10.2 mm for mutual information (MI) based FFD, 2.6 mm for local‐MI FFD, and 5.1 mm for normalized‐MI Demons). The resulting deformations resolved realistic deformation in clinical data with sub‐voxel TRE (&lt;2 mm) in cases of cervical, thoracic, and lumbar spine and preserved topology with sub‐voxel invertibility (0.004 mm) and positive‐determinant nonsingular spatial‐Jacobians.Conclusion.A modality‐independent deformable registration method has been developed, incorporating the constrained symmetric energy functional and the Huber metric to yield stable and accurate registration, with computational efficiency improved using the GN optimization. The approach yielded registration accuracy suitable to application in image‐guided spine‐surgery across each region of the spine under realistic modes of deformation.S. Vogt and G. Kleinszig is with Siemens Healthcare XP.

MIND Demons for MR-to-CT Deformable Image Registration In Image-Guided Spine Surgery.

Localization of target anatomy and critical structures defined in preoperative MR images can be achieved by means of multi-modality deformable registration to intraoperative CT. We propose a symmetric diffeomorphic deformable registration algorithm incorporating a modality independent neighborhood descriptor (MIND) and a robust Huber metric for MR-to-CT registration. The method, called MIND Demons, solves for the deformation field between two images by optimizing an energy functional that incorporates both the forward and inverse deformations, smoothness on the velocity fields and the diffeomorphisms, a modality-insensitive similarity function suitable to multi-modality images, and constraints on geodesics in Lagrangian coordinates. Direct optimization (without relying on an exponential map of stationary velocity fields used in conventional diffeomorphic Demons) is carried out using a Gauss-Newton method for fast convergence. Registration performance and sensitivity to registration parameters were analyzed in simulation, in phantom experiments, and clinical studies emulating application in image-guided spine surgery, and results were compared to conventional mutual information (MI) free-form deformation (FFD), local MI (LMI) FFD, and normalized MI (NMI) Demons. The method yielded sub-voxel invertibility (0.006 mm) and nonsingular spatial Jacobians with capability to preserve local orientation and topology. It demonstrated improved registration accuracy in comparison to the reference methods, with mean target registration error (TRE) of 1.5 mm compared to 10.9, 2.3, and 4.6 mm for MI FFD, LMI FFD, and NMI Demons methods, respectively. Validation in clinical studies demonstrated realistic deformation with sub-voxel TRE in cases of cervical, thoracic, and lumbar spine. A modality-independent deformable registration method has been developed to estimate a viscoelastic diffeomorphic map between preoperative MR and intraoperative CT. The method yields registration accuracy suitable to application in image-guided spine surgery across a broad range of anatomical sites and modes of deformation.

Feasibility of Multimodal Deformable Registration for Head and Neck Tumor Treatment Planning

To investigate the feasibility of using deformable registration in clinical practice to fuse MR and CT images of the head and neck for treatment planning. A state-of-the-art deformable registration algorithm was optimized, evaluated, and compared with rigid registration. The evaluation was based on manually annotated anatomic landmarks and regions of interest in both modalities. We also developed a multiparametric registration approach, which simultaneously aligns T1- and T2-weighted MR sequences to CT. This was evaluated and compared with single-parametric approaches. Our results show that deformable registration yielded a better accuracy than rigid registration, without introducing unrealistic deformations. For deformable registration, an average landmark alignment of approximatively 1.7 mm was obtained. For all the regions of interest excluding the cerebellum and the parotids, deformable registration provided a median modified Hausdorff distance of approximatively 1 mm. Similar accuracies were obtained for the single-parameter and multiparameter approaches. This study demonstrates that deformable registration of head-and-neck CT and MR images is feasible, with overall a significanlty higher accuracy than for rigid registration.

Multiatlas segmentation of thoracic and abdominal anatomy with level set-based local search.

Segmentation of organs at risk (OARs) remains one of the most time‐consuming tasks in radiotherapy treatment planning. Atlas‐based segmentation methods using single templates have emerged as a practical approach to automate the process for brain or head and neck anatomy, but pose significant challenges in regions where large interpatient variations are present. We show that significant changes are needed to autosegment thoracic and abdominal datasets by combining multi‐atlas deformable registration with a level set‐based local search. Segmentation is hierarchical, with a first stage detecting bulk organ location, and a second step adapting the segmentation to fine details present in the patient scan. The first stage is based on warping multiple presegmented templates to the new patient anatomy using a multimodality deformable registration algorithm able to cope with changes in scanning conditions and artifacts. These segmentations are compacted in a probabilistic map of organ shape using the STAPLE algorithm. Final segmentation is obtained by adjusting the probability map for each organ type, using customized combinations of delineation filters exploiting prior knowledge of organ characteristics. Validation is performed by comparing automated and manual segmentation using the Dice coefficient, measured at an average of 0.971 for the aorta, 0.869 for the trachea, 0.958 for the lungs, 0.788 for the heart, 0.912 for the liver, 0.884 for the kidneys, 0.888 for the vertebrae, 0.863 for the spleen, and 0.740 for the spinal cord. Accurate atlas segmentation for abdominal and thoracic regions can be achieved with the usage of a multi‐atlas and perstructure refinement strategy. To improve clinical workflow and efficiency, the algorithm was embedded in a software service, applying the algorithm automatically on acquired scans without any user interaction.PACS numbers: 87.57.nm, 87.57.N‐, 87.57.Q‐

Multimodal deformable registration of traumatic brain injury MR volumes via the Bhattacharyya distance.

An important problem of neuroimaging data analysis for traumatic brain injury (TBI) is the task of coregistering MR volumes acquired using distinct sequences in the presence of widely variable pixel movements which are due to the presence and evolution of pathology. We are motivated by this problem to design a numerically stable registration algorithm which handles large deformations. To this end, we propose a new measure of probability distributions based on the Bhattacharyya distance, which is more stable than the widely used mutual information due to better behavior of the square root function than the logarithm at zero. Robustness is illustrated on two TBI patient datasets, each containing 12 MR modalities. We implement our method on graphics processing units (GPU) so as to meet the clinical requirement of time-efficient processing of TBI data. We find that 6 sare required to register a pair of volumes with matrix sizes of 256 × 256 × 60 on the GPU. In addition to exceptional time efficiency via its GPU implementation, this methodology provides a clinically informative method for the mapping and evaluation of anatomical changes in TBI.

SU-E-J-109: Accurate Contour Transfer Between Different Image Modalities Using a Hybrid Deformable Image Registration and Fuzzy Connected Image Segmentation Method.

To develop and evaluate a tool that can improve the accuracy of contour transfer between different image modalities under challenging conditions of low image contrast and large image deformation, comparing to a few commonly used methods, for radiation treatment planning. The software tool includes the following steps and functionalities: (1) accepting input of images of different modalities, (2) converting existing contours on reference images (e.g., MRI) into delineated volumes and adjusting the intensity within the volumes to match target images (e.g., CT) intensity distribution for enhanced similarity metric, (3) registering reference and target images using appropriate deformable registration algorithms (e.g., B-spline, demons) and generate deformed contours, (4) mapping the deformed volumes on target images, calculating mean, variance, and center of mass as the initialization parameters for consecutive fuzzy connectedness (FC) image segmentation on target images, (5) generate affinity map from FC segmentation, (6) achieving final contours by modifying the deformed contours using the affinity map with a gradient distance weighting algorithm. The tool was tested with the CT and MR images of four pancreatic cancer patients acquired at the same respiration phase to minimize motion distortion. Dice's Coefficient was calculated against direct delineation on target image. Contours generated by various methods, including rigid transfer, auto-segmentation, deformable only transfer and proposed method, were compared. Fuzzy connected image segmentation needs careful parameter initialization and user involvement. Automatic contour transfer by multi-modality deformable registration leads up to 10% of accuracy improvement over the rigid transfer. Two extra proposed steps of adjusting intensity distribution and modifying the deformed contour with affinity map improve the transfer accuracy further to 14% averagely. Deformable image registration aided by contrast adjustment and fuzzy connectedness segmentation improves the contour transfer accuracy between multi-modality images, particularly with large deformation and low image contrast.

MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration

Deformable registration of images obtained from different modalities remains a challenging task in medical image analysis. This paper addresses this important problem and proposes a modality independent neighbourhood descriptor (MIND) for both linear and deformable multi-modal registration. Based on the similarity of small image patches within one image, it aims to extract the distinctive structure in a local neighbourhood, which is preserved across modalities. The descriptor is based on the concept of image self-similarity, which has been introduced for non-local means filtering for image denoising. It is able to distinguish between different types of features such as corners, edges and homogeneously textured regions. MIND is robust to the most considerable differences between modalities: non-functional intensity relations, image noise and non-uniform bias fields. The multi-dimensional descriptor can be efficiently computed in a dense fashion across the whole image and provides point-wise local similarity across modalities based on the absolute or squared difference between descriptors, making it applicable for a wide range of transformation models and optimisation algorithms. We use the sum of squared differences of the MIND representations of the images as a similarity metric within a symmetric non-parametric Gauss-Newton registration framework. In principle, MIND would be applicable to the registration of arbitrary modalities. In this work, we apply and validate it for the registration of clinical 3D thoracic CT scans between inhale and exhale as well as the alignment of 3D CT and MRI scans. Experimental results show the advantages of MIND over state-of-the-art techniques such as conditional mutual information and entropy images, with respect to clinically annotated landmark locations.

Deformable registration of high-resolution and cine MR tongue images.

This work investigates a novel 3D multimodal deformable registration method to align high-resolution magnetic resonance imaging (MRI) with cine MRI of the tongue for better visual and motion analysis. Both modalities have different strengths to characterize and analyze the tongue structure or motion. Visual and motion analysis of combined anatomical and temporal information can synergistically improve the utility of each modality. An automated multimodal registration method is presented utilizing structural information computed from the 3D Harris operator to encode spatial and geometric cues into the computation of mutual information. The robustness and accuracy of the proposed method have been demonstrated using experiments on clinical datasets and yielded better performance compared to the conventional method and an average error comparable to the inter-observer variability.

Model‐independent, multimodality deformable image registration by local matching of anatomical features and minimization of elastic energy

With respect to the demands of adaptive and 4D-radiotherapy applications, an algorithm is proposed for a fully automatic, multimodality deformable registration that follows the concept of translational relocation of regularly distributed image subvolumes governed by local anatomical features. Thereby, the problem of global deformable registration is broken down to multiple independent local registration steps which allows for straightforward parallelization of the algorithm. In a subsequent step, possible local misregistrations are corrected for by minimization of the elastic energy of the displacement field under consideration of image information. The final displacement field results from interpolation of the subvolume shift vectors. The algorithm can employ as a similarity measure both the correlation coefficient and mutual information. The latter allows the application to intermodality deformable registration problems. The typical calculation time on a modern multiprocessor PC is well below 1 min, which facilitates almost-interactive, "online" usage. CT-to-MRI and CT-to-cone-beam-CT registrations of head-and-neck data sets are presented, as well as inhale-to-exhale registrations of lung CT data sets. For quantitative evaluation of registration accuracy, a virtual thorax phantom was developed; additionally, a landmark-based evaluation on four lung respiratory-correlated CT data sets was performed. This consistently resulted in average registration residuals on the order of the voxel size or less (3D-residuals approximately 1-2 mm). Summarizing, the presented algorithm allows an accurate multimodality deformable registration with calculation times well below 1 min, and thus bears promise as a versatile basic tool in adaptive and 4D-radiotherapy applications.

TH‐D‐M100F‐04: A Biomechanical Phantom for Validation of Deformable Multimodality Image Algorithms

Purpose: Quantitative validation of multimodality deformable registration and segmentation algorithms is a challenging problem. To address this issue, we built a deformable multimodal imaging phantom system based on a preserved swine lung and a computer controlled airflow system. Comparing to most other digital and physical phantoms, this phantom is truly deformable, MRI compatible, biologically similar to human tissues, and more representative of clinical conditions. The phantom was imaged with CT and MRI protocols and yielded high resolution benchmark images. Our goal is to utilize these benchmark images to provide realistic and robust validation of medical image processing algorithms in radiotherapy applications, including multimodal deformable image registration, segmentation, tissue, as well as 4D‐CT imaging and reconstruction methods. Method and Materials: After the preserved swine lung leakage problems were addressed, the lung was inflated using a computer controlled airflow system and then imaged. 3D‐CT and MRI scans were taken at certain tidal volumes. For 4D‐CT imaging, the lung was scanned according to 4DCT protocol (Low, Med Phys v30, p1254–63, 2003) while the airflow cycle in the lung was computer‐controlled. For demonstration, we performed deformable CT‐CT registration and CT‐MR manual fusion on the collected images. 3D‐CT images were acquired at different spatial resolutions, up to image dimension of 1024×1024×530 and voxel volume of 0.244×0.244×0.67 mm. Results: 3D‐CT images allow us to clearly identify up to the 6th airway bifurcation. MR images were also of adequate quality. Conclusion: We have built the deformable phantom system and obtained images of high resolution with 3D‐CT, 4D‐CT and MRI. The images are expected to be very useful for validation of different medical image processing algorithms. We conclude that this system is promising tool for investigating and validating deformable image algorithms for radiotherapy.