Joint Image Registration Research Articles

Dynamic CBCT imaging using prior model-free spatiotemporal implicit neural representation (PMF-STINR)

Objective. Dynamic cone-beam computed tomography (CBCT) can capture high-spatial-resolution, time-varying images for motion monitoring, patient setup, and adaptive planning of radiotherapy. However, dynamic CBCT reconstruction is an extremely ill-posed spatiotemporal inverse problem, as each CBCT volume in the dynamic sequence is only captured by one or a few x-ray projections, due to the slow gantry rotation speed and the fast anatomical motion (e.g. breathing). Approach. We developed a machine learning-based technique, prior-model-free spatiotemporal implicit neural representation (PMF-STINR), to reconstruct dynamic CBCTs from sequentially acquired x-ray projections. PMF-STINR employs a joint image reconstruction and registration approach to address the under-sampling challenge, enabling dynamic CBCT reconstruction from singular x-ray projections. Specifically, PMF-STINR uses spatial implicit neural representations to reconstruct a reference CBCT volume, and it applies temporal INR to represent the intra-scan dynamic motion of the reference CBCT to yield dynamic CBCTs. PMF-STINR couples the temporal INR with a learning-based B-spline motion model to capture time-varying deformable motion during the reconstruction. Compared with the previous methods, the spatial INR, the temporal INR, and the B-spline model of PMF-STINR are all learned on the fly during reconstruction in a one-shot fashion, without using any patient-specific prior knowledge or motion sorting/binning. Main results. PMF-STINR was evaluated via digital phantom simulations, physical phantom measurements, and a multi-institutional patient dataset featuring various imaging protocols (half-fan/full-fan, full sampling/sparse sampling, different energy and mAs settings, etc). The results showed that the one-shot learning-based PMF-STINR can accurately and robustly reconstruct dynamic CBCTs and capture highly irregular motion with high temporal (∼ 0.1 s) resolution and sub-millimeter accuracy. Significance. PMF-STINR can reconstruct dynamic CBCTs and solve the intra-scan motion from conventional 3D CBCT scans without using any prior anatomical/motion model or motion sorting/binning. It can be a promising tool for motion management by offering richer motion information than traditional 4D-CBCTs.

An improved multi-modal joint segmentation and registration model based on Bhattacharyya distance measure

Joint image segmentation and registration of multi-modality images is a crucial step in the field of image prepossessing. The sensitivity of joint segmentation and registration models to noise is a significant challenge. During the registration process of multi-modal images, the similarity measure plays a vital role in measuring the results as a standard. Accordingly, an improved joint model for registering and segmenting multi-modality images is proposed, by utilising the Bhattacharyya distance measure to achieve improved noise robustness of the proposed model as compared to the existing model using the mutual information metric. The proposed model is applied to various medical and synthetic noisy images of multiple modalities. Moreover, the dataset images used in this study have been obtained from well-known, freely available BRATS 2015 and CHAOS datasets, where the proposed model produces satisfactory results as compared to the existing model. Experimental results show that the proposed model outperforms the existing model in terms of the Bhattacharyya distance measure in noisy images. Statistical analysis and comparison are performed through the relative reduction of the new distance measure, Dice similarity coefficient, Jaccard similarity coefficient and Hausdorff distance.

Joint synthesis and registration network for deformable MR-CBCT image registration for neurosurgical guidance

Objective. The accuracy of navigation in minimally invasive neurosurgery is often challenged by deep brain deformations (up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach). We propose a deep learning-based deformable registration method to address such deformations between preoperative MR and intraoperative CBCT. Approach. The registration method uses a joint image synthesis and registration network (denoted JSR) to simultaneously synthesize MR and CBCT images to the CT domain and perform CT domain registration using a multi-resolution pyramid. JSR was first trained using a simulated dataset (simulated CBCT and simulated deformations) and then refined on real clinical images via transfer learning. The performance of the multi-resolution JSR was compared to a single-resolution architecture as well as a series of alternative registration methods (symmetric normalization (SyN), VoxelMorph, and image synthesis-based registration methods). Main results. JSR achieved median Dice coefficient (DSC) of 0.69 in deep brain structures and median target registration error (TRE) of 1.94 mm in the simulation dataset, with improvement from single-resolution architecture (median DSC = 0.68 and median TRE = 2.14 mm). Additionally, JSR achieved superior registration compared to alternative methods—e.g. SyN (median DSC = 0.54, median TRE = 2.77 mm), VoxelMorph (median DSC = 0.52, median TRE = 2.66 mm) and provided registration runtime of less than 3 s. Similarly in the clinical dataset, JSR achieved median DSC = 0.72 and median TRE = 2.05 mm. Significance. The multi-resolution JSR network resolved deep brain deformations between MR and CBCT images with performance superior to other state-of-the-art methods. The accuracy and runtime support translation of the method to further clinical studies in high-precision neurosurgery.

Joint image pansharpening and registration via structure tensor total variation regularization

Joint image pansharpening and registration via structure tensor total variation regularization

Deep learning-based simultaneous registration and unsupervised non-correspondence segmentation of medical images with pathologies

PurposeThe registration of medical images often suffers from missing correspondences due to inter-patient variations, pathologies and their progression leading to implausible deformations that cause misregistrations and might eliminate valuable information. Detecting non-corresponding regions simultaneously with the registration process helps generating better deformations and has been investigated thoroughly with classical iterative frameworks but rarely with deep learning-based methods.MethodsWe present the joint non-correspondence segmentation and image registration network (NCR-Net), a convolutional neural network (CNN) trained on a Mumford–Shah-like functional, transferring the classical approach to the field of deep learning. NCR-Net consists of one encoding and two decoding parts allowing the network to simultaneously generate diffeomorphic deformations and segment non-correspondences. The loss function is composed of a masked image distance measure and regularization of deformation field and segmentation output. Additionally, anatomical labels are used for weak supervision of the registration task. No manual segmentations of non-correspondences are required.ResultsThe proposed network is evaluated on the publicly available LPBA40 dataset with artificially added stroke lesions and a longitudinal optical coherence tomography (OCT) dataset of patients with age-related macular degeneration. The LPBA40 data are used to quantitatively assess the segmentation performance of the network, and it is shown qualitatively that NCR-Net can be used for the unsupervised segmentation of pathologies in OCT images. Furthermore, NCR-Net is compared to a registration-only network and state-of-the-art registration algorithms showing that NCR-Net achieves competitive performance and superior robustness to non-correspondences.ConclusionNCR-Net, a CNN for simultaneous image registration and unsupervised non-correspondence segmentation, is presented. Experimental results show the network’s ability to segment non-correspondence regions in an unsupervised manner and its robust registration performance even in the presence of large pathologies.

A Nash game based variational model for joint image intensity correction and registration to deal with varying illumination

Registration aligns features of two related images so that information can be compared and/or fused in order to highlight differences and complement information. In real life images where bias field is present, this undesirable artefact causes inhomogeneity of image intensities and hence leads to failure or loss of accuracy of registration models based on minimization of the differences of the two image intensities. Here, we propose a non-linear variational model for joint image intensity correction (illumination and translation) and registration and reformulate it in a game framework. While a non-potential game offers flexible reformulation and can lead to better fitting errors, proving the solution existence for a non-convex model is non-trivial. Here we establish an existence result using the Schauder’s fixed point theorem. To solve the model numerically, we use an alternating minimization algorithm in the discrete setting. Finally numerical results can show that the new model outperforms existing models.

Joint image segmentation and registration based on a dynamic level set approach using truncated hierarchical B-splines

We present a novel approach for joint image segmentation and nonrigid registration using bidirectional composition based level set formulation. This efficient framework incorporates automatic structural analysis from image segmentation into the registration framework. This method has shown an improved performance as compared to carrying out segmentation and registration separately. Unlike previous approaches, the implicit level set function defining the segmentation contour and the spatial transformation function that maps the deformation for image registration are both defined using B-splines. This joint level set framework uses a variational form of an atlas-based segmentation together with large deformation based nonrigid registration. In addition, a bidirectional composition framework is introduced to incorporate a more symmetric update. The minimization of the variational form is accomplished by dynamic evaluations on a set of successively refined adaptive grids at multiple image resolutions. The improvement in the description of the segmentation result using higher order splines leads to a better accuracy of both the image segmentation and registration process. The performance of the proposed method is demonstrated on synthetic and medical images to show the improvement as compared to other registration methods.

Coupling brain-tumor biophysical models and diffeomorphic image registration

We present SIBIA (Scalable Integrated Biophysics-based Image Analysis), a framework for joint image registration and biophysical inversion and we apply it to analyze MR images of glioblastomas (primary brain tumors). We have two applications in mind. The first one is normal-to-abnormal image registration in the presence of tumor-induced topology differences. The second one is biophysical inversion based on single-time patient data. The underlying optimization problem is highly non-linear and non-convex and has not been solved before with a gradient-based approach.Given the segmentation of a normal brain MRI and the segmentation of a cancer patient MRI, we determine tumor growth parameters and a registration map so that if we “grow a tumor” (using our tumor model) in the normal brain and then register it to the patient image, then the registration mismatch is as small as possible. This “coupled problem” two-way couples the biophysical inversion and the registration problem. In the image registration step we solve a large-deformation diffeomorphic registration problem parameterized by an Eulerian velocity field. In the biophysical inversion step we estimate parameters in a reaction–diffusion tumor growth model that is formulated as a partial differential equation (PDE). In SIBIA, we couple these two sub-components in an iterative manner. We first presented the components of SIBIA in “Gholami et al., Framework for Scalable Biophysics-based Image Analysis, IEEE/ACM Proceedings of the SC2017”, in which we derived parallel distributed memory algorithms and software modules for the decoupled registration and biophysical inverse problems. In this paper, our contributions are the introduction of a PDE-constrained optimization formulation of the coupled problem, and the derivation of a Picard iterative solution scheme. We perform extensive tests to experimentally assess the performance of our method on synthetic and clinical datasets. We demonstrate the convergence of the SIBIA optimization solver in different usage scenarios. We demonstrate that using SIBIA, we can accurately solve the coupled problem in three dimensions (2563 resolution) in a few minutes using 11 dual-x86 nodes.

Joint cardiac and respiratory motion estimation for motion-corrected cardiac PET-MR

Respiratory and cardiac motion can strongly impair cardiac PET image quality and tracer uptake quantification. Standard gating techniques can minimize these motion artefacts but suffer from low signal-to-noise ratio because only a small percentage of the total data is utilized. Motion correction approaches have been proposed to overcome this problem but require accurate knowledge of such physiological motion. Here we present a joint PET-MR motion estimation approach which combines complimentary dynamic image information from simultaneously acquired MR and PET to ensure improved cardiac and respiratory motion estimation for motion-corrected image reconstruction (MCIR) of PET images. A 3D triple-echo Dixon MR scan is used both for calculation of MR-based attenuation correction (AC) maps and estimation of physiological motion. PET listmode data is obtained simultaneously to the MR acquisition which is used for a joint motion estimation and reconstruction of the final MCIR PET. In a first step, dynamic cardiac and respiratory motion resolved 4D MR and PET images are reconstructed. These image series are used in a joint image registration to estimate non-rigid cardiac and respiratory motion fields. In a second step, the motion fields are utilized in a MR MCIR to obtain cardiac and respiratory resolved dynamic MR-based AC maps. In the last step, the non-rigid motion fields and the dynamic AC maps are applied in a PET MCIR to obtain the final motion-corrected PET images. PET-MR data has been obtained in six patients without any known heart disease. Motion amplitudes were between 5.6 and 16 mm, with higher values in the basal compared to the mid-ventricular and apical segments. The proposed joint PET-MR motion estimation provided more accurate motion estimation than using either modality separately. The underestimation of PET uptake due to respiratory and cardiac motion artefacts in the AC maps was up to 17%. The average increase in uptake values using MCIR was 23% ± 10% (p < 0.0001), with values of 28% ± 11% (p < 0.0001) for basal, 21% ± 8% (p < 0.0001) for mid-cavity and 17% ± 7% (p < 0.0001) for apical segments. With the proposed scheme we could ensure high PET image quality and improve local PET uptake quantification by up to 30%. Attenuation correction and motion information was obtained from the same PET-MR raw data, which was obtained during free-breathing to minimize scan times and to increase patient comfort.

A Joint Image Registration and Superresolution Method Using a Combinational Continuous Generative Model

In the superresolution (SR) technique, as an inverse problem, the generative model forms low-resolution images from a high-resolution image. This process consists of three principal operations, namely, warping, blurring, and down-sampling. Unlike the traditional SR methods in which a discrete generative model is used, in this paper, a continuous generative model has been proposed for the SR technique. Rather than using a bilinear interpolation for warping operation, we use a Gaussian kernel for interpolation, as it has lower aliasing effect. Within the framework of the proposed continuous generative model, the three principal operations are combined into a unified operation and hence performed simultaneously. According to this combinational operation, a simultaneous image registration and SR method is proposed. Unlike previous simultaneous methods, we neither calculate the Jacobian matrix numerically (to increase the accuracy of calculation) nor derive it treating the three principal operations separately (to reduce the error propagation). We develop a new approach to derive the Jacobian matrix analytically by combining the three principal operations. The results show that our proposed method is more efficient than the recently proposed simultaneous methods.

Joint image registration and point spread function estimation for the super-resolution of satellite images

Image registration and point spread function (PSF) estimation are both key steps in image super-resolution (SR). Traditionally, these two steps are treated independently, which is adequate for natural images. However, for satellite images, which commonly suffer from focal plane distortion and unrecorded spacecraft jitter, it is always difficult to achieve satisfactory image registration or PSF estimation. Consequently, the errors brought by these two processes significantly affect each other and degrade the quality of the subsequent high-resolution (HR) reconstruction. In this paper, a novel joint image registration and PSF estimation method is proposed to produce HR images from a set of degraded low-resolution (LR) satellite images. The joint SR approach is formulated as a convex optimization problem which minimizes the combination of these two parts. It is aimed at achieving PSF estimation and registration simultaneously and progressively, to handle the error in different levels. In addition, the proposed method adopts a more generic observation model, including both geometric motion and radiation difference, which makes the model more universal. Moreover, an iterative scheme based on alternating minimization (AM) is developed to solve the presented cost function via simultaneous low-rank and total variation (LRTV) regularizations. The experimental results confirm the effectiveness of the proposed method on both simulated data and real satellite images.

A level-set approach to joint image segmentation and registration with application to CT lung imaging

Automated analysis of structural imaging such as lung Computed Tomography (CT) plays an increasingly important role in medical imaging applications. Despite significant progress in the development of image registration and segmentation methods, lung registration and segmentation remain a challenging task. In this paper, we present a novel image registration and segmentation approach, for which we develop a new mathematical formulation to jointly segment and register three-dimensional lung CT volumes. The new algorithm is based on a level-set formulation, which merges a classic Chan–Vese segmentation with the active dense displacement field estimation. Combining registration with segmentation has two key advantages: it allows to eliminate the problem of initializing surface based segmentation methods, and to incorporate prior knowledge into the registration in a mathematically justified manner, while remaining computationally attractive. We evaluate our framework on a publicly available lung CT data set to demonstrate the properties of the new formulation. The presented results show the improved accuracy for our joint segmentation and registration algorithm when compared to registration and segmentation performed separately.

On the Gamma-limit of joint image segmentation and registration functionals based on phase fields

A classical task in image processing is the following: Given two images, identify the structures inside (for instance detect all image edges or all homogeneous regions; this is called segmentation) and find a deformation which maps the structures in one image onto the corresponding ones in the other image (called registration). In medical imaging, for instance, one might segment the organs in two patient images and then identify corresponding organs in both images for automated comparison purposes. The image segmentation is classically performed variationally using the Mumford–Shah model, and the obtained structures are then mapped onto each other by minimizing a registration energy in which the deformation is regularized via elasticity. Experimentally it often seems more robust to perform segmentation and registration simultaneously so that both can benefit from each other. The question to be examined here is how phase field approximations of the Mumford–Shah model behave if used for the joint segmentation and registration problem.We mathematically analyze corresponding generic phase field models and reveal interesting phenomena that rule out some of the models. These phenomena are characteristic of coupling phase fields with deformations and thus are interesting in their own right. In essence, region-based segmentation and registration problems can well be approximated using phase fields, while edge-based approaches typically suffer from different types of vanishing or newly appearing edges. We conjecture how the introduction of a different scaling could remedy those shortcomings.

Joint image registration and fusion method with a gradient strength regularization

Image registration is an essential process for image fusion, and fusion performance can be used to evaluate registration accuracy. We propose a maximum likelihood (ML) approach to joint image registration and fusion instead of treating them as two independent processes in the conventional way. To improve the visual quality of a fused image, a gradient strength (GS) regularization is introduced in the cost function of ML. The GS of the fused image is controllable by setting the target GS value in the regularization term. This is useful because a larger target GS brings a clearer fused image and a smaller target GS makes the fused image smoother and thus restrains noise. Hence, the subjective quality of the fused image can be improved whether the source images are polluted by noise or not. We can obtain the fused image and registration parameters successively by minimizing the cost function using an iterative optimization method. Experimental results show that our method is effective with transformation, rotation, and scale parameters in the range of [−2.0, 2.0] pixel, [−1.1 deg, 1.1 deg], and [0.95, 1.05], respectively, and variances of noise smaller than 300. It also demonstrated that our method yields a more visual pleasing fused image and higher registration accuracy compared with a state-of-the-art algorithm.

Joint Image Registration and Fusion for Panchromatic and Multispectral Images

Both image registration and fusion are essential steps to produce high-resolution multispectral images in remote sensing. Traditionally, they are viewed as two independent processes. As a result, the registration errors ignored in the fusion process can significantly affect the fusion quality. In this context, an iterative optimization approach, which jointly considers the registration and fusion processes, is proposed for panchromatic (PAN) and multispectral (MS) images. Given a registration method and a fusion method, the joint optimization process is described as finding the optimal registration parameters to gain the optimal fusion performance. In our approach, the downhill simplex algorithm is adopted to refine the registration parameters iteratively. Experiments on a set of PAN and MS images of ZY-3 and GeoEye-1 show that the proposed approach outperforms several competing ones in terms of registration accuracy and fusion quality.

A Joint Land Cover Mapping and Image Registration Algorithm Based on a Markov Random Field Model

Traditionally, image registration of multi-modal and multi-temporal images is performed satisfactorily before land cover mapping. However, since multi-modal and multi-temporal images are likely to be obtained from different satellite platforms and/or acquired at different times, perfect alignment is very difficult to achieve. As a result, a proper land cover mapping algorithm must be able to correct registration errors as well as perform an accurate classification. In this paper, we propose a joint classification and registration technique based on a Markov random field (MRF) model to simultaneously align two or more images and obtain a land cover map (LCM) of the scene. The expectation maximization (EM) algorithm is employed to solve the joint image classification and registration problem by iteratively estimating the map parameters and approximate posterior probabilities. Then, the maximum a posteriori (MAP) criterion is used to produce an optimum land cover map. We conducted experiments on a set of four simulated images and one pair of remotely sensed images to investigate the effectiveness and robustness of the proposed algorithm. Our results show that, with proper selection of a critical MRF parameter, the resulting LCMs derived from an unregistered image pair can achieve an accuracy that is as high as when images are perfectly aligned. Furthermore, the registration error can be greatly reduced.

A Gauss–Newton approach to joint image registration and intensity correction

We develop a new efficient numerical methodology for automated simultaneous registration and intensity correction of images. The approach separates the intensity correction term from the images being registered in a regularized expression. Our formulation is consistent with the existing non-parametric image registration techniques, however, an extra additive intensity correction term is carried throughout. An objective functional is formed for which the corresponding Hessian and Jacobian is computed and employed in a multi-level Gauss–Newton minimization approach. In this paper, our experiments are based on elastic regularization on the transformation and total variation on the intensity correction. Validations on dynamic contrast enhanced MR abdominal images for both real and simulated data verified the efficacy of the model.The pursued approach is flexible in which we can exploit various forms of regularization on the transformation and the intensity correction.

Joint Image Registration and Super-Resolution From Low-Resolution Images With Zooming Motion

This paper proposes a new framework for joint image registration and high-resolution (HR) image reconstruction from multiple low-resolution (LR) observations with zooming motion. Conventional super-resolution (SR) methods typically formulate the SR problem as a two-stage process, namely, image registration followed by HR reconstruction. An important step in image SR is the effective estimation of motion parameters. However, the registration algorithms in these two-stage processes experience various degrees of errors. This could degrade the quality of subsequent HR reconstruction. In view of this, this paper presents a new approach that performs joint image registration and SR reconstruction. The proposed iterative SR framework enables the HR image and motion parameters to be estimated simultaneously and progressively. This could increase the potential SR improvement as more accurate estimates of motion parameters could be obtained iteratively. Experimental results show that the proposed method is effective in performing image registration and SR for simulated and real-life images and videos.

Tracking Metastatic Brain Tumors in Longitudinal Scans via Joint Image Registration and Labeling.

The treatment of metastatic brain tumors with stereotactic radiosurgery requires that the clinician first locate the tumors and measure their volumes. Thoroughly searching a patient scan for brain tumors and delineating the lesions can be a long and difficult task when done manually and is also prone to human error. In this paper, we present an automated method for detecting changes in brain tumor lesions over longitudinal scans to aide the clinician's task of determining tumor volumes. Our approach jointly registers the current image with a previous scan while estimating changes in intensity correspondences due to tumor growth or regression. We combine the label map with correspondence changes with tumor segmentations from a previous scan to estimate the metastases in the new image. Alignment and tumor tracking results show promise on 28 registrations using real patient data.

A Maximum Likelihood Approach to Joint Image Registration and Fusion

Both image registration and fusion can be formulated as estimation problems. Instead of estimating the registration parameters and the true scene separately as in the conventional way, we propose a maximum likelihood approach for joint image registration and fusion in this paper. More precisely, the fusion performance is used as the criteria to evaluate the registration accuracy. Hence, the registration parameters can be automatically tuned so that both fusion and registration can be optimized simultaneously. The expectation maximization algorithm is employed to solve this joint optimization problem. The Cramer-Rao bound (CRB) is then derived. Our experiments use several types of sensory images for performance evaluation, such as visual images, IR thermal images, and hyperspectral images. It is shown that the mean square error of estimating the registration parameters using the proposed method is close to the CRBs. At the mean time, an improved fusion performance can be achieved in terms of the edge preservation measure Q(AB/F), compared to the Laplacian pyramid fusion approach.