Non-rigid Registration Approach Research Articles

An Adaptive Region-Based Transformer for Nonrigid Medical Image Registration With a Self-Constructing Latent Graph.

Nonrigid registration of medical images is formulated usually as an optimization problem with the aim of seeking out the deformation field between a referential-moving image pair. During the past several years, advances have been achieved in the convolutional neural network (CNN)-based registration of images, whose performance was superior to most conventional methods. More lately, the long-range spatial correlations in images have been learned by incorporating an attention-based model into the transformer network. However, medical images often contain plural regions with structures that vary in size. The majority of the CNN-and transformer-based approaches adopt embedding of patches that are identical in size, disallowing representation of the inter-regional structural disparities within an image. Besides, it probably leads to the structural and semantical inconsistencies of objects as well. To address this issue, we put forward an innovative module called region-based structural relevance embedding (RSRE), which allows adaptive embedding of an image into unequally-sized structural regions based on the similarity of self-constructing latent graph instead of utilizing patches that are identical in size. Additionally, a transformer is integrated with the proposed module to serve as an adaptive region-based transformer (ART) for registering medical images nonrigidly. As demonstrated by the experimental outcomes, our ART is superior to the advanced nonrigid registration approaches in performance, whose Dice score is 0.734 on the LPBA40 dataset with 0.318% foldings for deformation field, and is 0.873 on the ADNI dataset with 0.331% foldings.

Comparison of physics-based deformable registration methods for image-guided neurosurgery.

This paper compares three finite element-based methods used in a physics-based non-rigid registration approach and reports on the progress made over the last 15 years. Large brain shifts caused by brain tumor removal affect registration accuracy by creating point and element outliers. A combination of approximation- and geometry-based point and element outlier rejection improves the rigid registration error by 2.5 mm and meets the real-time constraints (4 min). In addition, the paper raises several questions and presents two open problems for the robust estimation and improvement of registration error in the presence of outliers due to sparse, noisy, and incomplete data. It concludes with preliminary results on leveraging Quantum Computing, a promising new technology for computationally intensive problems like Feature Detection and Block Matching in addition to finite element solver; all three account for 75% of computing time in deformable registration.

Registration of spatio-temporal point clouds of plants for phenotyping.

Plant phenotyping is a central task in crop science and plant breeding. It involves measuring plant traits to describe the anatomy and physiology of plants and is used for deriving traits and evaluating plant performance. Traditional methods for phenotyping are often time-consuming operations involving substantial manual labor. The availability of 3D sensor data of plants obtained from laser scanners or modern depth cameras offers the potential to automate several of these phenotyping tasks. This automation can scale up the phenotyping measurements and evaluations that have to be performed to a larger number of plant samples and at a finer spatial and temporal resolution. In this paper, we investigate the problem of registering 3D point clouds of the plants over time and space. This means that we determine correspondences between point clouds of plants taken at different points in time and register them using a new, non-rigid registration approach. This approach has the potential to form the backbone for phenotyping applications aimed at tracking the traits of plants over time. The registration task involves finding data associations between measurements taken at different times while the plants grow and change their appearance, allowing 3D models taken at different points in time to be compared with each other. Registering plants over time is challenging due to its anisotropic growth, changing topology, and non-rigid motion in between the time of the measurements. Thus, we propose a novel approach that first extracts a compact representation of the plant in the form of a skeleton that encodes both topology and semantic information, and then use this skeletal structure to determine correspondences over time and drive the registration process. Through this approach, we can tackle the data association problem for the time-series point cloud data of plants effectively. We tested our approach on different datasets acquired over time and successfully registered the 3D plant point clouds recorded with a laser scanner. We demonstrate that our method allows for developing systems for automated temporal plant-trait analysis by tracking plant traits at an organ level.

Registration of magnetic resonance and computed tomography images in patients with oral squamous cell carcinoma for three-dimensional virtual planning of mandibular resection and reconstruction

The aim of this study was to evaluate and present an automated method for registration of magnetic resonance imaging (MRI) and computed tomography (CT) or cone beam CT (CBCT) images of the mandibular region for patients with oral squamous cell carcinoma (OSCC). Registered MRI and (CB)CT could facilitate the three-dimensional virtual planning of surgical guides employed for resection and reconstruction in patients with OSCC with mandibular invasion. MRI and (CB)CT images were collected retrospectively from 19 patients. MRI images were aligned with (CB)CT images employing a rigid registration approach (stage 1), a rigid registration approach using a mandibular mask (stage 2), and two non-rigid registration approaches (stage 3). Registration accuracy was quantified by the mean target registration error (mTRE), calculated over a set of landmarks annotated by two observers. Stage 2 achieved the best registration result, with an mTRE of 2.5±0.7mm, which was comparable to the inter- and intra-observer variabilities of landmark placement in MRI. Stage 2 was significantly better aligned compared to all approaches in stage 3. In conclusion, this study demonstrated that rigid registration with the use of a mask is an appropriate image registration method for aligning MRI and (CB)CT images of the mandibular region in patients with OSCC.

3D Motion Estimation of Left Ventricular Dynamics Using MRI and Track-to-Track Fusion.

Objective: This study investigates the estimation of three dimensional (3D) left ventricular (LV) motion using the fusion of different two dimensional (2D) cine magnetic resonance (CMR) sequences acquired during routine imaging sessions. Although standard clinical cine CMR data is inherently 2D, the actual underlying LV dynamics lies in 3D space and cannot be captured entirely using single 2D CMR image sequences. By utilizing the image information from various short-axis and long-axis image sequences, the proposed method intends to estimate the dynamic state vectors consisting of the position and velocity information of the myocardial borders in 3D space. Method: The proposed method comprises two main components: tracking myocardial points in 2D CMR sequences and fusion of multiple trajectories correspond to the tracked points. The tracking which yields the set of corresponding temporal points representing the myocardial points is performed using a diffeomorphic nonrigid image registration approach. The trajectories obtained from each cine CMR sequence is then fused with the corresponding trajectories from other CMR views using an unscented Kalman smoother (UKS) and a track-to-track fusion algorithm. Results: We evaluated the proposed method by comparing the results against CMR imaging with myocardial tagging. We report a quantitative performance analysis by projecting the state vector estimates we obtained onto 2D tagged CMR images acquired from the same subjects and comparing them against harmonic phase estimates. The proposed algorithm yielded a competitive performance with a mean root mean square error of 1.3±0.5 pixels (1.8±0.6 mm) evaluated over 118 image sequences acquired from 30 subjects. Conclusion: This study demonstrates that fusing the information from short and long-axis views of CMR improves the accuracy of cardiac tissue motion estimation. Clinical Impact: The proposed method demonstrates that the fusion of tissue tracking information from long and short-axis views improves the binary classification of the automated regional function assessment.

Joint image segmentation and registration based on a dynamic level set approach using truncated hierarchical B-splines

We present a novel approach for joint image segmentation and nonrigid registration using bidirectional composition based level set formulation. This efficient framework incorporates automatic structural analysis from image segmentation into the registration framework. This method has shown an improved performance as compared to carrying out segmentation and registration separately. Unlike previous approaches, the implicit level set function defining the segmentation contour and the spatial transformation function that maps the deformation for image registration are both defined using B-splines. This joint level set framework uses a variational form of an atlas-based segmentation together with large deformation based nonrigid registration. In addition, a bidirectional composition framework is introduced to incorporate a more symmetric update. The minimization of the variational form is accomplished by dynamic evaluations on a set of successively refined adaptive grids at multiple image resolutions. The improvement in the description of the segmentation result using higher order splines leads to a better accuracy of both the image segmentation and registration process. The performance of the proposed method is demonstrated on synthetic and medical images to show the improvement as compared to other registration methods.

Nonrigid Medical Image Registration Using an Information Theoretic Measure Based on Arimoto Entropy with Gradient Distributions.

This paper introduces a new nonrigid registration approach for medical images applying an information theoretic measure based on Arimoto entropy with gradient distributions. A normalized dissimilarity measure based on Arimoto entropy is presented, which is employed to measure the independence between two images. In addition, a regularization term is integrated into the cost function to obtain the smooth elastic deformation. To take the spatial information between voxels into account, the distance of gradient distributions is constructed. The goal of nonrigid alignment is to find the optimal solution of a cost function including a dissimilarity measure, a regularization term, and a distance term between the gradient distributions of two images to be registered, which would achieve a minimum value when two misaligned images are perfectly registered using limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimization scheme. To evaluate the test results of our presented algorithm in non-rigid medical image registration, experiments on simulated three-dimension (3D) brain magnetic resonance imaging (MR) images, real 3D thoracic computed tomography (CT) volumes and 3D cardiac CT volumes were carried out on elastix package. Comparison studies including mutual information (MI) and the approach without considering spatial information were conducted. These results demonstrate a slight improvement in accuracy of non-rigid registration.

Joint non-rigid image registration and reconstruction for quantitative atomic resolution scanning transmission electron microscopy

Aberration corrected scanning transmission electron microscopes (STEM) enable to determine local strain fields, composition and bonding states at atomic resolution. The precision to locate atomic columns is often obstructed by scan artifacts limiting the quantitative interpretation of STEM datasets. Here, a novel bias-corrected non-rigid registration approach is presented that compensates for fast and slow scan artifacts in STEM image series. The bias-correction is responsible for the correction of the slow scan artifacts and based on a explicit coupling of the deformations of the individual images in a series via a minimization of the average deformation. This allows to reduce fast scan noise in an image series and slow scan distortions simultaneously. The novel approach is tested on synthetic and experimental images and its implication on atomic resolution strain and elemental mapping is discussed.

EasyFont

Generating personal handwriting fonts with large amounts of characters is a boring and time-consuming task. For example, the official standard GB18030-2000 for commercial font products consists of 27,533 Chinese characters. Consistently and correctly writing out such huge amounts of characters is usually an impossible mission for ordinary people. To solve this problem, we propose a system, EasyFont , to automatically synthesize personal handwriting for all (e.g., Chinese) characters in the font library by learning style from a small number (as few as 1%) of carefully-selected samples written by an ordinary person. Major technical contributions of our system are twofold. First, we design an effective stroke extraction algorithm that constructs best-suited reference data from a trained font skeleton manifold and then establishes correspondence between target and reference characters via a non-rigid point set registration approach. Second, we develop a set of novel techniques to learn and recover users’ overall handwriting styles and detailed handwriting behaviors. Experiments including Turing tests with 97 participants demonstrate that the proposed system generates high-quality synthesis results, which are indistinguishable from original handwritings. Using our system, for the first time, the practical handwriting font library in a user’s personal style with arbitrarily large numbers of Chinese characters can be generated automatically. It can also be observed from our experiments that recently-popularized deep learning based end-to-end methods are not able to properly handle this task, which implies the necessity of expert knowledge and handcrafted rules for many applications.

A NEW REGISTRATION APPROACH FOR DYNAMIC ANALYSIS OF CALCIUM SIGNALS IN ORGANS.

Wing disc pouches of fruit flies are a powerful genetic model for studying physiological intercellular calcium (Ca 2+) signals for dynamic analysis of cell signaling in organ development and disease studies. A key to analyzing spatial-temporal patterns of Ca 2+ signal waves is to accurately align the pouches across image sequences. However, pouches in different image frames may exhibit extensive intensity oscillations due to Ca 2+ signaling dynamics, and commonly used multimodal non-rigid registration methods may fail to achieve satisfactory results. In this paper, we develop a new two-phase non-rigid registration approach to register pouches in image sequences. First, we conduct segmentation of the region of interest. (i.e., pouches) using a deep neural network model. Second, we use a B-spline based registration to obtain an optimal transformation and align pouches across the image sequences. Evaluated using both synthetic data and real pouch data, our method considerably outperforms the state-of-the-art non-rigid registration methods.

Sketch-based implicit blending

Implicit models can be combined by using composition operators; functions that determine the resulting shape. Recently, gradient-based composition operators have been used to express a variety of behaviours including smooth transitions, sharp edges, contact surfaces, bulging, or any combinations. The problem for designers is that building new operators is a complex task that requires specialized technical knowledge. In this work, we introduce an automatic method for deriving a gradient-based implicit operator from 2D drawings that prototype the intended visual behaviour. To solve this inverse problem, in which a shape defines a function, we introduce a general template for implicit operators. A user's sketch is interpreted as samples in the 3D operator's domain. We fit the template to the samples with a non-rigid registration approach. The process works at interactive rates and can accommodate successive refinements by the user. The final result can be applied to 3D surfaces as well as to 2D shapes. Our method is able to replicate the effect of any blending operator presented in the literature, as well as generating new ones such as non-commutative operators. We demonstrate the usability of our method with examples in font-design, collision-response modeling, implicit skinning, and complex shape design.

Gaussian Process Morphable Models.

Models of shape variations have become a central component for the automated analysis of images. An important class of shape models are point distribution models (PDMs). These models represent a class of shapes as a normal distribution of point variations, whose parameters are estimated from example shapes. Principal component analysis (PCA) is applied to obtain a low-dimensional representation of the shape variation in terms of the leading principal components. In this paper, we propose a generalization of PDMs, which we refer to as Gaussian Process Morphable Models (GPMMs). We model the shape variations with a Gaussian process, which we represent using the leading components of its Karhunen-Loève expansion. To compute the expansion, we make use of an approximation scheme based on the Nyström method. The resulting model can be seen as a continuous analog of a standard PDM. However, while for PDMs the shape variation is restricted to the linear span of the example data, with GPMMs we can define the shape variation using any Gaussian process. For example, we can build shape models that correspond to classical spline models and thus do not require any example data. Furthermore, Gaussian processes make it possible to combine different models. For example, a PDM can be extended with a spline model, to obtain a model that incorporates learned shape characteristics but is flexible enough to explain shapes that cannot be represented by the PDM. We introduce a simple algorithm for fitting a GPMM to a surface or image. This results in a non-rigid registration approach whose regularization properties are defined by a GPMM. We show how we can obtain different registration schemes, including methods for multi-scale or hybrid registration, by constructing an appropriate GPMM. As our approach strictly separates modeling from the fitting process, this is all achieved without changes to the fitting algorithm. To demonstrate the applicability and versatility of GPMMs, we perform a set of experiments in typical usage scenarios in medical image analysis and computer vision: The model-based segmentation of 3D forearm images and the building of a statistical model of the face. To complement the paper, we have made all our methods available as open source.

Projective biomechanical depth matching for soft tissue registration in laparoscopic surgery.

A key component of computer- assisted surgery systems is the accurate and robust registration of preoperative planning data with intraoperative sensor data. In laparoscopic surgery, this image-based registration remains challenging due to soft tissue deformations. This paper presents a novel approach for biomechanical soft tissue registration of preoperative CT data with stereo endoscopic image data. The proposed method consists of two registrations steps. First, we use a 3D surface mosaic from partial surfaces reconstructed from stereo endoscopic images to initially align the biomechanical model with the intraoperative position and shape of the organ. After this initialization, the biomechanical model is projected onto newly captured surfaces, resulting in displacement boundary conditions, which in turn are used to update the biomechanical model. The method is evaluated in silico, using a human liver model, and in vivo, using porcine data. The quantitative in silico data shows a stable behaviour of the biomechanical model and root-mean-square deviation of volume vertices of under 3mm with adjusted biomechanical parameters. This work contributes a fully automatic featureless non-rigid registration approach. The results of the in silico and in vivo experiments suggest that our method is able to handle dynamic deformations during surgery. Additional experiments, especially regarding human tissue behaviour, are an important next step towards clinical applications.

Learning Non-rigid Deformations for Robust, Constrained Point-based Registration in Image-Guided MR-TRUS Prostate Intervention.

Accurate and robust non-rigid registration of pre-procedure magnetic resonance (MR) imaging to intra-procedure trans-rectal ultrasound (TRUS) is critical for image-guided biopsies of prostate cancer. Prostate cancer is one of the most prevalent forms of cancer and the second leading cause of cancer-related death in men in the United States. TRUS-guided biopsy is the current clinical standard for prostate cancer diagnosis and assessment. State-of-the-art, clinical MR-TRUS image fusion relies upon semi-automated segmentations of the prostate in both the MR and the TRUS images to perform non-rigid surface-based registration of the gland. Segmentation of the prostate in TRUS imaging is itself a challenging task and prone to high variability. These segmentation errors can lead to poor registration and subsequently poor localization of biopsy targets, which may result in false-negative cancer detection. In this paper, we present a non-rigid surface registration approach to MR-TRUS fusion based on a statistical deformation model (SDM) of intra-procedural deformations derived from clinical training data. Synthetic validation experiments quantifying registration volume of interest overlaps of the PI-RADS parcellation standard and tests using clinical landmark data demonstrate that our use of an SDM for registration, with median target registration error of 2.98 mm, is significantly more accurate than the current clinical method. Furthermore, we show that the low-dimensional SDM registration results are robust to segmentation errors that are not uncommon in clinical TRUS data.

Detection of Conversion from Mild Cognitive Impairment to Alzheimer's Disease Using Longitudinal Brain MRI.

Mild Cognitive Impairment (MCI) is an intermediate stage between healthy and Alzheimer's disease (AD). To enable early intervention it is important to identify the MCI subjects that will convert to AD in an early stage. In this paper, we provide a new method to distinguish between MCI patients that either convert to Alzheimer's Disease (MCIc) or remain stable (MCIs), using only longitudinal T1-weighted MRI. Currently, most longitudinal studies focus on volumetric comparison of a few anatomical structures, thereby ignoring more detailed development inside and outside those structures. In this study we propose to exploit the anatomical development within the entire brain, as found by a non-rigid registration approach. Specifically, this anatomical development is represented by the Stationary Velocity Field (SVF) from registration between the baseline and follow-up images. To make the SVFs comparable among subjects, we use the parallel transport method to align them in a common space. The normalized SVF together with derived features are then used to distinguish between MCIc and MCIs subjects. This novel feature space is reduced using a Kernel Principal Component Analysis method, and a linear support vector machine is used as a classifier. Extensive comparative experiments are performed to inspect the influence of several aspects of our method on classification performance, specifically the feature choice, the smoothing parameter in the registration and the use of dimensionality reduction. The optimal result from a 10-fold cross-validation using 36 month follow-up data shows competitive results: accuracy 92%, sensitivity 95%, specificity 90%, and AUC 94%. Based on the same dataset, the proposed approach outperforms two alternative ones that either depends on the baseline image only, or uses longitudinal information from larger brain areas. Good results were also obtained when scans at 6, 12, or 24 months were used for training the classifier. Besides the classification power, the proposed method can quantitatively compare brain regions that have a significant difference in development between the MCIc and MCIs groups.

Diffeomorphic Multi-Frame Non-Rigid Registration of Cell Nuclei in 2D and 3D Live Cell Images.

To gain a better understanding of cellular and molecular processes, it is important to quantitatively analyze the motion of subcellular particles in live cell microscopy image sequences. Since, generally, the subcellular particles move and cell nuclei move as well as deform, it is important to decouple the movement of particles from that of the cell nuclei using non-rigid registration methods. We have developed a diffeomorphic multi-frame approach for non-rigid registration of cell nuclei in 2D and 3D live cell fluorescence microscopy images. Our non-rigid registration approach is based on local optic flow estimation, exploits information from multiple consecutive image frames, and determines diffeomorphic transformations in the log-domain, which allows efficient computation of the inverse transformations. To register single images of an image sequence to a reference image, we use a temporally weighted mean image, which is constructed based on inverse transformations and multiple consecutive frames. Using multiple consecutive frames improves the registration accuracy compared to pairwise registration, and using a temporally weighted mean image significantly reduces the computation time compared with previous work. In addition, we use a flow boundary preserving method for regularization of computed deformation vector fields, which prevents from over-smoothing compared to standard Gaussian filtering. Our approach has been successfully applied to 2D and 3D synthetic as well as real live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise, multi-frame, and temporal groupwise registration has been carried out.

Dynamic elasticity model for inter-subject non-rigid registration of 3D MRI brain scans

Abstract The purpose of this paper is to present a new approach for inter-subject non-rigid registration of 3D MR brain images which could assist in automatic labeling of brain structures. A physical dynamic elasticity model (DEM) is developed which tends to represent the complex non-linear deformations as elastic waves. The transformation is governed by elastodynamics wave equation. The registration process ensues in a hierarchical fashion, thus reducing the risk of obtaining a local optimal transformation. Along with the correction of local misalignments, it also removes global shape differences without any prior initialization. The proposed scheme was compared against high ranking registration methods including: DROP, SyN, ART and DRAMMS. The results were quantitatively analyzed by computing and testing the statistical significance of the volume overlap measures and Hausdorff distance for segmented structures with DROP, SyN, ART, DRAMMS and DEM registration methods. Experimental results demonstrate that the proposed DEM registration method leads to very promising results when applied to the problem of inter-subject registration and that favorably compares against classical registration approaches. Since DEM registration method is able to reduce registration errors significantly, hence it could be used to automatically label the anatomical structures for clinical studies.

A novel learning-based dissimilarity metric for rigid and non-rigid medical image registration by using Bhattacharyya Distances

Image registration is one of the major technologies of the medical image processing and analysis. According to the nature of the transformation, image registration can be categorized into two main classes: Rigid Registration and Non-rigid Registration. In this paper, a novel learning-based dissimilarity metric and its approximation are proposed and respectively adopted to the rigid and non-rigid registration of medical images. This novel metric utilizes Bhattacharyya Distances to measure the dissimilarity of the testing image pairs by incorporating the expected intensity distributions which is learnt from the registered training image pairs. In order to test the robustness and accuracy of the proposed metric in the multi-modal rigid registration, one thousand eight hundred randomized CT–T1 registrations were performed and evaluated by the Retrospective Image Registration Evaluation (RIRE) project. For the non-rigid registration, the approximation of the proposed dissimilarity metric is adopted with the Markov Random Field (MRF) modeled non-rigid registration approach. Non-rigid registration experiments of 2D synthetic images with artificial deformation fields and 3D images from the Simulated Brain Database were performed and evaluated respectively in this paper. The experimental results of both rigid and non-rigid medical image registrations demonstrated that the proposed dissimilarity metric outperforms the state-of-the-art approach (Mutual Information) and the close related approach (Kullback–Leibler Distances).

A multicomponent approach to nonrigid registration of diffusion tensor images

Diffusion tensor imaging has shown promise in the early detection and diagnosis of a host of disorders and neurologic conditions. In this paper, we propose a nonrigid registration approach for diffusion tensor images using a multicomponent information-theoretic measure. Explicit orientation optimization is enabled by incorporating tensor reorientation, which is necessary for wrapping diffusion tensor images. Experimental results on diffusion tensor images indicate the feasibility of the proposed approach and a much better performance compared to the affine registration method based on mutual information in terms of registration accuracy in the presence of geometric distortion.

Feature-based multi-resolution registration of immunostained serial sections.

The form and exact function of the blood vessel network in some human organs, like spleen and bone marrow, are still open research questions in medicine. In this paper, we propose a method to register the immunohistological stainings of serial sections of spleen and bone marrow specimens to enable the visualization and visual inspection of blood vessels. As these vary much in caliber, from mesoscopic (millimeter-range) to microscopic (few micrometers, comparable to a single erythrocyte), we need to utilize a multi-resolution approach. Our method is fully automatic; it is based on feature detection and sparse matching. We utilize a rigid alignment and then a non-rigid deformation, iteratively dealing with increasingly smaller features. Our tool pipeline can already deal with series of complete scans at extremely high resolution, up to 620 megapixels. The improvement presented increases the range of represented details up to smallest capillaries. This paper provides details on the multi-resolution non-rigid registration approach we use. Our application is novel in the way the alignment and subsequent deformations are computed (using features, i.e. "sparse"). The deformations are based on all images in the stack ("global"). We also present volume renderings and a 3D reconstruction of the vascular network in human spleen and bone marrow on a level not possible before. Our registration makes easy tracking of even smallest blood vessels possible, thus granting experts a better comprehension. Aquantitative evaluation of our method and related state of the art approaches with seven different quality measures shows the efficiency of our method. We also provide z-profiles and enlarged volume renderings from three different registrations for visual inspection.