Multimodal Registration Research Articles

Joint groupwise registration and ADC estimation in the liver using a B-value weighted metric

PurposeThe purpose of this work is to develop a groupwise elastic multimodal registration algorithm for robust ADC estimation in the liver on multiple breath hold diffusion weighted images. MethodsWe introduce a joint formulation to simultaneously solve both the registration and the estimation problems. In order to avoid non-reliable transformations and undesirable noise amplification, we have included appropriate smoothness constraints for both problems. Our metric incorporates the ADC estimation residuals, which are inversely weighted according to the signal content in each diffusion weighted image. ResultsResults show that the joint formulation provides a statistically significant improvement in the accuracy of the ADC estimates. Reproducibility has also been measured on real data in terms of the distribution of ADC differences obtained from different b-values subsets. ConclusionsThe proposed algorithm is able to effectively deal with both the presence of motion and the geometric distortions, increasing accuracy and reproducibility in diffusion parameters estimation.

Improving lateral resolution and image quality of optical coherence tomography by the multi-frame superresolution technique for 3D tissue imaging.

The multi-frame superresolution technique is introduced to significantly improve the lateral resolution and image quality of spectral domain optical coherence tomography (SD-OCT). Using several sets of low resolution C-scan 3D images with lateral sub-spot-spacing shifts on different sets, the multi-frame superresolution processing of these sets at each depth layer reconstructs a higher resolution and quality lateral image. Layer by layer processing yields an overall high lateral resolution and quality 3D image. In theory, the superresolution processing including deconvolution can solve the diffraction limit, lateral scan density and background noise problems together. In experiment, the improved lateral resolution by ~3 times reaching 7.81 µm and 2.19 µm using sample arm optics of 0.015 and 0.05 numerical aperture respectively as well as doubling the image quality has been confirmed by imaging a known resolution test target. Improved lateral resolution on in vitro skin C-scan images has been demonstrated. For in vivo 3D SD-OCT imaging of human skin, fingerprint and retina layer, we used the multi-modal volume registration method to effectively estimate the lateral image shifts among different C-scans due to random minor unintended live body motion. Further processing of these images generated high lateral resolution 3D images as well as high quality B-scan images of these in vivo tissues.

Embedded registration of visible and infrared images in real time for noninvasive skin cancer screening

We present an embedded system architecture that implements real-time multimodal registration to enable dual-camera spatio-temporal feature extraction in a skin cancer screening application. We test the system on a combination of visible and long-wave infrared image sequences, but it can be easily extended to setups operating in different sections of the spectrum. Image registration is performed by matching common features between each frame of a visible image to each frame of an infrared image sequence to estimate a projective transformation between them. The parameters of this transformation are estimated recursively on line with the video, thus enabling image registration in real time. The algorithm is implemented using a combination of embedded software and dedicated hardware units on a heterogeneous reconfigurable system-on-a-chip. The hardware performs feature detection and extraction, while the software estimates the transformation parameters and maps each visible video frame onto the infrared image coordinates. Implemented on an FPGA, our prototype runs at 540 frames per second with a 135 MHz clock, consumes 1.8 W and utilizes 29% and 54% of the logic and multiplier resources of the chip, respectively.

A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time

This work presents a data-driven method to simulate, in real-time, the biomechanical behavior of the breast tissues in some image-guided interventions such as biopsies or radiotherapy dose delivery as well as to speed up multimodal registration algorithms. Ten real breasts were used for this work. Their deformation due to the displacement of two compression plates was simulated off-line using the finite element (FE) method. Three machine learning models were trained with the data from those simulations. Then, they were used to predict in real-time the deformation of the breast tissues during the compression. The models were a decision tree and two tree-based ensemble methods (extremely randomized trees and random forest). Two different experimental setups were designed to validate and study the performance of these models under different conditions. The mean 3D Euclidean distance between nodes predicted by the models and those extracted from the FE simulations was calculated to assess the performance of the models in the validation set. The experiments proved that extremely randomized trees performed better than the other two models. The mean error committed by the three models in the prediction of the nodal displacements was under 2 mm, a threshold usually set for clinical applications. The time needed for breast compression prediction is sufficiently short to allow its use in real-time (<0.2 s).

Multimodal image registration based on binary gradient angle descriptor.

Multimodal image registration plays an important role in image-guided interventions/therapy and atlas building, and it is still a challenging task due to the complex intensity variations in different modalities. The paper addresses the problem and proposes a simple, compact, fast and generally applicable modality-independent binary gradient angle descriptor (BGA) based on the rationale of gradient orientation alignment. The BGA can be easily calculated at each voxel by coding the quadrant in which a local gradient vector falls, and it has an extremely low computational complexity, requiring only three convolutions, two multiplication operations and two comparison operations. Meanwhile, the binarized encoding of the gradient orientation makes the BGA more resistant to image degradations compared with conventional gradient orientation methods. The BGA can extract similar feature descriptors for different modalities and enable the use of simple similarity measures, which makes it applicable within a wide range of optimization frameworks. The results for pairwise multimodal and monomodal registrations between various images (T1, T2, PD, T1c, Flair) consistently show that the BGA significantly outperforms localized mutual information. The experimental results also confirm that the BGA can be a reliable alternative to the sum of absolute difference in monomodal image registration. The BGA can also achieve an accuracy of [Formula: see text], similar to that of the SSC, for the deformable registration of inhale and exhale CT scans. Specifically, for the highly challenging deformable registration of preoperative MRI and 3D intraoperative ultrasound images, the BGA achieves a similar registration accuracy of [Formula: see text] compared with state-of-the-art approaches, with a computation time of 18.3s per case. The BGA improves the registration performance in terms of both accuracy and time efficiency. With further acceleration, the framework has the potential for application in time-sensitive clinical environments, such as for preoperative MRI and intraoperative US image registration for image-guided intervention.

Towards a patient-specific hepatic arterial modeling for microspheres distribution optimization in SIRT protocol.

Selective internal radiation therapy (SIRT) using Yttrium-90 loaded glass microspheres injected in the hepatic artery is an emerging, minimally invasive therapy of liver cancer. A personalized intervention can lead to high concentration dose in the tumor, while sparing the surrounding parenchyma. We propose a computational model for patient-specific simulation of entire hepatic arterial tree, based on liver, tumors, and arteries segmentation on patient's tomography. Segmentation of hepatic arteries down to a diameter of 0.5 mm is semi-automatically performed on 3D cone-beam CT angiography. The liver and tumors are extracted from CT-scan at portal phase by an active surface method. Once the images are registered through an automatic multimodal registration, extracted data are used to initialize a numerical model simulating liver vascular network. The model creates successive bifurcations from given principal vessels, observing Poiseuille's and matter conservation laws. Simulations provide a coherent reconstruction of global hepatic arterial tree until vessel diameter of 0.05 mm. Microspheres distribution under simple hypotheses is also quantified, depending on injection site. The patient-specific character of this model may allow a personalized numerical approximation of microspheres final distribution, opening the way to clinical optimization of catheter placement for tumor targeting.

Multimodality registration of two-dimensional echocardiography and cardiac CT for mitral valve diagnosis and surgical planning.

A registration method to fuse two-dimensional (2-D) echocardiography images with cardiac computed tomography (CT) volume is presented. The method consists of two major procedures: temporal and spatial registrations. In temporal registration, the echocardiography frames at similar cardiac phases as the CT volume were interpolated based on electrocardiogram signal information, and the noise of the echocardiography image was reduced using the speckle reducing anisotropic diffusion technique. For spatial registration, an intensity-based normalized mutual information method was applied with a pattern search optimization algorithm to produce an interpolated cardiac CT image. The proposed registration framework does not require optical tracking information. Dice coefficient and Hausdorff distance for the left atrium assessments were [Formula: see text] and [Formula: see text], respectively; for left ventricle, they were [Formula: see text] and [Formula: see text], respectively. There was no significant difference in the mitral valve annulus diameter measurement between the manually and automatically registered CT images. The transformation parameters showed small deviations ([Formula: see text] deviation in translation and [Formula: see text] for rotation) between manual and automatic registrations. The proposed method aids the physician in diagnosing mitral valve disease as well as provides surgical guidance during the treatment procedure.

Functional density and edge maps: Characterizing functional architecture in individuals and improving cross-subject registration

Population-level inferences and individual-level analyses are two important aspects in functional magnetic resonance imaging (fMRI) studies. Extracting reliable and informative features from fMRI data that capture biologically meaningful inter-subject variation is critical for aligning and comparing functional networks across subjects, and connecting the properties of functional brain organization with variations in behavior, cognition and genetics. In this study, we derive two new measures, which we term functional density map and edge map, and demonstrate their usefulness in characterizing the function of individual brains. Specifically, using data from the Human Connectome Project (HCP), we show that (1) both functional maps capture intrinsic properties of the functional connectivity pattern in individuals while exhibiting large variation across subjects; (2) functional maps derived from either resting-state or task-evoked fMRI can be used to accurately identify subjects from a population; and (3) cross-subject alignment using these functional maps considerably reduces functional variation and improves functional correspondence across subjects over state-of-the-art multimodal registration algorithms. Our results suggest that the proposed functional density and edge maps are promising features in characterizing the functional architecture in individuals and provide an alternative way to explore the functional variation across subjects.

An embedded system for image segmentation and multimodal registration in noninvasive skin cancer screening.

We present a heterogeneous architecture for image registration and multimodal segmentation on an embedded system for noninvasive skin cancer screening. The architecture combines Otsu thresholding and the random walker algorithm to perform image segmentation, and features a hardware implementation of the Harris corner detection algorithm to perform region-of-interest detection and image registration. Running on a Xilinx XC7Z020 reconfigurable system-on-a-chip, our prototype computes the initial segmentation of a 400×400-pixel region of interest in the visible spectrum in 12.1 seconds, and registers infrared images against this region at 540 frames per second, while consuming 1.9W.

Multimodal registration of optical microscopic and infrared spectroscopic images from different tissue sections: An application to colon cancer

Fourier transform infrared (FTIR) spectroscopic images provide rich information of the biochemical tissue composition that can be analyzed together with other microscopy modalities to perform objective pathological diagnoses. Hematoxylin and Eosin (H&E) stained images are the reference images that pathologists use to make a final diagnosis of most diseases, such as cancer. Therefore, H&E images may be the most interesting imaging modality to be fused with FTIR images. Unfortunately, H&E stain introduces severe confounding artifacts in the FTIR spectra. Thus, in repeatable studies different slices of tissue must be used to acquire images for each imaging modality, which must be aligned so that the different regions of tissue spatially match. The main objective of this manuscript is to establish a complete pipeline where the two types of images (H&E and FTIR) from different tissue sections are aligned or registered. The proposed automatic framework starts by obtaining grayscale images from both FTIR raw data and H&E images where analogous anatomical structures are easily distinguishable. In the first alignment step, a feature-based registration produces a fast coarse rigid alignment by using the Scale Invariant Feature Transform (SIFT) algorithm to automatically find and match relevant keypoints in both grayscale images. Due to the spatial variability between samples, different combinations of SIFT parameters are explored and the best combination is selected through the maximization of a similarity measure between the aligned images. In the second alignment step, an intensity-based registration refines the initial alignment and compensates for the local spatial differences between the tissue sections by iteratively estimating a non-rigid transformation. This methodology was used to register 47 colon samples from three different pathological groups (16 normal, 16 intermediate and 15 tumoral) with good overall results, which were quantitatively evaluated for both registration steps. In the first rigid alignment step, the global median of difference in positioning compared to a manual registration was under 1 pixel. In the second registration step, the global median gain in mutual information between the registered images was 0.125 bits. In contrast to existing approaches, the proposed method does not need a prior segmentation step that may introduce errors and reduce the spatial information content, which is crucial when different sections of tissue are used. It can improve the accuracy to combine the spatial information extracted from both the traditional H&E stained images and the emerging FTIR spectroscopy.

Guiding multimodal registration with learned optimization updates.

In this paper, we address the multimodal registration problem from a novel perspective, aiming to predict the transformation aligning images directly from their visual appearance. We formulate the prediction as a supervised regression task, with joint image descriptors as input and the output are the parameters of the transformation that guide the moving image towards alignment. We model the joint local appearance with context aware descriptors that capture both local and global cues simultaneously in the two modalities, while the regression function is based on the gradient boosted trees method capable of handling the very large contextual feature space. The good properties of our predictions allow us to couple them with a simple gradient-based optimization for the final registration. Our approach can be applied to any transformation parametrization as well as a broad range of modality pairs. Our method learns the relationship between the intensity distributions of a pair of modalities by using prior knowledge in the form of a small training set of aligned image pairs (in the order of 1-5 in our experiments). We demonstrate the flexibility and generality of our method by evaluating its performance on a variety of multimodal imaging pairs obtained from two publicly available datasets, RIRE (brain MR, CT and PET) and IXI (brain MR). We also show results for the very challenging deformable registration of Intravascular Ultrasound and Histology images. In these experiments, our approach has a larger capture range when compared to other state-of-the-art methods, while improving registration accuracy in complex cases.

Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity

Automatic registration of multimodal remote sensing data (e.g., optical, LiDAR, SAR) is a challenging task due to the significant non-linear radiometric differences between these data. To address this problem, this paper proposes a novel feature descriptor named the Histogram of Orientated Phase Congruency (HOPC), which is based on the structural properties of images. Furthermore, a similarity metric named HOPCncc is defined, which uses the normalized correlation coefficient (NCC) of the HOPC descriptors for multimodal registration. In the definition of the proposed similarity metric, we first extend the phase congruency model to generate its orientation representation, and use the extended model to build HOPCncc. Then a fast template matching scheme for this metric is designed to detect the control points between images. The proposed HOPCncc aims to capture the structural similarity between images, and has been tested with a variety of optical, LiDAR, SAR and map data. The results show that HOPCncc is robust against complex non-linear radiometric differences and outperforms the state-of-the-art similarities metrics (i.e., NCC and mutual information) in matching performance. Moreover, a robust registration method is also proposed in this paper based on HOPCncc, which is evaluated using six pairs of multimodal remote sensing images. The experimental results demonstrate the effectiveness of the proposed method for multimodal image registration.

THE IMPACT OF MATCHING FUNCTIONAL ON ATROPHY MEASUREMENT FROM GEODESIC SHOOTING IN DIFFEOMORPHISMS.

Longitudinal registration has been used to map brain atrophy and tissue loss patterns over time, in both healthy and demented subjects. However, we have not seen a thorough application of the geodesic shooting in diffeomorphisms framework for this task. The registration model is complex and several choices must be made that may significantly impact the quality of results. One of these decisions is which image matching functional should drive the registration. We investigate four matching functionals for atrophy quantification using geodesic shooting in diffeomorphisms. We check if the choice of matching functional has an impact on the correlation of atrophy scores with clinical variables. We also check the impact of matching functional choice on estimates of the N80 sample size for hypothetical clinical trials that test for slowing of brain atrophy. We find that the mutual information function, which has primarily been used for linear and multi-modal registration, achieves comparable correlation with clinical variables to other matching functionals while yielding better sample size estimates.

Atlas construction of cardiac fiber architecture using a multimodal registration approach

Cardiac fibers have a quite complex and interesting spatial structure and the myocardial fiber orientation provides a great effort on both cardiac research studies and further clinical applications. Diffusion magnetic resolution imaging is a unique imaging methodology for non-invasively investigating the myofiber architecture, e.g., diffusion tensor imaging. Accurate quantitative mapping of the cardiac fiber is still challenging. In this study, a statistical diffusion atlas for myofiber is constructed from seven subjects that were acquired by diffusion weighted imaging scans. The atlas reconstruction work is based on a multimodal statistical framework which involves most advances in image analysis methods: level-sets for image segmentation, multi-modal registration and symmetric diffeomorphic nonlinear registration, weighted diffusion tensor averaging and accurate quantitative analysis. Experimental results demonstrate that the myofiber helix angle variation is markedly correlated with the transmural myocardial wall depth and its values vary from 45° in the endocardium to −40° in the epicardium, whilst it is 26° in the endocardium and −29° in the epicardium from traditional averaged atlas. The proposed atlas construction approach can effectively retain the water diffusion anisotropic information, especially on the myocardial tissue edges, which is a common known limit of current techniques. Moreover, we find that fractional anisotropy (FA) for the proposed atlas (0.55 ± 0.10) was higher than that of averaged one (0.42 ± 0.05). Substantial differences were found between FA values (P=0.02). Mean diffusivity (MD) from proposed atlas (0.76 ± 0.05 × 10−3 mm2/s) was significantly lower than that of average atlas (0.80 ± 0.12 × 10−3 mm2/s), P=0.03. This study illustrates a quantitative description for the myofiber structure and may provide a powerful tool to assess the micro-structure of myocardial tissue.

Fast and robust multimodal image registration using a local derivative pattern.

Deformable multimodal image registration, which can benefit radiotherapy and image guided surgery by providing complementary information, remains a challenging task in the medical image analysis field due to the difficulty of defining a proper similarity measure. This article presents a novel, robust and fast binary descriptor, the discriminative local derivative pattern (dLDP), which is able to encode images of different modalities into similar image representations. dLDP calculates a binary string for each voxel according to the pattern of intensity derivatives in its neighborhood. The descriptor similarity is evaluated using the Hamming distance, which can be efficiently computed, instead of conventional L1 or L2 norms. For the first time, we validated the effectiveness and feasibility of the local derivative pattern for multimodal deformable image registration with several multi-modal registration applications. dLDP was compared with three state-of-the-art methods in artificial image and clinical settings. In the experiments of deformable registration between different magnetic resonance imaging (MRI) modalities from BrainWeb, between computed tomography and MRI images from patient data, and between MRI and ultrasound images from BITE database, we show our method outperforms localized mutual information and entropy images in terms of both accuracy and time efficiency. We have further validated dLDP for the deformable registration of preoperative MRI and three-dimensional intraoperative ultrasound images. Our results indicate that dLDP reduces the average mean target registration error from 4.12mm to 2.30mm. This accuracy is statistically equivalent to the accuracy of the state-of-the-art methods in the study; however, in terms of computational complexity, our method significantly outperforms other methods and is even comparable to the sum of the absolute difference. The results reveal that dLDP can achieve superior performance regarding both accuracy and time efficiency in general multimodal image registration. In addition, dLDP also indicates the potential for clinical ultrasound guided intervention.

Geometric Graph Matching Using Monte Carlo Tree Search

We present an efficient matching method for generalized geometric graphs. Such graphs consist of vertices in space connected by curves and can represent many real world structures such as road networks in remote sensing, or vessel networks in medical imaging. Graph matching can be used for very fast and possibly multimodal registration of images of these structures. We formulate the matching problem as asingle player game solved using Monte Carlo Tree Search, which automatically balances exploring new possible matches and extending existing matches. Our method can handle partial matches, topological differences, geometrical distortion, does not use appearance information and does not require an initial alignment. Moreover, our method is very efficient-it can match graphs with thousands of nodes, which is an order of magnitude better than the best competing method, and the matching only takes a few seconds.

Efficient Variational Approach to Multimodal Registration of Anatomical and Functional Intra-Patient Tumorous Brain Data.

This paper addresses the functional localization of intra-patient images of the brain. Functional images of the brain (fMRI and PET) provide information about brain function and metabolism whereas anatomical images (MRI and CT) supply the localization of structures with high spatial resolution. The goal is to find the geometric correspondence between functional and anatomical images in order to complement and fuse the information provided by each imaging modality. The proposed approach is based on a variational formulation of the image registration problem in the frequency domain. It has been implemented as a C/C[Formula: see text] library which is invoked from a GUI. This interface is routinely used in the clinical setting by physicians for research purposes (Inscanner, Alicante, Spain), and may be used as well for diagnosis and surgical planning. The registration of anatomic and functional intra-patient images of the brain makes it possible to obtain a geometric correspondence which allows for the localization of the functional processes that occur in the brain. Through 18 clinical experiments, it has been demonstrated how the proposed approach outperforms popular state-of-the-art registration methods in terms of efficiency, information theory-based measures (such as mutual information) and actual registration error (distance in space of corresponding landmarks).

Multimodal registration of SD-OCT volumes and fundus photographs using histograms of oriented gradients.

With availability of different retinal imaging modalities such as fundus photography and spectral domain optical coherence tomography (SD-OCT), having a robust and accurate registration scheme to enable utilization of this complementary information is beneficial. The few existing fundus-OCT registration approaches contain a vessel segmentation step, as the retinal blood vessels are the most dominant structures that are in common between the pair of images. However, errors in the vessel segmentation from either modality may cause corresponding errors in the registration. In this paper, we propose a feature-based registration method for registering fundus photographs and SD-OCT projection images that benefits from vasculature structural information without requiring blood vessel segmentation. In particular, after a preprocessing step, a set of control points (CPs) are identified by looking for the corners in the images. Next, each CP is represented by a feature vector which encodes the local structural information via computing the histograms of oriented gradients (HOG) from the neighborhood of each CP. The best matching CPs are identified by calculating the distance of their corresponding feature vectors. After removing the incorrect matches the best affine transform that registers fundus photographs to SD-OCT projection images is computed using the random sample consensus (RANSAC) method. The proposed method was tested on 44 pairs of fundus and SD-OCT projection images of glaucoma patients and the result showed that the proposed method successfully registers the multimodal images and produced a registration error of 25.34 ± 12.34 μm (0.84 ± 0.41 pixels).

Multimodal registration of remotely sensed images based on Jeffrey’s divergence

Entropy-based measures (e.g., mutual information, also known as Kullback-Leiber divergence), which quantify the similarity between two signals, are widely used as similarity measures for image registration. Although they are proven superior to many classical statistical measures, entropy-based measures, such as mutual information, may fail to yield the optimum registration if the multimodal image pair has insufficient scene overlap region. To overcome this challenge, we proposed using the symmetric form of Kullback-Leiber divergence, namely Jeffrey’s divergence, as the similarity measure in practical multimodal image registration tasks. Mathematical analysis was performed to investigate the causes accounting for the limitation of mutual information when dealing with insufficient scene overlap image pairs. Experimental registrations of SPOT image, Landsat TM image, ALOS PalSAR image, and DEM data were carried out to compare the performance of Jeffrey’s divergence and mutual information. Results indicate that Jeffrey’s divergence is capable of providing larger feasible search space, which is favorable for exploring optimum transformation parameters in a larger range. This superiority of Jeffrey’s divergence was further confirmed by a series of paradigms. Thus, the proposed model is more applicable for registering image pairs that are greatly misaligned or have an insufficient scene overlap region.

Real-Time Ultrasound Segmentation, Analysis and Visualisation of Deep Cervical Muscle Structure.

Despite widespread availability of ultrasound and a need for personalised muscle diagnosis (neck/back pain-injury, work related disorder, myopathies, neuropathies), robust, online segmentation of muscles within complex groups remains unsolved by existing methods. For example, Cervical Dystonia (CD) is a prevalent neurological condition causing painful spasticity in one or multiple muscles in the cervical muscle system. Clinicians currently have no method for targeting/monitoring treatment of deep muscles. Automated methods of muscle segmentation would enable clinicians to study, target, and monitor the deep cervical muscles via ultrasound. We have developed a method for segmenting five bilateral cervical muscles and the spine via ultrasound alone, in real-time. Magnetic Resonance Imaging (MRI) and ultrasound data were collected from 22 participants (age: 29.0±6.6, male: 12). To acquire ultrasound muscle segment labels, a novel multimodal registration method was developed, involving MRI image annotation, and shape registration to MRI-matched ultrasound images, via approximation of the tissue deformation. We then applied polynomial regression to transform our annotations and textures into a mean space, before using shape statistics to generate a texture-to-shape dictionary. For segmentation, test images were compared to dictionary textures giving an initial segmentation, and then we used a customized Active Shape Model to refine the fit. Using ultrasound alone, on unseen participants, our technique currently segments a single image in [Formula: see text] to over 86% accuracy (Jaccard index). We propose this approach is applicable generally to segment, extrapolate and visualise deep muscle structure, and analyse statistical features online.