Multimodal Registration Research Articles

Pose estimation of a markerless fiber bundle for endoscopic optical biopsy.

Purpose: We present a markerless vision-based method for on-the-fly three-dimensional (3D) pose estimation of a fiberscope instrument to target pathologic areas in the endoscopic view during exploration. Approach: A 2.5-mm-diameter fiberscope is inserted through the endoscope's operating channel and connected to an additional camera to perform complementary observation of a targeted area such as a multimodal magnifier. The 3D pose of the fiberscope is estimated frame-by-frame by maximizing the similarity between its silhouette (automatically detected in the endoscopic view using a deep learning neural network) and a cylindrical shape bound to a kinematic model reduced to three degrees-of-freedom. An alignment of the cylinder axis, based on Plücker coordinates from the straight edges detected in the image, makes convergence faster and more reliable. Results: The performance on simulations has been validated with a virtual trajectory mimicking endoscopic exploration and on real images of a chessboard pattern acquired with different endoscopic configurations. The experiments demonstrated a good accuracy and robustness of the proposed algorithm with errors of in distance position and in axis orientation for the 3D pose estimation, which reveals its superiority over previous approaches. This allows multimodal image registration with sufficient accuracy of . Conclusion: Our pose estimation pipeline was executed on simulations and patterns; the results demonstrate the robustness of our method and the potential of fiber-optical instrument image-based tracking for pose estimation and multimodal registration. It can be fully implemented in software and therefore easily integrated into a routine clinical environment.

An Anatomical Thermal 3D Model in Preclinical Research: Combining CT and Thermal Images.

Even though animal trials are a controversial topic, they provide knowledge about diseases and the course of infections in a medical context. To refine the detection of abnormalities that can cause pain and stress to the animal as early as possible, new processes must be developed. Due to its noninvasive nature, thermal imaging is increasingly used for severity assessment in animal-based research. Within a multimodal approach, thermal images combined with anatomical information could be used to simulate the inner temperature profile, thereby allowing the detection of deep-seated infections. This paper presents the generation of anatomical thermal 3D models, forming the underlying multimodal model in this simulation. These models combine anatomical 3D information based on computed tomography (CT) data with a registered thermal shell measured with infrared thermography. The process of generating these models consists of data acquisition (both thermal images and CT), camera calibration, image processing methods, and structure from motion (SfM), among others. Anatomical thermal 3D models were successfully generated using three anesthetized mice. Due to the image processing improvement, the process was also realized for areas with few features, which increases the transferability of the process. The result of this multimodal registration in 3D space can be viewed and analyzed within a visualization tool. Individual CT slices can be analyzed axially, sagittally, and coronally with the corresponding superficial skin temperature distribution. This is an important and successfully implemented milestone on the way to simulating the internal temperature profile. Using this temperature profile, deep-seated infections and inflammation can be detected in order to reduce animal suffering.

Deep action learning enables robust 3D segmentation of body organs in various CT and MRI images

In this study, we propose a novel point cloud based 3D registration and segmentation framework using reinforcement learning. An artificial agent, implemented as a distinct actor based on value networks, is trained to predict the optimal piece-wise linear transformation of a point cloud for the joint tasks of registration and segmentation. The actor network estimates a set of plausible actions and the value network aims to select the optimal action for the current observation. Point-wise features that comprise spatial positions (and surface normal vectors in the case of structured meshes), and their corresponding image features, are used to encode the observation and represent the underlying 3D volume. The actor and value networks are applied iteratively to estimate a sequence of transformations that enable accurate delineation of object boundaries. The proposed approach was extensively evaluated in both segmentation and registration tasks using a variety of challenging clinical datasets. Our method has fewer trainable parameters and lower computational complexity compared to the 3D U-Net, and it is independent of the volume resolution. We show that the proposed method is applicable to mono- and multi-modal segmentation tasks, achieving significant improvements over the state-of-the-art for the latter. The flexibility of the proposed framework is further demonstrated for a multi-modal registration application. As we learn to predict actions rather than a target, the proposed method is more robust compared to the 3D U-Net when dealing with previously unseen datasets, acquired using different protocols or modalities. As a result, the proposed method provides a promising multi-purpose segmentation and registration framework, particular in the context of image-guided interventions.

Deep Learning-Based Annotation Transfer between Molecular Imaging Modalities: An Automated Workflow for Multimodal Data Integration.

An ever-increasing array of imaging technologies are being used in the study of complex biological samples, each of which provides complementary, occasionally overlapping information at different length scales and spatial resolutions. It is important to understand the information provided by one technique in the context of the other to achieve a more holistic overview of such complex samples. One way to achieve this is to use annotations from one modality to investigate additional modalities. For microscopy-based techniques, these annotations could be manually generated using digital pathology software or automatically generated by machine learning (including deep learning) methods. Here, we present a generic method for using annotations from one microscopy modality to extract information from complementary modalities. We also present a fast, general, multimodal registration workflow [evaluated on multiple mass spectrometry imaging (MSI) modalities, matrix-assisted laser desorption/ionization, desorption electrospray ionization, and rapid evaporative ionization mass spectrometry] for automatic alignment of complex data sets, demonstrating an order of magnitude speed-up compared to previously published work. To demonstrate the power of the annotation transfer and multimodal registration workflows, we combine MSI, histological staining (such as hematoxylin and eosin), and deep learning (automatic annotation of histology images) to investigate a pancreatic cancer mouse model. Neoplastic pancreatic tissue regions, which were histologically indistinguishable from one another, were observed to be metabolically different. We demonstrate the use of the proposed methods to better understand tumor heterogeneity and the tumor microenvironment by transferring machine learning results freely between the two modalities.

Reducing motion artifact in sequential-scan dual-energy CT imaging by incorporating deformable registration within joint statistical image reconstruction

Dual-energy computed tomography (DECT) has been widely used to reconstruct basis components. In previous studies, ou DECT algorithm has shown high accuracy in stopping power ratio (SPR) estimation of fixed objects for proton radiotherapy planning. However, patient movement between sequential data acquisitions may lead to severe motion artifacts in the component images. In order to reduce or eliminate the motion artifacts in clinical applications, we combine a deformable registration method with an accurate joint statistical iterative reconstruction algorithm, dual-energy alternating minimization (DEAM). Image registration is a process of geometrically aligning two or more images. We implement a multi-modality symmetric deformable registration method based on Advanced Normalization Tools (ANTs) to automatically align the scans we acquire for the same patient. The precalculated registration mapping and its inverse are then embedded into each iteration of the DEAM algorithm. The performance of warped DEAM is quantitatively assessed. Theoretically, the performance of warped DEAM on moved patients should be comparable to the performance of the original DEAM algorithm on fixed objects. The warped DEAM algorithm reduces motion artifacts while preserving the accuracy of the iterative joint statistical CT reconstruction algorithm, which enables us to reconstruct accurate results from sequentially scanned dual-energy patient data.

Virtual brain grafting: Enabling whole brain parcellation in the presence of large lesions

Brain atlases and templates are at the heart of neuroimaging analyses, for which they facilitate multimodal registration, enable group comparisons and provide anatomical reference. However, as atlas-based approaches rely on correspondence mapping between images they perform poorly in the presence of structural pathology. Whilst several strategies exist to overcome this problem, their performance is often dependent on the type, size and homogeneity of any lesions present. We therefore propose a new solution, referred to as Virtual Brain Grafting (VBG), which is a fully-automated, open-source workflow to reliably parcellate magnetic resonance imaging (MRI) datasets in the presence of a broad spectrum of focal brain pathologies, including large, bilateral, intra- and extra-axial, heterogeneous lesions with and without mass effect.The core of the VBG approach is the generation of a lesion-free T1-weighted image, which enables further image processing operations that would otherwise fail. Here we validated our solution based on Freesurfer recon-all parcellation in a group of 10 patients with heterogeneous gliomatous lesions, and a realistic synthetic cohort of glioma patients (n = 100) derived from healthy control data and patient data.We demonstrate that VBG outperforms a non-VBG approach assessed qualitatively by expert neuroradiologists and Mann-Whitney U tests to compare corresponding parcellations (real patients U(6,6) = 33, z = 2.738, P < .010, synthetic-patients U(48,48) = 2076, z = 7.336, P < .001). Results were also quantitatively evaluated by comparing mean dice scores from the synthetic-patients using one-way ANOVA (unilateral VBG = 0.894, bilateral VBG = 0.903, and non-VBG = 0.617, P < .001). Additionally, we used linear regression to show the influence of lesion volume, lesion overlap with, and distance from the Freesurfer volumes of interest, on labeling accuracy.VBG may benefit the neuroimaging community by enabling automated state-of-the-art MRI analyses in clinical populations using methods such as FreeSurfer, CAT12, SPM, Connectome Workbench, as well as structural and functional connectomics. To fully maximize its availability, VBG is provided as open software under a Mozilla 2.0 license (https://github.com/KUL-Radneuron/KUL_VBG).

Learning 4D Infant Cortical Surface Atlas with Unsupervised Spherical Networks.

Spatiotemporal (4D) cortical surface atlas during infancy plays an important role for surface-based visualization, normalization and analysis of the dynamic early brain development. Conventional atlas construction methods typically rely on classical group-wise registration on sub-populations and ignore longitudinal constraints, thus having three main issues: 1) constructing templates at discrete time points; 2) resulting in longitudinal inconsistency among different age's atlases; and 3) taking extremely long runtime. To address these issues, in this paper, we propose a fast unsupervised learning-based surface atlas construction framework incorporating longitudinal constraints to enforce the within-subject temporal correspondence in the atlas space. To well handle the difficulties of learning large deformations, we propose a multi-level multimodal spherical registration network to perform cortical surface registration in a coarse-to-fine manner. Thus, only small deformations need to be estimated at each resolution level using the registration network, which further improves registration accuracy and atlas quality. Our constructed 4D infant cortical surface atlas based on 625 longitudinal scans from 291 infants is temporally continuous, in contrast to the state-of-the-art UNC 4D Infant Surface Atlas, which only provides the atlases at a few discrete sparse time points. By evaluating the intra- and inter-subject spatial normalization accuracy after alignment onto the atlas, our atlas demonstrates more detailed and fine-grained cortical patterns, thus leading to higher accuracy in surface registration.

Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration.

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.

MiRID: Multi-Modal Image Registration Using Modality-Independent and Rotation-Invariant Descriptor

Axiomatically, symmetry is a fundamental property of mathematical functions defining similarity measures, where similarity measures are important tools in many areas of computer science, including machine learning and image processing. In this paper, we investigate a new technique to measure the similarity between two images, a fixed image and a moving image, in multi-modal image registration (MIR). MIR in medical image processing is essential and useful in diagnosis and therapy guidance, but still a very challenging task due to the lack of robustness against the rotational variance in the image transformation process. Our investigation leads to a novel, local self-similarity descriptor, called the modality-independent and rotation-invariant descriptor (miRID). By relying on the mean of the intensity values, an miRID is simply computable and can effectively handle the complicated intensity relationship between multi-modal images. Moreover, it can also overcome the problem of rotational variance by sorting the numerical values, each of which is the absolute difference between each pixel’s intensity and the mean of all pixel intensities within a patch of the image. The experimental result shows that our method outperforms others in both multi-modal rigid and non-rigid image registrations.

In vivo Population Averaged Stereotaxic T2w MRI Brain Template for the Adult Yucatan Micropig.

Population averaged brain templates are an essential tool for imaging-based neuroscience research, providing investigators with information about the expected size and morphology of brain structures and the spatial relationships between them, within a demographic cross-section. This allows for a standardized comparison of neuroimaging data between subjects and provides neuroimaging software with a probabilistic framework upon which further processing and analysis can be based. Many different templates have been created to represent specific study populations and made publicly available for human and animal research. An increasingly studied animal model in the neurosciences that still lacks appropriate brain templates is the adult Yucatan micropig. In particular, T2-weighted templates are absent in this species as a whole. To address this need and provide a tool for neuroscientists wishing to pursue neuroimaging research in the adult micropig, we present the construction of population averaged (n = 16) T2-weighted MRI brain template for the adult Yucatan micropig. Additionally, we present initial analysis of T1-weighted (n = 3), and diffusion-weighted (n = 3) images through multimodal registration of these contrasts to our T2 template. The strategies used here may also be generalized to create similar templates for other study populations or species in need of template construction.

RGB-topography and X-rays image registration for idiopathic scoliosis children patient follow-up

Children diagnosed with a scoliosis pathology are exposed during their follow up to ionic radiations in each X-rays diagnosis. This exposure can have negative effects on the patient’s health and cause diseases in the adult age. In order to reduce X-rays scanning, recent systems provide diagnosis of scoliosis patients using solely RGB images. The output of such systems is a set of augmented images and scoliosis related angles. However, these angles, confuse the physicians due to their large number. Moreover, the lack of X-rays scans makes it impossible for the physician to compare RGB and X-rays images, and decide whether to reduce X-rays exposure or not. In this work, we exploit both RGB images of scoliosis captured during clinical diagnosis, and X-rays hard copies provided by patients in order to register both images and give a rich comparison of diagnoses. The work consists in, first, establishing the monomodal (RGB topography of the back) and multimodal (RGB and Xrays) image database, then registering images based on patient landmarks, and finally blending registered images for a visual analysis and follow up by the physician. Proposed registration is based on a rigid transformation that preserves the topology of the patient’s back. Parameters of the rigid transformation are estimated using a proposed angle minimization of Cervical vertebra 7, and Posterior Superior Iliac Spine landmarks of a source and target diagnoses. Experiments conducted on the constructed database show a better monomodal and multimodal registration using our proposed method compared to registration using an Equation System Solving based registration.

End-to-end multimodal image registration via reinforcement learning.

Multimodal image registration is a vital initial step in several medical image applications for providing complementary information from different data modalities. Since images with different modalities do not exhibit the same characteristics, finding their accurate correspondences remains a challenge. For convolutional multimodal registration methods, two components are quite significant: descriptive image feature as well as the suited similarity metric. However, these two components are often custom-designed and are infeasible to the high diversity of tissue appearance across modalities. In this paper, we translate image registration into a decision-making problem, where registration is achieved via an artificial agent trained by asynchronous reinforcement learning. More specifically, convolutional long-short-term-memory is incorporated after stacked convolutional layers in this method to extract spatial-temporal image features and learn the similarity metric implicitly. A customized reward function driven by landmark error is advocated to guide the agent to the correct registration direction. A Monte Carlo rollout strategy is also leveraged to perform as a look-ahead inference in the testing stage, to increase registration accuracy further. Experiments on paired CT and MR images of patients diagnosed as nasopharyngeal carcinoma demonstrate that our method achieves state-of-the-art performance in medical image registration.

A practical method for multimodal registration and assessment of whole-brain disease burden using PET, MRI, and optical imaging

Many neurological diseases present with substantial genetic and phenotypic heterogeneity, making assessment of these diseases challenging. This has led to ineffective treatments, significant morbidity, and high mortality rates for patients with neurological diseases, including brain cancers and neurodegenerative disorders. Improved understanding of this heterogeneity is necessary if more effective treatments are to be developed. We describe a new method to measure phenotypic heterogeneity across the whole rodent brain at multiple spatial scales. The method involves co-registration and localized comparison of in vivo radiologic images (e.g. MRI, PET) with ex vivo optical reporter images (e.g. labeled cells, molecular targets, microvasculature) of optically cleared tissue slices. Ex vivo fluorescent images of optically cleared pathology slices are acquired with a preclinical in vivo optical imaging system across the entire rodent brain in under five minutes, making this methodology practical and feasible for most preclinical imaging labs. The methodology is applied in various examples demonstrating how it might be used to cross-validate and compare in vivo radiologic imaging with ex vivo optical imaging techniques for assessing hypoxia, microvasculature, and tumor growth.

Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration.

Methods for deep learning based medical image registration have only recently approached the quality of classical model-based image alignment. The dual challenge of both a very large trainable parameter space and often insufficient availability of expert supervised correspondence annotations has led to slower progress compared to other domains such as image segmentation. Yet, image registration could also more directly benefit from an iterative solution than segmentation. We therefore believe that significant improvements, in particular for multi-modal registration, can be achieved by disentangling appearance-based feature learning and deformation estimation. In this work, we examine an end-to-end trainable, weakly-supervised deep learning-based feature extraction approach that is able to map the complex appearance to a common space. Our results on thoracoabdominal CT and MRI image registration show that the proposed method compares favourably well to state-of-the-art hand-crafted multi-modal features, Mutual Information-based approaches and fully-integrated CNN-based methods - and handles even the limitation of small and only weakly-labeled training data sets.

The developing Human Connectome Project (dHCP) automated resting-state functional processing framework for newborn infants

The developing Human Connectome Project (dHCP) aims to create a detailed 4-dimensional connectome of early life spanning 20–45 weeks post-menstrual age. This is being achieved through the acquisition of multi-modal MRI data from over 1000 in- and ex-utero subjects combined with the development of optimised pre-processing pipelines. In this paper we present an automated and robust pipeline to minimally pre-process highly confounded neonatal resting-state fMRI data, robustly, with low failure rates and high quality-assurance. The pipeline has been designed to specifically address the challenges that neonatal data presents including low and variable contrast and high levels of head motion. We provide a detailed description and evaluation of the pipeline which includes integrated slice-to-volume motion correction and dynamic susceptibility distortion correction, a robust multimodal registration approach, bespoke ICA-based denoising, and an automated QC framework. We assess these components on a large cohort of dHCP subjects and demonstrate that processing refinements integrated into the pipeline provide substantial reduction in movement related distortions, resulting in significant improvements in SNR, and detection of high quality RSNs from neonates.

An application independent review of multimodal 3D registration methods

Registration is a ubiquitous operation in Visual Computing, with applications in 3D object retrieval among others. Registration is the process of overlaying two or more datasets taken from different viewpoints, at different times or by different sensors into a common reference frame. Multimodal registration is a special case where the data to be matched do not belong to the same modality and is challenging due to the diverse nature of the modalities involved which makes the creation of a distance function harder. Due to the large number of possible modality combinations and application fields, a considerable number of multimodal registration techniques have been proposed in diverse fields, including medicine and archaeology. This survey aims to unify 3D multimodal registration techniques (i.e. where at least one of the modalities is in 3D) across application domains, with the hope of providing an application-independent view and the potential for cross-fertilization. The problem of 3D multimodal registration is explicitly defined and the various methods are systematically categorized and described in terms of a number of important properties. Methods with publicly available source code have been compared on common datasets. A discussion on trends, observations and challenges for further research concludes the review.

Multimodal affine registration for ICGA and MCSL fundus images of high myopia.

The registration between indocyanine green angiography (ICGA) and multi-color scanning laser (MCSL) imaging fundus images is vital for the joint linear lesion segmentation in ICGA and MCSL and the evaluation whether MCSL can replace ICGA as a non-invasive diagnosis for linear lesion. To our best knowledge, there are no studies focusing on the image registration between these two modalities. In this paper, we propose a framework based on convolutional neural networks for the multimodal affine registration between ICGA and MCSL images, which contains two parts: coarse registration stage and fine registration stage. In the coarse registration stage, the optic disc is segmented and its centroid is used as a matching point to perform coarse registration. The fine registration stage regresses affine parameters directly using jointly supervised and weakly-supervised loss function. Experimental results show the effectiveness of the proposed method, which lays a sound foundation for further evaluation of non-invasive diagnosis of linear lesion based on MCSL.

Dynamics of soil surface temperature with unmanned aerial systems

Thermographies are a source of rich information, valuable in precision agriculture tasks such as crop stress assessment, plant disease analysis, and soil moisture evaluation. Traditionally, practitioners obtain soil temperature from the ground or using satellites and other airborne methods, which are costly and offer limited spatial and temporal resolution. In this paper, we introduce a method to measure soil surface temperature dynamics with the use of an unmanned aerial system (UAS). In our approach, we fuse information from thermal and multispectral cameras with ambient variables to generate estimates for soil temperature using computational intelligence models. Using the images, we produce a spatial reconstruction using structure from motion (SfM). After the multimodal registration of the resulting geo-referenced orthomosaics, we characterize the dynamics of the soil surface temperature using the differences between consecutively captured temperature orthomosaics. In our results, we are capable of estimating soil surface temperature from a UAS flying at 30 m AGL with a RMSE of 3.24∘C ± 0.3 and 1.77∘C ± 0.2, at one standard deviation, for two test fields with average ground sampling distances below 6.0 cm/pixel, using a Random Forest regressor.

An Indirect Multimodal Image Registration and Completion Method Guided by Image Synthesis.

Multimodal registration is a challenging task due to the significant variations exhibited from images of different modalities. CT and MRI are two of the most commonly used medical images in clinical diagnosis, since MRI with multicontrast images, together with CT, can provide complementary auxiliary information. The deformable image registration between MRI and CT is essential to analyze the relationships among different modality images. Here, we proposed an indirect multimodal image registration method, i.e., sCT-guided multimodal image registration and problematic image completion method. In addition, we also designed a deep learning-based generative network, Conditional Auto-Encoder Generative Adversarial Network, called CAE-GAN, combining the idea of VAE and GAN under a conditional process to tackle the problem of synthetic CT (sCT) synthesis. Our main contributions in this work can be summarized into three aspects: (1) We designed a new generative network called CAE-GAN, which incorporates the advantages of two popular image synthesis methods, i.e., VAE and GAN, and produced high-quality synthetic images with limited training data. (2) We utilized the sCT generated from multicontrast MRI as an intermediary to transform multimodal MRI-CT registration into monomodal sCT-CT registration, which greatly reduces the registration difficulty. (3) Using normal CT as guidance and reference, we repaired the abnormal MRI while registering the MRI to the normal CT.

Multimodal imaging in radiotherapy: Focus on adaptive therapy and quality control

Improved computer resources in radiation oncology department have greatly facilitated the integration of multimodal imaging into the workflow of radiation therapy. Nowadays, physicians have highly informative imaging modalities of the anatomical region to be treated. These images contribute to the targeting accuracy with the current treatment device, impacting both segmentation or patient's positioning. Additionally, in a constant effort to deliver personalized care, many teams seek to confirm the benefits of adaptive radiotherapy. The published works highlight the importance of registration algorithms, particularly those of elastic or deformable registration necessary to take into account the anatomical evolutions of the patients during the course of their therapy. These algorithms, often considered as “black boxes”, tend to be better controlled and understood by physicists and physicians thanks to the generalization of evaluation and validation methods. Given the still significant development of medical imaging techniques, it is foreseeable that multimodal registration needs require more efficient algorithms well integrated within the flow of data.