Multimodal Image Registration Research Articles

Uncertainty estimation and evaluation of deformation image registration based convolutional neural networks.

Objective&#xD;Fast and accurate deformable image registration (DIR) that include DIR uncertainty estimation are essential for safe and reliable clinical deployment. While recent deep learning models have shown promise in predicting DIR with its uncertainty, challenges persist in proper uncertainty evaluation and hyperparameter optimization for these methods. This work aims to develop and evaluate a model that can perform fast DIR and predict its uncertainty in seconds.&#xD;Approach&#xD;In this study, we introduce a novel probabilistic multi-resolution image registration model utilizing convolutional neural networks (CNNs) to estimate a multivariate normal distributed dense displacement field (DDF) in a multimodal image registration problem. To assess the quality of the DDF distribution predicted by the model, we propose a new metric based on the Kullback-Leibler divergence. The performance of our approach was evaluated against three other DIR algorithms (VoxelMorph, Monte Carlo Drop-Out, and Monte Carlo B-splines) capable of predicting uncertainty. The evaluation of the models included not only the quality of the deformation but also the reliability of the estimated uncertainty. Our application investigated registration of a treatment planning computed tomography (CT) to follow-up cone beam CT for daily adaptive radiotherapy.&#xD;Main results&#xD;The hyperparameter tuning of the models showed that there is a trade-off between the reliability of the estimated uncertainty and the accuracy of the deformation. In the optimal trade-off our model excelled in contour propagation and uncertainty estimation (p < 0.01) compared to existing uncertainty estimation models. We obtained an average dice similarity coefficient of 0.89 and a KL-divergence of 0.15.&#xD;Significance&#xD;By addressing challenges in DIR uncertainty estimation and evaluation, our work showed that both the DIR and its uncertainty can be reliably predicted paving the way for safe deployment in clinical environment.&#xD.

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

The measurement of retinal blood flow (RBF) in capillaries can provide a powerful biomarker for the early diagnosis and treatment of ocular diseases. However, no single modality can determine capillary flowrates with high precision. Combining erythrocyte-mediated angiography (EMA) with optical coherence tomography angiography (OCTA) has the potential to achieve this goal, as EMA can measure the absolute RBF of retinal microvasculature and OCTA can provide the structural images of capillaries. However, multimodal retinal image registration between these two modalities remains largely unexplored. To fill this gap, we establish MEMO, the first public multimodal EMA and OCTA retinal image dataset. A unique challenge in multimodal retinal image registration between these modalities is the relatively large difference in vessel density (VD). To address this challenge, we propose a segmentation-based deep-learning framework (VDD-Reg), which provides robust results despite differences in vessel density. VDD-Reg consists of a vessel segmentation module and a registration module. To train the vessel segmentation module, we further designed a two-stage semi-supervised learning framework (LVD-Seg) combining supervised and unsupervised losses. We demonstrate that VDD-Reg outperforms existing methods quantitatively and qualitatively for cases of both small VD differences (using the CF-FA dataset) and large VD differences (using our MEMO dataset). Moreover, VDD-Reg requires as few as three annotated vessel segmentation masks to maintain its accuracy, demonstrating its feasibility.

New improved model for joint segmentation and registration of multi-modality images: application to medical images

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

Teaching a hands-on session on simultaneous Positron Emission Tomography and Magnetic Resonance image acquisition and quantification in a clinical setting

Positron Emission Tomography/Magnetic Resonance (PET/MR) imaging, is a novel imaging modality that combines the capabilities of two powerful imaging techniques in a single acquisition. This unique integration allows for simultaneous acquisition of both metabolic and structural data within a single imaging session. In this sense, PET/MR offers a comprehensive and innovative approach to medical imaging, but accompanied by its intrinsic physics and engineering complexity, involving intricate synchronization of high-performance detectors, electromagnetic shielding, and sophisticated correction algorithms, among others. In this hands-on session performed during the 6th INFIERI summer school (Madrid, Spain), we described an introduction to simultaneous PET/MR image acquisition and quantification in a real clinical to students from diverse backgrounds in physics and engineering. We first presented the fundamental aspects of PET/MR image acquisition, including the main theoretical characteristics of PET/MR physics and electronics, visiting the PET/MR facilities at Hospital Universitario HM Puerta del Sur (HM Hospitales, Móstoles, Madrid), and discussing the main PET/MR clinical and research applications. Then, we followed with a practical session on PET/MR image quantification, preparing the students to understand and solve examples on image reconstruction, multimodal data visualization, inhomogeneity correction, image registration, and attenuation correction exploiting the simplicity and usefulness of 3D Slicer.

Robust image registration for analysis of multisource eye fundus images

This study underscores the crucial role of image preprocessing in enhancing the outcomes of multimodal image registration tasks using scale-invariant feature selection. The primary focus is on registering two types of retinal images, assessing a methodology’s performance on a set of retinal image pairs, including those with and without microaneurysms. Each pair comprises a color optical image and a gray-level fluorescein image, presenting distinct characteristics and captured under varying conditions. The SIFT methodology, encompassing five stages, with preprocessing as the initial and pivotal stage, is employed for image registration. Out of 35 test retina image pairs, 33 (94.28%) were successfully registered, with the inability to extract features hindering automatic registration in the remaining pairs. Among the registered pairs, 42.42% were retinal images without microaneurysms, and 57.57% had microaneurysms. Instead of simultaneous registration of all channels, independent registration of preprocessed images in each channel proved more effective. The study concludes with an analysis of the fifth registration’s resulting image to detect abnormalities or pathologies, highlighting the challenges encountered in registering blue channel images due to high intrinsic noise.

MMIR: an open-source software for the registration of multimodal histological images

BackgroundMultimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality.ResultsWe developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system.ConclusionOur software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

An integration of meta-heuristic approach utilizing kernel principal component analysis for multimodal medical image registration

An integration of meta-heuristic approach utilizing kernel principal component analysis for multimodal medical image registration

BCSwinReg: A cross-modal attention network for CBCT-to-CT multimodal image registration

Computed tomography (CT) and cone beam computed tomography (CBCT) registration plays an important role in radiotherapy. However, the poor quality of CBCT makes CBCT-CT multimodal registration challenging. Effective feature fusion and mapping often lead to better registration results for multimodal registration. Therefore, we proposed a new backbone network BCSwinReg and a cross-modal attention module CrossSwin. Specifically, a cross-modal attention CrossSwin is designed to promote multi-modal feature fusion, map the multi-modal domain to the common domain, and thus helping the network learn the correspondence between images better. Furthermore, a new network, BCSwinReg, is proposed to discover correspondence through cross-attention exchange information, obtain multi-level semantic information through a multi-resolution strategy, and finally integrate the deformation of multi-resolutions by the divide-conquer cascade method. We performed experiments on the publicly available 4D-Lung dataset to demonstrate the effectiveness of CrossSwin and BCSwinReg. Compared with VoxelMorph, the BCSwinReg has obtained performance improvements of 3.3% in Dice Similarity Coefficient (DSC) and 0.19 in the average 95% Hausdorff distance (HD95).

Multi-Scale Tokens-Aware Transformer Network for Multi-Region and Multi-Sequence MR-to-CT Synthesis in a Single Model.

The superiority of magnetic resonance (MR)-only radiotherapy treatment planning (RTP) has been well demonstrated, benefiting from the synthesis of computed tomography (CT) images which supplements electron density and eliminates the errors of multi-modal images registration. An increasing number of methods has been proposed for MR-to-CT synthesis. However, synthesizing CT images of different anatomical regions from MR images with different sequences using a single model is challenging due to the large differences between these regions and the limitations of convolutional neural networks in capturing global context information. In this paper, we propose a multi-scale tokens-aware Transformer network (MTT-Net) for multi-region and multi-sequence MR-to-CT synthesis in a single model. Specifically, we develop a multi-scale image tokens Transformer to capture multi-scale global spatial information between different anatomical structures in different regions. Besides, to address the limited attention areas of tokens in Transformer, we introduce a multi-shape window self-attention into Transformer to enlarge the receptive fields for learning the multi-directional spatial representations. Moreover, we adopt a domain classifier in generator to introduce the domain knowledge for distinguishing the MR images of different regions and sequences. The proposed MTT-Net is evaluated on a multi-center dataset and an unseen region, and remarkable performance was achieved with MAE of 69.33 ± 10.39 HU, SSIM of 0.778 ± 0.028, and PSNR of 29.04 ± 1.32 dB in head & neck region, and MAE of 62.80 ± 7.65 HU, SSIM of 0.617 ± 0.058 and PSNR of 25.94 ± 1.02 dB in abdomen region. The proposed MTT-Net outperforms state-of-the-art methods in both accuracy and visual quality.

SuperCUT, an unsupervised multimodal image registration with deep learning for biomedical microscopy.

Numerous imaging techniques are available for observing and interrogating biological samples, and several of them can be used consecutively to enable correlative analysis of different image modalities with varying resolutions and the inclusion of structural or molecular information. Achieving accurate registration of multimodal images is essential for the correlative analysis process, but it remains a challenging computer vision task with no widely accepted solution. Moreover, supervised registration methods require annotated data produced by experts, which is limited. To address this challenge, we propose a general unsupervised pipeline for multimodal image registration using deep learning. We provide a comprehensive evaluation of the proposed pipeline versus the current state-of-the-art image registration and style transfer methods on four types of biological problems utilizing different microscopy modalities. We found that style transfer of modality domains paired with fully unsupervised training leads to comparable image registration accuracy to supervised methods and, most importantly, does not require human intervention.

Data Science for Health Image Alignment: A User-Friendly Open-Source ImageJ/Fiji Plugin for Aligning Multimodality/Immunohistochemistry/Immunofluorescence 2D Microscopy Images.

Most of the time, the deep analysis of a biological sample requires the acquisition of images at different time points, using different modalities and/or different stainings. This information gives morphological, functional, and physiological insights, but the acquired images must be aligned to be able to proceed with the co-localisation analysis. Practically speaking, according to Aristotle's principle, "The whole is greater than the sum of its parts", multi-modal image registration is a challenging task that involves fusing complementary signals. In the past few years, several methods for image registration have been described in the literature, but unfortunately, there is not one method that works for all applications. In addition, there is currently no user-friendly solution for aligning images that does not require any computer skills. In this work, DS4H Image Alignment (DS4H-IA), an open-source ImageJ/Fiji plugin for aligning multimodality, immunohistochemistry (IHC), and/or immunofluorescence (IF) 2D microscopy images, designed with the goal of being extremely easy to use, is described. All of the available solutions for aligning 2D microscopy images have also been revised. The DS4H-IA source code; standalone applications for MAC, Linux, and Windows; video tutorials; manual documentation; and sample datasets are publicly available.

L2NLF: a novel linear-to-nonlinear framework for multi-modal medical image registration.

In recent years, deep learning has ushered in significant development in medical image registration, and the method of non-rigid registration using deep neural networks to generate a deformation field has higher accuracy. However, unlike monomodal medical image registration, multimodal medical image registration is a more complex and challenging task. This paper proposes a new linear-to-nonlinear framework (L2NLF) for multimodal medical image registration. The first linear stage is essentially image conversion, which can reduce the difference between two images without changing the authenticity of medical images, thus transforming multimodal registration into monomodal registration. The second nonlinear stage is essentially unsupervised deformable registration based on the deep neural network. In this paper, a brand-new registration network, CrossMorph, is designed, a deep neural network similar to the U-net structure. As the backbone of the encoder, the volume CrossFormer block can better extract local and global information. Booster promotes the reduction of more deep features and shallow features. The qualitative and quantitative experimental results on T1 and T2 data of 240 patients' brains show that L2NLF can achieve excellent registration effect in the image conversion part with very low computation, and it will not change the authenticity of the converted image at all. Compared with the current state-of-the-art registration method, CrossMorph can effectively reduce average surface distance, improve dice score, and improve the deformation field's smoothness. The proposed methods have potential value in clinical application.

Adjacent Self-Similarity 3-D Convolution for Multimodal Image Registration

Adjacent Self-Similarity 3-D Convolution for Multimodal Image Registration

CRCS: An automatic image processing pipeline for hormone level analysis of Cushing’s disease

Due to the abnormal secretion of adreno-cortico-tropic-hormone (ACTH) by tumors, Cushing's disease leads to hypercortisonemia, a precursor to a series of metabolic disorders and serious complications. Cushing's disease has high recurrence rate, short recurrence time and undiscovered recurrence reason after surgical resection. Qualitative or quantitative automatic image analysis of histology images can potentially in providing insights into Cushing's disease, but still no software has been available to the best of our knowledge. In this study, we propose a quantitative image analysis-based pipeline CRCS, which aims to explore the relationship between the expression level of ACTH in normal cell tissues adjacent to tumor cells and the postoperative prognosis of patients. CRCS mainly consists of image-level clustering, cluster-level multi-modal image registration, patch-level image classification and pixel-level image segmentation on the whole slide imaging (WSI). On both image registration and classification tasks, our method CRCS achieves state-of-the-art performance compared to recently published methods on our collected benchmark dataset. In addition, CRCS achieves an accuracy of 0.83 for postoperative prognosis of 12 cases. CRCS demonstrates great potential for instrumenting automatic diagnosis and treatment for Cushing's disease.

RA-MMIR : Multi-modal image registration by Robust Adaptive Variation Attention Gauge Field

Multi-modal image registration finds extensive application in high-level vision tasks. Especially in adverse conditions, multi-modal image registration and fusion is powerful for high-level visual analysis, high-level visual tasks commonly prioritize the analysis of crucial target regions within images, however precise multi-modal image registration remains a challenge. To address this issue, we rethought the collaboration between image registration and high-level visual tasks, and propose a Robust Adaptive Variation Attention Gauge Field registration framework that allows for flexible attention to both the target regions and global areas. Among them, in order to improve the robustness and timeliness of the algorithm in extracting and describing target regions or global areas features in adverse conditions, we propose a Robust Adaptive Variation Attention for building the gauge field, and in order to make robust adaptive parameters to reach the global optimum, we propose a Quasi-Simulated Annealing method based on mini-batch. To make the spatial transformation better fit feature matching and image registration, we design a deep learning and modeling approach based on spatial similarity metrics. In high-level visual tasks-based experiments such as image fusion, object detection, and 3D reconstruction, our method had the best collaborative performance and showed the best registration results under adverse conditions. The code is at https://github.com/JuiHuiQ/RA-MMIR.

SOLID: a novel similarity metric for mono-modal and multi-modal deformable image registration

Medical image registration is an integral part of various clinical applications including image guidance, motion tracking, therapy assessment and diagnosis. We present a robust approach for mono-modal and multi-modal medical image registration. To this end, we propose the novel shape operator based local image distance (SOLID) which estimates the similarity of images by comparing their second-order curvature information. Our similarity metric is rigorously tailored to be suitable for comparing images from different medical imaging modalities or image contrasts. A critical element of our method is the extraction of local features using higher-order shape information, enabling the accurate identification and registration of smaller structures. In order to assess the efficacy of the proposed similarity metric, we have implemented a variational image registration algorithm that relies on the principle of matching the curvature information of the given images. The performance of the proposed algorithm has been evaluated against various alternative state-of-the-art variational registration algorithms. Our experiments involve mono-modal as well as multi-modal and cross-contrast co-registration tasks in a broad variety of anatomical regions. Compared to the evaluated alternative registration methods, the results indicate a very favorable accuracy, precision and robustness of the proposed SOLID method in various highly challenging registration tasks.

Volumetric rigid MR-CT registration for glioblastoma in radiation oncology: A Novel approach

Glioblastoma multiforme (GBM), is a highly aggressive and rapidly growing grade IV astrocytoma that originates from astrocytes and causes a diverse range of neurological problems by infiltratingsurrounding brain tissue. Image guided radiotherapy is a crucial treatment option for GBM; however, precise target delineation is critical for efficient treatment in the presence of noise and distortion in the magnetic resonance (MR) and computed tomography (CT) images. Multimodal image registration, which generates aligned or fused images from different modalities, can enhance target delineation accuracy. The objective of this study was to evaluate the effectiveness of volumetric multimodal MR-CT rigid registration algorithms for the patients with GBM and to enhance the success of radiotherapy treatment responses by employing visual inspection and statistical assessment of the registered images. Specialized algorithms were developed to attain volumetric registration, leading to an enhanced target delineation for radiation treatment. The crucial role of multimodal rigid registration in improving the accuracy of target delineation in the context of radiation oncology for GBM was underscored by this study.

DTR-GAN: An Unsupervised Bidirectional Translation Generative Adversarial Network for MRI-CT Registration

Medical image registration is a fundamental and indispensable element in medical image analysis, which can establish spatial consistency among corresponding anatomical structures across various medical images. Since images with different modalities exhibit different features, it remains a challenge to find their exact correspondence. Most of the current methods based on image-to-image translation cannot fully leverage the available information, which will affect the subsequent registration performance. To solve the problem, we develop an unsupervised multimodal image registration method named DTR-GAN. Firstly, we design a multimodal registration framework via a bidirectional translation network to transform the multimodal image registration into a unimodal registration, which can effectively use the complementary information of different modalities. Then, to enhance the quality of the transformed images in the translation network, we design a multiscale encoder–decoder network that effectively captures both local and global features in images. Finally, we propose a mixed similarity loss to encourage the warped image to be closer to the target image in deep features. We extensively evaluate methods for MRI-CT image registration tasks of the abdominal cavity with advanced unsupervised multimodal image registration approaches. The results indicate that DTR-GAN obtains a competitive performance compared to other methods in MRI-CT registration. Compared with DFR, DTR-GAN has not only obtained performance improvements of 2.35% and 2.08% in the dice similarity coefficient (DSC) of MRI-CT registration and CT-MRI registration on the Learn2Reg dataset but has also decreased the average symmetric surface distance (ASD) by 0.33 mm and 0.12 mm on the Learn2Reg dataset.

Evaluating the Hounsfield unit assignment and dose differences between CT-based standard and deep learning-based synthetic CT images for MRI-only radiation therapy of the head and neck.

Magnetic resonance image only (MRI-only) simulation for head and neck (H&N) radiotherapy (RT) could allow for single-image modality planning with excellent soft tissue contrast. In the MRI-only simulation workflow, synthetic computed tomography (sCT) is generated from MRI to provide electron density information for dose calculation. Bone/air regions produce little MRI signal which could lead to electron density misclassification in sCT. Establishing the dosimetric impact of this error could inform quality assurance (QA) procedures using MRI-only RT planning or compensatory methods for accurate dosimetric calculation. The aim of this study was to investigate if Hounsfield unit (HU) voxel misassignments from sCT images result in dosimetric errors in clinical treatment plans. Fourteen H&N cancer patients undergoing same-day CT and 3T MRI simulation were retrospectively identified. MRI was deformed to the CT using multimodal deformable image registration. sCTs were generated from T1w DIXON MRIs using a commercially available deep learning-based generator (MRIplanner, Spectronic Medical AB, Helsingborg, Sweden). Tissue voxel assignment was quantified by creating a CT-derived HU threshold contour. CT/sCT HU differences for anatomical/target contours and tissue classification regions including air (<250 HU), adipose tissue (-250 HU to -51 HU), soft tissue (-50 HU to 199 HU), spongy (200 HU to 499 HU) and cortical bone(>500 HU) were quantified. t-test was used to determine if sCT/CT HU differences were significant. The frequency of structures that had a HU difference>80 HU (the CT window-width setting for intra-cranial structures) was computed to establish structure classification accuracy. Clinical intensity modulated radiation therapy (IMRT) treatment plans created on CT were retrospectively recalculated on sCT images and compared using the gamma metric. The mean ratio of sCT HUs relative to CT for air, adipose tissue, soft tissue, spongy and cortical bone were 1.7±0.3, 1.1±0.1, 1.0±0.1, 0.9±0.1 and 0.8±0.1 (value of 1 indicates perfect agreement). T-tests (significance set at t=0.05) identified differences in HU values for air, spongy and cortical bone in sCT images compared to CT. The structures with sCT/CT HU differences>80 HU of note were the left and right (L/R) cochlea and mandible (>79% of the tested cohort), the oral cavity (for 57% of the tested cohort), the epiglottis (for 43% of the tested cohort) and the L/R TM joints (occurring>29% of the cohort). In the case of the cochlea and TM joints, these structures contain dense bone/air interfaces. In the case of the oral cavity and mandible, these structures suffer the additional challenge of being positionally altered in CT versus MRI simulation (due to a non-MR safe immobilizing bite block requiring absence of bite block in MR). Finally, the epiglottis HU assignment suffers from its small size and unstable positionality. Plans recalculated on sCT yielded global/local gamma pass rates of 95.5%±2% (3mm, 3%) and 92.7%±2.1% (2mm, 2%). The largest mean differences in D95, Dmean , D50 dose volume histogram (DVH) metrics for organ-at-risk (OAR) and planning tumor volumes (PTVs) were 2.3%±3.0% and 0.7%±1.9% respectively. In this cohort, HU differences of CT and sCT were observed but did not translate into a reduction in gamma pass rates or differences in average PTV/OAR dose metrics greater than 3%. For sites such as the H&N where there are many tissue interfaces we did not observe large scale dose deviations but further studies using larger retrospective cohorts are merited to establish the variation in sCT dosimetric accuracy which could help to inform QA limits on clinical sCT usage.

A Multistage Rigid-Affine-Deformable Network for Three-Dimensional Multimodal Medical Image Registration

Multimodal image registration is an important component of medical image processing, allowing the integration of complementary information from various imaging modalities to improve clinical applications like diagnosis and treatment planning. We proposed a novel multistage neural network for three-dimensional multimodal medical image registration, which addresses the challenge of larger rigid deformations commonly present in medical images due to variations in patient positioning in different scanners and rigid anatomical structures. This multistage network combines rigid, affine and deformable transformations in three stages. The network was trained unsupervised with Mutual Information and Gradient L2 loss. We compared the results of our proposed multistage network with a rigid-affine-deformable registration with the classical registration method NiftyReg as a baseline and a multistage network, which combines affine and deformable transformation, as a benchmark. To evaluate the performance of the proposed multistage network, we used four three-dimensional multimodal in vivo datasets: three renal MR datasets consisting of T1-weighted and T2-weighted MR scans and one liver dataset containing CT and T1-weighted MR scans. Experimental results showed that combining rigid, affine and deformable transformations in a multistage network leads to registration results with a high structural similarity, overlap of the corresponding structures (Dice: 76.7 ± 12.5, 61.1 ± 14.0, 64.8 ± 16.2, 68.1 ± 24.6 for the four datasets) and a low level of image folding (|J| ≤ 0: less than or equal to 1.1%), resulting in a medical plausible registration result.