Multimodal Registration Research Articles

DELR-Net: a network for 3D multimodal medical image registration in more lightweight application scenarios.

3D multimodal medical image deformable registration plays a significant role in medical image analysis and diagnosis. However, due to the substantial differences between images of different modalities, registration is challenging and requires high computational costs. Deep learning-based registration methods face these challenges. The primary aim of this paper is to design a 3D multimodal registration network that ensures high-quality registration results while reducing the number of parameters. This study designed a Dual-Encoder More Lightweight Registration Network (DELR-Net). DELR-Net is a low-complexity network that integrates Mamba and ConvNet. The State Space Sequence Module and the Dynamic Large Kernel block are used as the main components of the dual encoders, while the Dynamic Feature Fusion block is used as the main component of the decoder. This study conducted experiments on 3D brain MR images and abdominal MR and CT images. Compared to existing registration methods, DELR-Net achieved better registration results while maintaining a lower number of parameters. Additionally, generalization experiments on other modalities showed that DELR-Net has superior generalization capabilities. DELR-Net significantly improves the limitations of 3D multimodal medical image deformable registration, achieving better registration performance with fewer parameters.

SparseMorph: A weakly-supervised lightweight sparse transformer for mono- and multi-modal deformable image registration

PurposeDeformable image registration (DIR) is crucial for improving the precision of clinical diagnosis. Recent Transformer-based DIR methods have shown promising performance by capturing long-range dependencies. Nevertheless, these methods still grapple with high computational complexity. This work aims to enhance the performance of DIR in both computational efficiency and registration accuracy. MethodsWe proposed a weakly-supervised lightweight Transformer model, named SparseMorph. To reduce computational complexity without compromising the representative feature capture ability, we designed a sparse multi-head self-attention (SMHA) mechanism. To accumulate representative features while preserving high computational efficiency, we constructed a multi-branch multi-layer perception (MMLP) module. Additionally, we developed an anatomically-constrained weakly-supervised strategy to guide the alignment of regions-of-interest in mono- and multi-modal images. ResultsWe assessed SparseMorph in terms of registration accuracy and computational complexity.Within the mono-modal brain datasets IXI and OASIS, our SparseMorph outperforms the state-of-the-art method TransMatch with improvements of 3.2 % and 2.9 % in DSC scores for MRI-to-CT registration tasks, respectively. Moreover, in the multi-modal cardiac dataset MMWHS, our SparseMorph shows DSC score improvements of 9.7 % and 11.4 % compared to TransMatch in MRI-to-CT and CT-to-MRI registration tasks, respectively. Notably, SparseMorph attains these performance advantages while utilizing 33.33 % of the parameters of TransMatch. ConclusionsThe proposed weakly-supervised deformable image registration model, SparseMorph, demonstrates efficiency in both mono- and multi-modal registration tasks, exhibiting superior performance compared to state-of-the-art algorithms, and establishing an effective DIR method for clinical applications.

Multimodal registration network with multi-scale feature-crossing.

A critical piece of information for prostate intervention and cancer treatment is provided by the complementary medical imaging modalities of ultrasound (US) and magnetic resonance imaging (MRI). Therefore, MRI-US image fusion is often required during prostate examination to provide contrast-enhanced TRUS, in which image registration is a key step in multimodal image fusion. We propose a novel multi-scale feature-crossing network for the prostate MRI-US image registration task. We designed a feature-crossing module to enhance information flow in the hidden layer by integrating intermediate features between adjacent scales. Additionally, an attention block utilizing three-dimensional convolution interacts information between channels, improving the correlation between different modal features. We used 100 cases randomly selected from The Cancer Imaging Archive (TCIA) for our experiments. A fivefold cross-validation method was applied, dividing the dataset into five subsets. Four subsets were used for training, and one for testing, repeating this process five times to ensure each subset served as the test set once. We test and evaluate our technique using fivefold cross-validation. The cross-validation trials result in a median target registration error of 2.20 mm on landmark centroids and a median Dice of 0.87 on prostate glands, both of which were better than the baseline model. In addition, the standard deviation of the dice similarity coefficient is 0.06, which suggests that the model is stable. We propose a novel multi-scale feature-crossing network for the prostate MRI-US image registration task. A random selection of 100 cases from The Cancer Imaging Archive (TCIA) was used to test and evaluate our approach using fivefold cross-validation. The experimental results showed that our method improves the registration accuracy. After registration, MRI and TURS images were more similar in structure and morphology, and the location and morphology of cancer were more accurately reflected in the images.

Whale optimization algorithm based rigid and non-rigid registration

Image registration has become one of the most widely used transformation techniques in satellite and medical imaging nowadays is image registration. Mapping of two or more than two images are known as registration of images. Multimodal images are those that are processed using the same registration model but were taken with different devices. In the current work, we introduce a multimodal image registration framework on which we have applied two meta-heuristic algorithms: the Whale Optimization Algorithm (WOA) and Particle Swarm Optimization (PSO), to reduce processing time and enhance the performance of both rigid and non-rigid multimodal registration frameworks. The outcomes of WOA and PSO based framework has been compared with each other with respect to both rigid and non-rigid frameworks.

Validation of a novel AI-based automated multimodal image registration of CBCT and intraoral scan aiding presurgical implant planning.

The objective of this study is to assess accuracy, time-efficiency and consistency of a novel artificial intelligence (AI)-driven automated tool for cone-beam computed tomography (CBCT) and intraoral scan (IOS) registration compared with manual and semi-automated approaches. A dataset of 31 intraoral scans (IOSs) and CBCT scans was used to validate automated IOS-CBCT registration (AR) when compared with manual (MR) and semi-automated registration (SR). CBCT scans were conducted by placing cotton rolls between the cheeks and teeth to facilitate gingival delineation. The time taken to perform multimodal registration was recorded in seconds. A qualitative analysis was carried out to assess the correspondence between hard and soft tissue anatomy on IOS and CBCT. In addition, a quantitative analysis was conducted by measuring median surface deviation (MSD) and root mean square (RMS) differences between registered IOSs. AR was the most time-efficient, taking 51.4 ± 17.2 s, compared with MR (840 ± 168.9 s) and SR approaches (274.7 ± 100.7 s). Both AR and SR resulted in significantly higher qualitative scores, favoring perfect IOS-CBCT registration, compared with MR (p = .001). Additionally, AR demonstrated significantly superior quantitative performance compared with SR, as indicated by low MSD (0.04 ± 0.07 mm) and RMS (0.19 ± 0.31 mm). In contrast, MR exhibited a significantly higher discrepancy compared with both AR (MSD = 0.13 ± 0.20 mm; RMS = 0.32 ± 0.14 mm) and SR (MSD = 0.11 ± 0.15 mm; RMS = 0.40 ± 0.30 mm). The novel AI-driven method provided an accurate, time-efficient, and consistent multimodal IOS-CBCT registration, encompassing both soft and hard tissues. This approach stands as a valuable alternative to manual and semi-automated registration approaches in the presurgical implant planning workflow.

Indirect deformable image registration using synthetic image generated by unsupervised deep learning

Background and purpose3D image registration is now common in many medical domains. Multimodal registration implies the use of different imaging modalities, which results in lower accuracy compared to monomodal registration. The aim of this study was to propose a novel approach for deformable image registration (DIR) that incorporates an unsupervised deep learning (DL)-based generation step. The objective was to reduce the challenge of multimodal registration to monomodal registration. Material and methodsTwo datasets from prostate radiotherapy patients were used to evaluate the proposed method. The first dataset consisted of Computed Tomography (CT)/ Cone Beam Computed Tomography (CBCT) pairs from 23 patients using different CBCT devices. The second dataset included Magnetic Resonance Imaging (MRI)/CT pairs from two different care centers, utilizing different MRI devices (0.35 T MRIdian MR-Linac, 1.5 T GE lightspeed MRI). Following a preprocessing step essential for ensuring DL synthesis accuracy and standardizing the database, synthetic CTs (sCTreg) were generated using an unsupervised conditional Generative Adversarial Network (cGAN). The generated sCTs from CBCT or MRI were then utilized for deformable registration with CT scans. This registration method was compared to three standard methods: rigid registration, Elastix registration based on BSplines, and VoxelMorph-based registration (applied exclusively to CBCT/CT). The endpoints of comparison were the dice coefficients calculated between delineated structures for both datasets. ResultsFor both datasets, intermediary sCT generation provided the highest dice coefficients. Dices reached 0.85, 0.85 and 0.75 for the prostate, bladder and rectum for the dataset 1 and 0.90, 0.95 and 0.87 respectively for the dataset 2. When the sCT were not used, dices reached 0.66, 0.78, 0.66 for the dataset 1 and 0.93, 0.87 and 0.84 for the dataset 2. Furthermore, the evaluation of the impact of registration on sCT generation showed that lower Mean Absolute Errors were obtained when the registration was conducted with a sCT. ConclusionsUsing unsupervised deep learning to synthesize intermediate sCT has led to improved registration accuracy in radiotherapy applications employing two distinct imaging modalities.

A robust and accurate feature matching method for multi-modal geographic images spatial registration

ABSTRACT While the current research has achieved satisfactory results for the registration of single mode data, there has always been a significant challenge in the registration of multi-modal images due to the obvious nonlinear radiation differences caused by different imaging mechanisms and imaging time. For example, multi-temporal visible, visible-synthetic aperture radar, visible-near infrared, near infrared-short wave infrared, visible-MAP, etc. To address this problem, we propose a Robust and Accurate Feature Matching Method for Multi-modal Geographic Images Spatial Registration (RAMMR) to fully extract common key points between images, weaken the radiation difference between data, and finally accurately match more inliers to realize multi-modal image registration. Considering the influence of noise and edge information on key point extraction, RAMMR first constructs a new scale space by introducing the Side Window Filter (SWF); Then, we improve Harris algorithm to extract key points based on the SWF scale space; After that, we propose an enhanced log-polar descriptor based on the gradient angles and gradient amplitudes of the scale space, which effectively improves the quality of the descriptor and avoids the mismatch of key points; Based on the standard Euclidean distance, we design a re-match strategy to obtain the initial matching results, and Random Sample Consensus (RANSAC) is used to eliminate outliers. Finally, the affine transformation parameters are calculated based on inliers, and multi-modal image registration is realized. RAMMR is evaluated on different multi-modal datasets and compared with some state-of-art methods. The experimental results show that RAMMR accurately registers multi-modal geographic images and obtains comparative results compared with benchmark methods. Our source datasets are publicly available at https://github.com/RSmfmr/multimodal-dataset.

Overview and Benchmark on Multi-Modal Lidar Point Cloud Registration for Forest Applications

Abstract. Light Detection and Ranging (LIDAR) is widely acknowledged as a robust tool for monitoring forest structure, dynamics, and changes. To achieve a high-complete forest structural model, LiDAR data acquisition from both aerial (above-canopy) and terrestrial (below-canopy) platforms is commonplace. Consequently, in such multi-modal LiDAR cases, robust data registration is required for accurate forest analysis, such as biomass and canopy growth. Yet, multi-modal LiDAR registration remains a significant challenge due to differences in observation perspectives, deficient data overlap, and often inhomogeneity in point distributions and densities. The challenge increases in complex forest environments due to the abundance of unstable features (e.g., leaves) and occlusions. Thus, the dynamic nature of forest scenes needs to be considered when applying registration methods on forest point clouds. In this paper, we overview the latest advancements in registering forest point clouds from multi-modal data acquisitions, aiming to discuss the strengths and weaknesses of the most used LiDAR registration methods for forest applications. To support our investigations, we benchmark two multi-modal registration methods especially designed for forest mapping against traditional global and feature-based approaches. Experiment assessments were conducted using two point clouds acquired from a permanent laser scanning and airborne laser scanning systems at a boreal forest plot.

Learning Multi-Modal Cross-Scale Deformable Transformer Network for Unregistered Hyperspectral Image Super-resolution

Hyperspectral image super-resolution (HSI-SR) is a technology to improve the spatial resolution of HSI. Existing fusion-based SR methods have shown great performance, but still have some problems as follows: 1) existing methods assume that the auxiliary image providing spatial information is strictly registered with the HSI, but images are difficult to be registered finely due to the shooting platforms, shooting viewpoints and the influence of atmospheric turbulence; 2) most of the methods are based on convolutional neural networks (CNNs), which is effective for local features but cannot utilize the global features. To this end, we propose a multi-modal cross-scale deformable transformer network (M2DTN) to achieve unregistered HSI-SR. Specifically, we formulate a spectrum-preserving based spatial-guided registration-SR unified model (SSRU) from the view of the realistic degradation scenarios. According to SSRU, we propose multi-modal registration deformable module (MMRD) to align features between different modalities by deformation field. In order to efficiently utilize the unique information between different modals, we design multi-scale feature transformer (MSFT) to emphasize the spatial-spectral features at different scales. In addition, we propose the cross-scale feature aggregation module (CSFA) to accurately reconstruct the HSI by aggregating feature information at different scales. Experiments show that M2DTN outperforms the-state-of-the-art HSI-SR methods. Code is obtainable at https://github.com/Jiahuiqu/M2DTN.

MMORF—FSL’s MultiMOdal Registration Framework

Abstract We present MMORF—FSL’s MultiMOdal Registration Framework—a newly released nonlinear image registration tool designed primarily for application to magnetic resonance imaging (MRI) images of the brain. MMORF is capable of simultaneously optimising both displacement and rotational transformations within a single registration framework by leveraging rich information from multiple scalar and tensor modalities. The regularisation employed in MMORF promotes local rigidity in the deformation, and we have previously demonstrated how this effectively controls both shape and size distortion, leading to more biologically plausible warps. The performance of MMORF is benchmarked against three established nonlinear registration methods—FNIRT, ANTs, and DR-TAMAS—across four domains: FreeSurfer label overlap, diffusion tensor imaging (DTI) similarity, task-fMRI cluster mass, and distortion. The evaluation is based on 100 unrelated subjects from the Human Connectome Project (HCP) dataset registered to the Oxford-MultiModal-1 (OMM-1) multimodal template via either the T1w contrast alone or in combination with a DTI/DTI-derived contrast. Results show that MMORF is the most consistently high-performing method across all domains—both in terms of accuracy and levels of distortion. MMORF is available as part of FSL, and its inputs and outputs are fully compatible with existing workflows. We believe that MMORF will be a valuable tool for the neuroimaging community, regardless of the domain of any downstream analysis, providing state-of-the-art registration performance that integrates into the rich and widely adopted suite of analysis tools in FSL.

Multiscale biochemical mapping of the brain through deep-learning-enhanced high-throughput mass spectrometry

Spatial omics technologies can reveal the molecular intricacy of the brain. While mass spectrometry imaging (MSI) provides spatial localization of compounds, comprehensive biochemical profiling at a brain-wide scale in three dimensions by MSI with single-cell resolution has not been achieved. We demonstrate complementary brain-wide and single-cell biochemical mapping using MEISTER, an integrative experimental and computational mass spectrometry (MS) framework. Our framework integrates a deep-learning-based reconstruction that accelerates high-mass-resolving MS by 15-fold, multimodal registration creating three-dimensional (3D) molecular distributions and a data integration method fitting cell-specific mass spectra to 3D datasets. We imaged detailed lipid profiles in tissues with millions of pixels and in large single-cell populations acquired from the rat brain. We identified region-specific lipid contents and cell-specific localizations of lipids depending on both cell subpopulations and anatomical origins of the cells. Our workflow establishes a blueprint for future development of multiscale technologies for biochemical characterization of the brain.

Three-Dimensional Multi-Modality Registration for Orthopaedics and Cardiovascular Settings: State-of-the-Art and Clinical Applications.

The multimodal and multidomain registration of medical images have gained increasing recognition in clinical practice as a powerful tool for fusing and leveraging useful information from different imaging techniques and in different medical fields such as cardiology and orthopedics. Image registration could be a challenging process, and it strongly depends on the correct tuning of registration parameters. In this paper, the robustness and accuracy of a landmarks-based approach have been presented for five cardiac multimodal image datasets. The study is based on 3D Slicer software and it is focused on the registration of a computed tomography (CT) and 3D ultrasound time-series of post-operative mitral valve repair. The accuracy of the method, as a function of the number of landmarks used, was performed by analysing root mean square error (RMSE) and fiducial registration error (FRE) metrics. The validation of the number of landmarks resulted in an optimal number of 10 landmarks. The mean RMSE and FRE values were 5.26 ± 3.17 and 2.98 ± 1.68 mm, respectively, showing comparable performances with respect to the literature. The developed registration process was also tested on a CT orthopaedic dataset to assess the possibility of reconstructing the damaged jaw portion for a pre-operative planning setting. Overall, the proposed work shows how 3D Slicer and registration by landmarks can provide a useful environment for multimodal/unimodal registration.

L2NLF: a novel linear-to-nonlinear framework for multi-modal medical image registration.

In recent years, deep learning has ushered in significant development in medical image registration, and the method of non-rigid registration using deep neural networks to generate a deformation field has higher accuracy. However, unlike monomodal medical image registration, multimodal medical image registration is a more complex and challenging task. This paper proposes a new linear-to-nonlinear framework (L2NLF) for multimodal medical image registration. The first linear stage is essentially image conversion, which can reduce the difference between two images without changing the authenticity of medical images, thus transforming multimodal registration into monomodal registration. The second nonlinear stage is essentially unsupervised deformable registration based on the deep neural network. In this paper, a brand-new registration network, CrossMorph, is designed, a deep neural network similar to the U-net structure. As the backbone of the encoder, the volume CrossFormer block can better extract local and global information. Booster promotes the reduction of more deep features and shallow features. The qualitative and quantitative experimental results on T1 and T2 data of 240 patients' brains show that L2NLF can achieve excellent registration effect in the image conversion part with very low computation, and it will not change the authenticity of the converted image at all. Compared with the current state-of-the-art registration method, CrossMorph can effectively reduce average surface distance, improve dice score, and improve the deformation field's smoothness. The proposed methods have potential value in clinical application.

Radiation-Variation Insensitive Coarse-to-Fine Image Registration for Infrared and Visible Remote Sensing Based on Zero-Shot Learning

Infrared and visible remote sensing image registration is significant for utilizing remote sensing images to obtain scene information. However, it is difficult to establish a large number of correct matches due to the difficulty in obtaining similarity metrics due to the presence of radiation variation between heterogeneous sensors, which is caused by different imaging principles. In addition, the existence of sparse textures in infrared images as well as in some scenes and the small number of relevant trainable datasets also hinder the development of this field. Therefore, we combined data-driven and knowledge-driven methods to propose a Radiation-variation Insensitive, Zero-shot learning-based Registration (RIZER). First, RIZER, as a whole, adopts a detector-free coarse-to-fine registration framework, and the data-driven methods use a Transformer based on zero-shot learning. Next, the knowledge-driven methods are embodied in the coarse-level matches, where we adopt the strategy of seeking reliability by introducing the HNSW algorithm and employing a priori knowledge of local geometric soft constraints. Then, we simulate the matching strategy of the human eye to transform the matching problem into a model-fitting problem and employ a multi-constrained incremental matching approach. Finally, after fine-level coordinate fine tuning, we propose an outlier culling algorithm that only requires very few iterations. Meanwhile, we propose a multi-scene infrared and visible remote sensing image registration dataset. After testing, RIZER achieved a correct matching rate of 99.55% with an RMSE of 1.36 and had an advantage in the number of correct matches, as well as a good generalization ability for other multimodal images, achieving the best results when compared to some traditional and state-of-the-art multimodal registration algorithms.

A Multimodal Point Cloud-Based Method for Tumor Localization in Robotic Ultrasound-Guided Radiotherapy.

Objectives: Part of the tumor localization methods in radiotherapy have poor real-time performance and may generate additional radiation. We propose a multimodal point cloud-based method for tumor localization in robotic ultrasound-guided radiotherapy, which only irradiates computed tomography (CT) during radiotherapy planning to avoid additional radiation. Methods: The tumor position was determined using the CT point cloud, and the red green blue depth (RGBD) point cloud was used to determine body surface scanning location corresponding to the tumor location. The relationship between the CT point cloud and RGBD point cloud was established through multi-modal point cloud registration. The point cloud was then used for robot tumor localization through coordinate transformation between camera and robot. Results: The maximum mean absolute error of the tumor location in the X, Y, and Z directions of the robot coordinate system were 0.781, 1.334, and 1.490 mm, respectively. The average point-to-point translation mean absolute error between the actual and predicted positions of the localization points was 1.847 mm. The maximum error in the random positioning experiment was 1.77 mm. Conclusion: The proposed method is radiation free and has real-time performance, with tumor localization accuracy that meets the requirements of radiotherapy. The proposed method, which potentially reduces the risks associated with radiation exposure while ensuring efficient and accurate tumor localization, represents a promising advancement in the field of radiotherapy.

Correlation Attention Registration Based on Deep Learning from Histopathology to MRI of Prostate.

Deep learning offers a promising methodology for the registration of prostate cancer images from histopathology to MRI. We explored how to effectively leverage key information from images to achieve improved end-to-end registration. We developed an approach based on a correlation attention registration framework to register segmentation labels of histopathology onto MRI. The network was trained using paired prostate datasets of histopathology and MRI from the Cancer Imaging Archive. We introduced An L2-Pearson correlation layer to enhance feature matching. Furthermore, our model employed an enhanced attention regression network to distinguish between key and nonkey features. For data analysis, we used the Kolmogorov-Smirnov test and a one-sample t-test, with the statistical significance level for the one-sample t-test set at 0.001. Compared with two other models (ProsRegNet and CNNGeo), our model exhibited improved performance in Dice coefficient, with increases of 9.893% and 2.753%, respectively. The Hausdorff distance was reduced by approximately 50% and 50%, while the average label error (ALE) was reduced by 0.389% and 15.021%. The proposed improved multimodal prostate registration framework demonstrated high performance in statistical analysis. The results indicate that our enhanced strategy significantly improves registration performance and enables faster registration of histopathological images of patients undergoing radical prostatectomy to preoperative MRI. More accurate registration can prevent over-diagnosing low-risk cancers and frequent false positives due to observer differences.

Framework for Deep Learning Based Multi-Modality Image Registration of Snapshot and Pathology Images.

Multi-modality image registration is an important task in medical imaging because it allows for information from different domains to be correlated. Histopathology plays a crucial role in oncologic surgery as it is the gold standard for investigating tissue composition from surgically excised specimens. Research studies are increasingly focused on registering medical imaging modalities such as white light cameras, magnetic resonance imaging, computed tomography, and ultrasound to pathology images. The main challenge in registration tasks involving pathology images comes from addressing the considerable amount of deformation present. This work provides a framework for deep learning-based multi-modality registration of microscopic pathology images to another imaging modality. The proposed framework is validated on the registration of prostate ex-vivo white light camera snapshot images to pathology hematoxylin-eosin images of the same specimen. A pipeline is presented detailing data acquisition, protocol considerations, image dissimilarity, training experiments, and validation. A comprehensive analysis is done on the impact of pre-processing, data augmentation, loss functions, and regularization. This analysis is supplemented by clinically motivated evaluation metrics to avoid the pitfalls of only using ubiquitous image comparison metrics. Consequently, a robust training configuration capable of performing the desired registration task is found. Utilizing the proposed approach, we achieved a dice similarity coefficient of 0.96, a mutual information score of 0.54, a target registration error of 2.4 mm, and a regional dice similarity coefficient of 0.70.

Volumetric rigid MR-CT registration for glioblastoma in radiation oncology: A Novel approach

Glioblastoma multiforme (GBM), is a highly aggressive and rapidly growing grade IV astrocytoma that originates from astrocytes and causes a diverse range of neurological problems by infiltratingsurrounding brain tissue. Image guided radiotherapy is a crucial treatment option for GBM; however, precise target delineation is critical for efficient treatment in the presence of noise and distortion in the magnetic resonance (MR) and computed tomography (CT) images. Multimodal image registration, which generates aligned or fused images from different modalities, can enhance target delineation accuracy. The objective of this study was to evaluate the effectiveness of volumetric multimodal MR-CT rigid registration algorithms for the patients with GBM and to enhance the success of radiotherapy treatment responses by employing visual inspection and statistical assessment of the registered images. Specialized algorithms were developed to attain volumetric registration, leading to an enhanced target delineation for radiation treatment. The crucial role of multimodal rigid registration in improving the accuracy of target delineation in the context of radiation oncology for GBM was underscored by this study.

DTR-GAN: An Unsupervised Bidirectional Translation Generative Adversarial Network for MRI-CT Registration

Medical image registration is a fundamental and indispensable element in medical image analysis, which can establish spatial consistency among corresponding anatomical structures across various medical images. Since images with different modalities exhibit different features, it remains a challenge to find their exact correspondence. Most of the current methods based on image-to-image translation cannot fully leverage the available information, which will affect the subsequent registration performance. To solve the problem, we develop an unsupervised multimodal image registration method named DTR-GAN. Firstly, we design a multimodal registration framework via a bidirectional translation network to transform the multimodal image registration into a unimodal registration, which can effectively use the complementary information of different modalities. Then, to enhance the quality of the transformed images in the translation network, we design a multiscale encoder–decoder network that effectively captures both local and global features in images. Finally, we propose a mixed similarity loss to encourage the warped image to be closer to the target image in deep features. We extensively evaluate methods for MRI-CT image registration tasks of the abdominal cavity with advanced unsupervised multimodal image registration approaches. The results indicate that DTR-GAN obtains a competitive performance compared to other methods in MRI-CT registration. Compared with DFR, DTR-GAN has not only obtained performance improvements of 2.35% and 2.08% in the dice similarity coefficient (DSC) of MRI-CT registration and CT-MRI registration on the Learn2Reg dataset but has also decreased the average symmetric surface distance (ASD) by 0.33 mm and 0.12 mm on the Learn2Reg dataset.

Feature-based multimodal registration framework for vertebral pose estimation

PurposeThe reliable estimation of the vertebral body posture helps to aid a safe and effective spine surgery. The proposed work aims to present an MR to X-ray image registration to assess the 3D pose of the vertebral body during spine surgery. The 3D assessment of vertebral pose assists in analyzing the position and orientation of the vertebral body to provide information during various clinical diagnosis conditions such as curvature estimation and pedicle screw insertion surgery.MethodsThe proposed feature-based registration framework extracted vertebral end plates to avoid the mismatch between the intensities of MR and X-ray images. Using the projection matrix, the segmented MRI is forward projected and then registered to the X-ray image using binary image matching similarity and the CMA-ES optimizer.ResultsThe proposed method estimated the vertebral pose by registering the simulated X-ray onto pre-operative MRI. To evaluate the efficacy of the proposed approach, a certain number of experiments are carried out on the simulated dataset.ConclusionThe proposed method is a fast and accurate registration method that can provide 3D information about the vertebral body. This 3D information is useful to improve accuracy during various clinical diagnoses.