Multimodal Registration Research Articles

Improving the accuracy of the PET/MRI tridimensional multimodal rigid image registration based on the FATEMD

The subject matter of the article is the improvement in the accuracy of multimodal image registration between PET and MRI images in the medical field. The focus of the article pertains to the importance of these images in diagnosis, interpretation, and surgical intervention. This study increased the accuracy of PET/MRI multimodal image registration achieved through a new approach based on the multi-resolution image decomposition. The tasks to be solved are: The study proposes a new method, the fast and adaptive three-dimensional mode decomposition (FATEMD), to generate multi-resolution components for accurate registration. The method used: The study uses the FATEMD approach, which estimates the transformation parameters of the registration from the PET image and the residue of the second level of the MRI image that is obtained after the extraction of the first two tridimensional intrinsic mode functions (TIMFs). The following results were obtained: The proposed method of multimodal registration between PET and MRI images involves the use of the fast and adaptive three-dimensional mode decomposition (FATEMD) approach. This approach was tested on 25 pairs of images from the Vanderbilt database and was found to have improved accuracy compared to the usual method, as shown through comparative studies using measures of mutual information, normalized mutual information, and entropy correlation coefficient. Conclusion. The main objective achieved in the study was to enhance the accuracy of PET/MRI multimodal image registration through the application of the FATEMD decomposition method. This approach is novel compared to traditional methods as it involves estimating the transformation parameters from the PET image and the second level residue of the MRI image, resulting in more precise outcomes as opposed to using just the PET and MRI images alone. The integration of multiple imaging techniques, such as PET and MRI, provides healthcare professionals with a more comprehensive view of a patient's anatomy and physiology, leading to enhanced diagnosis and treatment planning.

Abstract No. 49 Motion Compensation in 3D MRI-US Fusion Using Fast Deformable Registration: A Feasibility Study for Real-Time Intervention

The heterogeneity of hepatocellular carcinoma may be better seen with magnetic resonance imaging (MRI) compared with CT due to its high soft tissue contrast, providing precise tumor targets during therapy, while ultrasound (US) remains as the imaging modality for real-time guidance. Image fusion methods existing in clinical workflows involve rigid registration only and fail to compensate for liver motion in US. In this work, we present a hybrid deformable fusion method to align pre-interventional 3D MRI and interventional 3D US in real-time. Multimodal pre-interventional MRI (pMRI) and US (pUS) volumes were obtained from 3 human volunteers using a simultaneous MRI-US acquisition system, with an MR-compatible, hands-free US probe. pMRI and pUS volumes were aligned using conventional deformable registration, as it is not time critical. Deep learning (DL)–based registration was used for real-time fusion of pUS to iUS, and consecutive iUS volumes in near real time. The predicted DL deformation fields were used to deform the pMRI to match each US volume. US data with respiration were collected at a temporal resolution of 4.2 volumes/sec. 1600 US volumes from one volunteer was used for DL training. The hybrid deformable registration method was evaluated for pMRI and 20 US volume alignments for each volunteer. Mean Euclidean distance error between expert placed landmarks and predicted positions of landmarks after image alignment were computed. Table 1 shows the mean landmark error (LE) and computation times for the hybrid (HDR) and conventional deformable registration (CDR) method aligning pMRI and 20 US volumes. Feasibility of a multimodal hybrid deformable registration method with clinically acceptable registration accuracy and low latency was shown. The method for motion compensation may improve tumor targeting in interventional procedures including liver ablation.

Lesion-specific 3D-printed moulds for image-guided tissue multi-sampling of ovarian tumours: A prospective pilot study.

High-Grade Serous Ovarian Carcinoma (HGSOC) is the most prevalent and lethal subtype of ovarian cancer, but has a paucity of clinically-actionable biomarkers due to high degrees of multi-level heterogeneity. Radiogenomics markers have the potential to improve prediction of patient outcome and treatment response, but require accurate multimodal spatial registration between radiological imaging and histopathological tissue samples. Previously published co-registration work has not taken into account the anatomical, biological and clinical diversity of ovarian tumours. In this work, we developed a research pathway and an automated computational pipeline to produce lesion-specific three-dimensional (3D) printed moulds based on preoperative cross-sectional CT or MRI of pelvic lesions. Moulds were designed to allow tumour slicing in the anatomical axial plane to facilitate detailed spatial correlation of imaging and tissue-derived data. Code and design adaptations were made following each pilot case through an iterative refinement process. Five patients with confirmed or suspected HGSOC who underwent debulking surgery between April and December 2021 were included in this prospective study. Tumour moulds were designed and 3D-printed for seven pelvic lesions, covering a range of tumour volumes (7 to 133 cm3) and compositions (cystic and solid proportions). The pilot cases informed innovations to improve specimen and subsequent slice orientation, through the use of 3D-printed tumour replicas and incorporation of a slice orientation slit in the mould design, respectively. The overall research pathway was compatible with implementation within the clinically determined timeframe and treatment pathway for each case, involving multidisciplinary clinical professionals from Radiology, Surgery, Oncology and Histopathology Departments. We developed and refined a computational pipeline that can model lesion-specific 3D-printed moulds from preoperative imaging for a variety of pelvic tumours. This framework can be used to guide comprehensive multi-sampling of tumour resection specimens.

Multimodal registration of ultrasound and MR images using weighted self-similarity structure vector

ProposeMultimodal registration of 2D Ultrasound (US) and 3D Magnetic Resonance (MR) for fusion navigation can improve the intraoperative detection accuracy of lesion. However, multimodal registration remains a challenge because of the poor US image quality. In the study, a weighted self-similarity structure vector (WSSV) is proposed to registrate multimodal images. MethodThe self-similarity structure vector utilizes the normalized distance of symmetrically located patches in the neighborhood to describe the local structure information. The texture weights are extracted using the local standard deviation to reduce the speckle interference in the US images. The multimodal similarity metric is constructed by combining a self-similarity structure vector with a texture weight map. ResultsExperiments were performed on US and MR images of the liver from 88 groups of data including 8 patients and 80 simulated samples. The average target registration error was reduced from 14.91 ± 3.86 mm to 4.95 ± 2.23 mm using the WSSV-based method. ConclusionsThe experimental results show that the WSSV-based registration method could robustly align the US and MR images of the liver. With further acceleration, the registration framework can be potentially applied in time-sensitive clinical settings, such as US-MR image registration in image-guided surgery.

PCNet: A structure similarity enhancement method for multispectral and multimodal image registration

Multispectral and multimodal images are of important usage in the field of multi-source visual information fusion. Due to the alternation or movement of image devices, the acquired multispectral and multimodal images are usually misaligned, and hence image registration is pre-requisite. Different from the registration of common images, the registration of multispectral or multimodal images is a challenging problem due to the nonlinear variation of intensity and gradient. To cope with this challenge, we propose the phase congruency network (PCNet) to enhance the structure similarity of multispectral or multimodal images. The images can then be aligned using the similarity-enhanced feature maps produced by the network. PCNet is constructed under the inspiration of the well-known phase congruency. The network embeds the phase congruency prior into two simple trainable layers and series of modified learnable Gabor kernels. Thanks to the prior knowledge, once trained, PCNet is applicable on a variety of multispectral and multimodal data such as flash/no-flash and RGB/NIR images without additional further tuning. The prior also makes the network lightweight. The trainable parameters of PCNet is 2400× and 1500×less than the deep-learning registration method deep homography network (DHN) and unsupervised deep homography network (UDHN), while its registration performance surpasses them. Experimental results validate that PCNet outperforms current state-of-the-art conventional multimodal registration algorithms. Besides, PCNet can act as a complementary part of the deep-learning registration methods, which significantly boosts their registration accuracy. On the Columbia imaging and vision laboratory (CAVE) multispectral dataset, the percentage of the number of images under 1 pixel average corner error (ACE) of UDHN is raised from 0.1% to 82.5% after the processing of PCNet.

QACL: Quartet attention aware closed-loop learning for abdominal MR-to-CT synthesis via simultaneous registration.

Synthesis of computed tomography (CT) images from magnetic resonance (MR) images is an important task to overcome the lack of electron density information in MR-only radiotherapy treatment planning (RTP). Some innovative methods have been proposed for abdominal MR-to-CT synthesis. However, it is still challenging due to the large misalignment between preprocessed abdominal MR and CT images and the insufficient feature information learned by models. Although several studies have used the MR-to-CT synthesis to alleviate the difficulty of multi-modal registration, this misalignment remains unsolved when training the MR-to-CT synthesis model. In this paper, we propose an end-to-end quartet attention aware closed-loop learning (QACL) framework for MR-to-CT synthesis via simultaneous registration. Specifically, the proposed quartet attention generator and mono-modal registration network form a closed-loop to improve the performance of MR-to-CT synthesis via simultaneous registration. In particular, a quartet-attention mechanism is developed to enlarge the receptive fields in networks to extract the long-range and cross-dimension spatial dependencies. Experimental results on two independent abdominal datasets demonstrate that our QACL achieves impressive results with MAE of 55.30±10.59 HU, PSNR of 22.85±1.43 dB, and SSIM of 0.83±0.04 for synthesis, and with Dice of 0.799±0.129 for registration. The proposed QACL outperforms the state-of-the-art MR-to-CT synthesis and multi-modal registration methods.

Contrast-agnostic deep learning–based registration pipeline: Validation in spinal cord multimodal MRI data

Medical image registration can be challenging, in that optimal solutions depend on the application domain (unimodal, multimodal, intra-subject and inter-subject), anatomical sites (e.g., brain, spinal cord (SC) and lungs), dimensionality of the data (2D, 3D and 4D), deformation constraints (rigid, affine and nonlinear) and computational time. Solutions that could accommodate a large variety of applications while producing satisfactory results are needed. SynthMorph was recently introduced as an unsupervised deep learning–based registration method. A particularly interesting feature is that training is performed on synthetic data so that registration becomes agnostic to image contrast and anatomy. However, SynthMorph is particularly sensitive to the initial closeness of the images. In this work, we extend the SynthMorph method by developing a cascaded pipeline of two models that can accommodate large and fine deformations, respectively. We also validate this pipeline for the registration of intra-subject multimodal and inter-subject uni/multimodal MRI data of the SC. This task is known to be particularly difficult due to the vicinity of multiple tissue types whose morphometrics can vary substantially across subjects and contrasts. Evaluation of the method was conducted on a publicly available dataset (spine-generic, 267 subjects) and was compared with a state-of-the-art benchmark: Spinal Cord Toolbox and Advanced Normalization Tools. Results demonstrate better registration accuracy compared with the benchmark and about 24–30 times faster on CPUs depending on the image size. This proposed pipeline provides an easy-to-use, accurate and fast solution for multimodal 3D registration. The code and trained models are freely available at https://github.com/ivadomed/multimodal-registration.

Image blurring and spectral drift in imaging spectrometer system with an acousto-optic tunable filter and its application in UAV remote sensing

ABSTRACT In the design and research of an unmanned aerial vehicle-borne hyperspectral imager with an acousto-optic tunable filter (AOTF), the depth of the field range of the imaging objective lens is fixed, the spectral range of 400–1000nm is wide, and the AOTF crystal has different deflection angles for diffracted light of different wavelengths. These features all cause problems such as image blurring and spectral drifts. Therefore, we propose a solution using a tunable zoom lens and multi-modal spectrum registration to solve these problems. We also propose an improved image definition evaluation function using the zoom lens to obtain clear horizontal lines and blurred vertical lines and, thereby, hope to overcome image blurring in an AOTF spectrometer. The zoom parameters corresponding to the clear image in each spectral segment of the AOTF spectrometer can be determined rapidly and accurately. However, it is difficult to establish a mathematical model of the drift, using wavelength as a variable, to compensate the drift due to the clear image of each spectrum obtained using the zoom lens. The calibration method completes the registration of the spectral data cube composed of clear images, solving the problem of spectral band drift. Moreover, it has been verified through UAV-borne hyperspectral imaging that image blurring and spectral drift of the AOTF imaging spectrometer were significantly reduced, providing opportunities for the application and research of AOTF-based hyperspectral imagers in more fields, such as remote sensing.

Effect of magnetic resonance imaging pre-processing on the performance of model-based prostate tumor probability mapping

Objective. Multi-parametric magnetic resonance imaging (mpMRI) has become an important tool for the detection of prostate cancer in the past two decades. Despite the high sensitivity of MRI for tissue characterization, it often suffers from a lack of specificity. Several well-established pre-processing tools are publicly available for improving image quality and removing both intra- and inter-patient variability in order to increase the diagnostic accuracy of MRI. To date, most of these pre-processing tools have largely been assessed individually. In this study we present a systematic evaluation of a multi-step mpMRI pre-processing pipeline to automate tumor localization within the prostate using a previously trained model. Approach. The study was conducted on 31 treatment-naïve prostate cancer patients with a PI-RADS-v2 compliant mpMRI examination. Multiple methods were compared for each pre-processing step: (1) bias field correction, (2) normalization, and (3) deformable multi-modal registration. Optimal parameter values were estimated for each step on the basis of relevant individual metrics. Tumor localization was then carried out via a model-based approach that takes both mpMRI and prior clinical knowledge features as input. A sequential optimization approach was adopted for determining the optimal parameters and techniques in each step of the pipeline. Main results. The application of bias field correction alone increased the accuracy of tumor localization (area under the curve (AUC) = 0.77; p-value = 0.004) over unprocessed data (AUC = 0.74). Adding normalization to the pre-processing pipeline further improved diagnostic accuracy of the model to an AUC of 0.85 (p-value = 0.000 12). Multi-modal registration of apparent diffusion coefficient images to T2-weighted images improved the alignment of tumor locations in all but one patient, resulting in a slight decrease in accuracy (AUC = 0.84; p-value = 0.30). Significance. Overall, our findings suggest that the combined effect of multiple pre-processing steps with optimal values has the ability to improve the quantitative classification of prostate cancer using mpMRI. Clinical trials: NCT03378856 and NCT03367702.

Is image-to-image translation the panacea for multimodal image registration? A comparative study.

Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of rigid registration of multimodal biomedical and medical 2D and 3D images. We compare the performance of four Generative Adversarial Network (GAN)-based I2I translation methods and one contrastive representation learning method, subsequently combined with two representative monomodal registration methods, to judge the effectiveness of modality translation for multimodal image registration. We evaluate these method combinations on four publicly available multimodal (2D and 3D) datasets and compare with the performance of registration achieved by several well-known approaches acting directly on multimodal image data. Our results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I2I translation approach. The evaluated representation learning method, which aims to find abstract image-like representations of the information shared between the modalities, manages better, and so does the Mutual Information maximisation approach, acting directly on the original multimodal images. We share our complete experimental setup as open-source (https://github.com/MIDA-group/MultiRegEval), including method implementations, evaluation code, and all datasets, for further reproducing and benchmarking.

Cross-Modality Image Registration Using a Training-Time Privileged Third Modality.

In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWI [Formula: see text]). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWI [Formula: see text]) are considered easier to register to T2w due to the availability of corresponding features. We propose a learning from privileged modality algorithm, using a training-only imaging modality DWI [Formula: see text], to support the challenging multi-modality registration problems. We present experimental results based on 369 sets of 3D multiparametric MRI images from 356 prostate cancer patients and report, with statistical significance, a lowered median target registration error of 4.34 mm, when registering the holdout DWI [Formula: see text] and T2w image pairs, compared with that of 7.96 mm before registration. Results also show that the proposed learning-based registration networks enabled efficient registration with comparable or better accuracy, compared with a classical iterative algorithm and other tested learning-based methods with/without the additional modality. These compared algorithms also failed to produce any significantly improved alignment between DWI [Formula: see text] and T2w in this challenging application.

Building a tissue-unbiased brain template of fiber orientation distribution and tractography with multimodal registration.

Brain templates provide an essential standard space for statistical analysis of brain structure and function. Despite recent advances, diffusion MRI still lacks a template of fiber orientation distribution (FOD) and tractography that is unbiased for both white and gray matter. Therefore, we aim to build up a set of such templates for better white-matter analysis and joint structural and functional analysis. We have developed a multimodal registration method to leverage the complementary information captured by T1 -weighted, T2 -weighted, and diffusion MRI, so that a coherent transformation is generated to register FODs into a common space and average them into a template. Consequently, the anatomically constrained fiber-tracking method was applied to the FOD template to generate a tractography template. Fiber-centered functional connectivity analysis was then performed as an example of the benefits of such an unbiased template. Our FOD template preserves fine structural details in white matter and also, importantly, clear folding patterns in the cortex and good contrast in the subcortex. Quantitatively, our templates show better individual-template agreement at the whole-brain scale and segmentation scale. The tractography template aligns well with the gray matter, which led to fiber-centered functional connectivity showing high cross-group consistency. We have proposed a novel methodology for building a tissue-unbiased FOD and anatomically constrained tractography template based on multimodal registration. Our templates provide a standard space and statistical platform for not only white-matter analysis but also joint structural and functional analysis, therefore filling an important gap in multimodal neuroimage analysis.

Efficient convolution-based pairwise elastic image registration on three multimodal similarity metrics

This paper proposes a complete convolutional formulation for 2D multimodal pairwise image registration problems based on free-form deformations. We have reformulated in terms of discrete 1D convolutions the evaluation of spatial transformations, the regularization term, and their gradients for three different multimodal registration metrics, namely, normalized cross correlation, mutual information, and normalized mutual information. A sufficient condition on the metric gradient is provided for further extension to other metrics. The proposed approach has been tested, as a proof of concept, on contrast-enhanced first-pass perfusion cardiac magnetic resonance images. Execution times have been compared with the corresponding execution times of the classical tensor product formulation, both on CPU and GPU. The speed-up achieved by using convolutions instead of tensor products depends on the image size and the number of control points considered, the larger those magnitudes, the greater the execution time reduction. Furthermore, the speed-up will be more significant when gradient operations constitute the major bottleneck in the optimization process.

Mutual Information Neural Estimation for Unsupervised Multi-Modal Registration of Brain Images.

Many applications in image-guided surgery and therapy require fast and reliable non-linear, multi-modal image registration. Recently proposed unsupervised deep learning-based registration methods have demonstrated superior per-formance compared to iterative methods in just a fraction of the time. Most of the learning-based methods have focused on mono-modal image registration. The extension to multi-modal registration depends on the use of an appropriate similarity function, such as the mutual information (MI). We propose guiding the training of a deep learning-based registration method with MI estimation between an image-pair in an end-to-end trainable network. Our results show that a small, 2-layer network produces competitive results in both mono- and multi-modal registration, with sub-second run-times. Comparisons to both iterative and deep learning-based methods show that our MI-based method produces topologically and qualitatively superior results with an extremely low rate of non-diffeomorphic transformations. Real-time clinical application will benefit from a better visual matching of anatomical structures and less registration failures/outliers.

Investigation of Silicon-Based Anodes for Li-Ion Batteries Using X-Rays and Neutron 3D/4D Imaging Techniques

Silicon-based anode materials are one of the most promising approaches to further increase the energy density of lithium-ion batteries. However, current materials are limited by poor cycling stability and rapid capacity fading, mainly caused by the massive volume expansion of Si during lithiation and subsequent strain on the material composite. [1] Furthermore, this electrode swelling also results in continuous solid electrolyte interface (SEI) growth, which hinders the migration of Lithium and leads to permanent capacity loss. [2] [3]To optimize these materials, analytical techniques able to probe the local 3D morphology and Li content are necessary. By applying methods such as neutron computed tomography (NCT), X-Ray computed tomography (XCT) and synchrotron scattering micro-tomography (SMT) structural ageing and changes in the distribution of lithium in the various components can be characterized and quantified for different charging states and recharge cycle numbers.This study investigates an anode material based on a dual phase alloy system of amorphous Silicon (a-Si) with crystalline iron silicide (c-FeSi2) and graphite. [4] [5] Li-ion battery coin cells containing this silicon-graphite composite anode material were industrially produced and aged by performing 1, 300, and 700 charge-discharge cycles, corresponding to remaining capacities of 100, 70 and 50 %, respectively. NCT and XCT scans were acquired at the NeXT instrument of the ILL in Grenoble (France) [6], and SMT measurements performed at ID31 of the neighboring ESRF. These datasets were reconstructed and evaluated using specially developed data processing pipelines. By performing multi-modal registration, the NCT and XCT scans were aligned and segmented to combine the complementary datasets and create a 4D model of the Li-ion battery coin cells.Figure 1 shows a sample of the combined NCT (cyan) and XCT datasets (red). Cropped horizontal slices of Li-Ion battery coin cells are depicted side by side with the main components labelled. Comparing one of the highly cycled cells with 50% remaining capacity (top) with the reference cell without electrolyte (bottom), trapped Lithium can be observed in the anode layers, visible as cyan colored blotchy areas in the otherwise dark component. The pixel size is approximately 5 µm.The change of lithium distribution in the components between different cycle numbers and charge states in the attenuation-based tomography techniques of NCT / XCT is quantified by modelling expected attenuation parameters to the observed values. The diffraction-based tomography method of SMT was modeled using expected lattice parameters. From this information, trapped Lithium can be identified and changes in the distribution are analyzed to point out possible degradation and failure mechanisms.We acknowledge financial support from the European Union’s Horizon 2020 research and Innovation program No. 875514 (ECO2LIB) as well as No. 847439 (InnovaXN) under the Marie Skłodowska-Curie grant agreement.[1] S. Tardif et al, https://doi.org/10.1021/acsnano.7b05796 [2] T. Vorauer et al, https://doi.org/10.1038/s42004-020-00386-x [3] P. Kumar et al, https://doi.org/10.1002/smll.201906812 [4] C. Berhaut et al, https://doi.org/10.1021/acsnano.9b05055 [5] C. Berhaut et al, https://doi.org/10.1016/j.ensm.2020.04.008 [6] A. Tengattini et al, https://doi.org/10.1016/j.nima.2020.163939 Figure 1

Multimodal registration across 3D point clouds and CT-volumes

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology.

An improved multi-modal joint segmentation and registration model based on Bhattacharyya distance measure

Joint image segmentation and registration of multi-modality images is a crucial step in the field of image prepossessing. The sensitivity of joint segmentation and registration models to noise is a significant challenge. During the registration process of multi-modal images, the similarity measure plays a vital role in measuring the results as a standard. Accordingly, an improved joint model for registering and segmenting multi-modality images is proposed, by utilising the Bhattacharyya distance measure to achieve improved noise robustness of the proposed model as compared to the existing model using the mutual information metric. The proposed model is applied to various medical and synthetic noisy images of multiple modalities. Moreover, the dataset images used in this study have been obtained from well-known, freely available BRATS 2015 and CHAOS datasets, where the proposed model produces satisfactory results as compared to the existing model. Experimental results show that the proposed model outperforms the existing model in terms of the Bhattacharyya distance measure in noisy images. Statistical analysis and comparison are performed through the relative reduction of the new distance measure, Dice similarity coefficient, Jaccard similarity coefficient and Hausdorff distance.

Point cloud registration and change detection in urban environment using an onboard Lidar sensor and MLS reference data

This paper presents a new method for urban scene analysis, which comprises 3D point cloud registration and change detection through fusing Lidar point clouds with significantly different density characteristics. The introduced method is able to extract dynamic scene segments (traffic participants or urban renewal regions) and seasonal changes (vegetation regions) from instant 3D (i3D) measurements captured by a Rotating Multi-beam (RMB) Lidar sensor mounted onto the top of a moving vehicle. As reference data, we rely on a dense point cloud-based environment model provided by Mobile Laser Scanning (MLS) systems. The proposed approach is composed of new solutions for two main subtasks. First, a novel multimodal point cloud registration algorithm is introduced, which can improve the alignment of the sparse i3D measurements to the dense MLS data, where conventional point level registration or keypoint/segment matching strategies fail. Second, an efficient Markov Random Field-based change extraction step is implemented between the registered point clouds, which exploits that due to geometric considerations of mapping with the given sensor configuration, the essence of the problem can be solved quickly in the 2D range image domain without information loss. Experimental evaluation is conducted on a new Benchmark set that contains three different heavy traffic road sections in city center areas covering in total nearly 1 km long pathway sections. Test data consists of relevant industrial measurements provided by a state-of-the-art RMB scanner (with a point density of around 50-500 points/m2) and an up-to-date MLS system (more than 5000 points/m2). The clear advantages of the new method are quantitatively demonstrated against various reference techniques. In comparison to six different point cloud registration methods, the median value of point level distances is decreased by 1–2 orders of magnitude by the proposed approach. Regarding change detection, the new method outperforms the reference models either in F1-scores (by around 10–25%) or in computational complexity (10–1000 times faster).

Automatic 3D MRI-Ultrasound Registration for Image Guided Arthroscopy

Registration of partial view intra-operative ultrasound (US) to pre-operative MRI is an essential step in image-guided minimally invasive surgery. In this paper, we present an automatic, landmark-free 3D multimodal registration of pre-operative MRI to 4D US (high-refresh-rate 3D-US) for enabling guidance in knee arthroscopy. We focus on the problem of initializing registration in the case of partial views. The proposed method utilizes a pre-initialization step of using the automatically segmented structures from both modalities to achieve a global geometric initialization. This is followed by computing distance maps of the procured segmentations for registration in the distance space. Following that, the final local refinement between the MRI-US volumes is achieved using the LC2 (Linear correlation of linear combination) metric. The method is evaluated on 11 cases spanning six subjects, with four levels of knee flexion. A best-case error of 1.41 mm and 2.34∘ and an average registration error of 3.45 mm and 7.76∘ is achieved in translation and rotation, respectively. An inter-observer variability study is performed, and a mean difference of 4.41 mm and 7.77∘ is reported. The errors obtained through the developed registration algorithm and inter-observer difference values are found to be comparable. We have shown that the proposed algorithm is simple, robust and allows for the automatic global registration of 3D US and MRI that can enable US based image guidance in minimally invasive procedures.

Reattachable fiducial skin marker for automatic multimodality registration

PurposeFusing image information has become increasingly important for optimal diagnosis and treatment of the patient. Despite intensive research towards markerless registration approaches, fiducial marker-based methods remain the default choice for a wide range of applications in clinical practice. However, as especially non-invasive markers cannot be positioned reproducibly in the same pose on the patient, pre-interventional imaging has to be performed immediately before the intervention for fiducial marker-based registrations.MethodsWe propose a new non-invasive, reattachable fiducial skin marker concept for multi-modal registration approaches including the use of electromagnetic or optical tracking technologies. We furthermore describe a robust, automatic fiducial marker localization algorithm for computed tomography (CT) and magnetic resonance imaging (MRI) images. Localization of the new fiducial marker has been assessed for different marker configurations using both CT and MRI. Furthermore, we applied the marker in an abdominal phantom study. For this, we attached the marker at three poses to the phantom, registered ten segmented targets of the phantom’s CT image to live ultrasound images and determined the target registration error (TRE) for each target and each marker pose.ResultsReattachment of the marker was possible with a mean precision of 0.02 mm ± 0.01 mm. Our algorithm successfully localized the marker automatically in all (n=201) evaluated CT/MRI images. Depending on the marker pose, the mean (n=10) TRE of the abdominal phantom study ranged from 1.51 ± 0.75 mm to 4.65 ± 1.22 mm.ConclusionsThe non-invasive, reattachable skin marker concept allows reproducible positioning of the marker and automatic localization in different imaging modalities. The low TREs indicate the potential applicability of the marker concept for clinical interventions, such as the puncture of abdominal lesions, where current image-based registration approaches still lack robustness and existing marker-based methods are often impractical.