Registration Task Research Articles

IMFNet: Interpretable Multimodal Fusion for Point Cloud Registration

The existing state-of-the-art point descriptor relies on structure information only, which omits the texture information. However, texture information is crucial for our humans to distinguish a scene part. Moreover, the current learning-based point descriptors are all black boxes which are unclear how the original points contribute to the final descriptor. This paper proposes a new multimodal fusion method to generate a point cloud registration descriptor by considering structure and texture information. Specifically, a novel attention-fusion module is designed to extract the weighted texture information for descriptor extraction. In addition, we propose an interpretable module to explain our neural network by visually showing the original points contributing to the final descriptor. We use the descriptor's channel value as the loss to backpropagate to the target layer and consider the gradient as the significance of this point to the final descriptor. This paper moves one step further to explainable deep learning in the registration task. Comprehensive experiments on 3DMatch, 3DLoMatch and KITTI demonstrate that the multimodal fusion descriptor achieves state-of-the-art accuracy and improves the descriptor's distinctiveness. We also demonstrate our interpretable module in explaining the registration descriptor extraction.

MID: A Novel Mountainous Remote Sensing Imagery Registration Dataset Assessed by a Coarse-to-Fine Unsupervised Cascading Network

The geometric registration of mountainous remote sensing images is always a challenging project, as terrain fluctuations increase the complexity. Deep learning, with its superior computing power and data-driven nature, promises to solve this problem. However, the lack of an appropriate dataset limits the development of deep learning technology for mountainous remote sensing image registration, which it still an unsolved problem in photogrammetry and remote sensing. To remedy this problem, this paper presents a manually constructed imagery dataset of mountainous regions, called the MID (mountainous imagery dataset). To create the MID, we use 38 images from the Gaofen-2 satellite developed by China and generated 4093 pairs of reference and sensed image patches, making this the first real mountainous dataset to our knowledge. Simultaneously, we propose a fully unsupervised, convolutional-network-based iterative registration scheme for the MID. First, the large and global deformation of the reference and sensed images is reduced using an affine registration module, generating the coarse alignment. Then, the local and varied distortions are learned and eliminated progressively using a hybrid dilated convolution (HDC)-based encoder–decoder module with multistep iterations, achieving fine registration results. The HDC aims to increase the receptive field without blocking the artifacts, allowing for the continuous characteristics of the mountainous images of a local region to be represented. We provide a performance analysis of some typical registration algorithms and the developed approach for the MID. The proposed scheme gives the highest registration precision, achieving the subpixel alignment of mountainous remote sensing images. Additionally, the experimental results demonstrate the usability of the MID, which can lay a foundation for the development of deep learning technology in large mountainous remote sensing image registration tasks.

Rethinking of learning-based 3D keypoints detection for large-scale point clouds registration

The main solution for large-scale point clouds registration is to first obtain a set of matched 3D keypoint pairs and then accomplish the point cloud registration task based on these matched keypoint pairs. However, at present, many methods study the feature descriptors in the point clouds registration task, but few methods discuss the 3D keypoints detection issues. The commonly used 3D keypoints detection strategy is the voxel-grid-based downsampling method, and the detected 3D keypoints are usually with a relatively huge amount and also with no explicit geometrical properties, which finally leads to a low inlier ratio. In this study, we rethink the 3D keypoints detection problem for large-scale point clouds with deep learning. Specifically, we discuss four kinds of 3D keypoints detection methods based on the joint keypoint detection and description learning framework D3Feat, and carry out extensive analyses on both the indoor large-scale point clouds dataset 3DMatch and the outdoor large-scale point clouds dataset KITTI Odometry. Experimental results demonstrate that the Multi-layer Perceptron (MLP) based method achieves the best inlier ratios under the different numbers of extracted 3D keypoints on both the indoor and outdoor large-scale point clouds. Further, we test these four kinds of keypoints detection methods under the application of large-scale point clouds registration, and the MLP-based method also achieves the state-of-the-art registration performance.

⊥-loss: A symmetric loss function for magnetic resonance imaging reconstruction and image registration with deep learning.

Convolutional neural networks (CNNs) are increasingly adopted in medical imaging, e.g., to reconstruct high-quality images from undersampled magnetic resonance imaging (MRI) acquisitions or estimate subject motion during an examination. MRI is naturally acquired in the complex domain C, obtaining magnitude and phase information in k-space. However, CNNs in complex regression tasks are almost exclusively trained to minimize the L2 loss or maximizing the magnitude structural similarity (SSIM), which are possibly not optimal as they do not take full advantage of the magnitude and phase information present in the complex domain. This work identifies that minimizing the L2 loss in the complex field has an asymmetry in the magnitude/phase loss landscape and is biased, underestimating the reconstructed magnitude. To resolve this, we propose a new loss function for regression in the complex domain called ⊥-loss, which adds a novel phase term to established magnitude loss functions, e.g., L2 or SSIM. We show ⊥-loss is symmetric in the magnitude/phase domain and has favourable properties when applied to regression in the complex domain. Specifically, we evaluate the ⊥+ℓ2-loss and ⊥+SSIM-loss for complex undersampled MR image reconstruction tasks and MR image registration tasks. We show that training a model to minimize the ⊥+ℓ2-loss outperforms models trained to minimize the L2 loss and results in similar performance compared to models trained to maximize the magnitude SSIM while offering high-quality phase reconstruction. Moreover, ⊥-loss is defined in Rn, and we apply the loss function to the R2 domain by learning 2D deformation vector fields for image registration. We show that a model trained to minimize the ⊥+ℓ2-loss outperforms models trained to minimize the end-point error loss.

DeepMatch: Toward Lightweight in Point Cloud Registration.

From source to target, point cloud registration solves for a rigid body transformation that aligns the two point clouds. IterativeClosest Point (ICP) and other traditional algorithms require a long registration time and are prone to fall into local optima. Learning-based algorithms such as Deep ClosestPoint (DCP) perform better than those traditional algorithms and escape from local optimality. However, they are still not perfectly robust and rely on the complex model design due to the extracted local features are susceptible to noise. In this study, we propose a lightweight point cloud registration algorithm, DeepMatch. DeepMatch extracts a point feature for each point, which is a spatial structure composed of each point itself, the center point of the point cloud, and the farthest point of each point. Because of the superiority of this per-point feature, the computing resources and time required by DeepMatch to complete the training are less than one-tenth of other learning-based algorithms with similar performance. In addition, experiments show that our algorithm achieves state-of-the-art (SOTA) performance on both clean, with Gaussian noise and unseen category datasets. Among them, on the unseen categories, compared to the previous best learning-based point cloud registration algorithms, the registration error of DeepMatch is reduced by two orders of magnitude, achieving the same performance as on the categories seen in training, which proves DeepMatch is generalizable in point cloud registration tasks. Finally, only our DeepMatch completes 100% recall on all three test sets.

Multimodal registration across 3D point clouds and CT-volumes

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology.

Nonfinite-modality data augmentation for brain image registration

Brain image registration is fundamental for brain medical image analysis. However, the lack of paired images with diverse modalities and corresponding ground truth deformations for training hinder its development. We propose a novel nonfinite-modality data augmentation for brain image registration to combat this. Specifically, some available whole-brain segmentation masks, including complete fine brain anatomical structures, are collected from the actual brain dataset, OASIS-3. One whole-brain segmentation mask can generate many nonfinite-modality brain images by randomly merging some fine anatomical structures and subsequently sampling the intensities for each fine anatomical structure using random Gaussian distribution. Furthermore, to get more realistic deformations as the ground truth, an improved 3D Variational Auto-encoder (VAE) is proposed by introducing the intensity-level reconstruction loss and the structure-level reconstruction loss. Based on the generated images and trained improved 3D VAE, a new Synthetic Nonfinite-Modality Brain Image Dataset (SNMBID) is created. Experiments show that pre-training on SNMBID can improve the accuracy of registration. Notably, SNMBID can be a landmark for evaluating other brain registration methods, and the model trained on the SNMBID can be a baseline for the brain image registration task. Our code is available at https://github.com/MangoWAY/SMIBID_BrainRegistration.

Multi-Intensity Optimization-Based CT and Cone Beam CT Image Registration

Cancer is a highly lethal disease that is mainly treated by image-guided radiotherapy. Because the low dose of cone beam CT is less harmful to patients, cone beam CT images are often used for target delineation in image-guided radiotherapy of various cancers, especially in breast and lung cancer. However, breathing and heartbeat can cause position errors in images taken during different periods, and the low dose of cone beam CT also results in insufficient imaging clarity, rendering existing registration methods unable to meet the CT and cone beam CT registration tasks. In this paper, we propose a novel multi-intensity optimization-based CT and cone beam CT registration method. First, we use a multi-weighted mean curvature filtering algorithm to preserve the multi-intensity details of the input image pairs. Then, the strong edge retention results are registered using and intensity-based method to obtain the multi-intensity registration results. Next, a novel evaluation method called intersection mutual information is proposed to evaluate the registration accuracy of the different multi-intensity registration results. Finally, we determine the optimal registration transformation by intersection mutual information and apply it to the input image pairs to obtain the final registration results. The experimental results demonstrate the excellent performance of the proposed method, meeting the requirements of image-guided radiotherapy.

Multi-strategy mutual learning network for deformable medical image registration

Deformable medical image registration plays a vital role in clinical diagnosis, monitoring treatment, and postoperative recovery. Nevertheless, the existing registration algorithms rely on a single network or training strategy to complete the registration task. Even the most advanced registration algorithms cannot capture sufficiently compelling feature information by a single network or strategy.This paper proposes a new unsupervised and deformable medical image registration network framework to get the compelling feature. The network uses 2D sub-images of 3D images as additional constraint information to supplement the training process. Therefore, the network’s two-dimensional and three-dimensional data images can interactively learn each other’s characteristic information. In addition, we propose to perform secondary registration on the concentrated part of the registration area according to the characteristics of the input image. It enables the complete image and critical regions to learn their characteristic information interactively. This paper uses the image pyramid to integrate the two mutual learning strategies, thus proposing a multi-strategy mutual learning network(MMLN) and conducting many evaluation tests on the public data set OASIS and LPBA40. The test results show that the network can achieve better registration performance than other learning-based methods and traditional algorithms.

Multispectral image registration based on an improved scale-invariant feature transform algorithm

There are often significant intensity variations between multispectral images, making automatic registration tasks difficult. Traditional feature matching methods, such as the scale-invariant feature transform (SIFT), are often sensitive to nonlinear variations of intensity between multispectral images. To solve this problem, an improved SIFT algorithm is introduced. First, the contrast limited adaptive histogram equalization algorithm is introduced in the feature extraction stage to improve the feature point extraction results. Then, the Sobel operator is used to enhance the main direction consistency between homologous feature point pairs. The experimental results suggest that the method can obtain reliable registration results on unmanned aerial vehicle multispectral images.

Self-supervised rigid transformation equivariance for accurate 3D point cloud registration

Transformation equivariance has been widely investigated in 3D point cloud representation learning for more informative descriptors, which formulates the change of the representation with respect to the transformation of the input point clouds explicitly. In this paper, we extend this property to the task of 3D point cloud registration and propose a rigid transformation equivariance (RTE) for accurate 3D point cloud registration. Specifically, RTE formulates the change of the relative pose explicitly with respect to the rigid transformation of the input point clouds. To exploit RTE, we adopt a Siamese structure network with two shared registration branches. One focuses on the input pair of point clouds, and the other one focuses on the new pair achieved by applying two random rigid transformations to the input point clouds respectively. Since the change of the two output relative poses has been predicted according to RTE, a new additional self-supervised loss is obtained to supervise the training. This general network structure can be integrated with most learning-based point cloud registration frameworks easily to improve the performance. Our method adopts the state-of-the-art virtual point-based pipelines as our shared branches, in which we propose a data-driven matching based on learned cost volume (LCV) rather than traditional hand-crafted matching strategies. Experimental evaluations on both synthetic datasets and real datasets validate the effectiveness of our proposed framework. The source code will be made public.

Feature matching based on spatial clustering for aerial image registration with large view differences

Over the past few years, drones have been widely used in many fields, while the registration of aerial images is the basis of applications. In this paper, the RFM-SCAN algorithm is applied to the registration task of aerial images with large view differences, used to remove the mismatches in the coarse matching results of feature points. We make some improvements to address the problems in practice. To address the issue of data redundancy in the feature vectors used for clustering, our algorithm uses principal component analysis to lower the dimensionality of the feature vectors, which reduces the data redundancy and improves the clustering performance. To address the problem that the clustering results are sensitive to spatial location and easily classify isolated positive matches as mismatches, our algorithm recalls isolated positive matches by motion consistency constraint. We respectively conduct qualitative and quantitative experiments, which show that our improved algorithm has excellent performance compared to existing state-of-the-art algorithms.

Deep hashing for global registration of untracked 2D laparoscopic ultrasound to CT

PurposeThe registration of Laparoscopic Ultrasound (LUS) to CT can enhance the safety of laparoscopic liver surgery by providing the surgeon with awareness on the relative positioning between critical vessels and a tumour. In an effort to provide a translatable solution for this poorly constrained problem, Content-based Image Retrieval (CBIR) based on vessel information has been suggested as a method for obtaining a global coarse registration without using tracking information. However, the performance of these frameworks is limited by the use of non-generalisable handcrafted vessel features.MethodsWe propose the use of a Deep Hashing (DH) network to directly convert vessel images from both LUS and CT into fixed size hash codes. During training, these codes are learnt from a patient-specific CT scan by supplying the network with triplets of vessel images which include both a registered and a mis-registered pair. Once hash codes have been learnt, they can be used to perform registration with CBIR methods.ResultsWe test a CBIR pipeline on 11 sequences of untracked LUS distributed across 5 clinical cases. Compared to a handcrafted feature approach, our model improves the registration success rate significantly from 48% to 61%, considering a 20 mm error as the threshold for a successful coarse registration.ConclusionsWe present the first DH framework for interventional multi-modal registration tasks. The presented approach is easily generalisable to other registration problems, does not require annotated data for training, and may promote the translation of these techniques.

Vibration-based identification of engine valve clearance using a convolutional neural network

Contemporary operation-related requirements for combustion engines force the necessity of ongoing assessment of their in operation technical condition (e.g. marine engines). The engine efficiency and durability depend on a variety of parameters. One of them is valve clearance. As has been proven in the paper, the assessment of the valve clearance can be based on vibration signals, which is not a problem in terms of signal measurement and processing and is not invasive into the engine structure. The authors described the experimental research aiming at providing information necessary to develop and validate the proposed method. Active experiments were used with the task of valve clearance and registration of vibrations using a three-axis transducer placed on the engine cylinder head. The tests were carried out during various operating conditions of the engine set by 5 rotational speeds and 5 load conditions. In order to extract the training examples, fragments of the signal related to the closing of individual valves were divided into 11 shorter portions. From each of them, an effective value of the signal was determined. Obtained total 32054 training vectors for each valve related to 4 classes of valve clearance including very sensitive clearance above 0.8 mm associat-ed with high dynamic interactions in cylinder head. In the paper, the authors propose to use a convolutional network CNN to assess the correct engine valve clearance. The obtained results were compared with other methods of machine learning (pattern recognition network, random forest). Finally, using CNN the valve clearance class identification error was less than 1% for the intake valve and less than 3.5% for the exhaust valve. Developed method replaces the existing standard methods based on FFT and STFT combined with regression calculation where approximation error is up to 10%. Such results are more useful for further studies related not only to classification, but also to the prediction of the valve clearance condition in real engine operations.

A TIR-Visible Automatic Registration and Geometric Correction Method for SDGSAT-1 Thermal Infrared Image Based on Modified RIFT

High-resolution thermal infrared (TIR) remote sensing images can more accurately retrieve land surface temperature and describe the spatial pattern of urban thermal environment. The Thermal Infrared Spectrometer (TIS), which has high spatial resolution among spaceborne thermal infrared sensors at present, and global data acquisition capability, is one of the sensors equipped in the SDGSAT-1. It is an important complement to the existing international mainstream satellites. In order to produce standard data products, rapidly and accurately, the automatic registration and geometric correction method needs to be developed. Unlike visible–visible image registration, thermal infrared images are blurred in edge details and have obvious non-linear radiometric differences from visible images, which make it challenging for the TIR-visible image registration task. To address these problems, homomorphic filtering is employed to enhance TIR image details and the modified RIFT algorithm is proposed to achieve TIR-visible image registration. Different from using MIM for feature description in RIFT, the proposed modified RIFT uses the novel binary pattern string to descriptor construction. With sufficient and uniformly distributed ground control points, the two-step orthorectification framework, from SDGSAT-1 TIS L1A image to L4 orthoimage, are proposed in this study. The first experiment, with six TIR-visible image pairs, captured in different landforms, is performed to verify the registration performance, and the result indicates that the homomorphic filtering and modified RIFT greatly increase the number of corresponding points. The second experiment, with one scene of an SDGSAT-1 TIS image, is executed to test the proposed orthorectification framework. Subsequently, 52 GCPs are selected manually to evaluate the orthorectification accuracy. The result indicates that the proposed orthorectification framework is helpful to improve the geometric accuracy and guarantee for the subsequent thermal infrared applications.

Spatiotemporal 3D image registration for mesoscale studies of brain development

Comparison of brain samples representing different developmental stages often necessitates registering the samples to common coordinates. Although the available software tools are successful in registering 3D images of adult brains, registration of perinatal brains remains challenging due to rapid growth-dependent morphological changes and variations in developmental pace between animals. To address these challenges, we introduce CORGI (Customizable Object Registration for Groups of Images), an algorithm for the registration of perinatal brains. First, we optimized image preprocessing to increase the algorithm’s sensitivity to mismatches in registered images. Second, we developed an attention-gated simulated annealing procedure capable of focusing on the differences between perinatal brains. Third, we applied classical multidimensional scaling (CMDS) to align (“synchronize”) brain samples in time, accounting for individual development paces. We tested CORGI on 28 samples of whole-mounted perinatal mouse brains (P0–P9) and compared its accuracy with other registration algorithms. Our algorithm offers a runtime of several minutes per brain on a laptop and automates such brain registration tasks as mapping brain data to atlases, comparing experimental groups, and monitoring brain development dynamics.

Deep learning-based simultaneous registration and unsupervised non-correspondence segmentation of medical images with pathologies

PurposeThe registration of medical images often suffers from missing correspondences due to inter-patient variations, pathologies and their progression leading to implausible deformations that cause misregistrations and might eliminate valuable information. Detecting non-corresponding regions simultaneously with the registration process helps generating better deformations and has been investigated thoroughly with classical iterative frameworks but rarely with deep learning-based methods.MethodsWe present the joint non-correspondence segmentation and image registration network (NCR-Net), a convolutional neural network (CNN) trained on a Mumford–Shah-like functional, transferring the classical approach to the field of deep learning. NCR-Net consists of one encoding and two decoding parts allowing the network to simultaneously generate diffeomorphic deformations and segment non-correspondences. The loss function is composed of a masked image distance measure and regularization of deformation field and segmentation output. Additionally, anatomical labels are used for weak supervision of the registration task. No manual segmentations of non-correspondences are required.ResultsThe proposed network is evaluated on the publicly available LPBA40 dataset with artificially added stroke lesions and a longitudinal optical coherence tomography (OCT) dataset of patients with age-related macular degeneration. The LPBA40 data are used to quantitatively assess the segmentation performance of the network, and it is shown qualitatively that NCR-Net can be used for the unsupervised segmentation of pathologies in OCT images. Furthermore, NCR-Net is compared to a registration-only network and state-of-the-art registration algorithms showing that NCR-Net achieves competitive performance and superior robustness to non-correspondences.ConclusionNCR-Net, a CNN for simultaneous image registration and unsupervised non-correspondence segmentation, is presented. Experimental results show the network’s ability to segment non-correspondence regions in an unsupervised manner and its robust registration performance even in the presence of large pathologies.

SCRnet: A Spatial Consistency Guided Network Using Contrastive Learning for Point Cloud Registration

Point cloud registration is used to find a rigid transformation from the source point cloud to the target point cloud. The main challenge in the point cloud registration is in finding correct correspondences in complex scenes that may contain many noise and repetitive structures. At present, many existing methods use outlier rejections to help the network obtain more accurate correspondences, but they often ignore the spatial consistency between keypoints. Therefore, to address this issue, we propose a spatial consistency guided network using contrastive learning for point cloud registration (SCRnet), in which its overall stage is symmetrical. SCRnet consists of four blocks, namely feature extraction block, confidence estimation block, contrastive learning block and registration block. Firstly, we use mini-PointNet to extract coarse local and global features. Secondly, we propose confidence estimation block, which formulate outlier rejection as confidence estimation problem of keypoint correspondences. In addition, the local spatial features are encoded into the confidence estimation block, which makes the correspondence possess local spatial consistency. Moreover, we propose contrastive learning block by constructing positive point pairs and hard negative point pairs and using Point-Pair-INfoNCE contrastive loss, which can further remove hard outliers through global spatial consistency. Finally, the proposed registration block selects a set of matching points with high spatial consistency and uses these matching sets to calculate multiple transformations, then the best transformation can be identified by initial alignment and Iterative Closest Point (ICP) algorithm. Extensive experiments are conducted on KITTI and nuScenes dataset, which demonstrate the high accuracy and strong robustness of SCRnet on point cloud registration task.

Coordinate Translator for Learning Deformable Medical Image Registration.

The majority of deep learning (DL) based deformable image registration methods use convolutional neural networks (CNNs) to estimate displacement fields from pairs of moving and fixed images. This, however, requires the convolutional kernels in the CNN to not only extract intensity features from the inputs but also understand image coordinate systems. We argue that the latter task is challenging for traditional CNNs, limiting their performance in registration tasks. To tackle this problem, we first introduce Coordinate Translator, a differentiable module that identifies matched features between the fixed and moving image and outputs their coordinate correspondences without the need for training. It unloads the burden of understanding image coordinate systems for CNNs, allowing them to focus on feature extraction. We then propose a novel deformable registration network, im2grid, that uses multiple Coordinate Translator's with the hierarchical features extracted from a CNN encoder and outputs a deformation field in a coarse-to-fine fashion. We compared im2grid with the state-of-the-art DL and non-DL methods for unsupervised 3D magnetic resonance image registration. Our experiments show that im2grid outperforms these methods both qualitatively and quantitatively.

NonRegSRNet: A Nonrigid Registration Hyperspectral Super-Resolution Network

Due to the limitations of imaging systems, satellite hyperspectral imagery (HSI), which yields rich spectral information in many channels, often suffers from poor spatial resolution. HSI super-resolution (SR) refers to the fusion of high spatial resolution multispectral imagery (MSI) and low spatial resolution HSI to generate HSI that has both a high spatial and high spectral resolution. However, most existing SR methods assume that the two original images used are perfectly registered: in reality, nonrigid deformation areas can exist locally in the two images even if prior registration of the control points has been carried out. To address this problem, we propose a novel unsupervised spectral unmixing and image deformation correction network&#x2014;NonRegSRNet&#x2014;with multimodal and multitask learning that can be used for the joint registration of HSI and MSI and to produce SR imagery. More specifically, NonRegSRNet integrates the dense registration and SR tasks into a unified model that includes a triplet convolutional neural network. This allows these two tasks to complement each other so that better registration and SR results can be achieved. Furthermore, because the point spread function (PSF) and spectral response function (SRF) are often unavailable, two special convolutional layers are designed to adaptively learn the parameters of the PSF and SRF, which makes the proposed model more adaptable. Experimental results demonstrate that the proposed method has the ability to produce highly accurate and stable reconstructed images under complex nonrigid deformation conditions. (Code available at <uri>https://github.com/saber-zero/NonRegSRNet</uri>)