Feature-based Registration Method Research Articles

TransMatch: A Transformer-Based Multilevel Dual-Stream Feature Matching Network for Unsupervised Deformable Image Registration.

Feature matching, which refers to establishing the correspondence of regions between two images (usually voxel features), is a crucial prerequisite of feature-based registration. For deformable image registration tasks, traditional feature-based registration methods typically use an iterative matching strategy for interest region matching, where feature selection and matching are explicit, but specific feature selection schemes are often useful in solving application-specific problems and require several minutes for each registration. In the past few years, the feasibility of learning-based methods, such as VoxelMorph and TransMorph, has been proven, and their performance has been shown to be competitive compared to traditional methods. However, these methods are usually single-stream, where the two images to be registered are concatenated into a 2-channel whole, and then the deformation field is output directly. The transformation of image features into interimage matching relationships is implicit. In this paper, we propose a novel end-to-end dual-stream unsupervised framework, named TransMatch, where each image is fed into a separate stream branch, and each branch performs feature extraction independently. Then, we implement explicit multilevel feature matching between image pairs via the query-key matching idea of the self-attention mechanism in the Transformer model. Comprehensive experiments are conducted on three 3D brain MR datasets, LPBA40, IXI, and OASIS, and the results show that the proposed method achieves state-of-the-art performance in several evaluation metrics compared to the commonly utilized registration methods, including SyN, NiftyReg, VoxelMorph, CycleMorph, ViT-V-Net, and TransMorph, demonstrating the effectiveness of our model in deformable medical image registration.

Feature-Based Electromagnetic Tracking Registration Using Bioelectric Sensing

Catheter tracking is essential during minimally invasive endovascular procedures, and Electromagnetic (EM) tracking is a widely used technology for this purpose. When preoperative patient images are available, they can be used to guide EM-tracked interventions. However, a registration step between preoperative images and intraoperative EM tracking space is usually required. Most existing solutions for this registration process require manual interactions, which can add additional steps to the workflow. In this paper, a novel automatic feature-based registration method is proposed, based on electric sensing of vascular geometry by the catheter, also known as Bioelectric sensing. The technique employs the Bioelectric sensing capabilities of the catheter to identify vascular features, such as bifurcations, aneurysms, or stenosis. The known EM position of these features is then utilized to register the EM tracking space and the preoperative images. The registration is refined using iterative closest point (ICP) registration algorithms. Unlike existing solutions, the proposed method does not require external markers, interventional imaging, or additional surgeon actions, and hence does not impact the interventional workflow.

A flexible calibration method with multi-stage optimization for the axial error of mobile mapping systems

Mobile mapping systems, which are equipped with a laser scanner, panoramic camera, and inertial navigation system, can flexibly and efficiently obtain precision point clouds. However, these systems require re-installation for each mission, which causes displacement errors in the instruments, resulting in discrepancies between multiple collections of data. The existing calibration methods usually require additional control targets settled in a managed field survey. Furthermore, the point clouds obtained with mobile mapping systems do not follow the common assumption of rigid transformation. To alleviate the above problems, we propose a flexible in-situ calibration method that only requires data collected in a round-trip survey pattern and conducts a self-calibration of the placement parameters with meticulously designed exact point matching. Specifically, because the accuracy of the existing feature-based registration method is not sufficient to acquire the exact correspondences of point clouds and to prune the outliers in correspondence searching. We propose novel pruning and refining approaches to obtain exact point-to-point correspondences between the round-trip point clouds. From systematic analyses of the coupling between different factors, we then construct a simplified error model that takes into account the a priori rotation- and translation-weighted observations. Finally, to alleviate the correlation effects between different parameters, we decouple the least-squares solver into several stages and obtain robust self-calibration parameters. Surveying experiments show that the flexible calibration method proposed in this paper is both effective and feasible in improving the accuracy of point clouds in overlapping areas, and can reduce the inconsistency of the calibrated point clouds by 37.02%.

Surface topography data fusion of additive manufacturing based on confocal and focus variation microscopy.

In this paper, two innovative data fusion methods are proposed for reconstructing the surfaces produced by directed energy deposition (DED) additive manufacturing. The surface topographic data were obtained from confocal laser scanning microscopy (CLSM) and focus variation microscopy (FV). The first method (competitive data fusion) aims to improve the data quality by combining both the advantages of the CLSM and FV techniques, while the second method (cooperative data integration) is designed for generating a single representation that contains not only global information but also local details. The results show that both fusion methods achieved satisfactory results: in the competitive fusion, the fused data preserved the characteristics of FV data while its vertical resolution is also improved by integrating the short waves from the CLSM data; the cooperative data fusion achieved one pixel precision of the surface registration which adopted the feature-based registration method with the help of color image information. The computational complexity is reduced from O((m×n)2) to O(m×n + k). Both proposed data fusion methods provided innovative solutions for the microscopic surface reconstruction and surface representation in multiscales in the field of additive manufacturing.

Color fundus image registration using a learning-based domain-specific landmark detection methodology

Medical imaging, and particularly retinal imaging, allows to accurately diagnose many eye pathologies as well as some systemic diseases such as hypertension or diabetes. Registering these images is crucial to correctly compare key structures, not only within patients, but also to contrast data with a model or among a population. Currently, this field is dominated by complex classical methods because the novel deep learning methods cannot compete yet in terms of results and commonly used methods are difficult to adapt to the retinal domain. In this work, we propose a novel method to register color fundus images based on previous works which employed classical approaches to detect domain-specific landmarks. Instead, we propose to use deep learning methods for the detection of these highly-specific domain-related landmarks. Our method uses a neural network to detect the bifurcations and crossovers of the retinal blood vessels, whose arrangement and location are unique to each eye and person. This proposal is the first deep learning feature-based registration method in fundus imaging. These keypoints are matched using a method based on RANSAC (Random Sample Consensus) without the requirement to calculate complex descriptors. Our method was tested using the public FIRE dataset, although the landmark detection network was trained using the DRIVE dataset. Our method provides accurate results, a registration score of 0.657 for the whole FIRE dataset (0.908 for category S, 0.293 for category P and 0.660 for category A). Therefore, our proposal can compete with complex classical methods and beat the deep learning methods in the state of the art.

An accurate and robust registration framework based on outlier removal and feature point adjustment for remote sensing images

ABSTRACT The reliability of feature matching can decide the accuracy and robustness of the feature-based registration result. Aiming at the problem that the number of final feature matches preserved by many popular outlier removal methods is small, and the position accuracy of final feature matches is not high enough, we propose an accurate and robust image registration framework based on outlier removal and feature point adjustment in this paper. This framework increases the number and improves the position accuracy of inliers while eliminating most outliers. The increased number of inliers improves the robustness of image registration, and high accurate inliers improves the accuracy of image registration. Firstly, the initial feature matches are extracted by a commonly used feature-based registration method, such as the scale-invariant feature transform (SIFT)-based method. Then, outliers of the initial feature matches are eliminated by a frequency domain similarity measure, called PHase-based Structural SIMilarity (PH-SSIM) proposed in this paper. Considering the inherent error of the feature matches that still exist after the outlier elimination, a PH-SSIM-based feature point adjustment strategy is designed to fine-adjust the position of the preserved feature points in the reference image. Finally, the registration parameters are calculated by the fine-adjusted feature matches. The proposed framework has been evaluated by several remote sensing images with different resolution, grey-scale, texture, and scene, and compared with four state-of-the-art image registration methods. Experimental result demonstrates the high accuracy and robustness of the proposed method.

A Closed-Form Solution to Linear Feature-Based Registration of LiDAR Point Clouds

Due to the high complexity of geo-spatial entities and the limited field of view of LiDAR equipment, pairwise registration is a necessary step for integrating point clouds from neighbouring LiDAR stations. Considering that accurate extraction of point features is often difficult without the use of man-made reflectors, and the initial approximate values for the unknown transformation parameters must be estimated in advance to ensure the correct operation of those iterative methods, a closed-form solution to linear feature-based registration of point clouds is proposed in this study. Plücker coordinates are used to represent the linear features in three-dimensional space, whereas dual quaternions are employed to represent the spatial transformation. Based on the theory of least squares, an error norm (objective function) is first constructed by assuming that each pair of corresponding linear features is equivalent after registration. Then, by applying the extreme value analysis to the objective function, detailed derivations of the closed-form solution to the proposed linear feature-based registration method are given step by step. Finally, experimental tests are conducted on a real dataset. The derived experimental result demonstrates the feasibility of the proposed solution: By using eigenvalue decomposition to replace the linearization of the objective function, the proposed solution does not require any initial estimates of the unknown transformation parameters, which assures the stability of the registration method.

A hybrid feature-based patient-to-image registration method for robot-assisted long bone osteotomy.

The purpose of this study is to provide a simple, feasible and effective patient-to-image registration method for robot-assisted long bone osteotomy, which has rarely been systematically reported. The practical requirement is to meet the accuracy of 1mm or even higher without bone-implanted markers. A hybrid feature-based registration method termed CR-RAMSICP is proposed. Point-based coarse registration (CR) is accomplished relying on the optical retro-reflective markers attached to the tracked rigid body fixed out of the bone. In surface-based fine registration, an improved iterative closest point (ICP) algorithm based on the range-adaptive matching strategy (termed RAMSICP) is presented to cope with the robust precise matching between the asymmetric patient and image point clouds, which avoidsconverging to a local minimum. A series of registration experiments based on the isolated porcine iliums are carried out. The results illustrate that CR-RAMSICP not only significantly outperforms CR and CR-ICP in the accuracy and reproducibility, but also exhibits better robustness to the CR errors and less sensitiveness to the distribution and number of fiducial points located in the patient point cloud than CR-ICP. The proposed registration method CR-RAMSICP canstably satisfythedesiredregistration accuracy without the use of bone-implanted markers like fiducial screws. Besides, the RAMSICP algorithm used in fine registration is convenient for programming because any complex metrics or models are not involved.

Direct 3D ultrasound fusion for transesophageal echocardiography

BackgroundReal-time three-dimensional transesophageal echocardiography (3D TEE) has been increasingly used in clinic for fast 3D analysis of cardiac anatomy and function. However, 3D TEE still suffers from the limited field of view (FoV). It is challenging to adopt conventional multi-view fusion methods to 3D TEE images because feature-based registration methods tend to fail in the ultrasound scenario, and conventional intensity-based methods have poor convergence properties and require an iterative coarse-to-fine strategy. MethodsA novel multi-view registration and fusion method is proposed to enlarge the FoV of 3D TEE images efficiently. A direct method is proposed to solve the registration problem in the Lie algebra space. Fast implementation is realized by searching voxels on three orthogonal planes between two volumes. Besides, a weighted-average 3D fusion method is proposed to fuse the aligned images seamlessly. For a sequence of 3D TEE images, they are fused incrementally. ResultsQualitative and quantitative results of in-vivo experiments indicate that the proposed registration algorithm outperforms a state-of-the-art PCA-based registration method in terms of accuracy and efficiency. Image registration and fusion performed on 76 in-vivo 3D TEE volumes from nine patients show apparent enlargement of FoV (enlarged around two times) in the obtained fused images. ConclusionsThe proposed methods can fuse 3D TEE images efficiently and accurately so that the whole Region of Interest (ROI) can be seen in a single frame. This research shows good potential to assist clinical diagnosis, preoperative planning, and future intraoperative guidance with 3D TEE.

Adaptive Multi-View Image Mosaic Method for Conveyor Belt Surface Fault Online Detection

In order to improve the accuracy and real-time of image mosaic, realize the multi-view conveyor belt surface fault online detection, and solve the problem of longitudinal tear of conveyor belt, we in this paper propose an adaptive multi-view image mosaic (AMIM) method based on the combination of grayscale and feature. Firstly, the overlapping region of two adjacent images is preliminarily estimated by establishing the overlapping region estimation model, and then the grayscale-based method is used to register the overlapping region. Secondly, the image of interest (IOI) detection algorithm is used to divide the IOI and the non-IOI. Thirdly, only for the IOI, the feature-based partition and block registration method is used to register the images more accurately, the overlapping region is adaptively segmented, the speeded up robust features (SURF) algorithm is used to extract the feature points, and the random sample consensus (RANSAC) algorithm is used to achieve accurate registration. Finally, the improved weighted smoothing algorithm is used to fuse the two adjacent images. The experimental results showed that the registration rate reached 97.67%, and the average time of stitching was less than 500 ms. This method is accurate and fast, and is suitable for conveyor belt surface fault online detection.

An Accurate Registration Method Based on Global Mixed Structure Similarity (GMSIM) for Remote Sensing Images

Although remote sensing image registration has been studied for several years, achieving accurate image registration remains a challenging task due to the complicated conditions surrounding remote sensing images. To improve the accuracy and robustness of image registration, we proposed a registration method based on the global mixed structure similarity (GMSIM) measure. This measure mixes the structure similarity in both the frequency domain and the intensity domain because phase-based structure similarity in the frequency domain is sensitive to intensity contrast and spatial translation, and gray-based structure similarity in the intensity domain is efficient to structure change. Feature-based registration methods are used to generate the initial registration parameters. After that, we calculate the final registration parameters by maximizing GMSIM. Quantum-behaved particle swarm optimization (QPSO) is utilized to solve the optimal results of GMSIM due to its high efficiency. The proposed method has been evaluated on several remote sensing images differing in scale, gray, and scene and compared with three state-of-the-art registration methods. Experimental results demonstrate the high accuracy of the proposed scheme.

Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration.

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.

Robust global registration of point clouds by closed-form solution in the frequency domain

Point cloud registration is invariably an essential and challenging task in the fields of photogrammetry and computer vision to align multiple point clouds to a united reference frame. In this paper, we propose a novel global registration method using a robust phase correlation method for registration of low-overlapping point clouds, which is less sensitive to noise and outliers than feature-based registration methods. The proposed point cloud registration is achieved by converting the estimation of rotation, scaling, and translation in the spatial domain to a problem of correlating low-frequency components in the frequency domain. Specifically, it consists of three core steps: transformation from the spatial domain to the frequency domain, decoupling of rotation, scaling, and translation, and adapted phase correlation for robust shift estimation. In the first step, unstructured and unordered 3D points are transformed from the spatial domain to the frequency domain via 3D Fourier transformation, following a voxelization and binarization process. In the second step, rotation, scaling, and translation are decoupled by sequential operations, including Fourier transform, resampling strategies, and Fourier-Mellin transform. In the third step, the estimation of transformation parameters is transformed into shift estimation tasks. The shift estimation task is solved by a robust phase correlation method, in which low-frequency components are matched by decomposing the normalized cross-power spectrum and linearly fitting the decomposed signals with a closed-form solution by a ℓ1-norm-based robust estimator. Experiments were conducted using three different datasets of urban and natural scenarios. Results demonstrate the efficiency of the proposed method, with the majority of rotation and translation errors reaching less than 0.2 degree and 0.5 m, respectively. Additionally, it is also validated by experiments that the proposed method is robust to noise and versatile to datasets with wide ranges of overlaps and various geometric characteristics.

Modality-Free Feature Detector and Descriptor for Multimodal Remote Sensing Image Registration

The nonlinear radiation distortions (NRD) among multimodal remote sensing images bring enormous challenges to image registration. The traditional feature-based registration methods commonly use the image intensity or gradient information to detect and describe the features that are sensitive to NRD. However, the nonlinear mapping of the corresponding features of the multimodal images often results in failure of the feature matching, as well as the image registration. In this paper, a modality-free multimodal remote sensing image registration method (SRIFT) is proposed for the registration of multimodal remote sensing images, which is invariant to scale, radiation, and rotation. In SRIFT, the nonlinear diffusion scale (NDS) space is first established to construct a multi-scale space. A local orientation and scale phase congruency (LOSPC) algorithm are then used so that the features of the images with NRD are mapped to establish a one-to-one correspondence, to obtain sufficiently stable key points. In the feature description stage, a rotation-invariant coordinate (RIC) system is adopted to build a descriptor, without requiring estimation of the main direction. The experiments undertaken in this study included one set of simulated data experiments and nine groups of experiments with different types of real multimodal remote sensing images with rotation and scale differences (including synthetic aperture radar (SAR)/optical, digital surface model (DSM)/optical, light detection and ranging (LiDAR) intensity/optical, near-infrared (NIR)/optical, short-wave infrared (SWIR)/optical, classification/optical, and map/optical image pairs), to test the proposed algorithm from both quantitative and qualitative aspects. The experimental results showed that the proposed method has strong robustness to NRD, being invariant to scale, radiation, and rotation, and the achieved registration precision was better than that of the state-of-the-art methods.

Configuration and Registration of Multi-Camera Spectral Image Database of Icon Paintings

At the cost of added complexity and time, hyperspectral imaging provides a more accurate measure of the scene’s irradiance compared to an RGB camera. Several camera designs with more than three channels have been proposed to improve the accuracy. The accuracy is often evaluated based on the estimation quality of the spectral data. Currently, such evaluations are carried out with either simulated data or color charts to relax the spatial registration requirement between the images. To overcome this limitation, this article presents an accurately registered image database of six icon paintings captured with five cameras with different number of channels, ranging from three (RGB) to more than a hundred (hyperspectral camera). Icons are challenging topics because they have complex surfaces that reflect light specularly with a high dynamic range. Two contributions are proposed to tackle this challenge. First, an imaging configuration is carefully arranged to control the specular reflection, confine the dynamic range, and provide a consistent signal-to-noise ratio for all the camera channels. Second, a multi-camera, feature-based registration method is proposed with an iterative outlier removal phase that improves the convergence and the accuracy of the process. The method was tested against three other approaches with different features or registration models.

A Novel Coarse-to-Fine Scheme for Remote Sensing Image Registration Based on SIFT and Phase Correlation

Automatic image registration has been wildly used in remote sensing applications. However, the feature-based registration method is sometimes inaccurate and unstable for images with large scale difference, grayscale and texture differences. In this manuscript, a coarse-to-fine registration scheme is proposed, which combines the advantage of feature-based registration and phase correlation-based registration. The scheme consists of four steps. First, feature-based registration method is adopted for coarse registration. A geometrical outlier removal method is applied to improve the accuracy of coarse registration, which uses geometric similarities of inliers. Then, the sensed image is modified through the coarse registration result under affine deformation model. After that, the modified sensed image is registered to the reference image by extended phase correlation. Lastly, the final registration results are calculated by the fusion of the coarse registration and the fine registration. High universality of feature-based registration and high accuracy of extended phase correlation-based registration are both preserved in the proposed method. Experimental results of several different remote sensing images, which come from several published image registration papers, demonstrate the high robustness and accuracy of the proposed method. The evaluation contains root mean square error (RMSE), Laplace mean square error (LMSE) and red–green image registration results.

ORTHOGRAPHIC REFLECTANCE IMAGE FOR PLANAR TARGET LOCALIZATION IN LOW DENSITY TLS POINT CLOUDS

Abstract. Point cloud registration is important and essential task for terrestrial laser scanning applications. Point clouds acquired at different positions exhibit significant variation in point density. Most registration methods implicitly assume dense and uniform distributed point clouds, which is hardly the case in large-scale surveying. The accuracy and robustness of feature extraction are greatly influenced by the point density, which undermines the feature-based registration methods. We show that the accuracy and robustness of target localization dramatically decline with decreasing point density. A methodology for localization of artificial planar targets in low density point clouds is presented. An orthographic image of the target is firstly generated and the potential position of the target center is interactively selected. Then the 3D position of the target center is estimated by a non-linear least squares adjustment. The presented methodology enables millimeter level accuracy of target localization in point clouds with 30mm sample interval. The robustness and effectiveness of the methodology is demonstrated by the experimental results.

Accurate registration of in vivo time-lapse images.

In vivo imaging experiments often require automated detection and tracking of changes in the specimen. These tasks can be hindered by variations in the position and orientation of the specimen relative to the microscope, as well as by linear and nonlinear tissue deformations. We propose a feature-based registration method, coupled with optimal transformations, designed to address these problems in 3D time-lapse microscopy images. Features are detected as local regions of maximum intensity in source and target image stacks, and their bipartite intensity dissimilarity matrix is used as an input to the Hungarian algorithm to establish initial correspondences. A random sampling refinement method is employed to eliminate outliers, and the resulting set of corresponding features is used to determine an optimal translation, rigid, affine, or B-spline transformation for the registration of the source and target images. Accuracy of the proposed algorithm was tested on fluorescently labeled axons imaged over a 68-day period with a two-photon laser scanning microscope. To that end, multiple axons in individual stacks of images were traced semi-manually and optimized in 3D, and the distances between the corresponding traces were measured before and after the registration. The results show that there is a progressive improvement in the registration accuracy with increasing complexity of the transformations. In particular, sub-micrometer accuracy (2-3 voxels) was achieved with the regularized affine and B-spline transformations.

Towards Accurate and Robust Multi-Modal Medical Image Registration Using Contrastive Metric Learning

Multi-modal medical image registration takes an essential role in image-based clinical diagnosis and surgical planning. It is not trivial due to appearance variations across different modalities. Rigidly aligning two images is used to register rigid body structure, and it is also usually the first step for deformable registration with a large discrepancy. In the field of computer vision, one well-established method for image alignment is to find corresponding points from two images, and image alignment is based on identified corresponding points. Our method lies in this category. Feature representation is crucial in finding corresponding points. However, conventional feature representation like SIFT does not take multi-modal information into account, and thus, it fails. In this paper, we propose a Convolution Neural Network Feature-based Registration (CNNFR) method for aligning the multi-modal medical image. The important component in this method is learning keypoint descriptors using contrastive metric learning, which minimizes the difference between two feature representations from two corresponding points and maximizes difference of two feature representation from two distant points. Also, a transfer learning-based CNNRF (TrCNNRF) is proposed to improve the generalization learning performance when the training data are insufficient. Experimental results demonstrate that the proposed methods can achieve superior performance regarding both accuracy and robustness, which can be used to rigidly register multi-modal images and provide an initial estimation for non-rigid registration in clinical practices.

An Extension of Phase Correlation-Based Image Registration to Estimate Similarity Transform Using Multiple Polar Fourier Transform

Image registration is a core technology of many different image processing areas and is widely used in the remote sensing community. The accuracy of image registration largely determines the effect of subsequent applications. In recent years, phase correlation-based image registration has drawn much attention because of its high accuracy and efficiency as well as its robustness to gray difference and even slight changes in content. Many researchers have reported that the phase correlation method can acquire a sub-pixel accuracy of 1 / 10 or even 1 / 100 . However, its performance is acquired only in the case of translation, which limits the scope of the application of the method. However, there are few reports on the estimation of scales and angles based on the phase correlation method. To take advantage of the high accuracy property and other merits of phase correlation-based image registration and extend it to estimate the similarity transform, we proposed a novel algorithm, the Multilayer Polar Fourier Transform (MPFT), which uses a fast and accurate polar Fourier transform with different scaling factors to calculate the log-polar Fourier transform. The structure of the polar grids of MPFT is more similar to the one of the log-polar grid. In particular, for rotation estimation only, the polar grid of MPFT is the calculation grid. To validate its effectiveness and high accuracy in estimating angles and scales, both qualitative and quantitative experiments were carried out. The quantitative experiments included a numerical simulation as well as synthetic and real data experiments. The experimental results showed that the proposed method, MPFT, performs better than the existing phase correlation-based similarity transform estimation methods, the Pseudo-polar Fourier Transform (PPFT) and the Multilayer Fractional Fourier Transform method (MLFFT), and the classical feature-based registration method, Scale-Invariant Feature Transform (SIFT), and its variant, ms-SIFT.