Multimodal Remote Sensing Image Registration Research Articles

Multimodal remote sensing image registration based on adaptive multi-scale PIIFD

Multimodal remote sensing image registration based on adaptive multi-scale PIIFD

Multimodal Remote Sensing Image Registration Based on Adaptive Spectrum Congruency

Multimodal Remote Sensing Image Registration Based on Adaptive Spectrum Congruency

SRTPN: Scale and Rotation Transform Prediction Net for Multimodal Remote Sensing Image Registration

How to recover geometric transformations is one of the most challenging issues in image registration. To alleviate the effect of large geometric distortion in multimodal remote sensing image registration, a scale and rotate transform prediction net is proposed in this paper. First, to reduce the scale between the reference and sensed images, the image scale regression module is constructed via CNN feature extraction and FFT correlation, and the scale of sensed image can be recovered roughly. Second, the rotation estimate module is developed for predicting the rotation angles between the reference and the scale-recovered images. Finally, to obtain the accurate registration results, LoFTR is employed to match the geometric-recovered images. The proposed registration network was evaluated on GoogleEarth, HRMS, VIS-NIR and UAV datasets with contrast differences and geometric distortions. The experimental results show that the number of correct matches of our model reached 74.6%, and the RMSE of the registration results achieved 1.236, which is superior to the related methods.

Advances and Challenges in Multimodal Remote Sensing Image Registration

Over the past few decades, with the rapid development of global aerospace and aerial remote sensing technology, the types of sensors have evolved from the traditional monomodal sensors (e.g., optical sensors) to the new generation of multimodal sensors e.g., multispectral, hyperspectral, light detection and ranging (LiDAR) and synthetic aperture radar (SAR) sensors. These advanced devices can dynamically provide various and abundant multimodal remote sensing images with different spatial, temporal, and spectral resolutions according to different application requirements. Since then, it is of great scientific significance to carry out the research of multimodal remote sensing image registration, which is a crucial step for integrating the complementary information among multimodal data and making comprehensive observations and analysis of the Earths surface. In this work, we will present our own contributions to the field of multimodal image registration, summarize the advantages and limitations of existing multimodal image registration methods, and then discuss the remaining challenges and make a forward-looking prospect for the future development of the field.

Unsupervised Multimodal Remote Sensing Image Registration via Domain Adaptation

Unsupervised Multimodal Remote Sensing Image Registration via Domain Adaptation

Self-Supervised Keypoint Detection and Cross-Fusion Matching Networks for Multimodal Remote Sensing Image Registration

Remote sensing image matching is the basis upon which to obtain integrated observations and complementary information representation of the same scene from multiple source sensors, which is a prerequisite for remote sensing tasks such as remote sensing image fusion and change detection. However, the intricate geometric and radiometric differences between the multimodal images render the registration quite challenging. Although multimodal remote sensing image matching methods have been developed in recent decades, most classical and deep learning based techniques cannot effectively extract high repeatable keypoints and discriminative descriptors for multimodal images. Therefore, we propose a two-step “detection + matching” framework in this paper, where each step consists of a deep neural network. A self-supervised detection network is first designed to generate similar keypoint feature maps between multimodal images, which is used to detect highly repeatable keypoints. We then propose a cross-fusion matching network, which aims to exploit global optimization and fusion information for cross-modal feature descriptors and matching. The experiments show that the proposed method has superior feature detection and matching performance compared with current state-of-the-art methods. Specifically, the keypoint repetition rate of the detection network and the NN mAP of the matching network are 0.435 and 0.712 on test datasets, respectively. The proposed whole pipeline framework is evaluated, which achieves an average M.S. and RMSE of 0.298 and 3.41, respectively. This provides a novel solution for the joint use of multimodal remote sensing images for observation and localization.

A robust multimodal remote sensing image registration method and system using steerable filters with first- and second-order gradients

Co-registration of multimodal remote sensing (RS) images (e.g., optical, infrared, LiDAR, and SAR) is still an ongoing challenge because of nonlinear radiometric differences (NRD) and significant geometric distortions (e.g., scale and rotation changes) between these images. In this paper, a robust matching method based on the Steerable filters is proposed consisting of two critical steps. First, to address severe NRD, a novel structural descriptor named the Steerable Filters of first- and second-Order Channels (SFOC) is constructed, which combines the first- and second-order gradient information by using the steerable filters with a multi-scale strategy to depict more discriminative structure features of images. Then, a fast similarity measure is established called Fast Normalized Cross-Correlation (Fast-NCCSFOC), which employs the Fast Fourier Transform (FFT) technique and the integral image to improve the matching efficiency. Furthermore, to achieve reliable registration performance, a coarse-to-fine multimodal registration system is designed consisting of two pivotal modules. The local coarse registration is first conducted by involving both detection of interest points (IPs) and local geometric correction, which effectively utilizes the prior georeferencing information of RS images to address global geometric distortions. In the fine registration stage, the proposed SFOC is used to resist significant NRD, and to detect control points (CPs) between multimodal images by a template matching scheme. The performance of the proposed matching method has been evaluated with many different kinds of multimodal RS images. The results show its superior matching performance compared with the state-of-the-art methods. Moreover, the designed registration system also outperforms the popular commercial software (e.g., ENVI, ERDAS, and PCI) in both registration accuracy and computational efficiency. Our system is available at https://github.com/yeyuanxin110/SFOC-Multimodal_Remote_Sensing_Image_Registration_System.

Robust Multimodal Remote Sensing Image Registration Based on Local Statistical Frequency Information

Multimodal remote sensing image registration is a prerequisite for comprehensive application of remote sensing image data. However, inconsistent imaging environment and conditions often lead to obvious geometric deformations and significant contrast differences between multimodal remote sensing images, which makes the common feature extraction extremely difficult, resulting in their registration still being a challenging task. To address this issue, a robust local statistics-based registration framework is proposed, and the constructed descriptors are invariant to contrast changes and geometric transformations induced by imaging conditions. Firstly, maximum phase congruency of local frequency information is performed by optimizing the control parameters. Then, salient feature points are located according to the phase congruency response map. Subsequently, the geometric and contrast invariant descriptors are constructed based on a joint local frequency information map that combines Log-Gabor filter responses over multiple scales and orientations. Finally, image matching is achieved by finding the corresponding descriptors; image registration is further completed by calculating the transformation between the corresponding feature points. The proposed registration framework was evaluated on four different multimodal image datasets with varying degrees of contrast differences and geometric deformations. Experimental results demonstrated that our method outperformed several state-of-the-art methods in terms of robustness and precision, confirming its effectiveness.

Heterogeneous self-supervised interest point matching for multi-modal remote sensing image registration

ABSTRACT The implementation of multi-modal remote sensing image registration is still a challenge in various applications. In this paper, a novel framework called Heterogeneous SuperPoint Network is proposed for multi-modal remote sensing image registration. Heterogeneous SuperPoint Network operates on full-sized images and produces multi-modal interest points accompanied with fixed length descriptors in a single forward pass. Moreover, we develop a strategy for training a base detector that used in conjunction with heterogeneous homographic adaptation to generate pseudo-ground truth interest point labels for unlabeled images in a self-supervised fashion. Then, the interest points and point correspondences obtained by Heterogeneous SuperPoint Network are used to do the final matching with the nearest neighbor (NN) matching and Random sample consensus (RANSAC). Various image sets have been considered in this paper for the evaluation of the proposed approach. Experimental results demonstrate the superior performance of the proposed method over the existing design for the whole data set.

Deep Feature Correlation Learning for Multi-Modal Remote Sensing Image Registration

Deep descriptors have advantages over handcrafted descriptors on local image patch matching. However, due to the complex imaging mechanism of remote sensing images and the significant differences in appearance between multi-modal images, existing deep learning descriptors are unsuitable for multi-modal remote sensing image registration directly. To solve this problem, this paper proposes a deep feature correlation learning network (Cnet) for multi-modal remote sensing image registration. Firstly, Cnet builds a feature learning network based on the deep convolutional network with the attention learning module, to enhance the feature representation by focusing on meaningful features. Secondly, this paper designs a novel feature correlation loss function for Cnet optimization. It focuses on the relative feature correlation between matching and non-matching samples, which can improve the stability of network training and decrease the risk of overfitting. Additionally, the proposed feature correlation loss with a scale factor can further enhance the network training and accelerate the network convergence. Extensive experimental results on image patch matching (Brown, HPatches), cross-spectral image registration (VIS-NIR), multi-modal remote sensing image registration, and single-modal remote sensing image registration have demonstrated the effectiveness and robustness of the proposed method.

Multimodal Remote Sensing Image Registration Methods and Advancements: A Survey

With rapid advancements in remote sensing image registration algorithms, comprehensive imaging applications are no longer limited to single-modal remote sensing images. Instead, multi-modal remote sensing (MMRS) image registration has become a research focus in recent years. However, considering multi-source, multi-temporal, and multi-spectrum input introduces significant nonlinear radiation differences in MMRS images for which researchers need to develop novel solutions. At present, comprehensive reviews and analyses of MMRS image registration methods are inadequate in related fields. Thus, this paper introduces three theoretical frameworks: namely, area-based, feature-based and deep learning-based methods. We present a brief review of traditional methods and focus on more advanced methods for MMRS image registration proposed in recent years. Our review or comprehensive analysis is intended to provide researchers in related fields with advanced understanding to achieve further breakthroughs and innovations.

Modality-Free Feature Detector and Descriptor for Multimodal Remote Sensing Image Registration

The nonlinear radiation distortions (NRD) among multimodal remote sensing images bring enormous challenges to image registration. The traditional feature-based registration methods commonly use the image intensity or gradient information to detect and describe the features that are sensitive to NRD. However, the nonlinear mapping of the corresponding features of the multimodal images often results in failure of the feature matching, as well as the image registration. In this paper, a modality-free multimodal remote sensing image registration method (SRIFT) is proposed for the registration of multimodal remote sensing images, which is invariant to scale, radiation, and rotation. In SRIFT, the nonlinear diffusion scale (NDS) space is first established to construct a multi-scale space. A local orientation and scale phase congruency (LOSPC) algorithm are then used so that the features of the images with NRD are mapped to establish a one-to-one correspondence, to obtain sufficiently stable key points. In the feature description stage, a rotation-invariant coordinate (RIC) system is adopted to build a descriptor, without requiring estimation of the main direction. The experiments undertaken in this study included one set of simulated data experiments and nine groups of experiments with different types of real multimodal remote sensing images with rotation and scale differences (including synthetic aperture radar (SAR)/optical, digital surface model (DSM)/optical, light detection and ranging (LiDAR) intensity/optical, near-infrared (NIR)/optical, short-wave infrared (SWIR)/optical, classification/optical, and map/optical image pairs), to test the proposed algorithm from both quantitative and qualitative aspects. The experimental results showed that the proposed method has strong robustness to NRD, being invariant to scale, radiation, and rotation, and the achieved registration precision was better than that of the state-of-the-art methods.

Fast and Robust Matching for Multimodal Remote Sensing Image Registration

While image registration has been studied in remote sensing community for decades, registering multimodal data [e.g., optical, LiDAR, SAR, and map] remains a challenging problem because of significant nonlinear intensity differences between such data. To address this problem, this paper presents a fast and robust matching framework integrating local descriptors for multimodal registration. In the proposed framework, a local descriptor, such as Histogram of Oriented Gradient (HOG), Local Self Similarity (LSS), or Speeded-Up Robust Feature (SURF), is first extracted at each pixel to form a pixel-wise feature representation of an image. Then we define a similarity measure based on the feature representation in frequency domain using the 3 Dimensional Fast Fourier Transform (3DFFT) technique, followed by a template matching scheme to detect control points between images. In this procedure, we also propose a novel pixel-wise feature representation using orientated gradients of images, which is named channel features of orientated gradients (CFOG). This novel feature is an extension of the pixel-wise HOG descriptors, and outperforms that both in matching performance and computational efficiency. The major advantage of the proposed framework includes: (1) structural similarity representation using the pixel-wise feature description and (2) high computational efficiency due to the use of 3DFFT. Experimental results on different types of multimodal images show the superior matching performance of the proposed framework than the state-of-the-art methods.The proposed matching framework have been used in the software products of a Chinese listed company. The matlab code is available in this manuscript.

Multimodal Remote Sensing Image Registration Based on Image Transfer and Local Features

Automatic registration is still a challenging problem for multimodal remote sensing images including optical, light detection and ranging, synthetic aperture radar images, and so on. Due to the differences in imaging principles, the gray value, texture, and landscape characteristic of these images are different in the local area. This also makes it difficult to obtain satisfactory results for the conventional image registration methods. In order to achieve registration of multimodal images to obtain complementary information, we apply the transfer algorithm based on a deep image analogy to the preprocessing of image registration. It eliminates the differences in multimodal remote sensing images by blending the original image structure and texture. The conventional local feature-based method is applied to match the original and generated images. Correspondences are increased and the registration error is reduced. The experiments demonstrate that our method can effectively deal with multimodal data and produce more accurate results. The algorithm is based on the joint of image deep semantic features and indirectly achieves matching of the original image pair. It provides a new solution to the problem of multimodal images registration.

A novel extended phase correlation algorithm based on Log-Gabor filtering for multimodal remote sensing image registration

ABSTRACTAutomatic registration of multimodal remote sensing images, which is a critical prerequisite in a range of applications (e.g. image fusion, image mosaic, and image analysis), continues to be a fundamental and challenging problem. In this paper, we propose a novel extended phase correlation algorithm based on Log-Gabor filtering (LGEPC) for the registration of images with nonlinear radiometric differences and geometric differences (e.g. rotation, scale, and translation). Our algorithm focuses on two problems that the traditional extended phase correlation algorithms cannot well handle: 1) significant nonlinear radiometric differences and 2) large-scale differences between image pairs. After an over-complete multi-scale atlas space of the original image is built based on the filtered magnitudes obtained by using Log-Gabor filters with different central frequencies, the phase correlation of the single scale images is extended by LGEPC to atlases phase correlation, which is conducive to solving the problem of large scale and rotation differences between the image pairs. Subsequently, LGEPC eliminates the interface of the significant nonlinear radiometric differences by superimposing multi-scale geometric structural spectra and carrying out the phase correlation module, so that the translation can be well determined. Our experiments on synthetic images demonstrated the rationality and effectiveness of LGEPC, and the experiments on a variety of multimodal images confirmed that LGEPC can ideally achieve pixel-wise registration accuracy for multimodal image pairs that conform to the similarity transformation model.

Mutual information based multi-modal remote sensing image registration using adaptive feature weight

ABSTRACTMulti-module images registration is a challenging task in image processing, and more especially in the field of remote sensing. In this letter, we strive to present a novel mutual information scheme for image registration in remote sensing scenario based on feature map technique. We firstly take saliency detection advantages to extract geographic pattern, and then utilize the efficient Laplacian of Gaussian(LOG) and Guided Filter methods to construct a new feature map based on different characteristic of multi-channel images. To avoid practical traps of sub-optimization, we propose an novel mutual information(MI) algorithm based on an adapted weight strategy. The proposed model divides an image into patches and assigns weighted values according to patch similarities in order to solve the optimization problem, improve accuracy and enhance performance. Note that, our proposed method incorporates the LOG and Guided Filter methods into image registration for the first time to construct a new feature map based on differences and similarities strategy. Experiments are conducted over island and coastline scenes, and reveal that our hybrid model has a significant performance and outperforms the state-of-the-art methods in remote sensing image registration.

Remote sensing image registration based on Gaussian-Hermite moments and the Pseudo-RANSAC algorithm

ABSTRACTThis study deals with an important issue that is often encountered with the registration of remote sensing images which are obtained at different times and/or through inter/intra sensors. Remote sensing images may differ significantly in gray-level characteristics and contrast, among other aspects. Thus, it may be difficult to apply directly area-based approaches which are dependent on image intensity values. In this work, a novel image registration approach based on Gaussian-Hermite moments and the Pseudo-RANSAC algorithm is proposed. The problem of intensity difference commonly incurred in multi-temporal or multimodal remote sensing image registration is tackled using features that are invariant to intensity mapping during the feature point matching process. In particular, the feature points are herein represented by a range of newly introduced Gaussian-Hermite moments, and the corresponding feature points in a certain reference image are sought with the Euclidean distance measure. Moreover, an improved RANdom SAmple Consensus (RANSAC) algorithm is presented, reducing computational time complexity while improving performance in stability and accuracy. The final warping of images according to their refined feature points is conducted with bilinear interpolation. The proposed approach has been successfully applied to register synthetic and real remote sensing images, demonstrating its efficacy with systematic experimental evaluations.

Multimodal Remote Sensing Image Registration With Accuracy Estimation at Local and Global Scales

This paper focuses on potential accuracy of remote sensing images registration. We investigate how this accuracy can be estimated without ground truth available and used to improve registration quality of mono- and multi-modal pair of images. At the local scale of image fragments, the Cramer-Rao lower bound (CRLB) on registration error is estimated for each local correspondence between coarsely registered pair of images. This CRLB is defined by local image texture and noise properties. Opposite to the standard approach, where registration accuracy is only evaluated at the output of the registration process, such valuable information is used by us as an additional input knowledge. It greatly helps detecting and discarding outliers and refining the estimation of geometrical transformation model parameters. Based on these ideas, a new area-based registration method called RAE (Registration with Accuracy Estimation) is proposed. In addition to its ability to automatically register very complex multimodal image pairs with high accuracy, the RAE method provides registration accuracy at the global scale as covariance matrix of estimation error of geometrical transformation model parameters or as point-wise registration Standard Deviation. This accuracy does not depend on any ground truth availability and characterizes each pair of registered images individually. Thus, the RAE method can identify image areas for which a predefined registration accuracy is guaranteed. The RAE method is proved successful with reaching subpixel accuracy while registering eight complex mono/multimodal and multitemporal image pairs including optical to optical, optical to radar, optical to Digital Elevation Model (DEM) images and DEM to radar cases. Other methods employed in comparisons fail to provide in a stable manner accurate results on the same test cases.

Automatic approach based on point and curve features for multimodal remote sensing image registration

Due to the significant spectral difference in multimodal image registration,the rate of correctly matched feature- points calculated with point- feature detecting and matching algorithms,such as speed up Robust features,SURF,is quite low in some difficult cases. In order to solve this issue,this paper proposes a novel image registration approach with the help of point and curve features. It adopts the location information indexed by the correct matching pair of feature- points and transformation information determined by the pair of correct matching curves simultaneously. Also,the feasibility and advantages of this algorithm were all confirmed by the experiments in this paper. The results show that this method can achieve automatically the registration of the multi- modal remote sensing images in the completely unattended case and align some kinds of remote sensing images automatically. In addition,it is more robust and reliable.