Synthetic Aperture Radar Image Registration Research Articles

A Sub-Second Method for SAR Image Registration Based on Hierarchical Episodic Control

For Synthetic Aperture Radar (SAR) image registration, successive processes following feature extraction are required by both the traditional feature-based method and the deep learning method. Among these processes, the feature matching process—whose time and space complexity are related to the number of feature points extracted from sensed and reference images, as well as the dimension of feature descriptors—proves to be particularly time consuming. Additionally, the successive processes introduce data sharing and memory occupancy issues, requiring an elaborate design to prevent memory leaks. To address these challenges, this paper introduces the OptionEM-based reinforcement learning framework to achieve end-to-end SAR image registration. This framework outputs registered images directly without requiring feature matching and the calculation of the transformation matrix, leading to significant processing time savings. The Transformer architecture is employed to learn image features, while a correlation network is introduced to learn the correlation and transformation matrix between image pairs. Reinforcement learning, as a decision process, can dynamically correct errors, making it more-efficient and -robust compared to supervised learning mechanisms such as deep learning. We present a hierarchical reinforcement learning framework combined with Episodic Memory to mitigate the inherent problem of invalid exploration in generalized reinforcement learning algorithms. This approach effectively combines coarse and fine registration, further enhancing training efficiency. Experiments conducted on three sets of SAR images, acquired by TerraSAR-X and Sentinel-1A, demonstrated that the proposed method’s average runtime is sub-second, achieving subpixel registration accuracy.

RTV-SIFT: Harnessing Structure Information for Robust Optical and SAR Image Registration

Registration of optical and synthetic aperture radar (SAR) images is challenging because extracting located identically and unique features on both images are tricky. This paper proposes a novel optical and SAR image registration method based on relative total variation (RTV) and scale-invariant feature transform (SIFT), named RTV-SIFT, to extract feature points on the edges of structures and construct structural edge descriptors to improve the registration accuracy. First, a novel RTV-Harris feature point detection method by combining the RTV and the multiscale Harris algorithm is proposed to extract feature points on both images’ significant structures. This ensures a high repetition rate of the feature points. Second, the feature point descriptors are constructed on enhanced phase congruency edge (EPCE), which combines the Sobel operator and maximum moment of phase congruency (PC) to extract edges from structured images that enhance robustness to nonlinear intensity differences and speckle noise. Finally, after coarse registration, the position and orientation Euclidean distance (POED) between feature points is utilized to achieve fine feature point matching to improve the registration accuracy. The experimental results demonstrate the superiority of the proposed RTV-SIFT method in different scenes and image capture conditions, indicating its robustness and effectiveness in optical and SAR image registration.

SAR and Optical Image Registration Based on Deep Learning with Co-Attention Matching Module

Image registration is the basis for the joint interpretation of synthetic aperture radar (SAR) and optical images. However, the significant nonlinear radiation difference (NRD) and the geometric imaging model difference render the registration quite challenging. To solve this problem, both traditional and deep learning methods are used to extract structural information with dense descriptions of the images, but they ignore that structural information of the image pair is coupled and often process images separately. In this paper, a deep learning-based registration method with a co-attention matching module (CAMM) for SAR and optical images is proposed, which integrates structural feature maps of the image pair to extract keypoints of a single image. First, joint feature detection and description are carried out densely in both images, for which the features are robust to radiation and geometric variation. Then, a CAMM is used to integrate both images’ structural features and generate the final keypoint feature maps so that the extracted keypoints are more distinctive and repeatable, which is beneficial to global registration. Finally, considering the difference in the imaging mechanism between SAR and optical images, this paper proposes a new sampling strategy that selects positive samples from the ground-truth position’s neighborhood and augments negative samples by randomly sampling distractors in the corresponding image, which makes positive samples more accurate and negative samples more abundant. The experimental results show that the proposed method can significantly improve the accuracy of SAR–optical image registration. Compared to the existing conventional and deep learning methods, the proposed method yields a detector with better repeatability and a descriptor with stronger modality-invariant feature representation.

Optical and SAR Image Registration Based on the Phase Congruency Framework

The improved phase congruency (PC) algorithms have been successfully applied to optical and synthetic aperture radar (SAR) image registration since they are insensitive to nonlinear radiometric and geometric differences. However, most of the algorithms are sensitive to large-scale differences and rotation differences between optical and SAR images. To tackle this, we propose a PC framework to register optical and SAR images. It is compatible with large-scale and rotation invariance. Firstly, a multi-scale Harris keypoint extraction method based on the maximum moment of PC (named PC-Harris) is proposed. The scale space is constructed by combining PC with the log-Gabor filter. Secondly, we propose a PC model to construct the feature descriptors. The orientation and amplitude responses are obtained based on the PC model. Meanwhile, the novel descriptor is constructed based on the polar coordinate system and thus can handle the scale and rotation differences between optical and SAR images. Finally, outliers are removed by the fast sample consensus (FSC). The experiments conducted on several optical and SAR images verify the effectiveness of the proposed framework.

Optical and SAR Image Registration Based on Multi-Scale Orientated Map of Phase Congruency

Optical and Synthetic Aperture Radar (SAR) images are highly complementary, and their registrations are a fundamental task for other remote sensing applications. Traditional feature-matching algorithms fail to solve the significant nonlinear radiation difference (NRD) caused by different sensors. To address this problem, a robust registration algorithm with the multi-scale orientated map of phase congruency (MSPCO) is proposed. First, a nonlinear diffusion scale space is established to obtain the scale invariance of feature points. Compared with the linear Gaussian scale space, the nonlinear diffusion scale space can better preserve the edge and texture information. Second, to ensure the quantity and repeatability of features, corner points and edge points are detected on the moment map of phase congruency, respectively, which is the foundation to the next feature matching. Third, the MSPCO descriptor is constructed via the orientation of phase congruency (PCO). PCO is highly robust to NRD, and the different scales of PCOs enhance the robustness of the descriptor. Finally, a feature-matching strategy based on an effective scale ratio is proposed, which reduces the number of comparisons among features and improves computational efficiency. The experimental results show that the proposed method is better than the existing feature-based methods in terms of the number of correct matches and registration accuracy. The registration accuracy is only inferior to that of the most advanced template matching method, and the accuracy difference is within 0.3 pixels, which fully demonstrates the robustness and accuracy of the proposed method in optical and SAR image registration.

A Survey on SAR and Optical Satellite Image Registration

After decades of research, automatic synthetic aperture radar (SAR)-optical registration remains an unsolved problem. SAR and optical satellites utilize different imaging mechanisms, resulting in imagery with dissimilar heterogeneous characteristics. Transforming and translating these characteristics into a shared domain has been the main challenge in SAR-optical matching for many years. Combining the two sensors will improve the quality of existing and future remote sensing applications across multiple industries. Several approaches have emerged as promising candidates in the search for combining SAR and optical imagery. In addition, recent research has indicated that machine learning-based approaches have great potential for filling the information gap posed by utilizing only one sensor type in Earth observation applications. However, several challenges remain, and combining them is a multi-step process where no one-size-fits-all approach is available. This article reviews traditional, state-of-the-art, and recent development trends in SAR-optical co-registration methods.

SAR and Optical Image Registration Based on Uniform Feature Points Extraction and Consistency Gradient Calculation

Synthetic aperture radar (SAR) satellites have an active sensor on board, which emits electromagnetic signals and measures the strength and time delay of the returned signal backscattered from ground objects. Optical images have rich spectral information, but it is easily affected by atmospheric attenuation and weather conditions. Thus, the study of the registration between these two images is of great significance. We present a novel method for SAR and optical image registration. In the stage of feature points extraction, the method combines phase consistency intensity screening and scale space grid division to obtain stable and uniform feature points from the image. During the stage of feature description, the method employs the extended phase consistency method to calculate the gradient amplitude and direction of the image, and improves the correctness of the main direction calculation and descriptor construction. Experimental results demonstrate its superior matching performance with respect to the state-of-the-art methods.

Self-Supervised SAR Image Registration With SAR-Superpoint and Transformation Aggregation

Owing to various factors, including severe speckle noise and orbit direction differences, performing multitemporal synthetic aperture radar (SAR) image registration with high accuracy and robustness may become difficult. Herein, an efficient self-supervised deep learning registration network for multitemporal SAR image registration, SAR-superpoint and transformation aggregation network (SSTA-Net), is proposed. The SSTA-Net consists of three parts: 1) the SAR-Superpoint detection network (SS-Net); 2) the transformation aggregation feature matching network (TA-Net); and 3) the unstable point removal module. Specifically, a pseudolabel generation method is adopted without additional annotations. It transfers the characteristics of real SAR data to synthetic data through a feature transition module, which can generate feature point labels for real SAR images for self-training SS-Net. Furthermore, a position–channel aggregation attention is proposed and embedded into the SS-Net to efficiently capture position and channel information and to increase the stability and accuracy of feature point identification. Finally, a unique transformation aggregation strategy is designed to improve the robustness of feature matching, and an unstable point removal module is adopted to eliminate the mismatched point pairs caused by orbit differences. Six sets of multitemporal SAR images were used to evaluate the registration performance of the SSTA-Net, and our model was also compared with the traditional and deep learning algorithms. The experimental results demonstrate that the SSTA-Net outperforms various state-of-the-art approaches for SAR image registration.

A Robust Multiscale Edge Detection Method for Accurate SAR Image Registration

Edge detection is a technique used to identify inherent structures within an image, and it is an essential requirement for synthetic aperture radar (SAR) applications. In particular, ratio-based edge detectors have been widely used in SAR image registration because of their ability to extract invariant features and reduce the effects of speckle noise. However, current edge detectors often struggle to accurately detect multi-scale objects and low-contrast structures. To address this issue, we present a robust multi-scale edge detector that uses a modified convolution kernel to improve the extensibility of edge features and aggregates multi-scale feature responses. We also propose a local scale estimation module to enhance edge responses in low-contrast areas and reduce noise effects. The experimental results demonstrate that our proposed method effectively preserves the integrity, continuity, and robustness of multi-scale and low-contrast structures. By incorporating our proposed edge detector into feature and template matching frameworks, we are able to significantly improve matching accuracy and outperform state-of-the-art SAR image registration methods.

CFOG-like image registration algorithm based on 3D-structural feature descriptor for suburban optical and SAR

Owing to nonlinear radiation differences between optical and synthetic aperture radar (SAR) images, and the inevitable SAR speckle noise, the registration of optical and SAR images is challenging work. This paper proposes a novel pixel-wise 3D-structural feature descriptor based on channel-features-of-oriented-gradients (CFOG), thus, it inherits the fast and robust advantages of CFOG. First, the Block-Harris and grid points feature extraction method is used to obtain sufficient uniformly distributed keypoints from the reference images. Then, the new feature descriptor based on the combination of first-order oriented gradients and second-order gradients (FOSG) is proposed. Specifically, two different operators, the Sobel and the ratio of exponentially weighted averages are utilized to calculate the gradient for the optical and SAR images, respectively. On this basis, the first-order oriented gradients and second-order gradients are calculated by a 3D-Gaussian filter and then normalized along the Z-direction to form the FOSG by superposition. Finally, template matching based on the sum of squared differences similarity metric is implemented in the frequency domain. Our method is experimentally compared with five state-of-the-art methods using six pairs of suburban optical and SAR images. The results show that the proposed algorithm is robust to nonlinear radiation differences and SAR speckle noise, and achieves high accuracy.

Rotation-Invariant Registration Algorithm of Optical and SAR Images Using Fast Bilateral Filters and Fourier-Based Descriptors

Determining precise main orientations for optical and synthetic aperture radar (SAR) image registration is challenging because of serious radiometric differences and multiplicative speckles. In this letter, a rotation-invariant registration algorithm of optical and SAR images is proposed based on fast bilateral filters and Fourier-based descriptors. First, two nonlinear scale spaces are, respectively, designed for optical and SAR images based on fast bilateral filters to preserve structural features of the images while smoothing. The consistent gradients of the images are calculated in the process using the Sobel operator and the ratio of exponentially weighted average operator, respectively. Then, a novel framework is proposed for rotation-invariant optical and SAR image registration. In the framework, Harris detectors are utilized to detect sufficient and reliable feature points in the scale spaces, and downsampled Fourier-based descriptors, achieving a high discriminability by avoiding the process of orientation assignment, are designed for rotation-invariant and fast feature description. Experiments on three pairs of optical and SAR images demonstrate the effectiveness of the proposed algorithm compared to existing state-of-the-art algorithms.

Mountainous SAR Image Registration Using Image Simulation and an L2E Robust Estimator

Synthetic Aperture Radar (SAR) is one of the most widely utilized methods to extract elevation information and identify large-scale deformations in mountainous areas. Homologous points in stereo SAR image pairs are difficult to identify due to complex geometric and radiometric distortions. In this paper, a new approach for mountainous area images is suggested. Firstly, a simulated SAR image and a look-up table based on DEM data are generated by a range-Doppler model and an empirical formula. Then, a point matching RPM-L2E algorithm is used to match images obtained by the simulation and in real-time to indirectly obtain the feature points of the real SAR images. Finally, the accurate registration of mountainous areas in the SAR images is achieved by a polynomial transform. Experimental verification is performed by using the data of mountainous SAR images from the same sensor and different sensors. When the registration accuracy of the method is compared with that of two state-of-the-art image registration algorithms, better outcomes are experimentally shown. The suggested approach can effectively solve the registration problem of SAR images of mountainous areas, and can overcome the disadvantages of poor adaptability and low accuracy of traditional SAR image registration methods for mountainous areas.

A Robust Strategy for Large-Size Optical and SAR Image Registration

The traditional template matching strategy of optical and synthetic aperture radar (SAR) is sensitive to the nonlinear transformation between two images. In some cases, the optical and SAR image pairs do not conform to the affine transformation condition. To address this issue, this study presents a novel template matching strategy which uses the One-Class Support Vector Machine (SVM) to remove outliers. First, we propose a method to construct the similarity map dataset using the SEN1-2 dataset for training the One-Class SVM. Second, a four-step strategy for optical and SAR image registration is presented in this paper. In the first step, the optical image is divided into some grids. In the second step, the strongest Harris response point is selected as the feature point in each grid. In the third step, we use Gaussian pyramid features of oriented gradients (GPOG) descriptor to calculate the similarity map in the search region. The trained One-Class SVM is used to remove outliers through similarity maps in the fourth step. Furthermore, the number of improve matches (NIM) and the rate of improve matches (RIM) are designed to measure the effect of image registration. Finally, this paper designs two experiments to prove that the proposed strategy can correctly select the matching points through similarity maps. The experimental results of the One-Class SVM in dataset show that the One-Class SVM can select the correct points in different datasets. The image registration results obtained on the second experiment show that the proposed strategy is robust to the nonlinear transformation between optical and SAR images.

MOTIF: MULTI-ORIENTATION TENSOR INDEX FEATURE DESCRIPTOR FOR SAR-OPTICAL IMAGE REGISTRATION

Abstract. The inherent speckle noise in synthetic aperture radar (SAR) images and the significant differences between SAR and optical images in nonlinear radiation give rise to the great difficulty in computing similarity between image features, improving detection accuracy of corresponding points and the efficiency of image matching, thus making the registration of SAR and optical images a long-standing challenging task. To address these issues, a new SAR-optical image registration method was proposed in this paper, namely, Multi-orientation Tensor Index Feature (MoTIF), which is characterized by a lightweight feature descriptor. Specifically, we firstly established a diffusion tensor model based on the information of image gradient orientation. Then, the model was parameterized using polar coordinates to help identify the MoTIF and get the array of indices of maximum value, with which we could draw a multi-orientation index map and thereupon construct the feature vector descriptors. To evaluate the proposed method, seven representative SAR-optical image pairs were tested along with a comparison with other four state-of-the-art methods. Results show that our MoTIF method outperforms the other methods in that it substantially de-speckles SAR images, overcomes nonlinear radiation distortions caused by the differences between SAR and optical images, and achieves high precision and efficiency in image registration. The average number of correct matches (NCM) of 151.0 and the root of mean-squared error (RMSE) of 1.66 pixels obtained by utilizing MoTIF with lower time consumption adds more evidence to its superior performance. The time consumption of the MoTIF method is better than that of the other four methods, and the calculation speed is 4 times faster than that of the LGHD method. Executable code and test data are published in the link https://skyearth.org/publication/project/MoTIF/

Robust registration of SAR and optical images based on deep learning and improved Harris algorithm

Traditional algorithms can achieve good results when registering homologous images, but it cannot reach satisfying results for registration between synthetic aperture radar (SAR) and optical images. The difficulty is that the image texture information and structures of different modalities is very different which leads to poor registration results. To solve this problem, we present a robust matching framework for registration between SAR and optical images. First, a novel deep learning network is utilized to generate high quality pseudo-optical images from SAR images. Next, feature points are detected and extracted using the multi-scale Harris algorithm. Then the feature points are constructed through the gradient position orientation histogram method. Finally, the actual position of the feature points will be reconstructed through a feedback mechanism for matching. Experimental results demonstrate its superior matching performance with respect to the state-of-the-art methods.

A Transformer-Based Coarse-to-Fine Wide-Swath SAR Image Registration Method under Weak Texture Conditions

As an all-weather and all-day remote sensing image data source, SAR (Synthetic Aperture Radar) images have been widely applied, and their registration accuracy has a direct impact on the downstream task effectiveness. The existing registration algorithms mainly focus on small sub-images, and there is a lack of available accurate matching methods for large-size images. This paper proposes a high-precision, rapid, large-size SAR image dense-matching method. The method mainly includes four steps: down-sampling image pre-registration, sub-image acquisition, dense matching, and the transformation solution. First, the ORB (Oriented FAST and Rotated BRIEF) operator and the GMS (Grid-based Motion Statistics) method are combined to perform rough matching in the semantically rich down-sampled image. In addition, according to the feature point pairs, a group of clustering centers and corresponding images are obtained. Subsequently, a deep learning method based on Transformers is used to register images under weak texture conditions. Finally, the global transformation relationship can be obtained through RANSAC (Random Sample Consensus). Compared with the SOTA algorithm, our method’s correct matching point numbers are increased by more than 2.47 times, and the root mean squared error (RMSE) is reduced by more than 4.16%. The experimental results demonstrate that our proposed method is efficient and accurate, which provides a new idea for SAR image registration.

Optical and SAR Image Registration Using Complexity Analysis and Binary Descriptor in Suburban Areas

Optical and synthetic aperture radar (SAR) image registration is a challenging task due to significant geometric and radiometric differences. In particular, the strong scattering phenomenon in SAR images can seriously affect the registration results. Accordingly, to solve the low repeatability of the key points in optical and SAR images, a complexity analysis scheme is proposed. In the first phase, the complexity distribution diagram is calculated by the threshold sliding window in the edge images obtained from the maximum moment of the phase congruency. Then morphological operation and connected regions are used to obtain the high-complexity regions and mask them to avoid the extraction of the interference points. Next to solve the limitations of the local self-similarity (LSS) descriptor in the optical and SAR image registration, such as poor discrimination, expensive computational complexity, and sensitivity to large geometric and radiometric difference, we propose a binary LSS descriptor (BLSS). We replace the correlation surface of the LSS with the local gradient orientation histogram. Furthermore, we construct descriptors based on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">XY</i> -coordinate system and convert the correlation of the regions to binary descriptors. Finally, the fast sample consensus (FSC) is used to remove false correspondences. The experiments conducted on several optical and SAR image pairs verify the effectiveness of the proposed algorithm.

KAZE-SAR: SAR Image Registration Using KAZE Detector and Modified SURF Descriptor for Tackling Speckle Noise

Synthetic aperture radar (SAR) image registration is still a challenging task in remote sensing. The scale-invariant feature transform (SIFT) method and its extensions are most widely used feature detectors in SAR images. However, due to the presence of nonlinear speckle noise, some wrong features are chosen which directly influence the feature matching process. In this article, for the first time, the KAZE algorithm with a modified version of speeded-up robust features (SURF) descriptor is used for SAR image registration. It uses discrete stochastic second order nonlinear partial differential equations (PDEs) to model the SAR image’s edge structure. KAZE uses nonlinear diffusion filtering to build up the scale levels of the SIFT descriptor. It preserves the edges while simultaneously decreasing the speckle noise in smoothing the image. Therefore, it captures the exact location of features in down-sampling and filtering stages of SIFT. Experimental results show that our proposed method has increased the localization accuracy and distinctiveness of feature extraction through the use of KAZE detector. Our method is also able to handle challenging cases in different image sources, different view angles, different image times, complex affine mapping, presence of blurring, and noise.

Combining Optimized SAR-SIFT Features and RD Model for Multisource SAR Image Registration

Multisource synthetic aperture radar (SAR) image registration is a difficult task in remote sensing due to the influence of speckle noise and geometric distortions between the images. The SAR scale-invariant feature transform (SAR-SIFT) is a prior option to extract features for automatic registration of SAR images. However, most registration methods based on the SAR-SIFT operator only use the image information (the geometric distribution of the pixels making up the feature) and do not consider the influences of the side-looking imagery mode and the topographic relief. In this article, we propose a novel registration method to address the above problems, which combines the optimized SAR-SIFT features and the imaging geometry of the SAR system. The major improvement of the algorithm is the use of the range–Doppler (RD) model in feature matching and geometric transformation estimation. A new local matching method aided by the RD model, transforming the global matching mode to the local one, is first introduced to avoid the gross errors and improve the matching efficiency. Then, an RD-model-based geometric transformation model is presented for multisource SAR images with obvious geometric distortions. In addition, the stationary wavelet transform (SWT) is brought in the feature extraction to optimize the SAR-SIFT keypoints to extract reliable and uniformed features. The experimental results fully demonstrate the applicability of the proposed method for registration of multisource SAR images in different acquiring configurations, especially with different passes.

Robust Optical and SAR Image Registration Based on Phase Congruency Scale Space

Automatic optical and synthetic aperture radar (SAR) image registration is still a challenging task due to significant non-linear radiation distortions (NRD) between such images and the speckle noise in the SAR images. This letter addresses this problem by proposing a novel descriptor named the Phase Space Feature Descriptor (PSFD), which is based on the Phase Congruency Scale Space. The proposed PSFD aims to capture the structural similarity between optical and SAR images. In the definition of the proposed similarity metric, we first extend the phase congruency model in different frequencies to generate the Phase Congruency Scale Space. The PSFD is robust against nonlinear radiation distortions because Phase Congruency Scale Space uses different frequencies of log Gabor wavelet information to filter out noise while extracting information. In the first experiment, compared to other methods, PSFD can locate the correct match point precisely in different scenes.In the second matching, the results show that PSFD is robust against NRD and the speckle noise of the SAR images, outperforming the other proposed algorithm.