3MRS: An Effective Coarse-to-Fine Matching Method for Multimodal Remote Sensing Imagery

Zhongli Fan,Li Zhang,Junjun Zhang,Yuxian Liu,Yuxuan Liu,Haibin Ai,Yushan Sun

doi:10.3390/rs14030478

Zhongli Fan, Li Zhang + Show 5 more

Open Access

https://doi.org/10.3390/rs14030478

Copy DOI

Abstract

The fusion of image data from multiple sensors is crucial for many applications. However, there are significant nonlinear intensity deformations between images from different kinds of sensors, leading to matching failure. To address this need, this paper proposes an effective coarse-to-fine matching method for multimodal remote sensing images (3MRS). In the coarse matching stage, feature points are first detected on a maximum moment map calculated with a phase congruency model. Then, feature description is conducted using an index map constructed by finding the index of the maximum value in all orientations of convolved images obtained using a set of log-Gabor filters. At last, several matches are built through image matching and outlier removal, which can be used to estimate a reliable affine transformation model between the images. In the stage of fine matching, we develop a novel template matching method based on the log-Gabor convolution image sequence and match the template features with a 3D phase correlation matching strategy, given that the initial correspondences are achieved with the estimated transformation. Results show that compared with SIFT, and three state-of-the-art methods designed for multimodal image matching, PSO-SIFT, HAPCG, and RIFT, only 3MRS successfully matched all six types of multimodal remote sensing image pairs: optical–optical, optical–infrared, optical–depth, optical–map, optical–SAR, and day–night, with each including ten different image pairs. On average, the number of correct matches (NCM) of 3MRS was 164.47, 123.91, 4.88, and 4.33 times that of SIFT, PSO-SIFT, HAPCG, and RIFT for the successfully matched image pairs of each method. In terms of accuracy, the root-mean-square error of correct matches for 3MRS, SIFT, PSO-SIFT, HAPCG, and RIFT are 1.47, 1.98, 1.79, 2.83, and 2.45 pixels, respectively, revealing that 3MRS got the highest accuracy. Even though the total running time of 3MRS was the longest, the efficiency for obtaining one correct match is the highest considering the most significant number of matches. The source code of 3MRS and the experimental datasets and detailed results are publicly available.

Highlights

channel feature of orientated gradients (CFOG) constructs descriptors in a pixel-by-pixel manner, enhancing the ability to describe detailed structures in images, and its matching performance is significantly better than sparse feature descriptors
We propose a coarse-to-fine multimodal remote sensing image matching method (3MRS) based on the 2D phase congruency model to overcome the large nonlinear intensity deformations (NID)
In order to verify the superior of 3MRS, we compare it with four state-of-the-art algorithms: scale-invariant feature transform (SIFT), PSO-SIFT, histogram of absolute phase consistency gradients (HAPCG), and radiation-variation insensitive feature transform (RIFT)

Summary

Introduction

With the fast development of sensor manufacture and space delivery technology, a multiple platform Earth observation system has been formed, providing various remote sensing data of different spatial, spectral resolutions, and different modalities. The joint use of the multimodal images from different types of sensor data can benefit many applications, such as change detection [1,2,3,4], object detection [5,6,7], and land use and Remote Sens. A fundamental prerequisite for the joint use of multimodal images is image matching, which finds accurate correspondences on two or more images with overlapped areas. It is crucial to accomplish the task of multimodal remote sensing images matching accurately and robustly

Methods

Results

Discussion

Conclusion