Set Of Putative Correspondences Research Articles

MGCNet: Multi-granularity consensus network for remote sensing image correspondence pruning

Correspondence pruning aims to remove false correspondences (outliers) from an initial putative correspondence set. This process holds significant importance and serves as a fundamental step in various applications within the fields of remote sensing and photogrammetry. The presence of noise, illumination changes, and small overlaps in remote sensing images frequently result in a substantial number of outliers within the initial set, thereby rendering the correspondence pruning notably challenging. Although the spatial consensus of correspondences has been widely used to determine the correctness of each correspondence, achieving uniform consensus can be challenging due to the uneven distribution of correspondences. Existing works have mainly focused on either local or global consensus, with a very small perspective or large perspective, respectively. They often ignore the moderate perspective between local and global consensus, called group consensus, which serves as a buffering organization from local to global consensus, hence leading to insufficient correspondence consensus aggregation. To address this issue, we propose a multi-granularity consensus network (MGCNet) to achieve consensus across regions of different scales, which leverages local, group, and global consensus to accomplish robust and accurate correspondence pruning. Specifically, we introduce a GroupGCN module that randomly divides the initial correspondences into several groups and then focuses on group consensus and acts as a buffer organization from local to global consensus. Additionally, we propose a Multi-level Local Feature Aggregation Module that adapts to the size of the local neighborhood to capture local consensus and a Multi-order Global Feature Module to enhance the richness of the global consensus. Experimental results demonstrate that MGCNet outperforms state-of-the-art methods on various tasks, highlighting the superiority and great generalization of our method. In particular, we achieve 3.95% and 8.5% mAP5° improvement without RANSAC on the YFCC100M dataset in known and unknown scenes for pose estimation, compared to the second-best models (MSA-LFC and CLNet). Source code: https://github.com/1211193023/MGCNet.

VSFormer: Visual-Spatial Fusion Transformer for Correspondence Pruning

Correspondence pruning aims to find correct matches (inliers) from an initial set of putative correspondences, which is a fundamental task for many applications. The process of finding is challenging, given the varying inlier ratios between scenes/image pairs due to significant visual differences. However, the performance of the existing methods is usually limited by the problem of lacking visual cues (e.g., texture, illumination, structure) of scenes. In this paper, we propose a Visual-Spatial Fusion Transformer (VSFormer) to identify inliers and recover camera poses accurately. Firstly, we obtain highly abstract visual cues of a scene with the cross attention between local features of two-view images. Then, we model these visual cues and correspondences by a joint visual-spatial fusion module, simultaneously embedding visual cues into correspondences for pruning. Additionally, to mine the consistency of correspondences, we also design a novel module that combines the KNN-based graph and the transformer, effectively capturing both local and global contexts. Extensive experiments have demonstrated that the proposed VSFormer outperforms state-of-the-art methods on outdoor and indoor benchmarks. Our code is provided at the following repository: https://github.com/sugar-fly/VSFormer.

SC 2-PCR++: Rethinking the Generation and Selection for Efficient and Robust Point Cloud Registration.

Outlier removal is a critical part of feature-based point cloud registration. In this article, we revisit the model generation and selection of the classic RANSAC approach for fast and robust point cloud registration. For the model generation, we propose a second-order spatial compatibility (SC 2) measure to compute the similarity between correspondences. It takes into account global compatibility instead of local consistency, allowing for more distinctive clustering between inliers and outliers at an early stage. The proposed measure promises to find a certain number of outlier-free consensus sets using fewer samplings, making the model generation more efficient. For the model selection, we propose a new Feature and Spatial consistency constrained Truncated Chamfer Distance (FS-TCD) metric for evaluating the generated models. It considers the alignment quality, the feature matching properness, and the spatial consistency constraint simultaneously, enabling the correct model to be selected even when the inlier rate of the putative correspondence set is extremely low. Extensive experiments are carried out to investigate the performance of our method. In addition, we also experimentally prove that the proposed SC 2 measure and the FS-TCD metric are general and can be easily plugged into deep learning based frameworks.

Learning Second-Order Attentive Context for Efficient Correspondence Pruning

Correspondence pruning aims to search consistent correspondences (inliers) from a set of putative correspondences. It is challenging because of the disorganized spatial distribution of numerous outliers, especially when putative correspondences are largely dominated by outliers. It's more challenging to ensure effectiveness while maintaining efficiency. In this paper, we propose an effective and efficient method for correspondence pruning. Inspired by the success of attentive context in correspondence problems, we first extend the attentive context to the first-order attentive context and then introduce the idea of attention in attention (ANA) to model second-order attentive context for correspondence pruning. Compared with first-order attention that focuses on feature-consistent context, second-order attention dedicates to attention weights itself and provides an additional source to encode consistent context from the attention map. For efficiency, we derive two approximate formulations for the naive implementation of second-order attention to optimize the cubic complexity to linear complexity, such that second-order attention can be used with negligible computational overheads. We further implement our formulations in a second-order context layer and then incorporate the layer in an ANA block. Extensive experiments demonstrate that our method is effective and efficient in pruning outliers, especially in high-outlier-ratio cases. Compared with the state-of-the-art correspondence pruning approach LMCNet, our method runs 14 times faster while maintaining a competitive accuracy.

Correspondence Attention Transformer: A Context-Sensitive Network for Two-View Correspondence Learning

Seeking reliable correspondences then recovering camera poses from a set of putative correspondences extracted from two images of the same scene is a fundamental problem in computer vision. Recent advances have demonstrated that this problem can be effectively solved by using a deep architecture based on the multi-layer perceptron, where the context normalization is designed to make the network permutation-equivariant and embed global information in the sparse point data. However, the context normalization simply normalizes the feature maps according to their distribution and treats each correspondence equally, leading to difficulties in adequately capturing scene geometry encoded by the inliers, especially in case of severe outliers. To address this issue, this paper designs a context-sensitive network based on the self-attention mechanism, termed as correspondence attention transformer (CAT), to enhance the consistent geometry information of inliers and simultaneously suppress outliers during embedding global information. In particular, we design an attention-style structure to aggregate features from all correspondences, i.e., a spatial attention namely CAT-S, which provides each correspondence with information exchange from others in the putative set. To capture the contextual information in a more comprehensive and robust way, we also introduce a multi-head mechanism in our structure to exploit the geometrical context from different aspects. Moreover, considering the high memory request in spatial attention, we propose a covariance normalized channel attention CAT-C in our framework, which can largely reduce the memory consumption and parameter scale, but it asks for eigenvalue decomposition in each attention block thus resulting in more runtime. Anyway, these two attention mechanisms can realize information exchange from the spatial or channel aspect, which both contribute to constructing the geometrical context between inliers and encourage the network to pay more attention to the feature subset about potential inliers. Extensive experiments have been conducted over both indoor and outdoor datasets on the tasks of camera pose estimation, outlier removal, and image registration, which demonstrate the superiority of our method that realizes a large performance improvement compared with the current state-of-the-art approaches.

DeTarNet: Decoupling Translation and Rotation by Siamese Network for Point Cloud Registration

Point cloud registration is a fundamental step for many tasks. In this paper, we propose a neural network named DetarNet to decouple the translation t and rotation R, so as to overcome the performance degradation due to their mutual interference in point cloud registration. First, a Siamese Network based Progressive and Coherent Feature Drift (PCFD) module is proposed to align the source and target points in high-dimensional feature space, and accurately recover translation from the alignment process. Then we propose a Consensus Encoding Unit (CEU) to construct more distinguishable features for a set of putative correspondences. After that, a Spatial and Channel Attention (SCA) block is adopted to build a classification network for finding good correspondences. Finally, the rotation is obtained by Singular Value Decomposition (SVD). In this way, the proposed network decouples the estimation of translation and rotation, resulting in better performance for both of them. Experimental results demonstrate that the proposed DetarNet improves registration performance on both indoor and outdoor scenes. Our code will be available in https://github.com/ZhiChen902/DetarNet.

RANet: A relation-aware network for two-view correspondence learning

Finding true correspondences from a set of putative correspondences is a basic task in computer vision. Recent advances have demonstrated that Multi Layer Perceptrons (MLPs) can handle the unordered correspondence learning problem by training a deep classifier. However, MLPs ignore the relationship between correspondences, such as geometric information, spatial information and inlier distribution information, which will lead to difficulties in modeling complex global context. To solve this issue, we propose a Relation-Aware Network (RANet), by capturing rich global information from channel and spatial dimensions, to establish reliable correspondences for feature matching. Specifically, we firstly present a Global Context Attention block by a two-branch attention structure to cooperate with MLPs for contextual information extraction. Then, we design a Relation-Aware Filter block by further exploring the channel and spatial relationship information with different connection manners. Finally, we combine two blocks to obtain an enhanced basic block with strong feature representation capacity due to the acquisition of global contextual information. Our experiments have been conducted over both indoor and outdoor datasets on the tasks of outlier removal and camera pose estimation, which demonstrate the superiority of our network that achieves the best performance compared with the state-of-the-art approaches.

Cascade Network with Guided Loss and Hybrid Attention for Finding Good Correspondences

Finding good correspondences is a critical prerequisite in many feature based tasks. Given a putative correspondence set of an image pair, we propose a neural network which finds correct correspondences by a binary-class classifier and estimates relative pose through classified correspondences. First, we analyze that due to the imbalance in the number of correct and wrong correspondences, the loss function has a great impact on the classification results. Thus, we propose a new Guided Loss that can directly use evaluation criterion (Fn-measure) as guidance to dynamically adjust the objective function during training. We theoretically prove that the perfect negative correlation between the Guided Loss and Fn-measure, so that the network is always trained towards the direction of increasing Fn-measure to maximize it. We then propose a hybrid attention block to extract feature, which integrates the Bayesian attentive context normalization (BACN) and channel-wise attention (CA). BACN can mine the prior information to better exploit global context and CA can capture complex channel context to enhance the channel awareness of the network. Finally, based on our Guided Loss and hybrid attention block, a cascade network is designed to gradually optimize the result for more superior performance. Experiments have shown that our network achieves the state-of-the-art performance on benchmark datasets. Our code will be available in https://github.com/wenbingtao/GLHA.

MANet: Multi-Scale Attention Network for Correspondence Learning

Establishing reliable correspondences from a putative correspondence set is a challenging task. Most of state-of-the-art methods utilize the local context and global context to address the task. However, the local and global context often contains large number of outliers, which have a negative impact on capturing scene geometry. In this paper, we propose a Multi-scale Attention Network (called MANet), which introduces the attention mechanism for feature matching, to improve the ability of capturing scene geometry. Specifically, we first fuse the features of low and high levels by an multi-scale strategy network to enhance the representative ability of features. Then, we propose an attentive PointCN block and an attentive pooling layer, to discriminatively capture global context and local context information, respectively. We demonstrate through extensive experiments on both indoor and outdoor datasets that MANet provides a significant improvement in the performance of the two-view geometry and correspondences accuracy compared to the state-of-the-art methods.

Weakly supervised learning for image keypoint matching using graph convolutional networks

Matching between two sets of features from a pair of images is a fundamental and critical step in most computer vision tasks. Existing attempts typically establish a set of putative correspondences with the nearest-neighbor rule in feature spaces and then try to find a subset of reliable matches. However, when there are large camera angles, repetitive structures, and illumination changes existing in the two images of the same scene, recently proposed feature matching approaches do not work well to find good correspondences, especially with a higher proportion of false-positive matches in the putative set. To address these problems, we propose a novel weakly supervised Graph Convolutional Siamese Network Matcher, called GCSNMatcher, to learn the correct correspondences for image feature matching. In particular, GCSNMatcher can directly work on unstructured keypoint sets and further exploit geometric information among sparse interest points by constructing dynamic neighborhood graph structures to enhance the ability of the feature representation of each keypoint. With channel-wised symmetric aggregation operations in our graph convolutional neural networks, the performance of our matcher does not vary under different permutations of unordered keypoint sets. Empirical studies on Yahoo’s YFCC100M benchmark dataset demonstrate that our matcher can give a more robust performance for image matching tasks than those state-of-the-art methods, even when it is trained on small datasets.

A GMS-Guided Approach for 2D Feature Correspondence Selection

Feature correspondence selection, which aims to seek as many true matches (i.e., inliers) as possible from a given putative set while minimizing false matches (i.e., outliers), is crucial to many feature-matching based tasks in computer vision. It remains a challenging problem how to deal with putative sets with low inlier ratios. To address this problem, in this paper, we propose a novel correspondence selection strategy, which is guided by Grid-based Motion Statistics (GMS). We first adopt the GMS to generate a small correspondence set with a high inlier ratio. Then, an accurate geometric model is built using the above correspondence set. Finally, the built geometric model is used to filter the given putative correspondence set to obtain true correspondences. The experimental results on benchmark datasets demonstrate that our proposed approach outperforms the state-of-the-art approaches for putative sets with various inlier ratios, especially for cases with low inlier ratios.

Fast point matching using corresponding circles

Point matching via corresponding circles (ICC) is a technique for removing outliers (mismatches) from given putative point correspondences in image pairs. It can be used as a basis for a wide range of applications including structure-from-motion, 3D reconstruction, tracking, image retrieval, registration, and object recognition. In this paper, we propose a new method called Fast Identification of point correspondences by Corresponding Circles (FICC) that improves the quality of the rejection mismatches and reduces the cost of computing it. In particular, we propose a new strategy that aims to take better advantage of the corresponding circles and reduces the number of putative points correspondences tested by the corresponding circles in each iteration rather than all set of putative correspondences, as in the original ICC. This reduces the computing time and together with a more efficient tool for rejecting mismatches which leads to significant gains in efficiency. We provide comparative results illustrating the improvements obtained by FICC over ICC, and we compare with many state-of-the-art methods.

Fast non-rigid image feature matching for agricultural UAV via probabilistic inference with regularization techniques

Image feature matching is one of the most important fundamental technologies for agricultural unmanned aerial vehicle (UAV). There has been some attempt on this task, but the performance and efficiency are still not satisfactory due to the complexity of UAV images, especially undergo non-rigid transformation. In this paper, we propose a probabilistic method to address these problems. We start by creating a set of putative correspondences based on the feature similarity and then focus on removing outliers from the putative set and estimating the transformation as well. This is formulated as a maximum likelihood estimation of a Bayesian model with latent variables indicating whether matches in the putative set are inliers or outliers. We enforce three effective regularization techniques on the correspondence in a reproducing kernel Hilbert space simultaneously, which helps to find an optimal solution. The problem is finally solved by using the expectation–maximization algorithm, and the closed-form solution of the transformation is derived in the maximization step. Moreover, a fast implementation based on sparse approximation is given which reduces the method computation complexity to quadratic without performance sacrifice. Extensive experiments on real farmland images demonstrate accurate and robust results of the proposed method which outperforms current state-of-the-art methods, particularly in case of severe outliers.

Gaussian field consensus: A robust nonparametric matching method for outlier rejection

In this paper, we propose a robust method, called Gaussian Field Consensus (GFC), for outlier rejection from given putative point set matching correspondences. Finding correct correspondences (inliers) is a key component in many computer vision and pattern recognition tasks, and the goal of outlier (mismatch) rejection is to fit the transformation function that maps one feature point set to another. Our GFC starts by inputting a putative correspondence set which is contaminated by many outliers, and the main task of our GFC is to identify the underlying true correspondences from outliers. Then we formulate this challenging problem by Gaussian Field nonparametric matching model which bases on the exponential distance loss and kernel method in a reproducing kernel Hilbert space. Next, We introduce a local linear constraint based on the regularization theory to preserve the topological structure of the feature points. Moreover, the sparse approximation is used to reduce the search space, in this way, we can handle a large number of points easily. Finally, we test the GFC method on several real image datasets in the presence of outliers, where the experimental results show that our proposed method outperforms current state-of-the-art methods in most test scenarios.

Retinal image registration via feature-guided Gaussian mixture model.

Registration of retinal images taken at different times, from different perspectives, or with different modalities is a critical prerequisite for the diagnoses and treatments of various eye diseases. This problem can be formulated as registration of two sets of sparse feature points extracted from the given images, and it is typically solved by first creating a set of putative correspondences and then removing the false matches as well as estimating the spatial transformation between the image pairs or solved by estimating the correspondence and transformation jointly involving an iteration process. However, the former strategy suffers from missing true correspondences, and the latter strategy does not make full use of local appearance information, which may be problematic for low-quality retinal images due to a lack of reliable features. In this paper, we propose a feature-guided Gaussian mixture model (GMM) to address these issues. We formulate point registration as the estimation of a feature-guided mixture of densities: A GMM is fitted to one point set, such that both the centers and local features of the Gaussian densities are constrained to coincide with the other point set. The problem is solved under a unified maximum-likelihood framework together with an iterative expectation-maximization algorithm initialized by the confident feature correspondences, where the image transformation is modeled by an affine function. Extensive experiments on various retinal images show the robustness of our approach, which consistently outperforms other state-of-the-art methods, especially when the data is badly degraded.

Nonrigid Feature Matching for Remote Sensing Images via Probabilistic Inference With Global and Local Regularizations

In this letter, we propose a probabilistic method for the feature matching of remote sensing images which undergo nonrigid transformations. We start by creating a set of putative correspondences based on the feature similarity and then focus on removing outliers from the putative set and estimating the transformation as well. This is formulated as a maximum likelihood estimation of a Bayesian model with latent variables indicating whether matches in the putative set are inliers or outliers. We impose nonparametric global geometrical constraints on the correspondence using Tikhonov regularizers in a reproducing kernel Hilbert space. We also introduce a local geometrical constraint to preserve local structures among neighboring feature points. The problem is solved by using the expectation–maximization algorithm, and the closed-form solution of the transformation is derived in the maximization step. Moreover, a fast implementation based on sparse approximation is given which reduces the method computation complexity to linearithmic without performance sacrifice. Extensive experiments on real remote sensing images demonstrate accurate results of the proposed method which outperforms current state-of-the-art methods, particularly in case of severe outliers.

Robust Feature Matching for Remote Sensing Image Registration via Locally Linear Transforming

Feature matching, which refers to establishing reliable correspondence between two sets of features (particularly point features), is a critical prerequisite in feature-based registration. In this paper, we propose a flexible and general algorithm, which is called locally linear transforming (LLT), for both rigid and nonrigid feature matching of remote sensing images. We start by creating a set of putative correspondences based on the feature similarity and then focus on removing outliers from the putative set and estimating the transformation as well. We formulate this as a maximum-likelihood estimation of a Bayesian model with hidden/latent variables indicating whether matches in the putative set are outliers or inliers. To ensure the well-posedness of the problem, we develop a local geometrical constraint that can preserve local structures among neighboring feature points, and it is also robust to a large number of outliers. The problem is solved by using the expectation–maximization algorithm (EM), and the closed-form solutions of both rigid and nonrigid transformations are derived in the maximization step. In the nonrigid case, we model the transformation between images in a reproducing kernel Hilbert space (RKHS), and a sparse approximation is applied to the transformation that reduces the method computation complexity to linearithmic. Extensive experiments on real remote sensing images demonstrate accurate results of LLT, which outperforms current state-of-the-art methods, particularly in the case of severe outliers (even up to 80%).

Image Feature Matching via Progressive Vector Field Consensus

In this letter, we propose a simple yet effective approach, named Progressive Vector Field Consensus (PVFC), for addressing the problem of finding more true feature correspondences between images. The key idea is to progressively perform feature matching based on Vector Field Consensus, and hence greatly boost the number of true matches as well as avoid false matches. More specifically, it uses matching results on a small putative correspondence set with high inlier ratio to guide the matching on a large putative correspondence set which probably covers the whole true correspondences. We model the transformation between images in a reproducing kernel Hilbert space, and a sparse approximation is applied to the transformation to avoid high computational complexity. Our results quantitatively show that our PVFC outperforms state-of-the-art methods, both in accuracy and in efficiency. Moreover, the progressive framework is general and can be applied to other cases for robust estimation.

Automatic visible and infrared face registration based on silhouette matching and robust transformation estimation

Registration of multi-sensor data (particularly visible color sensors and infrared sensors) is a prerequisite for multimodal image analysis such as image fusion. In this paper, we proposed an automatic registration technique for visible and infrared face images based on silhouette matching and robust transformation estimation. The key idea is to represent a (visible or infrared) face image by its silhouette which is extracted from the image’s edge map and consists of a set of discrete points, and then align the two silhouette point sets by using their feature similarity and spatial geometrical information. More precisely, our algorithm first matches the silhouette point sets by their local shape features such as shape context, which creates a set of putative correspondences that may contaminated by outliers. Next, we estimate the accurate transformation from the putative correspondence set under a robust maximum likelihood framework combining with the EM algorithm, where the transformation between the image pair is modeled by a parametric model such as the rigid or affine transformation. The qualitative and quantitative comparisons on a publicly available database demonstrate that our method significantly outperforms other state-of-the-art visible/infrared face registration methods. As a result, our method will be beneficial for fusion-based face recognition.

Robust point matching via vector field consensus.

In this paper, we propose an efficient algorithm, called vector field consensus, for establishing robust point correspondences between two sets of points. Our algorithm starts by creating a set of putative correspondences which can contain a very large number of false correspondences, or outliers, in addition to a limited number of true correspondences (inliers). Next, we solve for correspondence by interpolating a vector field between the two point sets, which involves estimating a consensus of inlier points whose matching follows a nonparametric geometrical constraint. We formulate this a maximum a posteriori (MAP) estimation of a Bayesian model with hidden/latent variables indicating whether matches in the putative set are outliers or inliers. We impose nonparametric geometrical constraints on the correspondence, as a prior distribution, using Tikhonov regularizers in a reproducing kernel Hilbert space. MAP estimation is performed by the EM algorithm which by also estimating the variance of the prior model (initialized to a large value) is able to obtain good estimates very quickly (e.g., avoiding many of the local minima inherent in this formulation). We illustrate this method on data sets in 2D and 3D and demonstrate that it is robust to a very large number of outliers (even up to 90%). We also show that in the special case where there is an underlying parametric geometrical model (e.g., the epipolar line constraint) that we obtain better results than standard alternatives like RANSAC if a large number of outliers are present. This suggests a two-stage strategy, where we use our nonparametric model to reduce the size of the putative set and then apply a parametric variant of our approach to estimate the geometric parameters. Our algorithm is computationally efficient and we provide code for others to use it. In addition, our approach is general and can be applied to other problems, such as learning with a badly corrupted training data set.