Point Matching Problem Research Articles

Evaluating Learning-based Tie Point Matching for Geometric Processing of Off-Track Satellite Stereo

Abstract. Tie-point matching of off-track stereo images is a very challenging task, which can impact bias compensation and digital surface model (DSM) generation. Compared to in-track stereo images, off-track stereo images are more complex primarily due to the radiometric differences caused by sun illumination, sensor responses, atmospheric conditions, and seasonal land cover variations, and secondly due to the longer baseline and larger intersection angle. These challenges significantly limit the use of the vast number of images in satellite archives for automated geometric processing and mapping. Recent advances in deep learning (DL) based matching show promising results against images with diverse illuminations, viewing angles and scales through learning examples. This paper evaluates the potentials of addressing the tie point matching problems in off-track satellite stereo images. Specifically, we focus on stereo pairs that failed or underperformed in classic matching algorithms (i.e., SIFT (scale invariant feature transform)), and evaluate the DL-based tie points matchers by its resulting geometric accuracy in relative orientation, and the generated DSM. The experiments are carried out using 40 off-track satellite stereo pairs from four different regions around the world. We conclude that DL-based methods provide a significant higher success rate in matching challenging multi-temporal stereo pairs, even if their matching accuracy is slightly lower than classic algorithms.

Constrained many-to-many point matching in two dimensions

In the minimum-weight many-to-many point matching problem, we are given a set R of red points and a set B of blue points in the plane, of total size N, and we want to pair up each point in R to one or more points in B and vice versa so that the sum of distances between the paired points is minimized. This problem can be solved in O(N3)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$O(N^3)$$\\end{document} time by using a reduction to the minimum-weight perfect matching problem, and thus, it is not fast enough to be used for on-line systems where a large number of tunes need to be compared. Motivated by similarity problems in music theory, in this paper we study several constrained minimum-weight many-to-many point matching problems in which the allowed pairings are given by geometric restrictions, i.e., a bichromatic pair can be matched if and only if the corresponding points satisfy a specific condition of closeness. We provide algorithms to solve these constrained versions in O(N) time when the sets R and B are given ordered by abscissa.

Feature Point Detection and Description Networks Based on Asymmetric Convolution and the Cross-ResolutionImage-Matching Method

Image matching can be transformed into the problem of feature point detection and matching of images. The current neural network methods have a weak detection effect on feature points and cannot extract enough sparse and uniform feature points. In order to improve the detection and description ability of feature points, this paper proposes a self-supervised feature point detection and description network based on asymmetric convolution: ACPoint. Specifically, first, feature point pseudolabels are learned from an unlabeled dataset, and pseudolabels are used for supervised learning; then, the learned model is used to update pseudolabels. Through multiple iterations of model training and label updating, high-quality labels and high-accuracy models are obtained adaptively. The asymmetric convolution feature point (ACPoint) network adopts an asymmetric convolution module to simultaneously train three convolution branches to learn more feature information, which uses two one-dimensional convolutions to enhance the backbone of square convolution from both horizontal and vertical directions and improve the representation of local features during inference. Based on the ACPoint network, a cross-resolution image-matching method is proposed. Experiments show that our proposed network model has higher localization accuracy and homography estimation ability on the HPatches dataset.

Point matching based on affine invariant centroid trees

Object detection can be formulated as a point matching problem when objects are modeled by point sets. Moments, which have been widely used for point matching, are limited to affine transformations as their support point sets cannot keep invariant. To address this problem, we developed an affine invariant centroid tree (AICT) to obtain a rigorous affine invariant support point set (SPS). The algorithm is constructed by a recursive process: the point set is first divided by the vector from the certain point to the centroid of the point set, and the centroids of subsets are used to generate vectors for renewed partitions. In addition, the centroids of the subsets are stored to form an AICT. The AICT represents the inherent structure of the point set. It is highly tolerant to noise and outliers due to the partitions on the whole point set. More importantly, it is affine invariant owing to the affine invariance of partition. Therefore, we can get rigorous affine invariant descriptors while moments are combined with AICT. The experimental results on synthesized and real data verify that our proposed algorithm outperforms the state-of-the-art point matching methods including shape context, iterative closet point, and the method adopting thin plate spline for rigid robust point matching (TPS-RPM).

Matching of corroded defects in onshore pipelines based on In-Line Inspections and Voronoi partitions

Onshore pipelines are usually subjected to a corrosion attack. Regular inspections known as In-Line inspections (ILI) are commonly used with magnetic (MFL) or ultrasonic (UT) tools to prevent any failure. New defects will appear between consecutive inspections due to the aggressiveness of the surroundings and the detection thresholds associated with the defects’ depth. This work focuses on the matching problem between two inspections, aiming to identify the degradation increments and the position of new defects. Typically, it is linked to the well-known point matching problem in pattern recognition, where the objective is finding the best affine transformation between two sets of points in a plane. This work presents an alternative using Voronoi cells to filter possible matches and an iterative approach to determine the best affine transformation, considering the uncertainty in any direction. The approach was implemented for a real pipeline 45 km long and for synthetic corrosion defects, allowing us to identify possible matches easily. Based on the new and old defects, some insights about the probability of detection and false alarm are deduced. For this purpose, experimental probability and results from recognized exponential and log–logistic functions were considered.

Searching Dense Point Correspondences via Permutation Matrix Learning

Although 3D point cloud data has received widespread attentions as a general form of 3D signal expression, applying point clouds to the task of dense correspondence estimation between 3D shapes has not been investigated widely. Furthermore, even in the few existing 3D point cloud-based methods, an important and widely acknowledged principle, i.e . one-to-one matching, is usually ignored. In response, this paper presents a novel end-to-end learning-based method to estimate the dense correspondence of 3D point clouds, in which the problem of point matching is formulated as a zero-one assignment problem to achieve a permutation matching matrix to implement the one-to-one principle fundamentally. Note that the classical solutions of this assignment problem are always non-differentiable, which is fatal for deep learning frameworks. Thus we design a special matching module, which solves a doubly stochastic matrix at first and then projects this obtained approximate solution to the desired permutation matrix. Moreover, to guarantee end-to-end learning and the accuracy of the calculated loss, we calculate the loss from the learned permutation matrix but propagate the gradient to the doubly stochastic matrix directly which bypasses the permutation matrix during the backward propagation. Our method can be applied to both non-rigid and rigid 3D point cloud data and extensive experiments show that our method achieves state-of-the-art performance for dense correspondence learning.

A Two-Step Method for Remote Sensing Images Registration Based on Local and Global Constraints

In this article, we propose an effective method for remote sensing image registration. Point features are robust to remote sensing images with low quality, small overlapping area, and local deformation. Therefore, we extract point features from remote sensing images and convert the problem of remote sensing image registration into the problem of feature point matching. A correspondence set constructed solely on the similar of features often contains many false correspondences or outliers, so our key idea is to remove the mismatches in the initial correspondence set and obtain a stable correspondence through a two-step strategy. First, we use two constraints to construct the optimization model which can solve in linear time. The first constraint is that the topology of the points and their neighbors can be maintained after the spatial transformation. Another constraint is that the feature distance of the correct matches are similar to the neighbors. Then, we design a strategy to increase the number of inliers and raise the precision by a global constraint calculated from the solution in the previous step. Experiments on a variety of remote sensing image datasets demonstrate that our method is more robust and accurate than state-of-the-art methods.

Robust point cloud registration based on topological graph and Cauchy weighted [formula omitted]-norm

Point Cloud Registration (PCR) is a fundamental and important issue in photogrammetry and computer vision. Its goal is to find rigid transformations that register multiple 3D point sets. This paper proposes a robust and efficient PCR method based on topological graph and Cauchy weighted lq-norm. Our method does not require initializations and is highly robust to outliers and partial overlaps. It contains two major steps: (1) correspondence-based coarse registration, which is called Weighted lq Coarse Registration (WlqCR). In the WlqCR method, we represent feature point sets as topological graphs and transform the point matching problem to an edge matching problem. We build a mathematical model for edge correspondence maximization. We also present an edge voting strategy to distinguish potential correct matches from mismatches. Then, we define a concept called edge vector, which has a property that it is invariant to translations. Based on this property, six Degrees of Freedoms (DoF) PCR problem can be simplified into two three DoF subproblems, i.e., rotation estimation and translation estimation. (2) fine registration based on Weighted lq Iterative Closest Point (WlqICP). We propose a new ICP method called WlqICP, which is much more robust to partial overlaps compared with traditional ICP. In both rotation estimation and WlqICP, we use a new Cauchy weighted lq-norm (0<q<1) instead of l2-norm for object function construction, which has a high degree of robustness. Extensive experiments on both simulated and real data demonstrate the power of the proposed method, i.e., our method is more robust (is able to tolerate up to 99% of outliers) and much faster than compared state-of-the-art methods (WlqCR is almost two orders of magnitude faster than RANdom SAmple Consesus (RANSAC) and its variants under 95% of outliers). The source code will be made publicly available in http://www.escience.cn/people/lijiayuan/index.html.

A Mixture Likelihood Model of the Anisotropic Gaussian and Uniform Distributions for Accurate Oblique Image Point Matching

In this letter, we propose a mixture likelihood model for accurate oblique image point matching. The basic prior assumption is that the noises are anisotropic with zero mean and different covariances in $x$ - and $y$ -directions for inliers, while the outliers have uniform distribution, which is more suitable for tilted scenes or viewpoint changes. Furthermore, the oblique image point matching problem is formulated as an improved maximum a posteriori (IMAP) estimation of a Bayesian model. In this model, based on the vector field interpolation framework, we combined the mixture likelihood model and our previous adaptive image mismatch removal method, where a two-order term of the regularization coefficient is introduced into the regularized risk function, and a parameter self-adaptive Gaussian kernel function is imposed to construct the regularization term. Subsequently, the expectation–maximization algorithm is utilized to solve the IMAP estimation, in which all the latent variances are able to obtain excellent estimation. Experimental results on real data sets verified that our method was superior to some similar methods in terms of precision and also had better self-adaptability characteristic than some hypothesis-and-verify methods. More experiments on viewpoint changes demonstrated our method’s effectiveness without loss of precision–recall tradeoffs, besides significant efficiency improvement.

Automatic calibration method for the full parameter of a camera-projector system

We propose an automatic calibration method using grid-structured light for the full parameter of a camera-projector system, including principal points, equivalent focal length, image distortion coefficients, the rotation matrix, and translation vectors between the camera and projector. Grid-structured light is projected onto a board, camera image intersection points are extracted, and three-dimensional intersection points are computed according to a homography matrix. Finally, the full parameter of a camera-projector system is solved based on stereo vision. No manual intervention is required during image processing, which simplifies the operations and improves efficiency. The image-processing kernel problem involves both automatic detection and intersection point matching. (1) The proposed intersection point detection method utilizes multiscale fusion. At each level of the image pyramid, intersection points are searched according to gray distribution and geometrical characteristics. With the gray-gravity method, coordinates are achieved with subpixel intersection point precision. Therefore, the location precision exceeds 0.5 pixel. (2) The proposed matching method employs belief propagation. Taking intersection points as nodes, a Bayesian network is established according to the Markov random field hypothesis. The image intersection point matching problem between a camera and the projector is then transformed into a maximum a posteriori estimation problem. Ultimately, 15 images are used to calibrate the full parameter of a camera-projector system. The results indicate that the reprojection error exceeds 0.15 pixel.

Low‐rank path‐following algorithm for 3D similarity registration

To address the 3D point matching problem where the pose difference between two point sets is unknown, the authors propose a path following (PF)–based algorithm. This method works by reducing the objective function of robust point matching (RPM) algorithm to a function of point correspondence variable and then using PF for optimisation. By using the 3D similarity transformation which has few parameters, authors’ method needs no regularisation on transformation and, therefore, can handle the case when the pose difference between two point sets is unknown. The authors also propose a novel convex term for use in the PF algorithm which is based on the low‐rank nature of authors’ objective function and leads to a PF algorithm which converges quickly. Experimental results demonstrated better robustness of the proposed method over state‐of‐the‐art methods and authors’ method is also efficient.

GridDS: a hybrid data structure for residue computation in point set matching

Registration of 3D point clouds is a problem that arises in a variety of research areas such as computer vision, computer graphics and computational geometry. This situation causes most papers in the area to focus on solving practical problems by using data structures often developed in theoretical contexts. Consequently, discrepancies arise between asymptotic cost and experimental performance. The point cloud registration or matching problem encompasses many different steps. Among them, the computation of the distance between two point sets (often refereed to as residue computation) is crucial and can be seen as an aggregate of range searching or nearest neighbor searching. In this paper, we aim at providing theoretical analysis and experimental performance of range searching and nearest neighbor data structures applied to 3D point cloud registration. Performance of widely used data structures such as compressed octrees, KDtrees, BDtrees and regular grids is reported. Additionally, we present a new hybrid data structure named GridDS, which combines a regular grid with some preexisting “inner” data structure in order to meet the best asymptotic bounds while also obtaining the best performance. The experimental evaluation in both synthetic and real data demonstrates that the hybrid data structures built using GridDS improve the running times of the single data structures. Thus, as we have studied the performances of the state-of-the-art techniques managing to improve their respective running times thanks to GridDS, this paper presents the best running time for point cloud residue computation up to date.

Geodesic-like features for point matching

Point matching problem seeks the optimal correspondences between two sets of points via minimizing the dissimilarities of the corresponded features. The features are widely represented by a graph model consisting of nodes and edges, where each node represents one key point and each edge describes the pair-wise relations between its end nodes. The edges are typically measured depending on the Euclidian distances between their end nodes, which is, however, not suitable for objects with non-rigid deformations. In this paper, we notice that all the key points are spanning on a manifold which is the surface of the target object. The distance measurement on a manifold, geodesic distance, is robust under non-rigid deformations. Hence, we first estimate the manifold depending on the key points and concisely represent the estimation by a graph model called the Geodesic Graph Model (GGM). Then, we calculate the distance measurement on GGM, which is called the geodesic-like distance, to approximate the geodesic distance. The geodesic-like distance can better tackle non-rigid deformations. To further improve the robustness of the geodesic-like distance, a weight setting process and a discretization process are proposed. The discretization process produces the geodesic-like features for the point matching problem. We conduct multiple experiments over widely used datasets and demonstrate the effectiveness of our method.

An Efficient Globally Optimal Algorithm for Asymmetric Point Matching.

Although the robust point matching algorithm has been demonstrated to be effective for non-rigid registration, there are several issues with the adopted deterministic annealing optimization technique. First, it is not globally optimal and regularization on the spatial transformation is needed for good matching results. Second, it tends to align the mass centers of two point sets. To address these issues, we propose a globally optimal algorithm for the robust point matching problem in the case that each model point has a counterpart in scene set. By eliminating the transformation variables, we show that the original matching problem is reduced to a concave quadratic assignment problem where the objective function has a low rank Hessian matrix. This facilitates the use of large scale global optimization techniques. We propose a modified normal rectangular branch-and-bound algorithm to solve the resulting problem where multiple rectangles are simultaneously subdivided to increase the chance of shrinking the rectangle containing the global optimal solution. In addition, we present an efficient lower bounding scheme which has a linear assignment formulation and can be efficiently solved. Extensive experiments on synthetic and real datasets demonstrate the proposed algorithm performs favorably against the state-of-the-art methods in terms of robustness to outliers, matching accuracy, and run-time.

Learning coherent vector fields for robust point matching under manifold regularization

In this paper, we propose a robust method for coherent vector field learning with outliers (mismatches) using manifold regularization, called manifold regularized coherent vector field (MRCVF). The method could remove outliers from inliers (correct matches) and learn coherent vector fields fitting for the inliers with graph Laplacian constraint. In the proposed method, we first formulate the point matching problem as learning a corresponding vector field based on a mixture model (MM). Manifold regularization term is added to preserve the intrinsic geometry of the mapped point set of vector fields. More specially, the optimal mapping function is obtained by solving a weighted Laplacian regularized least squares (LapRLS) in a reproducing kernel Hilbert space (RKHS) with a matrix-valued kernel. Moreover, we use the Expectation Maximization (EM) optimization algorithm to update the unknown parameters in each iteration. The experimental results on the synthetic data set, real image data sets, and non-rigid images quantitatively demonstrate that our proposed method is robust to outliers, and it outperforms several state-of-the-art methods in most scenarios.

A Feature Point Matching Based on Spatial Order Constraints Bilateral-Neighbor Vote.

Feature point matching is a fundamental and challenging problem in many computer vision applications. In this paper, a robust feature point matching algorithm named spatial order constraints bilateral-neighbor vote (SOCBV) is proposed to remove outliers for a set of matches (including outliers) between two images. A directed k nearest neighbor (knn) graph of match sets is generated, and the problem of feature point matching is formulated as a binary discrimination problem. In the discrimination process, the class labeled matrix is built via the spatial order constraints defined on the edges that connect a point to its knn. Then, the posterior inlier class probability of each match is estimated with the knn density estimation and spatial order constraints. The vote of each match is determined by averaging all posterior class probabilities that originate from its associative inliers set and is used for removing outliers. The algorithm iteratively removes outliers from the directed graph and recomputes the votes until the stopping condition is satisfied. Compared with other popular algorithms, such as RANSAC, RSOC, GTM, SOC and WGTM, experiments under various testing data sets demonstrate strong robustness for the proposed algorithm.

Robust track-to-track association in the presence of sensor biases and missed detections

The paper addresses the problem of robust track-to-track association in the presence of sensor biases and missed detections. Under the condition of large range biases with sensors, it is validated that the structural difference between two sets of local tracks from different sensors can be described by a non-rigid transformation. After that, we turn the robust track-to-track association problem into the non-rigid point matching problem in the framework of TPS-RPM (Thin Plate Spline-Robust Point Matching). Further, to improve the performance of the track-to-track association, the structural feature is introduced for each local track, and the structural similarity is incorporated by regularizing the energy function of the TPS-RPM algorithm. Simulation results demonstrate the effectiveness of the proposed approaches compared with competing algorithms.

Detecting Point Pattern of Multiple Line Segments Using Hough Transformation

Surface defects in manufacturing often exhibit particular spatial patterns. These patterns contain valuable information about the manufacturing process and can help to identify potential root causes. In this paper, we present a new method to detect the point patterns that consist of multiple line segments. The basic idea is that by using the Hough transformation, we convert the point pattern detection problem into a simple point matching problem. Compared with the existing point pattern matching methods, the proposed method does not require training data and is relatively easy to implement and compute. The details of the detection algorithm are presented and the parameter selection and performance evaluation of this method are investigated. Case studies are presented to validate the effectiveness of this method.

Robust feature point matching with sparse model.

Feature point matching that incorporates pairwise constraints can be cast as an integer quadratic programming (IQP) problem. Since it is NP-hard, approximate methods are required. The optimal solution for IQP matching problem is discrete, binary, and thus sparse in nature. This motivates us to use sparse model for feature point matching problem. The main advantage of the proposed sparse feature point matching (SPM) method is that it generates sparse solution and thus naturally imposes the discrete mapping constraints approximately in the optimization process. Therefore, it can optimize the IQP matching problem in an approximate discrete domain. In addition, an efficient algorithm can be derived to solve SPM problem. Promising experimental results on both synthetic points sets matching and real-world image feature sets matching tasks show the effectiveness of the proposed feature point matching method.

Affine Image Registration Based on Geometric Inference

Image registration is widely used in applications for mapping one image to another. As it is often formulated as a point matching problem, in this paper, a novel method, called the Geometric Inference (GI) algorithm, is proposed for feature point based image registration. Firstly, according to affine distance invariant, the global geometric relationship between collinear correspondences is deduced and used for collinear point matching. Secondly, utilizing affine area invariant, geometric relationship between noncollinear correspondences is inferred and used for noncollinear point matching. Finally, the best affine transformation can be discovered from the correspondences composed of the collinear and noncollinear corresponding point pairs. Experiments on synthesized and real data demonstrate that GI is well-adapt to image registration as it is fast and robust to missing points, outliers, and noise.