Feature Correspondences Research Articles

Geographic Semantic Network for Cross-View Image Geo-Localization

The task of cross-view image geo-localization aims to determine the geo-location (Global Positioning System (GPS) coordinates) of a query ground-view image by matching the image with GPS-tagged aerial (or satellite) images in the reference dataset. Due to the dramatic domain gap between the ground and aerial images, the problem is challenging. The existing approaches mainly adopt convolutional neural networks (CNNs) to learn discriminative features. However, these CNN-based methods mainly leverage appearance and semantic information but fail to jointly model the appearance, positional, and orientation properties of scene objects, which belong to the spatial hierarchy. Since spatial hierarchy information is crucial for cross-view feature correspondence, in this article, we propose an end-to-end network architecture, dubbed GeoNet. GeoNet consists of a ResNetX module and a GeoCaps module. On the one hand, the ResNetX module is developed to learn powerful intermediate feature maps and allows the stable propagation of gradients in deep CNNs. On the other hand, the GeoCaps module utilizes the capsule network to encapsulate the intermediate feature maps into several capsules, whose length and orientation represent the existence probability and spatial hierarchy information of scene objects, respectively. Moreover, by using a dynamic routing-by-agreement mechanism, the GeoCaps module is capable of modeling parts-to-whole relationships between scene objects, which is viewpoint invariant and capable of bridging the cross-view domain gap. In addition to GeoNet, we introduce a simple yet effective metric learning method, based on which two weighted soft margin loss functions with online batch hard sample mining are devised. These functions not only speed up convergence but also improve the generalization ability of the network. Extensive experiments on three well-known datasets demonstrate that our GeoNet not only achieves state-of-the-art results for the ground-to-aerial and aerial-to-ground geo-localization tasks but also outperforms competing approaches for the few-shot geo-localization task.

Robust Feature Matching via Local Consensus

Feature matching is the foundation and key task of remote sensing image registration, which is to establish a reliable point corresponding relationship between the feature points of two images. In this article, a simple and effective local consensus method for rigid and nonrigid feature matching is proposed and applied to solve the problem of high outliers ratio caused by nonrigid transformation, nonlinear radiation difference, and speckle noise in the remote sensing image registration task. We first establish the putative feature correspondences according to the similarity between local descriptors and then use local consensus constraints (including neighborhood consensus and motion vector consensus) to remove outliers. The specific steps are given as follows. First, we use the neighborhood consensus constraint of feature points to carry out preliminary filtering to remove outliers with obvious errors and retain a large number of inliers, so as to obtain a clean reliable set. Then, the reliable set space is grided into several nonoverlapping cells, and the estimated motion vector is calculated for each cell. By taking the comprehensive deviation between the ordinary motion vectors and estimated motion vectors, we transform the matching problem into a mathematical optimization model and derive a closed-form solution with linear time and linear space complexities. In this way, our method can also significantly increase the speed of operation without sacrificing accuracy. A large number of feature matching experiments on remote sensing prove that our method is superior to existing methods and also has good results in the general scene.

Ego-Motion Estimation With Stereo Cameras Using Efficient 3D–2D Edge Correspondences

This article presents a new edge-based ego-motion estimation method for stereo cameras using 3D–2D edge alignment. To filter out prominent edgelets from Canny edge extractor, a region growing algorithm is developed to remove cluttered edges out of raw edgelets. Edge matching between stereo image pairs is guided by sparse feature correspondence using optical flow. The disparity is then refined using an inverse compositional algorithm, which is far more computationally efficient than searching along the epipolar line. The relative camera motion is solved by 3D–2D edge alignment based on an iterative closest point (ICP) algorithm, minimizing the point-to-tangent distance. A gradient-orientation-based similarity measurement is proposed, and the optimal correspondence is searched between the 3-D edge point projected onto the current image and the extracted edge pixels in the current frame with 1-nearest neighbor. In addition, the inverse depth of edge points in the reference frame is optimized. Experiments show that the proposed method can provide centimeter-level relative pose error (RPE), which has a better performance than the point-based approach efficient perspective-n-point (EPnP) and the edge-based approach direct edge-alignment (DE-A).

Learning Enriched Features for Image Super Resolution

In recent years, significant progress has been made in image super-resolution (SR) methods based on convolutional neural networks. However, most of them do not fully utilize multi-scale feature correspondence in the image SR process, resulting in blurred and artifact detail restoration, especially for SR tasks with larger scaling factors (i.e. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 4$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 8$ </tex-math></inline-formula> ). A multi-scale feature-enhanced SR network (MFENet) is proposed to solve the problems mentioned above. Specifically, the super-resolution method is enhanced by utilizing multi-scale feature correspondence in two ways. On the one hand, we propose an effective multi-scale non-local attention module and it can fully exploit a large number of low-level features at different scales of a single low-resolution (LR) feature map by exhaustively evaluating the correlations in local feature, in-scale non-local feature, and cross-scale non-local feature. On the other hand, the mapping from high-resolution (HR) to LR images is explored by a feedback branch we introduced, that provides an additional constraint for images reconstruction. It can penalize incorrect SR predictions to achieve better SR performance. Experimental results on 5 benchmark datasets show that our method outperforms state-of-the-art methods in terms of both subjective visual quality and objective quantitative metrics.

Pro-UIGAN: Progressive Face Hallucination From Occluded Thumbnails.

In this paper, we study the task of hallucinating an authentic high-resolution (HR) face from an occluded thumbnail. We propose a multi-stage Progressive Upsampling and Inpainting Generative Adversarial Network, dubbed Pro-UIGAN, which exploits facial geometry priors to replenish and upsample ( 8× ) the occluded and tiny faces ( 16×16 pixels). Pro-UIGAN iteratively (1) estimates facial geometry priors for low-resolution (LR) faces and (2) acquires non-occluded HR face images under the guidance of the estimated priors. Our multi-stage hallucination network upsamples and inpaints occluded LR faces via a coarse-to-fine fashion, significantly reducing undesirable artifacts and blurriness. Specifically, we design a novel cross-modal attention module for facial priors estimation, in which an input face and its landmark features are formulated as queries and keys, respectively. Such a design encourages joint feature learning across the input facial and landmark features, and deep feature correspondences will be discovered by attention. Thus, facial appearance features and facial geometry priors are learned in a mutually beneficial manner. Extensive experiments show that our Pro-UIGAN attains visually pleasing completed HR faces, thus facilitating downstream tasks, i.e., face alignment, face parsing, face recognition as well as expression classification.

Feature matching for multi-epoch historical aerial images

Historical imagery is characterized by high spatial resolution and stereoscopic acquisitions, providing a valuable resource for recovering 3D land-cover information. Accurate geo-referencing of diachronic historical images by means of self-calibration remains a bottleneck because of the difficulty to find sufficient amount of feature correspondences under evolving landscapes. In this research, we present a fully automatic approach to detecting feature correspondences between historical images taken at different times (i.e., inter-epoch), without auxiliary data required. Based on relative orientations computed within the same epoch (i.e., intra-epoch), we obtain DSMs (Digital Surface Model) and incorporate them in a rough-to-precise matching. The method consists of: (1) an inter-epoch DSMs matching to roughly co-register the orientations and DSMs (i.e, the 3D Helmert transformation), followed by (2) a precise inter-epoch feature matching using the original RGB images. The innate ambiguity of the latter is largely alleviated by narrowing down the search space using the co-registered data. With the inter-epoch feature correspondences, we refine the image orientations and quantitatively evaluate the results (1) with DoD (Difference of DSMs), (2) with ground check points, and (3) by quantifying ground displacement due to an earthquake. We demonstrate that our method: (1) can automatically georeference diachronic historical images; (2) can effectively mitigate systematic errors induced by poorly estimated camera parameters; (3) is robust to drastic scene changes. Compared to the state-of-the-art, our method improves the image georeferencing accuracy by a factor of 2. The proposed methods are implemented in MicMac, a free, open-source photogrammetric software.

For Tom Inge (1936–2021)

I met M. Thomas Inge in 1989 at the conference of the Popular Culture Association of the South after I chaired a panel on the antebellum comic writer George Washington Harris. I had tried to put the panel into MLA's national conference but was turned down. Another fan of Harris and his creation, Sut Lovingood, Tom arranged to have the panel added to the regional conference: typical of Tom's generosity. After the panel, he suggested a collection of essays on Harris, which I agreed to work on if he helped; that was one of the best deals I ever made, for it was the start of a collaboration that was part mentorship, part friendship. I doubt my career path would have been the same if I hadn't tried to shoehorn Harris and his rowdy character Sut into MLA. Tom and I also shared a passion for the work of William Faulkner, yet another fan of Harris and Sut.Tom was an indefatigable scholar and author of over fifty books. Twenty-five years ago, I once had a copy of his bibliography, which was then more than forty pages long. It is no exaggeration to say that he was a force majeure in American literature, especially in Southern literature, humor studies, and comic books. He was a founding member of the American Humor Studies Association, started American Humor: An Interdisciplinary Newsletter in 1974 with Larry Mintz, and later was the editor of Studies in American Humor.But it was in the arena of comic books and popular culture that Tom truly stood out. His book Comics as Culture (1990) was foundational for the now burgeoning academic field of comic books and graphic novels. I have this memory of him saying that he had to rent a place to hold his collection of original comic artifacts. I understand Virginia Commonwealth University will be the repository for the collection, making it an instant Mecca for research on comics in American culture. As mentioned in the VCU Libraries tribute to Dr. Inge, “The collection features correspondence from noted artists and writers such as Art Spiegelman, Mort Walker, Bruce Duncan, and Harold Foster in addition to manuscripts of Inge's scholarly works. … In 2018, he received the Lynn Bartholome Eminent Scholar Award, the highest honor offered by the national Popular Culture Association that recognizes distinguished scholars who have made significant and lasting contributions to the study of popular and American culture” (“Thomas Inge, Pioneering Comics Scholar and Former Professor, Dies at 85,” VCU Libraries, May 24, 2021, https://www.library.vcu.edu/).Tom's sense of the comic in American literature was large: he once coaxed me into participating in a special issue he was editing on humor by Nathaniel Hawthorne, not the only time I benefited from his projects. Edgar Allan Poe is perhaps more obvious than Hawthorne as a subject for unexpected humor as a theme and a topic, and here Tom's extensive collection demonstrates its impact. Tom gave the Poe Museum thirty-six comic books, including issues of Classics Illustrated, Edgar Allan Poe's Haunt of Horror, Roman Dirge's Lenore, and Weird Tales of Horror. He also donated a page of original artwork for a Zippy the Pinhead comic strip and a Lenore doll based on the character in the Lenore comics. Some of his other comics formed the core of the Poe Museum's exhibit “Poe in Comics.” Similarly, items from his collection created “Mark Twain in the Comics: An Exhibition” for the 2009 State of Mark Twain Studies.In many ways, Tom was a legend in the part of academia that worked on comics and pop culture, or Southern literature, or humor studies in general, but what I will remember most is how approachable Tom was, always friendly and always gentlemanly in his demeanor. I was lucky our paths crossed, for he provided a model of how scholars in the humanities should work and play.

High-precision 3D shape measurement of rigid moving objects based on the Hilbert transform.

Phase-shifting profilometry (PSP) is a three-dimensional (3D) measurement method of point-to-point calculation. The consistency of object position is the prerequisite to ensure the successful application of PSP in moving objects. The position information of an object can be well characterized by the modulation patterns, and hence a high-quality modulation pattern is the guarantee of pixel-matching accuracy. In this paper, a generic modulation pattern enhancement method for rigid moving objects based on the Hilbert transform is proposed. First, the Hilbert transform is employed to suppress the zero-frequency components of the fringe pattern, and a hybrid digital filter window is applied to filter out the positive fundamental frequency components for a higher signal-to-noise ratio. Then the grid-based motion statistics for fast, ultra-robust feature correspondence algorithm is used to match the high-quality modulation patterns between two adjacent frames, and the object positions in the three deformed patterns are made consistent by image clipping. Finally, the three-step PSP is used to reconstruct the 3D shape of the measured object. Experimental results demonstrate that the proposed method can substantially improve the quality of the modulation pattern, achieve high-precision pixel matching, and ultimately reduce the motion-introduced phase error.

Tightly-coupled camera/LiDAR integration for point cloud generation from GNSS/INS-assisted UAV mapping systems

Unmanned aerial vehicles (UAVs) equipped with integrated global navigation satellite systems/inertial navigation systems (GNSS/INS) together with cameras and/or LiDAR sensors are being widely used for topographic mapping in a variety of applications such as precision agriculture, coastal monitoring, and archaeological documentation. Integration of image-based and LiDAR point clouds can provide a comprehensive 3D model of the area of interest. For such integration, ensuring a good alignment between data from the different sources is critical. Although many works have been conducted on this topic, there is still a need for a rigorous integration approach that minimizes the discrepancy between camera and LiDAR data caused by inaccurate system calibration parameters and/or trajectory artifacts. This study proposes an automated tightly-coupled camera/LiDAR integration workflow for GNSS/INS-assisted UAV systems. The proposed strategy is conducted in three main steps. First, an image-based point cloud is generated using a LiDAR/GNSS/INS-assisted structure from motion (SfM) strategy. Then, feature correspondences between image-based and LiDAR point clouds are automatically identified. Finally, an integrated-bundle adjustment procedure including image points, LiDAR raw measurements, and GNSS/INS information is conducted to minimize the discrepancy between point clouds from different sensors while estimating system calibration parameters and refining the trajectory information. The proposed SfM strategy and integration framework are evaluated using five datasets. The SfM results show that using LiDAR data can facilitate feature matching and further increase the number of reconstructed 3D points. The experimental results also illustrate that the developed automated camera/LiDAR integration strategy is capable of accurately estimating system calibration parameters to achieve good alignment among camera/LiDAR data from single/multiple systems. An absolute accuracy in the range of 3–5 cm is achieved for the image/LiDAR point clouds after the integration process.

Learning Two-View Correspondences and Geometry via Local Neighborhood Correlation.

Seeking quality feature correspondences (also known as matches) is a foundational step in computer vision. In our work, a novel and effective network with a stable local constraint, named the Local Neighborhood Correlation Network (LNCNet), is proposed to capture abundant contextual information of each correspondence in the local region, followed by calculating the essential matrix and camera pose estimation. Firstly, the k-Nearest Neighbor () algorithm is used to divide the local neighborhood roughly. Then, we calculate the local neighborhood correlation matrix () between the selected correspondence and other correspondences in the local region, which is used to filter outliers to obtain more accurate local neighborhood information. We cluster the filtered information into feature vectors containing richer neighborhood contextual information so that they can be used to more accurately determine the probability of correspondences as inliers. Extensive experiments have demonstrated that our proposed LNCNet performs better than some state-of-the-art networks to accomplish outlier rejection and camera pose estimation tasks in complex outdoor and indoor scenes.

Toward 3D object reconstruction from stereo images

Inferring the complete 3D shape of an object from an RGB image has shown impressive results, however, existing methods rely primarily on recognizing the most similar 3D model from the training set to solve the problem. These methods suffer from poor generalization and may lead to low-quality reconstructions for unseen objects. Nowadays, stereo cameras are pervasive in emerging devices such as dual-lens smartphones and robots, which enables the use of the two-view nature of stereo images to explore the 3D structure and thus improve the reconstruction performance. In this paper, we propose a new deep learning framework for reconstructing the 3D shape of an object from a pair of stereo images, which reasons about the 3D structure of the object by taking bidirectional disparities and feature correspondences between the two views into account. Besides, we present a large-scale synthetic benchmarking dataset, namely StereoShapeNet, containing 1,052,976 pairs of stereo images rendered from ShapeNet along with the corresponding bidirectional depth and disparity maps. Experimental results on the StereoShapeNet benchmark demonstrate that the proposed framework outperforms the state-of-the-art methods. The project page is available athttps://haozhexie.com/project/stereo-3d-reconstruction.

The shape matching element method

We introduce a new method for direct physics-based animation of volumetric curved models, represented using NURBS surfaces. Our technical contribution is the Shape Matching Element Method (SEM). SEM is a completely meshless algorithm, the first to simultaneously be robust to gaps and overlaps in geometry, be compatible with standard constitutive models and time integration schemes, support contact and frictional interactions and to preserve feature correspondence during simulation which enables editable simulated output. We demonstrate the efficacy of our algorithm by producing compelling physics-based animations from a variety of curved input models.

Self-Localization of Mobile Robots Using a Single Catadioptric Camera with Line Feature Extraction.

This paper presents a novel self-localization technique for mobile robots using a central catadioptric camera. A unified sphere model for the image projection is derived by the catadioptric camera calibration. The geometric property of the camera projection model is utilized to obtain the intersections of the vertical lines and ground plane in the scene. Different from the conventional stereo vision techniques, the feature points are projected onto a known planar surface, and the plane equation is used for depth computation. The 3D coordinates of the base points on the ground are calculated using the consecutive image frames. The derivation of motion trajectory is then carried out based on the computation of rotation and translation between the robot positions. We develop an algorithm for feature correspondence matching based on the invariability of the structure in the 3D space. The experimental results obtained using the real scene images have demonstrated the feasibility of the proposed method for mobile robot localization applications.

Fast and robust online three-dimensional measurement based on feature correspondence

Online three-dimensional (3D) measurement plays an important role in industry. When phase-shifting profilometry is employed in online 3D measurement, pixel matching is an important step to keep objects at the same coordinate value. However, the correlation operation and marker feature matching algorithms may take a long time, increasing the complexity. So a fast and robust online 3D measurement based on feature correspondence is proposed. In this method, only one frame of the sinusoidal fringe pattern is projected onto the measured object, and image correction technique is employed to rectify pixel size. Then five frames of deformed patterns with equivalent displacement are captured by the camera, and the corresponding modulation patterns are extracted. The oriented fast and rotated brief feature algorithm is used to extract the matching pair of feature points, and the improved grid-based motion statistical feature algorithm can better eliminate the false match to achieve pixel matching. In this way, five frames of deformed patterns with an equivalent shifted-phase can be extracted. Finally, the 3D shape of the measured object is reconstructed by the five-step equivalent phase-shifting algorithm. Experimental results verify the effectiveness and feasibility of the proposed method.

Indoor camera pose estimation via style‐transfer 3D models

AbstractMany vision‐based indoor localization methods require tedious and comprehensive pre‐mapping of built environments. This research proposes a mapping‐free approach to estimating indoor camera poses based on a 3D style‐transferred building information model (BIM) and photogrammetry technique. To address the cross‐domain gap between virtual 3D models and real‐life photographs, a CycleGAN model was developed to transform BIM renderings into photorealistic images. A photogrammetry‐based algorithm was developed to estimate camera pose using the visual and spatial information extracted from the style‐transferred BIM. The experiments demonstrated the efficacy of CycleGAN in bridging the cross‐domain gap, which significantly improved performance in terms of image retrieval and feature correspondence detection. With the 3D coordinates retrieved from BIM, the proposed method can achieve near real‐time camera pose estimation with an accuracy of 1.38 m and 10.1° in indoor environments.

Fast and Robust Certifiable Estimation of the Relative Pose Between Two Calibrated Cameras

This work contributes an efficient algorithm to compute the Relative Pose problem (RPp) between calibrated cameras and certify the optimality of the solution, given a set of pair-wise feature correspondences affected by noise and probably corrupted by wrong matches. We propose a family of certifiers that is shown to increase the ratio of detected optimal solutions. This set of certifiers is incorporated into a fast essential matrix estimation pipeline that, given any initial guess for the RPp, refines it iteratively on the product space of 3D rotations and 2-sphere. In addition, this fast certifiable pipeline is integrated into a robust framework that combines Graduated Non-convexity and the Black-Rangarajan duality between robust functions and line processes. We proved through extensive experiments on synthetic and real data that the proposed framework provides a fast and robust relative pose estimation. We make the code publicly available \url{https://github.com/mergarsal/FastCertRelPose.git}.

Mining consistent correspondences using co-occurrence statistics

In this paper, we propose a mismatch removal method, which mines consistent image feature correspondences using co-occurrence statistics. The proposed method relies on a co-occurrence matrix that counts the number of pixel value pairs co-occurring within the images. Specifically, we propose to integrate the co-occurrence statistics with local spatial information, to preserve the consensus of neighborhood elements. Then, a new measure based on co-occurrence statistics is defined for correspondence similarity, to preserve the consensus of neighborhood topology. After that, with the consensus of neighborhood elements and neighborhood topology, the mismatch removal problem is formulated into a mathematical model, which has a closed-form solution. Extensive experiments show that the proposed method is able to achieve superior or competitive performance on matching accuracy over several state-of-the-art competing methods. In addition, we further exploit the consensus of neighborhood elements and neighborhood topology to propose a novel guided sampling method, which can significantly improve the quality of sampling minimal subsets over state-of-the-arts for two-view geometric model fitting.

Adversarial Partial Multi-Label Learning with Label Disambiguation

Partial multi-label learning (PML), which tackles the problem of learning multi-label prediction models from instances with overcomplete noisy annotations, has recently started gaining attention from the research community. In this paper, we propose a novel adversarial learning model, PML-GAN, under a generalized encoder-decoder framework for partial multi-label learning. The PML-GAN model uses a disambiguation network to identify irrelevant labels and uses a multi-label prediction network to map the training instances to their disambiguated label vectors, while deploying a generative adversarial network as an inverse mapping from label vectors to data samples in the input feature space. The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels in a bi-directional mapping. Extensive experiments are conducted on both synthetic and real-world partial multi-label datasets, while the proposed model demonstrates the state-of-the-art performance.

Loop-Closure Detection Using Local Relative Orientation Matching

Loop-closure detection (LCD), which aims to recognize a previously visited location, is a crucial component of the simultaneous localization and mapping system. In this paper, a novel appearance-based LCD method is presented. In particular, we propose a simple yet surprisingly useful feature matching algorithm for real-time geometrical verification of candidate loop-closures, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">local relative orientation</i> matching (LRO). It aims to efficiently establish reliable feature correspondences based on preserving local topological structures between the query image and candidate frame. To effectively retrieve candidate loop closures, we introduce the aggregated selective match kernel framework into the LCD task, which can effectively represent images and reduce the quantization noise of the traditional bag-of-words framework. In addition, the SuperPoint neural network is employed to extract reliable interest points and feature descriptors. Extensive experimental results demonstrate that our LRO can significantly improve the LCD performance, and the proposed overall LCD method can achieve much better performance over the current state-of-the-art on six publicly available datasets.

Odometry Estimation from Sparse LiDAR Point Cloud Constrained by Image Feature Correspondence

Estimation of translation between consecutive frames, i.e., odometry, plays an important role in autonomous navigation. This paper presents an odometry estimation method using sparse LiDAR points and image feature points. In case of sparse LiDAR measurements, it is difficult to accurately estimate depth at image feature points. Image feature points with low-accuracy depth cause misconvergence in odometry optimization. To improve the robustness to the misconvergence, a new method with a Gaussian process that estimates not only the depth at image feature points but also the variance is proposed. By using this variance, it estimates the residual of image features in the world coordinate with depth, or in the image coordinate without depth. This allows more accurate and robust estimation than conventional methods in case of sparse LiDAR points. In an experiment with simulated sparse LiDAR points from the KITTI dataset, the proposed method is confirmed to estimate the odometry more accurately than conventional methods.