Relative Pose Estimation Research Articles

A fast, lightweight deep learning vision pipeline for autonomous UAV landing support with added robustness

Despite massive development in aerial robotics, precise and autonomous landing in various conditions is still challenging. This process is affected by many factors, such as terrain shape, weather conditions, and the presence of obstacles. This paper describes a deep learning-accelerated image processing pipeline for accurate detection and relative pose estimation of the UAV with respect to the landing pad. Moreover, the system provides increased safety and robustness by implementing human presence detection and error estimation for both landing target detection and pose computation. Human presence and landing pad location are performed by estimating the presence probability via segmentation. This is followed by the landing pad keypoints’ location regression algorithm, which, in addition to coordinates, provides the uncertainty of presence for each defined landing pad landmark. To perform the aforementioned tasks, a set of lightweight neural network models was selected and evaluated. The resulting measurements of the system’s performance and accuracy are presented for each component individually and for the whole processing pipeline. The measurements are performed using onboard embedded UAV hardware and confirm that the method can provide accurate, low-latency feedback information for safe landing support.

DHM-Net: Deep Hypergraph Modeling for Robust Feature Matching.

We present a novel deep hypergraph modeling architecture (called DHM-Net) for feature matching in this paper. Our network focuses on learning reliable correspondences between two sets of initial feature points by establishing a dynamic hypergraph structure that models group-wise relationships and assigns weights to each node. Compared to existing feature matching methods that only consider pair-wise relationships via a simple graph, our dynamic hypergraph is capable of modeling nonlinear higher-order group-wise relationships among correspondences in an interaction capturing and attention representation learning fashion. Specifically, we propose a novel Deep Hypergraph Modeling block, which initializes an overall hypergraph by utilizing neighbor information, and then adopts node-to-hyperedge and hyperedge-to-node strategies to propagate interaction information among correspondences while assigning weights based on hypergraph attention. In addition, we propose a Differentiation Correspondence-Aware Attention mechanism to optimize the hypergraph for promoting representation learning. The proposed mechanism is able to effectively locate the exact position of the object of importance via the correspondence aware encoding and simple feature gating mechanism to distinguish candidates of inliers. In short, we learn such a dynamic hypergraph format that embeds deep group-wise interactions to explicitly infer categories of correspondences. To demonstrate the effectiveness of DHM-Net, we perform extensive experiments on both real-world outdoor and indoor datasets. Particularly, experimental results show that DHM-Net surpasses the state-of-the-art method by a sizable margin. Our approach obtains an 11.65% improvement under error threshold of 5° for relative pose estimation task on YFCC100M dataset. Code will be released at https://github.com/CSX777/DHM-Net.

Dual-correlate optimized coarse-fine strategy for monocular laparoscopic videos feature matching via multilevel sequential coupling feature descriptor

Feature matching of monocular laparoscopic videos is crucial for visualization enhancement in computer-assisted surgery, and the keys to conducting high-quality matches are accurate homography estimation, relative pose estimation, as well as sufficient matches and fast calculation. However, limited by various monocular laparoscopic imaging characteristics such as highlight noises, motion blur, texture interference and illumination variation, most exiting feature matching methods face the challenges of producing high-quality matches efficiently and sufficiently. To overcome these limitations, this paper presents a novel sequential coupling feature descriptor to extract and express multilevel feature maps efficiently, and a dual-correlate optimized coarse-fine strategy to establish dense matches in coarse level and adjust pixel-wise matches in fine level. Firstly, a novel sequential coupling swin transformer layer is designed in feature descriptor to learn and extract multilevel feature representations richly without increasing complexity. Then, a dual-correlate optimized coarse-fine strategy is proposed to match coarse feature sequences under low resolution, and the correlated fine feature sequences is optimized to refine pixel-wise matches based on coarse matching priors. Finally, the sequential coupling feature descriptor and dual-correlate optimization are merged into the Sequential Coupling Dual-Correlate Network (SeCo DC-Net) to produce high-quality matches. The evaluation is conducted on two public laparoscopic datasets: Scared and EndoSLAM, and the experimental results show the proposed network outperforms state-of-the-art methods in homography estimation, relative pose estimation, reprojection error, matching pairs number and inference runtime. The source code is publicly available at https://github.com/Iheckzza/FeatureMatching.

HYBRID DEEP LEARNING APPROACH FOR VEHICLE’S RELATIVE ATTITUDE ESTIMATION USING MONOCULAR CAMERA

Abstract. Relative pose estimation using a monocular camera is one of the most common approaches for aiding vehicle’s navigation. It involves determining the position and orientation of a vehicle relative to its surroundings using only a single camera. This can be achieved through four main steps: feature detection and matching, motion estimation, filtering and optimization, and scale estimation. Feature tracking involves detecting and tracking distinctive visual features in the environment, such as corners or edges, and using their relative motion to estimate the camera's movement. This approach can be prone to errors due to feature detection and tracking difficulties, as well as issues with moving objects, occlusions, and changes in lighting conditions. These typical computer vision approaches are computationally intensive and may require significant processing power as well, which limits their real time application. This paper proposes a hybrid deep neural network approach for estimating the relative attitude of a vehicle using a monocular camera to aid in vehicle navigation. The proposed neural network adopts a relatively shallow architecture to minimize the computational cost and to meet the real-time requirements of low-cost processing systems. The network is trained using the KITTI dataset and can estimate the relative attitude of the vehicle with a RMSE of relative orientation of 0.017 degrees per frame. The processing time of the proposed approach is around 28 ms per frame including both the tracking and network prediction steps, which is significantly faster than the typical estimation pipelines. The results show that the proposed approach is a viable alternative to conventional computer vision methods and can significantly reduce computational costs, deal with the confusing scenarios of the moving objects while maintaining a good accuracy in estimating ego-motion.

Robust spacecraft relative pose estimation via CNN-aided line segments detection in monocular images

Autonomous spacecraft relative navigation via monocular images became a hot topic in the past few years and, recently, received a further push thanks to the constantly growing field of artificial neural networks and the publication of several spaceborne image datasets. Despite the proliferation of spacecraft relative-state initialization algorithms developed, most architectures adopt computationally expensive solutions relying on convolutional neural networks (CNNs) that provide accurate output at the cost of a high computational burden that seems unfeasible for current spaceborne hardware. The paper addresses this issue by proposing a novel pose initialization algorithm based on lightweight CNNs. Inspired by previous state-of-the-art algorithms, the developed architecture leverages a fast and accurate target detection CNN followed by a line segment detection CNN capable of running with low inference time on mobile devices. The line segments and their junctions are grouped into complex geometrical groups, reducing the solution search space, and subsequently, they are adopted to extract the final pose estimate. As a main outcome, the analyses demonstrate that the lightweight architecture developed scores high accuracy in the pose estimation task, with a mean estimation error of less than 10 cm in translation and 2.5°in rotation. The baseline algorithm scores a mean SLAB error of 0.04552 with a standard deviation of 0.22972 in the test dataset. Detailed analyses demonstrate that the uncertainties on the overall pose score are driven mainly by errors in the relative attitude, which gives the highest contribution to the pose error metric adopted. The analyses on the error distributions point out that the uncertainties on the estimated relative position are higher in the camera boresight axis direction. Concerning the relative attitude, the algorithm proposed has higher uncertainties in estimating directions of the target x and y axes due to ambiguities related to the target geometry. Notably, the target detection CNN trained in this work outperforms the previous top scores in the benchmark dataset. The performances of the proposed algorithm have been investigated further by analyzing the effects on the accuracy due to the relative distance and the presence of background in the images. Lastly, the paper delves into the possibility of adopting a sub-portion of the 2D-to-3D match matrix made by the most complex perceptual groups identified that positively affects the overall run-time, pointing out the performances in terms of accuracy of the estimates and providing a comparison of both the baseline and the reduced match matrix versions against state-of-the-art algorithms concerning relative position and attitude errors and solution availability, highlighting the high accuracy and solution availability of the proposed architectures.

Certifiable planar relative pose estimation with gravity prior

In this work we propose a certifiable solver for the relative pose problem between two calibrated cameras under the assumptions that the unknown 3D points lay on an unknown plane and the axis of rotation is given, e.g. by an IMU. The problem is stated in terms of the rotation, translation and plane parameters and solved iteratively by an on-manifold optimization. Since the problem is nonconvex, we then try to certify this solution as the global optimum. For that, we leverage four different definitions for the search space that provide us with different certification capabilities. Since the formulations lack the Linear Independence Constraint Qualification and two of them have more constraints than variables, we cannot derive a closed-form certifier. Instead, we leverage the iterative algorithm proposed in our previous work Garcia-Salguero and Gonzalez-Jimenez (2023) that does not assume any condition on the problem formulation. Our evaluation on synthetic and real data shows that the smaller formulations are enough to certify most of the solutions, whereas the redundant ones certify all of them, including problem instances with highly noisy data. Code can be found in https://github.com/mergarsal.

A Robust and Integrated Visual Odometry Framework Exploiting the Optical Flow and Feature Point Method.

In this paper, we propose a robust and integrated visual odometry framework exploiting the optical flow and feature point method that achieves faster pose estimate and considerable accuracy and robustness during the odometry process. Our method utilizes optical flow tracking to accelerate the feature point matching process. In the odometry, two visual odometry methods are used: global feature point method and local feature point method. When there is good optical flow tracking and enough key points optical flow tracking matching is successful, the local feature point method utilizes prior information from the optical flow to estimate relative pose transformation information. In cases where there is poor optical flow tracking and only a small number of key points successfully match, the feature point method with a filtering mechanism is used for posing estimation. By coupling and correlating the two aforementioned methods, this visual odometry greatly accelerates the computation time for relative pose estimation. It reduces the computation time of relative pose estimation to 40% of that of the ORB_SLAM3 front-end odometry, while ensuring that it is not too different from the ORB_SLAM3 front-end odometry in terms of accuracy and robustness. The effectiveness of this method was validated and analyzed using the EUROC dataset within the ORB_SLAM3 open-source framework. The experimental results serve as supporting evidence for the efficacy of the proposed approach.

UAV navigation in large-scale GPS-denied bridge environments using fiducial marker-corrected stereo visual-inertial localisation

The use of Unmanned Aerial Vehicles (UAVs) for bridge inspection has gained popularity recently; however, accurately localising the UAV in GPS-denied areas is still challenging, which hinders the development of fully autonomous UAV-assisted bridge inspection solutions. This paper proposes a fiducial marker-corrected stereo visual-inertial localisation (FMC-SVIL) method, running on a resource-constrained onboard computer, to estimate UAV's global pose underneath bridge girders. The proposed FMC-SVIL utilises an optimised stereo visual-inertial odometry for continuous relative pose estimation between consecutive camera frames and an improved AprilTag2-based measurement algorithm for accurate global referencing and periodic pose corrections. The method is validated through extensive experiments, and the results show that the FMC-SVIL achieved UAV localisation with a root mean square error of 0.416 m in sunny conditions and 0.340 m in cloudy conditions. FMC-SVIL outperforms the leading vision-based simultaneous localisation and mapping (SLAM) algorithms for flights over multiple bridge spans.

SSL-Net: Sparse semantic learning for identifying reliable correspondences

Feature matching aims to identify reliable correspondences between two sets of given initial feature points, which is of considerable importance to photogrammetry and computer vision. In this study, we propose an innovative sparse semantic learning-based network, named SSL-Net, for feature matching. Specifically, SSL-Net includes a novel sparsity constraint (SC) block, which builds a sparse graph for sparse semantic learning. The SC block adopts a region-to-whole learning strategy to measure the confidence of nodes in the sparse graph. It helps the sparse graph preserve the semantic information of positive influence while rejecting unnecessary ones, thereby suppressing the negative influence of incorrect correspondences. In addition, SSL-Net also includes a channel-spatial attention feature gathering block, which gathers features along the spatial direction and channel dimension of correspondences. To mitigate the existence of label ambiguity, we incorporate the accommodation factor into the loss function of SSL-Net for feature matching. As a result, our network outperforms the state-of-the-art method by a considerable margin. Notably, SSL-Net achieves a 9.05% improvement under an error threshold of 5° over the state-of-the-art method for the relative pose estimation task on the YFCC100M dataset. Our code will be available at https://github.com/guobaoxiao/SSL-Net.

A graph optimization approach to range-based relative location

With the development of multi-robot cluster system, the research of multi-node relative pose estimation is in the ascendant. However, for the environment without GNSS or fixed anchors, the problem of relative pose estimation is still challenging. In this paper, a general algorithm and system framework based on graph optimization algorithm is proposed to fuse inter- and intra-robot measurements, as well as mobility models with sliding windows for relative attitude estimation. In order to verify the feasibility and universality of the algorithm, the algorithm has used to design a relative positioning system based on UWB positioning with a “single-node multi-tags” system structure, which can simultaneously obtain reliable relative positions and directions. The design of multi-source fusion can further improve the accuracy and robustness of relative positioning. Experiment results shows that in the harsh environment without GNSS and fixed anchor points, the relative pose estimation accuracy of this system can reach centi-meter level, which may be the best potential choice for relative positioning of multi robot cluster in complex environment.

Pose Graph Relocalization with Deep Object Detection and BIM-Supported Object Landmark Dictionary

Indoor localization is a prerequisite for autonomous robot applications in the construction industry. However, traditional localization techniques rely on low-level features and do not exploit construction-related semantics. They also are sensitive to environmental factors such as illumination and reflection rate, and therefore suffer from unexpected drifts and failures. This study proposes a pose graph relocalization framework that utilizes object-level landmarks to enhance a traditional visual localization system. The proposed framework builds an object landmark dictionary from Building Information Model (BIM) as prior knowledge. Then a multimodal deep neural network (DNN) is proposed to realize 3D object detection in real time, followed by instance-level object association with false-positive rejection, and relative pose estimation with outlier removal. Finally, a keyframe-based graph optimization is performed to rectify the drifts of traditional visual localization. The proposed framework was validated using a mobile platform with red-green-blue-depth (RGB-D) and inertial sensors, and the test scene was an indoor office environment with furnishing elements. The object detection model achieved 62.9% mean average precision (mAP). The relocalization technique reduced translational drifts by 64.67% and rotational drifts by 41.59% compared with traditional visual–inertial odometry.

FMAP: Learning robust and accurate local feature matching with anchor points

Local feature matching involves the task of establishing the pixel-wise correspondences between a pair of images. As an integral component of plentiful computer vision applications (e.g., visual localization), this task has been successfully performed using Transformer-based methods. However, these methods typically extract numerous keypoints from sparse texture regions to construct a densely-connected graph neural network (GNN) for long-range feature aggregation, which inevitably triggers redundant message exchange and hampers the learning process. Furthermore, they employ transformer encoder layers that consider images as 1D sequences, leaving them incapable of extracting multiscale local structural information from the images, which is critical for establishing correspondence in image pairs with significantly scales shifts. In this study, we develop FMAP, an innovative detector-free approach that enables accurate local feature matching. For the first issue, FMAP employs an anchor points feature aggregation module (APAM) that captures representative keypoints and discards the extraneous keypoints to build a sparsified GNN for compact yet clean message exchange, with the key insight that the keypoints containing abundant visual information are distinguishable from their neighbors. For the second issue, FMAP proposes a global–local multiscale perception module (GMPM), which incorporates abundant multiscale local context information into global feature representation by employing multiple depth-wise convolutions with varying kernel sizes, thereby generating discriminative features that are robust to scale shifts. In addition, the depth-wise convolutions are utilized in the feed-forward network of the GMPM to further fuse the global context information and local feature representation. Extensive experiments on several standard benchmarks demonstrate that the proposed FMAP method significantly outperforms state-of-the-art methods. Compared to the cutting-edge methods MatchFormer, QuadTree, and TopicFM in relative pose estimation task, FMAP surpasses them by 2.27%, 0.58%, and 1.08% in terms of AUC@5°. Besides, FMAP noticeably outperforms the baseline LoFTR by (2.38%,1.89%,1.45%) in terms of AUC@(5°, 10°, 20°). Moreover, we integrate FMAP into an official visual localization framework and conduct a visual localization experiment, with the results showing that FMAP exceeds LoFTR by 2.3% in terms of AP.

CoPR: Toward Accurate Visual Localization With Continuous Place-Descriptor Regression

Visual Place Recognition (VPR) is an image-based localization method that estimates the camera location of a query image by retrieving the most similar reference image from a map of geo-tagged reference images. In this work, we look into two fundamental bottlenecks for its localization accuracy: reference map sparseness and viewpoint invariance. Firstly, the reference images for VPR are only available at sparse poses in a map, which enforces an upper bound on the maximum achievable localization accuracy through VPR. We therefore propose Continuous Place-descriptor Regression (CoPR) to densify the map and improve localization accuracy. We study various interpolation and extrapolation models to regress additional VPR feature descriptors from only the existing references. Secondly, we compare different feature encoders and show that CoPR presents value for all of them. We evaluate our models on three existing public datasets and report on average around 30% improvement in VPR-based localization accuracy using CoPR, on top of the 15% increase by using a viewpoint-variant loss for the feature encoder. The complementary relation between CoPR and Relative Pose Estimation is also discussed.

RGB-D 데이터를 이용한 비전 기반 로봇 간 상대 위치 및 자세 추정 기법 연구

In this paper, a vision-based robot-to-robot relative pose estimation method is proposed for markerless wheeled mobile robots using an RGB-D camera. The proposed method comprises three parts: (i) object detection using a machine vision technique with RGB data, (ii) relative position estimation averaging over the point cloud with outlier rejection, and (iii) relative orientation estimation using point cloud scan matching through iterative closest point (ICP) methods. The neighboring or target robots are detected using YOLOv3-tiny so that the position of the robot can be determined in pixel coordinates. To estimate the 3D position of the target robot in the camera coordinates of the ego-robot, the object detection results are used along with the depth information and the precalibrated pinhole camera model. To estimate the orientation of the target robot, a point cloud is generated using the depth image, downsampled, and then used for scan matching by using ICP methods. For experimental verification, the proposed relative pose estimation method was implemented as a robot operating system package and tested on a small-scale differential-driving wheeled mobile robot, TurtleBot3; the estimation results were compared with the ground truth in real-world environments.

ConvMatch: Rethinking Network Design for Two-View Correspondence Learning

Multilayer perceptron (MLP) has been widely used in two-view correspondence learning for only unordered correspondences provided, and it extracts deep features from individual correspondence effectively. However, the problem of lacking context information limits its performance and hence, many extra complex blocks are designed to capture such information in the follow-up studies. In this paper, from a novel perspective, we design a correspondence learning network called ConvMatch that for the first time can leverage convolutional neural network (CNN) as the backbone to capture better context, thus avoiding the complex design of extra blocks. Specifically, with the observation that sparse motion vectors and dense motion field can be converted into each other with interpolating and sampling, we regularize the putative motion vectors by estimating dense motion field implicitly, then rectify the errors caused by outliers in local areas with CNN, and finally obtain correct motion vectors from the rectified motion field. Extensive experiments reveal that ConvMatch with a simple CNN backbone consistently outperforms state-of-the-arts including MLP-based methods for relative pose estimation and homography estimation, and shows promising generalization ability to different datasets and descriptors. Our code is publicly available at https://github.com/SuhZhang/ConvMatch.

Context-enhanced motion coherence modeling for global outlier rejection

Feature matching is an essential step in a wide range of photogrammetry and computer vision tasks but limited by the ambiguities of local descriptors. Hence, numerous false feature matches (outliers) will inevitably be generated, especially in complex scenarios. Motion coherence can establish the statistical relationships among sparse motions to remove outliers, assuming that the true matches are coherent, and the false matches are randomly scattered. However, existing methods model motion coherence either in a local spatial context or without considering rich matching priors, thereby leading to numerous matching failures when outlier rates are high. In this study, we propose a context-enhanced motion coherence modeling (CoMo) method to distinguish consistent correct motions from erroneous matches for robust outlier rejection. The CoMo method deploys a consistency-aware motion descriptor to encode the consistency-related matching priors of feature matches as a high-dimensional representation, which can provide a rich context for differentiating heterogeneous motions. Based on this discriminative descriptor, we further introduce deformable affine transformation (DAT) as a proxy for motion and fit the coherent motions from the candidate matches with a globally smooth function under the truncated least squares estimation framework. Extensive experiments on multiple large datasets (including the Image Matching Challenge at CVPR 2020) demonstrate that the CoMo method can effectively model motion coherence from noisy candidate matches and outperform other state-of-the-art methods in outlier rejection and relative camera pose estimation. The code is available at https://github.com/geovsion/CoMo.

Trifocal Relative Pose From Lines at Points.

We present a method for solving two minimal problems for relative camera pose estimation from three views, which are based on three view correspondences of (i) three points and one line and the novel case of (ii) three points and two lines through two of the points. These problems are too difficult to be efficiently solved by the state of the art Gröbner basis methods. Our method is based on a new efficient homotopy continuation (HC) solver framework MINUS, which dramatically speeds up previous HC solving by specializing hc methods to generic cases of our problems. We characterize their number of solutions and show with simulated experiments that our solvers are numerically robust and stable under image noise, a key contribution given the borderline intractable degree of nonlinearity of trinocular constraints. We show in real experiments that (i) sift feature location and orientation provide good enough point-and-line correspondences for three-view reconstruction and (ii) that we can solve difficult cases with too few or too noisy tentative matches, where the state of the art structure from motion initialization fails.

On Making SIFT Features Affine Covariant

An approach is proposed for recovering affine correspondences (ACs) from orientation- and scale-covariant, e.g., SIFT, features exploiting pre-estimated epipolar geometry. The method calculates the affine parameters consistent with the epipolar geometry from the point coordinates and the scales and rotations which the feature detector obtains. The proposed closed-form solver returns a single solution and is extremely fast, i.e., 0.5 upmu seconds on average. Possible applications include estimating the homography from a single upgraded correspondence and, also, estimating the surface normal for each correspondence found in a pre-calibrated image pair (e.g., stereo rig). As the second contribution, we propose a minimal solver that estimates the relative pose of a vehicle-mounted camera from a single SIFT correspondence with the corresponding surface normal obtained from, e.g., upgraded ACs. The proposed algorithms are tested both on synthetic data and on a number of publicly available real-world datasets. Using the upgraded features and the proposed solvers leads to a significant speed-up in the homography, multi-homography and relative pose estimation problems with better or comparable accuracy to the state-of-the-art methods.

AeroNet: An efficient relative localization and object detection network for cooperative aerial-ground unmanned vehicles

This paper proposes an efficient relative localization and object detection network (AeroNet) based on incremental learning for minimalistic high-speed cooperative navigation of aerial-ground unmanned vehicles in cluttered environments. Due to highly limited computation capability and memory resources in micro-UAVs, YOLO series are applied as the baseline of object detection network, and a lightweight backbone is built based on the depthwise separable convolution. To improve the real-time performance, the detection head is formulated with broad learning system. Besides, 6D relative pose estimation is achieved via equation fitting of an elliptical cooperative mark. To verify the effectiveness of AeroNet, experiments are conducted on Intel NUC and NVIDIA TX2 with our self-collected dataset. Results show that AeroNet can progressively increase the accuracy of object detection to 89%, and the computational time is only 76ms on Intel NUC and 28ms on Nvidia TX2, respectively, which meet the need of real-time requirement of on-board calculation in micro-UAV avionics systems.

Lidar Pose Tracking of a Tumbling Spacecraft Using the Smoothed Normal Distribution Transform

Lidar sensors enable precise pose estimation of an uncooperative spacecraft in close range. In this context, the iterative closest point (ICP) is usually employed as a tracking method. However, when the size of the point clouds increases, the required computation time of the ICP can become a limiting factor. The normal distribution transform (NDT) is an alternative algorithm which can be more efficient than the ICP, but suffers from robustness issues. In addition, lidar sensors are also subject to motion blur effects when tracking a spacecraft tumbling with a high angular velocity, leading to a loss of precision in the relative pose estimation. This work introduces a smoothed formulation of the NDT to improve the algorithm’s robustness while maintaining its efficiency. Additionally, two strategies are investigated to mitigate the effects of motion blur. The first consists in un-distorting the point cloud, while the second is a continuous-time formulation of the NDT. Hardware-in-the-loop tests at the European Proximity Operations Simulator demonstrate the capability of the proposed methods to precisely track an uncooperative spacecraft under realistic conditions within tens of milliseconds, even when the spacecraft tumbles with a significant angular rate.