Perspective-n-Point Research Articles

Drone Detection and Pose Estimation Using Relational Graph Networks

With the upsurge in use of Unmanned Aerial Vehicles (UAVs), drone detection and pose estimation by using optical sensors becomes an important research subject in cooperative flight and low-altitude security. The existing technology only obtains the position of the target UAV based on object detection methods. To achieve better adaptability and enhanced cooperative performance, the attitude information of the target drone becomes a key message to understand its state and intention, e.g., the acceleration of quadrotors. At present, most of the object 6D pose estimation algorithms depend on accurate pose annotation or a 3D target model, which costs a lot of human resource and is difficult to apply to non-cooperative targets. To overcome these problems, a quadrotor 6D pose estimation algorithm was proposed in this paper. It was based on keypoints detection (only need keypoints annotation), relational graph network and perspective-n-point (PnP) algorithm, which achieves state-of-the-art performance both in simulation and real scenario. In addition, the inference ability of our relational graph network to the keypoints of four motors was also evaluated. The accuracy and speed were improved significantly compared with the state-of-the-art keypoints detection algorithm.

DeepHMap++: Combined Projection Grouping and Correspondence Learning for Full DoF Pose Estimation

In recent years, estimating the 6D pose of object instances with convolutional neural network (CNN) has received considerable attention. Depending on whether intermediate cues are used, the relevant literature can be roughly divided into two broad categories: direct methods and two-stage pipelines. For the latter, intermediate cues, such as 3D object coordinates, semantic keypoints, or virtual control points instead of pose parameters are regressed by CNN in the first stage. Object pose can then be solved by correspondence constraints constructed with these intermediate cues. In this paper, we focus on the postprocessing of a two-stage pipeline and propose to combine two learning concepts for estimating object pose under challenging scenes: projection grouping on one side, and correspondence learning on the other. We firstly employ a local-patch based method to predict projection heatmaps which denote the confidence distribution of projection of 3D bounding box’s corners. A projection grouping module is then proposed to remove redundant local maxima from each layer of heatmaps. Instead of directly feeding 2D–3D correspondences to the perspective-n-point (PnP) algorithm, multiple correspondence hypotheses are sampled from local maxima and its corresponding neighborhood and ranked by a correspondence–evaluation network. Finally, correspondences with higher confidence are selected to determine object pose. Extensive experiments on three public datasets demonstrate that the proposed framework outperforms several state of the art methods.

LAP-SLAM: A Line-Assisted Point-Based Monocular VSLAM

While the performance of the state-of-the-art point-based VSLAM (vision simultaneous localization and mapping) systems in well textured sequences is impressive, their performance in poorly textured situations is not satisfactory enough. A sensible alternative or addition is to consider lines. In this paper, we propose a novel line-assisted point-based VSLAM algorithm (LAP-SLAM). Our algorithm uses lines without descriptor matching, and the lines are used to assist the computation conducted by points. To the best of our knowledge, this paper proposes a new way to include line information in VSLAM. The basic idea is to use the collinear relationship of points to optimize the current point-based VSLAM algorithm. In LAP-SLAM, we propose a practical algorithm to match lines and compute the collinear relationship of points, a line-assisted bundle adjustment approach and a modified perspective-n-point (PnP) approach. We built our system based on the architecture and pipeline of ORB-SLAM. We evaluate the proposed method on a diverse range of indoor sequences in the TUM dataset and compare with point-based and point-line-based methods. The results show that the accuracy of our algorithm is close to point-line-based VSLAM systems with a much faster speed.

Image Uncertainty-Based Absolute Camera Pose Estimation with Fibonacci Outlier Elimination

Estimating the pose of a camera is a vital requirement in real-world applications like virtual reality, structure from motion, vision-assisted robot localization and manipulation. The existing Perspective-n-Point (PnP) based pose estimation algorithms have poor accuracy in presence of noise and outliers. Hence, they are combined with the Random Sample Consensus strategies to eliminate the outliers prior to pose estimation and to produce accurate results at the expense of computation time. With this concern, an Image Uncertainty-based Perspective-n-Point (IUPnP) with Fibonacci-based outlier rejection is proposed to accurately estimate the absolute pose of a calibrated camera with a minimum computation load. The uncertainties of the spherically normalized camera coordinates are formulated in the tangent space of the camera coordinate system and the initial pose is estimated using Singular Value Decomposition. The correspondences with tangent space residual exceeding the threshold values, are classified as outliers and then rejected iteratively. In order to prevent the inlier rejections and to improve the pose estimates, the threshold values are updated eventually using the Fibonacci technique. Finally, the estimated pose values are refined, using Gauss-Newton optimization. The proposed IUPnP algorithm is tested with synthetic data and real image data to validate its performance by comparing with the existing PnP algorithms in terms of accuracy. The results show that the proposed technique produces better pose estimates for correspondences with 50% outliers, than the state-of-art techniques do.

Recovering 6D object pose from RGB indoor image based on two-stage detection network with multi-task loss

Object pose estimation from an RGB image has recently become a common problem owing to its widespread application. The advent of convolutional neural networks has impelled significant progress in object detection. However, most available methods do not involve a category-level pose estimation approach. This study presents an end-to-end 6D category-level pose estimation based on a two-stage bounding-box recognition backbone architecture. Our network directly outputs the 6D pose without requiring multiple stages or additional post-processing such as a Perspective-n-Point (PnP). The two-stage CNN architecture and our loss function render multi-task joint training effective and efficient. We improve the pose estimation accuracy by replacing fully connected layers with fully convolutional layers. Fully convolutional networks require fewer parameters and are less susceptible to overfitting. Moreover, we transform the pose estimation problem into classification and regression tasks using our network; these are called Pose-cls and Pose-reg, respectively. We also present qualitative and quantitative results on real data from the SUN RGB-D dataset. The experiments demonstrate the effectiveness of our algorithms compared to other state-of-the-art methods.

Efficient Model-Based Object Pose Estimation Based on Multi-Template Tracking and PnP Algorithms

Three-Dimensional (3D) object pose estimation plays a crucial role in computer vision because it is an essential function in many practical applications. In this paper, we propose a real-time model-based object pose estimation algorithm, which integrates template matching and Perspective-n-Point (PnP) pose estimation methods to deal with this issue efficiently. The proposed method firstly extracts and matches keypoints of the scene image and the object reference image. Based on the matched keypoints, a two-dimensional (2D) planar transformation between the reference image and the detected object can be formulated by a homography matrix, which can initialize a template tracking algorithm efficiently. Based on the template tracking result, the correspondence between image features and control points of the Computer-Aided Design (CAD) model of the object can be determined efficiently, thus leading to a fast 3D pose tracking result. Finally, the 3D pose of the object with respect to the camera is estimated by a PnP solver based on the tracked 2D-3D correspondences, which improves the accuracy of the pose estimation. Experimental results show that the proposed method not only achieves real-time performance in tracking multiple objects, but also provides accurate pose estimation results. These advantages make the proposed method suitable for many practical applications, such as augmented reality.

A universal, closed-form approach for absolute pose problems

We propose a general approach for absolute pose problems including the well known perspective-n-point (PnP) problem, its generalized variant (GPnP) with and without scale, and the pose from 2D line correspondences (PnL). These have received a tremendous attention in the computer vision community during the last decades. However, it was only recently that efficient, globally optimal, closed-form solutions have been proposed, which can handle arbitrary numbers of correspondences including minimal configurations as well as over-constrained cases with linear complexity. We follow the general scheme by eliminating the linear parameters first, which results in a least squares error function that only depends on the non-linear rotation and a small symmetric coefficient matrix of fixed size. Then, in a second step the rotation is solved with algorithms which are derived using methods from algebraic geometry such as the Gröbner basis method. We propose a unified formulation based on a representation with orthogonal complements which allows to combine different types of constraints elegantly in one single framework. We show that with our unified formulation existing polynomial solvers can be interchangeably applied to problem instances other than those they were originally proposed for. It becomes possible to compare them on various registrations problems with respect to accuracy, numerical stability, and computational speed. Our compression procedure not only preserves linear complexity, it is even faster than previous formulations. For the second step we also derive an own algebraic equation solver, which can additionally handle the registration from 3D point-to-point correspondences, where other rotation solvers fail. Finally, we also present a marker-based SLAM approach with automatic registration to a target coordinate system based on partial and distributed reference information. It represents an application example that goes beyond classical camera pose estimation from image measurements and also serves for evaluation on real data.

A simple, robust and fast method for the perspective-n-point Problem

In this work, we present a simple, robust and fast method to the perspective-n-point (PnP) problem for determining the position and orientation of a calibrated camera from known reference points. Our method transfers the pose estimation problem into an optimal problem, and only needs to solve a seventh-order and a fourth-order univariate polynomial, respectively, which makes the processes more easily understood and significantly improves the performance. Additionally, the number of solutions of the proposed method is substantially smaller than existing methods. Experiment results show that the proposed method can stably handle all 3D point configurations, including the ordinary 3D case, the quasi-singular case, and the planar case, and it offers accuracy comparable or better than that of the state-of-art methods, but at much lower computational cost.

An efficient vision-based pose estimation algorithm using the assistant reference planes based on the perspective projection rays

The vision-based pose estimation using feature point markers is known as the PnP (perspective-n-point) problem. Usually the pose estimation algorithms use the pinhole imaging camera model. However, the traditional pinhole calibration may generate errors. It is a perspective projection algorithm using geometrical approximations to calibrate intrinsic and extrinsic parameters. It does not consider the lens system and only gives a single lumped result for the multiple optical elements. This leads to the accuracy reduction of pose estimation. To solve the problem, the pose estimation algorithm using a new camera model is presented in this paper. Two assistant reference planes are used to describe the camera model. The mapping relationship between the image plane and two assistant reference planes are established based on the perspective projection rays. Four coplanar feature points are used to solve the object pose. The expression of each perspective projection ray could be obtained through the mapping relationship between the image plane and two assistant reference planes. The geometrical constraints formed by the four points are then expressed with the perspective projection rays. The coordinates of feature points in the measurement system are calculated. The object pose is finally obtained. Both noise analysis and accuracy evaluation experiments were used to verify this pose estimation algorithm. Experiment results show that the algorithm in this paper can effectively reduce the influence of the pose estimation data noise and improve the pose estimation accuracy.

Acquisition and Registration of Human Point Cloud Based on Depth Camera

Registration of human cloud points is a key step in 3D human reconstruction. So, this paper proposes using depth camera to acquire human point cloud. Then, using feature matching and PnP (Perspective-n-Point) to solve the camera pose.Finally, use least square to optimize camera pose and implement point cloud registration. Experimental results show the effectiveness of this method.

Automatic Landing Assist System Using IMU+PnP for Robust Positioning of Fixed-Wing UAVs

This research proposes using a robust Perspective-n-Point (PnP) solution for an automatic landing assistant system for landing a fixed-wing, unmanned, aerial vehicle (UAV) on a runway. Specifically, we attack the problems of: 1) the difficulty in localizing markers on the ground; and 2) multiple candidate poses from PnP algorithms. The former issue can be resolved based on a least-square-based calibration between the camera and the inertial moment unit (IMU) plus geometrical information with consideration given to Lie’s algebra: SO(3). The latter issue has been presented during a long history in the pose estimation field. For an aerial vehicle that can freely move, we propose to resolve this problem using a fusion algorithm between the IMU and PnP, based on object space collinearity. We experiment and analyze that this fusion solution is among the best methods to enhance runway positioning accuracy. Furthermore, discussion based on availability of equipment is presented.

RETRACTED ARTICLE: Fast and robust absolute camera pose estimation with known focal length

Some 3D computer vision techniques such as structure from motion (SFM) and augmented reality (AR) depend on a specific perspective-n-point (PnP) algorithm to estimate the absolute camera pose. However, existing PnP algorithms are difficult to achieve a good balance between accuracy and efficiency, and most of them do not make full use of the internal camera information such as focal length. In order to attack these drawbacks, we propose a fast and robust PnP (FRPnP) method to calculate the absolute camera pose for 3D compute vision. In the proposed FRPnP method, we firstly formulate the PnP problem as the optimization problem in the null space that can avoid the effects of the depth of each 3D point. Secondly, we can easily get the solution by the direct manner using singular value decomposition. Finally, the accurate information of camera pose can be obtained by optimization strategy. We explore four ways to evaluate the proposed FRPnP algorithm with synthetic dataset, real images, and apply it in the AR and SFM system. Experimental results show that the proposed FRPnP method can obtain the best balance between computational cost and precision, and clearly outperforms the state-of-the-art PnP methods.

2DTriPnP: A Robust Two-Dimensional Method for Fine Visual Localization Using Google Streetview Database

The complete camera pose (location + orientation) of the Google street view (GSV) images is provided by Google. Hence, one can utilize this information to localize a query camera based on the projective geometry. The existing literature works either perform image retrieval-based rough location recognition or require high-computational power/specific features for three-dimensional fine localization. In this paper, we propose a robust 2-D method for outdoor image-based localization using the GSV database. Having found the nearest neighboring images (best matches) in the GSV database using image retrieval techniques or the GPS circle information, the proposed method can be applied for robust fine localization of pedestrians/vehicles. The proposed method first finds the common features among the three views, i.e., query view and two from the best matches. Next, for each common feature, a 2-D triangulation is performed using the retrieved database images to find the feature world coordinates. We call this procedure “2DTri.” Afterward, a novel set of nonlinear equations is solved to estimate the fine location of the query. The novel set of equations can be interpreted as a 2-D version of the well-known perspective n-point (PnP) problem, which we call “2DPnP.” Hence, the proposed method is named “2DTriPnP.” The 2DPnP step is performed in a robust way, which is more accurate and considerably less complex compared to the conventional RANSAC-based robust methods. 2DTriPnP will be demonstrated experimentally to show better localization performance compared to other state-of-the-art methods.

An Improved Orthogonal Iterative Algorithm for Monocular Camera Pose Estimation

In this paper, we focus on the classical perspective-n-point(PnP) problem in monocular camera pose estimation and propose an improved orthogonal iterative algorithm. The key idea is to integrate the steps in each iteration, and extract the repetitive calculation of each iteration. The repetitive computation in each iteration can be abstracted and done before iteration process. The computational complexity of each iteration is reduced from toO(n) to O(1) . So that, more iterations can be done in a short time and the accuracy is improved as well. The simulation results and real experiments results show the efficiency and accuracy of our entry accepted iterative algorithms, our algorithm can maintain high pose estimation accuracy. This algorithm is more likely to converge to correct pose in a short run time, which makes it more applicable for real time applications.

Pose Estimation from Line Correspondences: A Complete Analysis and a Series of Solutions.

In this paper we deal with the camera pose estimation problem from a set of 2D/3D line correspondences, which is also known as PnL (Perspective-n-Line) problem. We carry out our study by comparing PnL with the well-studied PnP (Perspective-n-Point) problem, and our contributions are three-fold: (1) We provide a complete 3D configuration analysis for P3L, which includes the well-known P3P problem as well as several existing analyses as special cases. (2) By exploring the similarity between PnL and PnP, we propose a new subset-based PnL approach as well as a series of linear-formulation-based PnL approaches inspired by their PnP counterparts. (3) The proposed linear-formulation-based methods can be easily extended to deal with the line and point features simultaneously.

MLPNP – A REAL-TIME MAXIMUM LIKELIHOOD SOLUTION TO THE PERSPECTIVE-N-POINT PROBLEM

In this paper, a statistically optimal solution to the Perspective-n-Point (PnP) problem is presented. Many solutions to the PnP problem are geometrically optimal, but do not consider the uncertainties of the observations. In addition, it would be desirable to have an internal estimation of the accuracy of the estimated rotation and translation parameters of the camera pose. Thus, we propose a novel maximum likelihood solution to the PnP problem, that incorporates image observation uncertainties and remains real-time capable at the same time. Further, the presented method is general, as is works with 3D direction vectors instead of 2D image points and is thus able to cope with arbitrary central camera models. This is achieved by projecting (and thus reducing) the covariance matrices of the observations to the corresponding vector tangent space.

MLPNP – A REAL-TIME MAXIMUM LIKELIHOOD SOLUTION TO THE PERSPECTIVE-N-POINT PROBLEM

Abstract. In this paper, a statistically optimal solution to the Perspective-n-Point (PnP) problem is presented. Many solutions to the PnP problem are geometrically optimal, but do not consider the uncertainties of the observations. In addition, it would be desirable to have an internal estimation of the accuracy of the estimated rotation and translation parameters of the camera pose. Thus, we propose a novel maximum likelihood solution to the PnP problem, that incorporates image observation uncertainties and remains real-time capable at the same time. Further, the presented method is general, as is works with 3D direction vectors instead of 2D image points and is thus able to cope with arbitrary central camera models. This is achieved by projecting (and thus reducing) the covariance matrices of the observations to the corresponding vector tangent space.

Robust world-centric stereo EKF localization with active loop closing for AUVs

Visual localization is a crucial task in Autonomous Underwater Vehicles (AUV) and it is usually complicated by the extreme irregularity of the natural aquatic environments, or by unfavorable water conditions. Visual Simultaneous Localization and Mapping (SLAM) approaches are widely used in land and represent the most precise techniques for localization, but applied underwater, they are still an open and ongoing challenge. This paper presents a general approach to visual 3D pose-based SLAM based on Extended Kalman Filters (EKF). This approach has a general design being applicable to any vehicle with up to 6 Degrees of freedom, so, it is particularly suitable for AUV. It uses only visual data coming from a stereo camera, all orientations involved in the system are represented in the quaternion space in order to avoid the gimbal lock singularities, and the sparsity of the covariance matrix is guaranteed during the whole trajectory since the state vector only includes the vehicle global pose. The vehicle pose is continuously predicted by means of a stereo visual odometer, and eventually corrected with the pose constraints given by a particularization of the Perspective N-Point problem (PNP) [1], applied to the registration of images that most likely close a loop. Experimental results show the important pose corrections given by the SLAM approach with respect to a ground truth, compared with the evident trajectory errors present in the visual odometer estimates.

A new 2-point absolute pose estimation algorithm under planar motion

Several pose estimation algorithms, such as n-point and perspective n-point (PnP), have been introduced over the last few decades to solve the relative and absolute pose estimation problems in robotics research. Since the n-point algorithms cannot decide the real scale of robot motion, the PnP algorithms are often addressed to find the absolute scale of motion. This paper introduce a new PnP algorithm which use only two 3D–2D correspondences by considering only planar motion. Experiment results prove that the proposed algorithm solves the absolute motion in real scale with high accuracy and less computational time compared to previous algorithms.

Bayesian perspective-plane (BPP) with maximum likelihood searching for visual localization

The proposed "Perspective-Plane" in this paper is similar to the well-known "Perspective-n-Point (PnP)" or "Perspective-n-Line (PnL)" problems in computer vision. However, it has broader applications and potentials, because planar scenes are more widely available than control points or lines in daily life. We address this problem in the Bayesian framework and propose the "Bayesian Perspective-Plane (BPP)" algorithm, which can deal with more generalized constraints rather than type-specific ones. The BPP algorithm consists of three steps: 1) plane normal computation by maximum likelihood searching from Bayesian formulation; 2) plane distance computation; and 3) visual localization. In the first step, computation of the plane normal is formulated within the Bayesian framework, and is solved by using the proposed Maximum Likelihood Searching Model (MLS-M). Two searching modes of 2D and 1D are discussed. MLS-M can incorporate generalized planar and out-of-plane deterministic constraints. With the computed normal, the plane distance is recovered from a reference length or distance. The positions of the object or the camera can be determined afterwards. Extensions of the proposed BPP algorithm to deal with un-calibrated images and for camera calibration are discussed. The BPP algorithm has been tested with both simulation and real image data. In the experiments, the algorithm was applied to recover planar structure and localize objects by using different types of constraints. The 2D and 1D searching modes were illustrated for plane normal computation. The results demonstrate that the algorithm is accurate and generalized for object localization. Extensions of the proposed model for camera calibration were also illustrated in the experiment. The potential of the proposed algorithm was further demonstrated to solve the classic Perspective-Three-Point (P3P) problem and classify the solutions in the experiment. The proposed BPP algorithm suggests a practical and effective approach for visual localization.