Camera Pose Estimation Method Research Articles

Depth-Based Efficient PnP: A Rapid and Accurate Method for Camera Pose Estimation

Depth-Based Efficient PnP: A Rapid and Accurate Method for Camera Pose Estimation

Point Cloud Plane Segmentation-Based Robust Image Matching for Camera Pose Estimation

The mainstream image matching method for recovering the motion of the camera is based on local feature matching, which faces the challenges of rotation, illumination, and the presence of dynamic objects. In addition, local feature matching relies on the distance between descriptors, which easily leads to lots of mismatches. In this paper, we propose a new robust image matching method for camera pose estimation, called IM_CPE. It is a novel descriptor matching method combined with 3-D point clouds for image matching. Specifically, we propose to extract feature points based on a pair of matched point cloud planes, which are generated and segmented based on depth images. Then, the feature points are matched based on the distance between their corresponding 3-D points on the point cloud planes and the distance between their descriptors. Moreover, the robustness of the matching can be guaranteed by the centroid distance of the matched point cloud planes. We evaluate the performance of IM_CPE using four well-known key point extraction algorithms, namely Scale-Invariant Feature Transform (SIFT), Speed Up Robust Feature (SURF), Features from Accelerated Segment Test (FAST), and Oriented FAST and Rotated Brief (ORB), with four sequences from the TUM RGBD dataset. According to the experimental results, compared to the original SIFT, SURF, FAST, and ORB algorithms, the NN_mAP performance of the four key point algorithms has been improved by 11.25%, 13.98%, 16.63%, and 10.53% on average, respectively, and the M.Score has also been improved by 25.15%, 23.05%, 22.28%, and 11.05% on average, respectively. The results show that the IM_CPE can be combined with the existing key points extraction algorithms and the IM_CPE can significantly improve the performance of these key points algorithms.

Spaceborne pose determination based on image to 3D digital surface model matching

Developing an accurate onboard-camera pose estimation is one major challenge of satellite systems, and the attempt of improving remote sensing camera pose accuracy never ceases. The camera pose can be recovered by aligning a captured 2D image and a 3D digital surface model of the corresponding scene. In this paper, a novel camera pose estimation method from captured images with the over known real scene 3D products is proposed to enhance remote sensing camera attitude accuracy. The purpose of this estimation is to determine the pose of a camera purely from an image based on a known 3D model, where 3D products of very high spatial resolution are projected onto image space by virtual camera system with error-contained initial exterior orientation parameters, and whether the pose of the camera can be determined precisely depends on the 2D–3D registration result. The process consists of two steps: (1) feature extraction and (2) similarity measure and registration. Furthermore, the proposed method revises the rotation matrix and translation vector by utilizing formulation based on quaternion representation of rotation, respectively. We evaluate our method on challenging simulation data and results show that acceptable accuracy of camera pose can be achieved.

Closed-form camera pose and plane parameters estimation for moments-based visual servoing of planar objects

Image moments are global descriptors of an image and can be used to achieve control-decoupling properties in visual servoing. However, only a few methods completely decouple the control. This study introduces a novel camera pose estimation method, which is a closed-form solution, based on the image moments of planar objects. Traditional position-based visual servoing estimates the pose of a camera relative to an object, but the pose estimation method directly estimates the pose of an initial camera relative to a desired camera. Because the estimation method is based on plane parameters, a plane parameters estimation method based on the 2D rotation, 2D translation, and scale invariant moments is also proposed. A completely decoupled position-based visual servoing control scheme from the two estimation methods above was adopted. The new scheme exhibited asymptotic stability when the object plane was in the camera field of view. Simulation results demonstrated the effectiveness of the two estimation methods and the advantages of the visual servo control scheme compared with the classical method.

Object-Based Visual Camera Pose Estimation From Ellipsoidal Model and 3D-Aware Ellipse Prediction

In this paper, we propose a method for initial camera pose estimation from just a single image which is robust to viewing conditions and does not require a detailed model of the scene. This method meets the growing need of easy deployment of robotics or augmented reality applications in any environments, especially those for which no accurate 3D model nor huge amount of ground truth data are available. It exploits the ability of deep learning techniques to reliably detect objects regardless of viewing conditions. Previous works have also shown that abstracting the geometry of a scene of objects by an ellipsoid cloud allows to compute the camera pose accurately enough for various application needs. Though promising, these approaches use the ellipses fitted to the detection bounding boxes as an approximation of the imaged objects. In this paper, we go one step further and propose a learning-based method which detects improved elliptic approximations of objects which are coherent with the 3D ellipsoids in terms of perspective projection. Experiments prove that the accuracy of the computed pose significantly increases thanks to our method. This is achieved with very little effort in terms of training data acquisition - a few hundred calibrated images of which only three need manual object annotation. Code and models are released at https://gitlab.inria.fr/tangram/3d-aware-ellipses-for-visual-localization

Relative Camera Pose Estimation using Synthetic Data with Domain Adaptation via Cycle-Consistent Adversarial Networks

Learning-based visual localization has become prospective over the past decades. Since ground truth pose labels are difficult to obtain, recent methods try to learn pose estimation networks using pixel-perfect synthetic data. However, this also introduces the problem of domain bias. In this paper, we first build a Tuebingen Buildings dataset of RGB images collected by a drone in urban scenes and create a 3D model for each scene. A large number of synthetic images are generated based on these 3D models. We take advantage of image style transfer and cycle-consistent adversarial training to predict the relative camera poses of image pairs based on training over synthetic environment data. We propose a relative camera pose estimation approach to solve the continuous localization problem for autonomous navigation of unmanned systems. Unlike those existing learning-based camera pose estimation methods that train and test in a single scene, our approach successfully estimates the relative camera poses of multiple city locations with a single trained model. We use the Tuebingen Buildings and the Cambridge Landmarks datasets to evaluate the performance of our approach in a single scene and across-scenes. For each dataset, we compare the performance between real images and synthetic images trained models. We also test our model in the indoor dataset 7Scenes to demonstrate its generalization ability.

Voxel-Based Scene Representation for Camera Pose Estimation of a Single RGB Image

Deep learning has been utilized in end-to-end camera pose estimation. To improve the performance, we introduce a camera pose estimation method based on a 2D-3D matching scheme with two convolutional neural networks (CNNs). The scene is divided into voxels, whose size and number are computed according to the scene volume and the number of 3D points. We extract inlier points from the 3D point set in a voxel using random sample consensus (RANSAC)-based plane fitting to obtain a set of interest points consisting of a major plane. These points are subsequently reprojected onto the image using the ground truth camera pose, following which a polygonal region is identified in each voxel using the convex hull. We designed a training dataset for 2D–3D matching, consisting of inlier 3D points, correspondence across image pairs, and the voxel regions in the image. We trained the hierarchical learning structure with two CNNs on the dataset architecture to detect the voxel regions and obtain the location/description of the interest points. Following successful 2D–3D matching, the camera pose was estimated using n-point pose solver in RANSAC. The experiment results show that our method can estimate the camera pose more precisely than previous end-to-end estimators.

Accurate Dynamic SLAM Using CRF-Based Long-Term Consistency.

Accurate camera pose estimation is essential and challenging for real world dynamic 3D reconstruction and augmented reality applications. In this article, we present a novel RGB-D SLAM approach for accurate camera pose tracking in dynamic environments. Previous methods detect dynamic components only across a short time-span of consecutive frames. Instead, we provide a more accurate dynamic 3D landmark detection method, followed by the use of long-term consistency via conditional random fields, which leverages long-term observations from multiple frames. Specifically, we first introduce an efficient initial camera pose estimation method based on distinguishing dynamic from static points using graph-cut RANSAC. These static/dynamic labels are used as priors for the unary potential in the conditional random fields, which further improves the accuracy of dynamic 3D landmark detection. Evaluation using the TUM and Bonn RGB-D dynamic datasets shows that our approach significantly outperforms state-of-the-art methods, providing much more accurate camera trajectory estimation in a variety of highly dynamic environments. We also show that dynamic 3D reconstruction can benefit from the camera poses estimated by our RGB-D SLAM approach.

An efficient and globally optimal method for camera pose estimation using line features

The accurate estimation of camera pose using numerous line correspondences in real time is a challenging task. This paper presents a non-iterative approach to solve the Perspective-n-Line (PnL) problem. The method can provide high speed and global optimality, as well as linear complexity. A nonlinear least squares (non-LLS) objective function is first formulated by parameterizing the rotation matrix with Cayley representation. A system of three third-order equations is then derived from its optimality conditions, and then, it is solved directly based on the Grobner basis technique. Finally, the camera pose can be easily obtained by back-substitution. A major advantage of the proposed method lies in scalability, as the size of the elimination template used in the Grobner basis technique is independent to the number of line correspondences. Extensive and detailed experiments on synthetic data and real images are conducted, demonstrating that the proposed method achieves an accuracy that is equivalent or superior to the leading methods, but with reduced computational requirements. The source code is available at https://github.com/dannyshin1/danny/tree/master/OPnL1 .

Prediction of terrain occlusion in Chang'e-4 mission

This paper proposes a precise terrain occlusion prediction algorithm based on stereo image sequences from the China's Chang'e-4 Yutu-2 rover’ visual system, which includes an on orbit stereo camera calibration method and a camera pose estimation method. The main advantage of the proposed algorithm is that the above two methods consider the problem that the coefficient matrix and observation vector should also be considered with regard to the error during optimization. In this case, the total least squares solution is applied instead of the ordinary least squares solution. Experimental results show that the proposed algorithm has higher precision (with average differences in the altitude angles and azimuthal angles of −0.37 degrees and −0.29 degrees) and application stability than the ordinary least squares method. The proposed algorithm has been successfully used in Chang'e-4 mission operations.

An anaglyph 2D-3D stereoscopic video visualization approach

In this paper, we propose a simple anaglyph 3D stereo generation algorithm from 2D video sequence with a monocular camera. In our novel approach, we employ camera pose estimation method to directly ...

Underwater Structure from Motion for Cameras Under Refractive Surfaces

Structure from Motion (SfM), as a three-dimensional (3D) reconstruction technique, can estimate the structure of an object by using a single moving camera. Cameras deployed in underwater environments are generally confined to waterproof housings. Thus, the light rays entering the camera are refracted twice; once at the interface between the water and the camera housing, and again at the interface between the camera housing and air. Images captured from scenes in underwater environments are prone to, and deteriorate, from distortion caused by this refraction. Severe distortions in geometric reconstruction would be caused if the refractive distortion is not properly addressed. Here, we propose a SfM approach to deal with the refraction in a camera system including a refractive surface. The impact of light refraction is precisely modeled in the refractive model. Based on the model, a new calibration and camera pose estimation method is proposed. This proposed method assists in accurate 3D reconstruction using the refractive camera system. Experiments, including simulations and real images, show that the proposed method can achieve accurate reconstruction, and effectively reduce the refractive distortion compared to conventional SfM.

Sick Moves! Motion Parameters as Indicators of Simulator Sickness.

We explore motion parameters, more specifically gait parameters, as an objective indicator to assess simulator sickness in Virtual Reality (VR). We discuss the potential relationships between simulator sickness, immersion, and presence. We used two different camera pose (position and orientation) estimation methods for the evaluation of motion tasks in a large-scale VR environment: a simple model and an optimized model that allows for a more accurate and natural mapping of human senses. Participants performed multiple motion tasks (walking, balancing, running) in three conditions: a physical reality baseline condition, a VR condition with the simple model, and a VR condition with the optimized model. We compared these conditions with regard to the resulting sickness and gait, as well as the perceived presence in the VR conditions. The subjective measures confirmed that the optimized pose estimation model reduces simulator sickness and increases the perceived presence. The results further show that both models affect the gait parameters and simulator sickness, which is why we further investigated a classification approach that deals with non-linear correlation dependencies between gait parameters and simulator sickness. We argue that our approach could be used to assess and predict simulator sickness based on human gait parameters and we provide implications for future research.

基于目标函数加权优化的相机位姿估计算法

基于目标函数加权优化的相机位姿估计算法

Research of the three-dimensional tracking and registration method based on multiobjective constraints in an AR system.

To match the virtual image and actual environment in an augmented reality (AR) system, it is necessary to complete the task of three-dimensional (3D) tracking registration. This paper proposes a new method for 3D tracking registration. Previous methods extract feature points in images to realize tracking registration. In this paper, the objects are extracted from the deep convolution neural network in the scene, and the camera pose is estimated by establishing the constraint relation of the objects. Then, 3D tracking and registration of the virtual object are realized. We design an improved single-shot multibox detector semantic segmentation network to identify and segment the scene and extract the pixel classification results of the objects in the scene. The effect of classification with this method is better. The depth of the extracted object is estimated based on the data from the left and right cameras, and the 2D image is converted into a 3D point cloud. A camera pose estimation method, combined with multiobjective information, is proposed. The camera transformation matrix is directly estimated by establishing a mathematical model. This method avoids the effect on the accuracy of the camera pose estimation when the feature points are not sufficient. Moreover, by assigning different weights to the point clouds of different objects, errors caused by the model can be reduced. The experimental results showed that the 3D registration method proposed in this paper is less than 2.5 pixels in the application scene of an augmented reality head-up display. This method had a better effect compared with that of existing methods and also improved driving safety.

Automatic Calibration of Odometry and Robot Extrinsic Parameters Using Multi-Composite-Targets for a Differential-Drive Robot with a Camera.

This paper simultaneously calibrates odometry parameters and the relative pose between a monocular camera and a robot automatically. Most camera pose estimation methods use natural features or artificial landmark tools. However, there are mismatches and scale ambiguity for natural features; the large-scale precision landmark tool is also challenging to make. To solve these problems, we propose an automatic process to combine multiple composite targets, select keyframes, and estimate keyframe poses. The composite target consists of an aruco marker and a checkerboard pattern. First, an analytical method is applied to obtain initial values of all calibration parameters; prior knowledge of the calibration parameters is not required. Then, two optimization steps are used to refine the calibration parameters. Planar motion constraints of the camera are introduced in these optimizations. The proposed solution is automatic; manual selection of keyframes, initial values, and robot construction within a specific trajectory are not required. The competing accuracy and stability of the proposed method under different target placements and robot paths are tested experimentally. Positive effects on calibration accuracy and stability are obtained when (1) composite targets are adopted; (2) two optimization steps are used; (3) plane motion constraints are introduced; and (4) target numbers are increased.

Adaptive Monocular Visual-Inertial SLAM for Real-Time Augmented Reality Applications in Mobile Devices.

Simultaneous localization and mapping (SLAM) is emerging as a prominent issue in computer vision and next-generation core technology for robots, autonomous navigation and augmented reality. In augmented reality applications, fast camera pose estimation and true scale are important. In this paper, we present an adaptive monocular visual–inertial SLAM method for real-time augmented reality applications in mobile devices. First, the SLAM system is implemented based on the visual–inertial odometry method that combines data from a mobile device camera and inertial measurement unit sensor. Second, we present an optical-flow-based fast visual odometry method for real-time camera pose estimation. Finally, an adaptive monocular visual–inertial SLAM is implemented by presenting an adaptive execution module that dynamically selects visual–inertial odometry or optical-flow-based fast visual odometry. Experimental results show that the average translation root-mean-square error of keyframe trajectory is approximately 0.0617 m with the EuRoC dataset. The average tracking time is reduced by 7.8%, 12.9%, and 18.8% when different level-set adaptive policies are applied. Moreover, we conducted experiments with real mobile device sensors, and the results demonstrate the effectiveness of performance improvement using the proposed method.

CONVOLUTIONAL NEURAL NETWORK FOR CAMERA POSE ESTIMATION FROM OBJECT DETECTIONS

Abstract. Known scene geometry and camera calibration parameters give important information to video content analysis systems. In this paper, we propose a novel method for camera pose estimation based on people observation in the input video captured by static camera. As opposed to previous techniques, our method can deal with false positive detections and inaccurate localization results. Specifically, the proposed method does not make any assumption about the utilized object detector and takes it as a parameter. Moreover, we do not require a huge labeled dataset of real data and train on the synthetic data only. We apply the proposed technique for camera pose estimation based on head observations. Our experiments show that the algorithm trained on the synthetic dataset generalizes to real data and is robust to false positive detections.

Mirage: an O(n) time analytical solution to 3D camera pose estimation with multi-camera support

SUMMARYMirage is a camera pose estimation method that analytically solves pose parameters in linear time for multi-camera systems. It utilizes a reference camera pose to calculate the pose by minimizing the 2D projection error between reference and actual pixel coordinates. Previously, Mirage has been successfully applied to trajectory tracking (visual servoing) problem. In this study, a comprehensive evaluation of Mirage is performed by particularly focusing on the area of camera pose estimation. Experiments have been performed using simulated and real data on noisy and noise-free environments. The results are compared with the state-of-the-art techniques. Mirage outperforms other methods by generating fast and accurate results in all tested environments.

Automated architectural reconstruction using reference planes under convex optimization

In this paper, a method for the automated reconstruction of architectures from two views of a monocular camera is proposed. While this research topic has been studied over the last few decades, we contend that a satisfactory approach has not yet been devised. Here, a new method to solve the same problem with several points of novelty is proposed. First, reference planes are automatically detected using color, straight lines, and edge/vanishing points. This approach is quite robust and fast even when different views and complicated conditions are presented. Second, the camera pose and 3D points are accurately estimated by a two-view geometry constraint in the convex optimization approach. It has been demonstrated that camera rotations are appropriately estimated, while translations induce a significant error in short baseline images. To overcome this problem, we rely only on reference planes to estimate image homography instead of using the conventional camera pose estimation method. Thus, the problem associated with short baseline images is adequately addressed. The 3D points and translation are then simultaneously triangulated. Furthermore, both the homography and 3D point triangulation are computed via the convex optimization approach. The error of back-projection and measured points is minimized in L ∞-norm so as to overcome the local minima problem of the canonical L 2-norm method. Consequently, extremely accurate homography and point clouds can be achieved with this scheme. In addition, a robust plane fitting method is introduced to describe a scene. The corners are considered as properties of the plane in order to limit the boundary. Thus, it is necessary to find the exact corresponding corner positions by searching along the epipolar line in the second view. Finally, the texture of faces is mapped from 2D images to a 3D plane. The simulation results demonstrate the effectiveness of the proposed method for scenic images in an outdoor environment.