Visual Pose Estimation Research Articles

Autonomous Positioning for Mobile Vehicles Based on Visual‐Inertial Fusion in Challenging Dark Roadway Scenes

ABSTRACTAccurate positioning for autonomous driven underground mining vehicles (UMVs) and coal mine robots (CMRs) is indeed one of the cores in the intelligentization of coal mining. Completely different from positioning on the ground and in parking scenes, there will be great difficulties in realizing the accurate active positioning for UMVs shuttled in dark and narrow roadways. We propose an effective visual and inertial fusion positioning method for autonomous CMRs and roadheaders in challenging roadway scenarios based on the odometer‐aided inertial navigation system and visual pose estimation system. Velocity information of the odometer is adapted to restrain the error accumulation of inertial positioning based on a Kalman filter. The hybrid visual feature detection algorithm is put forward to improve the accuracy and robustness of visual observation information in a dark environment. Autonomous experiments for CMRs and roadheaders are separately performed in the narrow roadway and dark passageway to demonstrate the applicability of our localization method. The proposed approach outperforms the subsystems and existing methods in accuracy and has outstanding stability.

Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation.

Dexterous robotic manipulation tasks depend on estimating the state of in-hand objects, particularly their orientation. Although cameras have been traditionally used to estimate the object's pose, tactile sensors have recently been studied due to their robustness against occlusions. This paper explores tactile data's temporal information for estimating the orientation of grasped objects. The data from a compliant tactile sensor were collected using different time-window sample sizes and evaluated using neural networks with long short-term memory (LSTM) layers. Our results suggest that using a window of sensor readings improved angle estimation compared to previous works. The best window size of 40 samples achieved an average of 0.0375 for the mean absolute error (MAE) in radians, 0.0030 for the mean squared error (MSE), 0.9074 for the coefficient of determination (R2), and 0.9094 for the explained variance score (EXP), with no enhancement for larger window sizes. This work illustrates the benefits of temporal information for pose estimation and analyzes the performance behavior with varying window sizes, which can be a basis for future robotic tactile research. Moreover, it can complement underactuated designs and visual pose estimation methods.

Exploration of Applying Pose Estimation Techniques in Table Tennis

The newly developed computer vision pose estimation technique in artificial intelligence (AI) is an emerging technology with potential advantages, such as high efficiency and contactless detection, for improving competitive advantage in the sports industry. The related literature is currently lacking an integrated and comprehensive discussion about the applications and limitations of using the pose estimation technique. The purpose of this study was to apply AI pose estimation techniques, and to discuss the concepts, possible applications, and limitations of these techniques in table tennis. This study implemented the OpenPose pose algorithm in a real-world video of a table tennis game. The research results show that the pose estimation algorithm performs well in estimating table tennis players’ poses from the video in a graphics processing unit (GPU)-accelerated environment. This study proposes an innovative two-stage AI pose estimation method for effectively addressing the current difficulties in applying AI to table tennis players’ pose estimation. Finally, this study provides several recommendations, benefits, and various perspectives (training vs. tactics) of table tennis and pose estimation limitations for the sports industry.

Estimating Ground Reaction Forces from Two-Dimensional Pose Data: A Biomechanics-Based Comparison of AlphaPose, BlazePose, and OpenPose.

The adoption of computer vision pose estimation approaches, used to identify keypoint locations which are intended to reflect the necessary anatomical landmarks relied upon by biomechanists for musculoskeletal modelling, has gained increasing traction in recent years. This uptake has been further accelerated by keypoint use as inputs into machine learning models used to estimate biomechanical parameters such as ground reaction forces (GRFs) in the absence of instrumentation required for direct measurement. This study first aimed to investigate the keypoint detection rate of three open-source pose estimation models (AlphaPose, BlazePose, and OpenPose) across varying movements, camera views, and trial lengths. Second, this study aimed to assess the suitability and interchangeability of keypoints detected by each pose estimation model when used as inputs into machine learning models for the estimation of GRFs. The keypoint detection rate of BlazePose was distinctly lower than that of AlphaPose and OpenPose. All pose estimation models achieved a high keypoint detection rate at the centre of an image frame and a lower detection rate in the true sagittal plane camera field of view, compared with slightly anteriorly or posteriorly located quasi-sagittal plane camera views. The three-dimensional ground reaction force, instantaneous loading rate, and peak force for running could be estimated using the keypoints of all three pose estimation models. However, only AlphaPose and OpenPose keypoints could be used interchangeably with a machine learning model trained to estimate GRFs based on AlphaPose keypoints resulting in a high estimation accuracy when OpenPose keypoints were used as inputs and vice versa. The findings of this study highlight the need for further evaluation of computer vision-based pose estimation models for application in biomechanical human modelling, and the limitations of machine learning-based GRF estimation models that rely on 2D keypoints. This is of particular relevance given that machine learning models informing athlete monitoring guidelines are being developed for application related to athlete well-being.

Application of Gesture Estimation Method Based on Computer

This paper will study the computer vision pose estimation algorithm and apply it to fall detection. At present, the problem of aging cannot be ignored. There are more and more elderly people living alone. Due to their poor physical fitness, they are very easy to fall in daily life and bring great harm to the body and mind of the elderly. Therefore, they are in great need of means to monitor the daily life of the elderly in real time and make judgments on fall events. Driven by the rapid development of computer vision technology, human pose estimation has been widely used in fields such as motion recognition, motion posture scoring, and assisted rehabilitation therapy. The core of pose estimation is to detect the joint information and bone information of the human body from the image, and connect them to form a bone image to help the computer understand the human body information. The use of pose estimation for fall detection requires the extraction of human joint point information and bone information. In order to extract human joint point coordinates and bone images from video sequences, this paper has done a lot of research on computer vision pose estimation. First, the background subtraction method is studied. After that, the openpose pose estimation algorithm is studied. then studied the attitude estimation of Azure Kinect DK depth camera, focusing on its TOF principle and body tracking principle. After comparison, it is found that traditional pose estimation such as openpose is based on optical RGB images. In this paper, we mainly use Azure Kinect DK depth camera for fall detection. In the fall detection method, this paper uses the threshold method combined with pattern recognition method to detect falls. The thresholding method uses the head fall speed and the body tilt angle. The tilt angle of the upper body is used to solve the bending and squatting errors. The pattern recognition method transforms fall detection into two classification problems, fall and no-fall, and uses a support vector machine binary classification method combined with a directed gradient histogram to extract features from skeletal images for classification. The final experimental data show that this paper implements fall detection based on human posture estimation using Azure Kinect DK depth camera with high accuracy and practicality.

Intelligent vehicle visual pose estimation algorithm based on deep learning and parallel computing for dynamic scenes

In view of the disadvantages of the existing pose estimation algorithm, which has low real-time performance and the positioning accuracy will be greatly reduced in dynamic scene, a compound deep learning and parallel computing algorithm (DP-PE) is proposed. The detection algorithm based on deep learning is used to detect dynamic objects in the environment, and the dynamic feature points are removed before the matching of feature points to reduce the impact of dynamic objects on the positioning accuracy; A method for distinguishing “pseudo-dynamic objects” is proposed to solve the problem that the stationary vehicles and pedestrians in the environment are regarded as dynamic objects. The parallel computing framework for feature point extraction and matching is established on CPU-GPU heterogeneous platform to speed up DP-PE; In the localization part of DP-PE, we propose a 3D interior point detection strategy to achieve parallel search of map points, and the saturated linear kernel function is used to act on reprojection error to realize the parallelization of pose optimization. We verify the algorithm on KITTI dataset, the experimental results show that average speedup ratio of feature point extraction and matching is 6.5 times, and the overall computational efficiency of DP-PE is about 7 times higher than that before acceleration, which can realize high precision and efficient pose estimation in dynamic scene.

DeepNet-Based 3D Visual Servoing Robotic Manipulation

The fourth industrial revolution (industry 4.0) demands high-autonomy and intelligence robotic manipulators. The goal is to accomplish autonomous manipulation tasks without human interventions. However, visual pose estimation of target object in 3D space is one of the critical challenges for robot-object interaction. Incorporating the estimated pose into an autonomous manipulation control scheme is another challenge. In this paper, a deep-ConvNet algorithm is developed for object pose estimation. Then, it is integrated into a 3D visual servoing to achieve a long-range mobile manipulation task using a single camera setup. The proposed system integrates (1) deep-ConvNet training using only synthetic single images, (2) 6DOF object pose estimation as sensing feedback, and (3) autonomous long-range mobile manipulation control. The developed system consists of two main steps. First, a perception network trains on synthetic datasets and then efficiently generalizes to real-life environment without postrefinements. Second, the execution step takes the estimated pose to generate continuous translational and orientational joint velocities. The proposed system has been experimentally verified and discussed using the Husky mobile base and 6DOF UR5 manipulator. Experimental findings from simulations and real-world settings showed the efficiency of using synthetic datasets in mobile manipulation task.

Influence factors of the accuracy of monocular vision pose estimation for spacecraft based on neural network

Accurate acquisition of spacecraft pose information is a key step in tasks such as space debris removal, rendezvous and docking, and on-orbit maintenance. The application of neural network has deepened for aerospace measurement and control. Studying the parameters’ influence and degree of imaging and neural network on the accuracy of spacecraft pose estimation is necessary and urgent, but there is still little research in related fields. Approximate Bayesian inference in the depth Gaussian process is utilized in this paper to explore above problems, and systematic verification experiments are carried based on the general neural network and spacecraft pose estimation data set. The verification results depict that the estimation errors of position and attitude can be both effectively reduced by selecting a camera whose radial distortion error is less than 240%. Selecting a camera with focal ratio error less than 2.0% can effectively reduce the position estimation error. Selecting a camera whose principal point error is less than 1.0% can effectively improve the accuracy of attitude estimation. The suppression of tangential distortion error does not significantly improve the estimation accuracy of both position and attitude. Parameters about weighted attenuation factor, the number of network layers and characteristic channels of neural network are all sensitive to the accuracy of pose estimation of spacecraft within a specific range. An excellent balance can be achieved between the estimation accuracy and computational overhead when the weight attenuation factor is set between 0.01 and 0.1, and the number of network layers and characteristic channels are set separately between 20 ∼ 40 and 70 ∼ 90. It can reach the accuracy of the traditional pose estimation algorithm, and has better processing time stability in real ground test system by using the network architecture of ResNet34 with weight attenuation factor of 0.01, characteristic channels of 90.

Monocular Visual Pose Estimation for Flexible Drogue by Decoupling the Deformation

During aerial refueling, the flexible structure of the drogue stabilizing umbrella will inevitably deform. Conventional feature correspondence based methods fail to work when the drogue suffers deformation. By decoupling the deformation, a monocular visual pose estimation method of flexible drogue is proposed to overcome the problem. First, the space invariant features of the drogue are nominated by analyzing the structure and deformation of the drogue. Subsequently, the deformation in the constraints on feature correspondence is decoupled by establishing the model coordinate system. Next, the initial solution is solved by means of the structural constraints between the model coordinate system and the world coordinate system of the drogue. After that, the optimal pose is obtained by the optimization of the rotation-invariant function. Ultimately, the effectiveness and robustness of the proposed method are verified. The results sufficiently show that our method provides excellent resistance to flexible drogue deformation along with an outstanding real-time performance. And our method outperforms the others in all evaluation metrics under identical conditions.

Benchmarking pose estimation for robot manipulation

Robot grasping and manipulation require estimation of 3D object poses. Recently, a number of methods and datasets for vision-based pose estimation have been proposed. However, it is unclear how well the performance measures developed for visual pose estimation predict success in robot manipulation. In this work, we introduce an approach that connects error in pose and success in robot manipulation, and propose a probabilistic performance measure of the task success rate. A physical setup is needed to estimate the probability densities from real world samples, but evaluation of pose estimation methods is offline using captured test images, ground truth poses and the estimated densities. We validate the approach with four industrial manipulation tasks and evaluate a number of publicly available pose estimation methods. The popular pose estimation performance measure, Average Distance of Corresponding model points (ADC), does not offer any quantitatively meaningful indication of the frequency of success in robot manipulation. Our measure is instead quantitatively informative: e.g., a score of 0.24 corresponds to average success probability of 24%.

Relative pose estimation of cooperative target based on monocular vision

With the vigorous development of the global logistics industry, warehousing automation has become a popular applied research direction in computer vision. In this paper, the fork and the fork hole’s relative posture during the forklift picking process were taken as the research object. The relative position of the two is solved based on the Aruco cooperative logo PnP algorithm. In this paper, the initial values of the relative poses of the two are addressed by the linear algorithm DLT, and the relative poses are further optimized by Bundle Adjustment (BA). The iterative error observation function is constructed based on the visual pose estimation model, and the Gauss-Newton method is adopted. The least-square fitting iterative process is carried out many times, and the relative position and posture of the fork and the fork hole with high precision are finally obtained.

Single view based nonlinear vision pose estimation from coplanar points

In this paper, an iterative algorithm is proposed to determine the pose of a planar target. The algorithm relies on the planar target whose size is known, and the pose of the target can be recovered by using coplanar points from only a single image. The iterative optimization algorithm includes two steps: first, the rough pose parameters are calculated by using the least square method based on the 2D-to-3D point correspondences, then, the rough pose is used as the initial guess for the optimal iterative algorithm which minimizes the reprojection errors for coplanar image points to determine the exact pose solution. Moreover, the corresponding image processing algorithm is addressed. The pose determination algorithm includes edge detection, line extraction, pose computing, etc. Simulation results concerning the effectiveness and accuracy of the method are presented; furthermore, experiments carried out on a variable geometry truss robot manipulator with eye-in-hand camera configuration for the autonomous capturing task demonstrate the robustness of the proposed method in the real environments.

An Improved Pose Estimation Method Based on Projection Vector With Noise Error Uncertainty

Aiming at the problem of anomalous and non-independent distribution of the image errors in the feature-based visual pose estimation, a method of monocular visual pose estimation based on the uncertainty of noise error established by projection vector is proposed. First, by using the covariance matrix to describe the uncertainty of the feature point direction and integrating the uncertainty of the feature point direction into the pose estimation, characteristic point measurement error with different degrees of directional uncertainty can be adapted that can makes the algorithm robust. Then, by introducing the projection vector and combining the depth information of each feature point to represent the collinearity error, the model nonlinear problem caused by the camera perspective projection can be eliminated that can make the algorithm have global convergence. Finally, we use global convergence theorem to prove the global convergence of the proposed algorithm. The results show that the proposed method has good robustness and convergence while adapting to different degrees of error uncertainty, which can meet practical engineering applications.

ANFIS-Based Visual Pose Estimation of Uncertain Robotic Arm Using Two Uncalibrated Cameras

This paper describes a new approach for the visual pose estimation of an uncertain robotic manipulator using ANFIS (Artificial Neuro-Fuzzy Inference System) and two uncalibrated cameras. The main emphasis of this work is on the ability to estimate the positioning accuracy and repeatability of a low-cost robotic arm with unknown parameters under uncalibrated vision system. The vision system is composed of two cameras; installed on the top and on the lateral side of the robot, respectively. These two cameras need no calibration; thus, they can be installed in any position and orientation with just the condition that the end-effector of the robot must remain always visible. A red-colored feature point is fixed on the end of the third robotic arm link. In this study, captured image data via two fixed-cameras vision system are used as the sensor feedback for the position tracking of an uncertain robotic arm. LabVolt R5150 manipulator in our laboratory is used as case study. The visual estimation system is trained using ANFIS with subtractive clustering method in MATLAB. In MATLAB, the robot, feature point and cameras are simulated as physical behaviors. To get the required data for ANFIS, the manipulator was maneuvered within its workspace using forward kinematics and the feature point image coordinates were acquired with the two cameras. Simulation experiments show that the location of the robotic arm can be trained in ANFIS using two uncalibrated cameras; and problems for computational complexity and calibration requirement of multi-view geometry can be eliminated. Observing Mean Square Error (MSE), Root Mean Square Error (RMSE), Error Mean and Standard Deviation Errors, the performance of the proposed approach is efficient for using as visual feedback in uncertain robotic manipulator. Further, the proposed approach using ANFIS and uncalibrated vision system has better in flexibility, user-friendly manner and computational concepts over conventional techniques.

Visual Tilt Estimation for Planar-Motion Methods in Indoor Mobile Robots

Visual methods have many applications in mobile robotics problems, such as localization, navigation, and mapping. Some methods require that the robot moves in a plane without tilting. This planar-motion assumption simplifies the problem, and can lead to improved results. However, tilting the robot violates this assumption, and may cause planar-motion methods to fail. Such a tilt should therefore be corrected. In this work, we estimate a robot’s tilt relative to a ground plane from individual panoramic images. This estimate is based on the vanishing point of vertical elements, which commonly occur in indoor environments. We test the quality of two methods on images from several environments: An image-space method exploits several approximations to detect the vanishing point in a panoramic fisheye image. The vector-consensus method uses a calibrated camera model to solve the tilt-estimation problem in 3D space. In addition, we measure the time required on desktop and embedded systems. We previously studied visual pose-estimation for a domestic robot, including the effect of tilts. We use these earlier results to establish meaningful standards for the estimation error and time. Overall, we find the methods to be accurate and fast enough for real-time use on embedded systems. However, the tilt-estimation error increases markedly in environments containing relatively few vertical edges.

General fusion frame of circles and points in vision pose estimation

A general fusion frame of circles and points in vision pose estimation is addressed. We present a very simple formula to solve camera pose with a single circle, and then develop a general fusion method to integrate solved poses of circles and points. Two steps including initial guess and non-linear optimization are used for vision pose estimation. Different situations such as one circle one point, circles with same rotation center axis and circles with different rotation center axis are discussed for initial guess. After that, a novel unified reprojection error for circles and points is defined to determine the optimal pose solution. Experiments with real images are carried out to validate the proposed method, and results show that the method is valid and high-accuracy available.

Estimation of Antenna Pose in the Earth Frame Using Camera and IMU Data from Mobile Phones.

The poses of base station antennas play an important role in cellular network optimization. Existing methods of pose estimation are based on physical measurements performed either by tower climbers or using additional sensors attached to antennas. In this paper, we present a novel non-contact method of antenna pose measurement based on multi-view images of the antenna and inertial measurement unit (IMU) data captured by a mobile phone. Given a known 3D model of the antenna, we first estimate the antenna pose relative to the phone camera from the multi-view images and then employ the corresponding IMU data to transform the pose from the camera coordinate frame into the Earth coordinate frame. To enhance the resulting accuracy, we improve existing camera-IMU calibration models by introducing additional degrees of freedom between the IMU sensors and defining a new error metric based on both the downtilt and azimuth angles, instead of a unified rotational error metric, to refine the calibration. In comparison with existing camera-IMU calibration methods, our method achieves an improvement in azimuth accuracy of approximately 1.0 degree on average while maintaining the same level of downtilt accuracy. For the pose estimation in the camera coordinate frame, we propose an automatic method of initializing the optimization solver and generating bounding constraints on the resulting pose to achieve better accuracy. With this initialization, state-of-the-art visual pose estimation methods yield satisfactory results in more than 75% of cases when plugged into our pipeline, and our solution, which takes advantage of the constraints, achieves even lower estimation errors on the downtilt and azimuth angles, both on average (0.13 and 0.3 degrees lower, respectively) and in the worst case (0.15 and 7.3 degrees lower, respectively), according to an evaluation conducted on a dataset consisting of 65 groups of data. We show that both of our enhancements contribute to the performance improvement offered by the proposed estimation pipeline, which achieves downtilt and azimuth accuracies of respectively 0.47 and 5.6 degrees on average and 1.38 and 12.0 degrees in the worst case, thereby satisfying the accuracy requirements for network optimization in the telecommunication industry.

Vision pose estimation from planar dual circles in a single image

A novel vision based pose estimation method for a single image with planar dual circles is addressed. We present a very simple formula to solve camera pose with a single circle, and then develop a fusion method to integrate solved poses of dual circles by using space geometry constraints. After that, a new definition of the difference quantity between two ellipses is proposed to evaluate reprojection errors of dual circles and determine the optimal and unique pose solution. Experiments with synthetic data and real images are carried out to validate the proposed method, and results show that the method has a high accuracy and good robustness.

Long-range stereo visual odometry for extended altitude flight of unmanned aerial vehicles

Visual Odometry is a key technology for robust and accurate navigation of unmanned aerial vehicles in a number of low altitude applications (<120 m), particularly in environments where access to a global positioning system is possible, but not guaranteed. Navigating via vision alone reduces dependence on a global positioning system and other global navigation satellite systems, enhancing navigation robustness even in the presence of jamming, spoofing or long dropouts. To date, however, most demonstrations of visual odometry are in close proximity to the ground or other structures and are often implemented as a monocular camera combined with inertial sensing, rather than vision alone, to account for scale drift. Stereo visual odometry has received little attention for applications beyond 30 m altitude due to the generally poor performance of stereo rigs for these extremely small baseline-to-depth ratios, otherwise termed long-range stereo. This paper demonstrates stereo visual pose estimation at altitudes of up to 120 m above ground level on a small fixed-wing unmanned aerial vehicle by adapting the traditional stereo visual odometry paradigm to explicitly account for inaccurate triangulation and poorly observed scale from the stereo baseline. In addition, issues related to long-range stereo such as biased sensing are investigated to justify the approach, and a novel bundle adjustment algorithm is presented capable of handling vibration induced structural deformation between the cameras. This is achieved by continually optimizing the stereo transform within a set of inequality bounds. Results are presented demonstrating the algorithm on field-gathered data from a 2 m wingspan fixed-wing unmanned aerial vehicle flying at 30-120 m altitude over a 6.5 km trajectory.

A multi-view stereo based 3D hand-held scanning system using visual-inertial navigation and structured light

3 Production Engineering Research Institute(PRI), LG Electronics, LG-ro 222, Jinwi-myeon, Pyeongtaek 451-713, Korea Abstract. This paper describes the implementation of a 3D handheld scanning system based on visual inertial pose estimation and structured light technique.3D scanning system is composed of stereo camera, inertial navigation system (INS) and illumination projector to collect high resolution data for close range applications. The proposed algorithm for visual pose estimation is either based on feature matching or using accurate target object. The integration of INS enables the scanning system to provide the fast and reliable pose estimation supporting visual pose estimates. Block matching algorithm was used to render two view 3D reconstruction. For multiview 3D approach, rough registration and final alignment of point clouds using iterative closest point algorithm further improves the scanning accuracy. The proposed system is potentially advantageous for the generation of 3D models in bio-medical applications.