Robot Pose Estimation Research Articles

GISR: Geometric Initialization and Silhouette-Based Refinement for Single-View Robot Pose and Configuration Estimation

GISR: Geometric Initialization and Silhouette-Based Refinement for Single-View Robot Pose and Configuration Estimation

Diverse Humanoid Robot Pose Estimation from Images Using Only Sparse Datasets

We present a novel dataset for humanoid robot pose estimation from images, addressing the critical need for accurate pose estimation to enhance human–robot interaction in extended reality (XR) applications. Despite the importance of this task, large-scale pose datasets for diverse humanoid robots remain scarce. To overcome this limitation, we collected sparse pose datasets for commercially available humanoid robots and augmented them through various synthetic data generation techniques, including AI-assisted image synthesis, foreground removal, and 3D character simulations. Our dataset is the first to provide full-body pose annotations for a wide range of humanoid robots exhibiting diverse motions, including side and back movements, in real-world scenarios. Furthermore, we introduce a new benchmark method for real-time full-body 2D keypoint estimation from a single image. Extensive experiments demonstrate that our extended dataset-based pose estimation approach achieves over 33.9% improvement in accuracy compared to using only sparse datasets. Additionally, our method demonstrates the real-time capability of 42 frames per second (FPS) and maintains full-body pose estimation consistency in side and back motions across 11 differently shaped humanoid robots, utilizing approximately 350 training images per robot.

The application prospects of robot pose estimation technology: exploring new directions based on YOLOv8-ApexNet.

Service robot technology is increasingly gaining prominence in the field of artificial intelligence. However, persistent limitations continue to impede its widespread implementation. In this regard, human motion pose estimation emerges as a crucial challenge necessary for enhancing the perceptual and decision-making capacities of service robots. This paper introduces a groundbreaking model, YOLOv8-ApexNet, which integrates advanced technologies, including Bidirectional Routing Attention (BRA) and Generalized Feature Pyramid Network (GFPN). BRA facilitates the capture of inter-keypoint correlations within dynamic environments by introducing a bidirectional information propagation mechanism. Furthermore, GFPN adeptly extracts and integrates feature information across different scales, enabling the model to make more precise predictions for targets of various sizes and shapes. Empirical research findings reveal significant performance enhancements of the YOLOv8-ApexNet model across the COCO and MPII datasets. Compared to existing methodologies, the model demonstrates pronounced advantages in keypoint localization accuracy and robustness. The significance of this research lies in providing an efficient and accurate solution tailored for the realm of service robotics, effectively mitigating the deficiencies inherent in current approaches. By bolstering the accuracy of perception and decision-making, our endeavors unequivocally endorse the widespread integration of service robots within practical applications.

BDR6D: Bidirectional Deep Residual Fusion Network for 6D Pose Estimation

Six-dimensional (6D) pose estimation is an important branch in the field of robotics focused on enhancing the ability of robots to manipulate and grasp objects. The latest research trend in 6D pose estimation is to directly predict the positions of two-dimensional (2D) keypoints from a single red, green, and blue (RGB) image through convolutional neural networks (CNNs) and establish a corresponding relationship with the three-dimensional (3D) keypoints of the model. Then, the perspective-n-point (PnP) algorithm is used to recover the 6D pose parameters. Currently, two challenges are encountered in pose estimation based on an RGB image. On the one hand, an RGB image lacks depth information, and it is thus difficult to directly obtain the corresponding geometric object information. On the other hand, when depth information is available, it is difficult to efficiently fuse the features of the RGB image with the features of the corresponding depth image. In this paper, we propose a bidirectional depth residual fusion network with a depth prediction (DP) network to estimate the 6D poses of objects (BDR6D). The BDR6D network predicts the depth information of objects using an RGB image, converts the depth information into point cloud information, and performs feature extraction and representation together with the RGB information during the feature extraction and representation stages. Specifically, the RGB image is fed into the BDR6D network, the DP network predicts the depth information of the objects in the image, and the depth map and RGB image are input into a point cloud network (PCN) and CNN, respectively, for feature extraction and representation. We build the bidirectional depth residual (BDR) structure so that the CNN and PCN can share information during feature extraction and representation. This approach allows the two networks to use each other’s local and global information to improve feature extraction and representation. For the keypoint selection stage, we propose an effective 2D keypoint selection method that considers the appearance and geometric information of the object of interest. We evaluate the proposed method with three benchmark datasets and compare it with other 6D pose estimation algorithms. The experimental results show that our method outperforms the state-of-the-art approach. Finally, we deploy our proposed method in conjunction with the Universal Robots 5 manipulator (UR5) robot to grasp and manipulate objects. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The purpose of this paper is to solve the problem of 6D pose estimation for robot grasping. The existing RGB image-based pose estimation approach faces two challenges. On the one hand, a single RGB image lacks depth information, so that it is difficult to directly obtain the corresponding geometric object information. On the other hand, when depth information is available, it is difficult to efficiently fuse the features of the RGB image with the features of the corresponding depth image. To solve the above problems, a novel network that can predict the depth information of objects from an RGB image and fuse the depth information with the RGB information to estimate the 6D pose of objects is proposed. Furthermore, we propose an effective 2D keypoint selection method that considers the appearance and geometric information of objects of interest. We evaluate the proposed approach based on three benchmark datasets and the UR5 robot platform and verify that our method is effective.

Visual-Based Kinematics and Pose Estimation for Skid-Steering Robots

To build commercial robots, skid-steering mechanical design is of increased popularity due to its manufacturing simplicity and unique mechanism. However, these also cause significant challenges on software and algorithm design, especially for the pose estimation (i.e., determining the robot’s rotation and position) of skid-steering robots, since they change their orientation with an inevitable skid. To tackle this problem, we propose a probabilistic sliding-window estimator dedicated to skid-steering robots, using measurements from a monocular camera, the wheel encoders, and optionally an inertial measurement unit (IMU). Specifically, we explicitly model the kinematics of skid-steering robots by both track instantaneous centers of rotation (ICRs) and correction factors, which are capable of compensating for the complexity of track-to-terrain interaction, the imperfectness of mechanical design, terrain conditions and smoothness, etc. To prevent performance reduction in robots’ long-term missions, the time-and location-varying kinematic parameters are estimated online along with pose estimation states in a tightly-coupled manner. More importantly, we conduct in-depth observability analysis for different sensors and design configurations in this paper, which provides us with theoretical tools in making the correct choice when building real commercial robots. In our experiments, we validate the proposed method by both simulation tests and real-world experiments, which demonstrate that our method outperforms competing methods by wide margins. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the problem of long-term pose estimation of the commonly commercial-used skid-steering robots with only low-cost sensors. Skid-steering robots change their orientation with a skid, which poses a significant challenge for pose estimation when using the wheel encoders. We propose to online estimate the robot’s kinematics, which succeeds in compensating for the complexity of track-to-terrain interaction, due to the slippage, the imperfectness of mechanical design, terrain conditions and smoothness. It is critical to estimate the kinematics and poses jointly to prevent performance reduction in robots’ long-term missions. We further theoretically analyze whether the kinematics parameters can be estimated under different sensor configurations, and find out the special degrade motions that make the parameters unobservable.

HiroPoseEstimation: A Dataset of Pose Estimation for Kid-Size Humanoid Robot

Pose estimation is a field of computer vision research that involves detecting, associating, and tracking data points on body parts. It is used for health monitoring, sign language understanding, human gesture control, elderly activities, sports, and humanoid robot pose estimation. The anatomy of a humanoid robot is similar to a human, which forms the basis for utilizing humanoid robot pose estimation. The Humanoid League is a major domain of the RoboCup competition, featuring soccer matches between humanoid robots. Pose estimation is used to measure the robot’s performance. Nevertheless, there have not been many research done on this subject. A new dataset model needs to be developed to solve the proposed problem. This work introduces HiroPoseEstimation, a kid-size humanoid robot dataset with several types of robots used in various poses based on movements in a soccer game. It is evaluated with both bottomup and top-down approaches using keypoint mask R-CNN and single-stage encoder-decoder model. Both methods demonstrate good performance on the proposed dataset.

A Robust Semi-Direct 3D SLAM for Mobile Robot Based on Dense Optical Flow in Dynamic Scenes.

Dynamic objects bring about a large number of error accumulations in pose estimation of mobile robots in dynamic scenes, and result in the failure to build a map that is consistent with the surrounding environment. Along these lines, this paper presents a robust semi-direct 3D simultaneous localization and mapping (SLAM) algorithm for mobile robots based on dense optical flow. First, a preliminary estimation of the robot's pose is conducted using the sparse direct method and the homography matrix is utilized to compensate for the current frame image to reduce the image deformation caused by rotation during the robot's motion. Then, by calculating the dense optical flow field of two adjacent frames and segmenting the dynamic region in the scene based on the dynamic threshold, the local map points projected within the dynamic regions are eliminated. On this basis, the robot's pose is optimized by minimizing the reprojection error. Moreover, a high-performance keyframe selection strategy is developed, and keyframes are inserted when the robot's pose is successfully tracked. Meanwhile, feature points are extracted and matched to the keyframes for subsequent optimization and mapping. Considering that the direct method is subject to tracking failure in practical application scenarios, the feature points and map points of keyframes are employed in robot relocation. Finally, all keyframes and map points are used as optimization variables for global bundle adjustment (BA) optimization, so as to construct a globally consistent 3D dense octree map. A series of simulations and experiments demonstrate the superior performance of the proposed algorithm.

Safe and Robust Map Updating for Long-Term Operations in Dynamic Environments.

Ensuring safe and continuous autonomous navigation in long-term mobile robot applications is still challenging. To ensure a reliable representation of the current environment without the need for periodic remapping, updating the map is recommended. However, in the case of incorrect robot pose estimation, updating the map can lead to errors that prevent the robot's localisation and jeopardise map accuracy. In this paper, we propose a safe Lidar-based occupancy grid map-updating algorithm for dynamic environments, taking into account uncertainties in the estimation of the robot's pose. The proposed approach allows for robust long-term operations, as it can recover the robot's pose, even when it gets lost, to continue the map update process, providing a coherent map. Moreover, the approach is also robust to temporary changes in the map due to the presence of dynamic obstacles such as humans and other robots. Results highlighting map quality, localisation performance, and pose recovery, both in simulation and experiments, are reported.

Learning-Based Visual-Strain Fusion for Eye-in-Hand Continuum Robot Pose Estimation and Control

Image processing has significantly extended the practical value of the eye-in-hand camera, enabling and promoting its applications for quantitative measurement. However, fully vision-based pose estimation methods sometimes encounter difficulties in handling cases with deficient features. In this article, we fuse visual information with the sparse strain data collected from a single-core fiber inscribed with fiber Bragg gratings (FBGs) to facilitate continuum robot pose estimation. An improved extreme learning machine algorithm with selective training data updates is implemented to establish and refine the FBG-empowered (F-emp) pose estimator <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">online</i> . The integration of F-emp pose estimation can improve sensing robustness by reducing the number of times that visual tracking is lost given moving visual obstacles and varying lighting. In particular, this integration solves pose estimation failures under full occlusion of the tracked features or complete darkness. Utilizing the fused pose feedback, a hybrid controller incorporating kinematics and data-driven algorithms is proposed to accomplish fast convergence with high accuracy. The online-learning error compensator can improve the target tracking performance with a 52.3%–90.1% error reduction compared with constant-curvature model-based control, without requiring fine model-parameter tuning and prior data acquisition.

Robot Pose Estimation and Normal Trajectory Generation on Curved Surface Using an Enhanced Non-Contact Approach.

The use of robots for machining operations has become very popular in the last few decades. However, the challenge of the robotic-based machining process, such as surface finishing on curved surfaces, still persists. Prior studies (non-contact- and contact-based) have their own limitations, such as fixture error and surface friction. To cope with these challenges, this study proposes an advanced technique for path correction and normal trajectory generation while tracking a curved workpiece's surface. Initially, a key-point selection approach is used to estimate a reference workpiece's coordinates using a depth measuring tool. This approach overcomes the fixture errors and enables the robot to track the desired path, i.e., where the surface normal trajectory is needed. Subsequently, this study employs an attached RGB-D camera on the end-effector of the robot for determining the depth and angle between the robot and the contact surface, which nullifies surface friction issues. The point cloud information of the contact surface is employed by the pose correction algorithm to guarantee the robot's perpendicularity and constant contact with the surface. The efficiency of the proposed technique is analyzed by carrying out several experimental trials using a 6 DOF robot manipulator. The results reveal a better normal trajectory generation than previous state-of-the-art research, with an average angle and depth error of 1.8 degrees and 0.4 mm.

Optimal Robot Pose Estimation Using Scan Matching by Turning Function

The turning function is a tool in image processing that measures the difference between two polygonal shapes. We propose a localization algorithm for the optimal pose estimation of autonomous mobile robots using the scan-matching method based on the turning function algorithm. There are several methodologies aimed at moving the robots in the right way and carrying out their missions well, which involves the integration of localization and control. In the proposed method, the localization problem is implemented in the form of an optimization problem. Afterwards, the turning function algorithm and the simplex method are applied to estimate the localization and orientation of the robot. The proposed algorithm first receives the polygons extracted from two sensors’ data and then allocates a histogram to each sensor scan. This algorithm attempts to maximize the similarity of the two histograms by converting them to a unified coordinate system. In this way, the estimate of the difference between the two situations is calculated. In more detail, the main objective of this study is to provide an algorithm aimed at reducing errors in the localization and orientation of mobile robots. The simulation results indicate the great performance of this algorithm. Experimental results on simulated and real datasets show that the proposed algorithms achieve better results in terms of both position and orientation metrics.

Lifelong mapping in the wild: Novel strategies for ensuring map stability and accuracy over time evaluated on thousands of robots

Lifelong mapping presents unique challenges to household robots which operate in the same environment over long durations. One is the growth of redundant information in the map as it evolves over time, which can easily overwhelm the limited computation resources of a household robot. Another is the possibility of mapping errors. An error in robot pose estimate, which if not corrected fast enough, will result in incorrect occupancy and semantic representation, rendering the map unusable. Finally, for a lifelong mapping system where the map is updated continuously, avoiding these errors altogether is infeasible. In this paper, we present a comprehensive overview of novel strategies for eliminating redundant information from the map and preventing and correcting mapping errors. We also present a detailed evaluation of these novel strategies on 10,000 robots running in indoor environments across different geographic locations of the world to demonstrate map stability and accuracy over time.

A Powered Floor System with Integrated Robot Localization

One of the most pressing issues in the field of mobile robotics is power delivery. In the past, we have proposed a powered floor based solution. In this article, we propose a system which combines a powered floor with a robot pose estimation system. The floor on which the mobile robots stand is composed of an array of interdigitated conductors that provide DC power supply; the stripes of conductors are interwoven similarly to what happens in a carpet, thus creating a sort of checkerboard of positive and negative pads. The robots are powered through sliding contacts. The power supply voltage is modulated with a binary encoding that uniquely identifies each conductor stripe power line; in this way, each robot is able to self-localize, by exploiting the information coming from the contacting pins. We describe the theoretical framework that allows concurrent power delivery and localization (with error boundaries). Then, we present the experimental evaluation that we performed using a prototype realization of the proposed powered floor system.

Dense Normal Based Degeneration-Aware 2-D Lidar Odometry for Correlative Scan Matching

Degeneracy detection and pose estimation of mobile robots have long been difficult task for laser scan matching techniques. Since the information perceived by lidar is underconstrained or deficient in degeneration scenes such as long corridors and lobbies, it is challenging for common hardware-limited lidar odometry to operate safely and stably. In this article, we propose a novel degeneration-aware correlative scan matching (CSM) algorithm for such situations. First, we carry out a low-dependency degeneration scene detection and analysis approach to determine the current degeneration situation accurately and instantly. Then, an environmentally sensitive CSM approach is designed based on real-time results, which dynamically adjusts the motion weight to achieve the autonomous adaptability to a variety of environments. Extensive experiments on both datasets and real-world scenarios demonstrate the superiority of the proposed algorithm in both degeneration scene detection and lidar odometry. In comparison to the CSM algorithm, the proposed algorithm not only minimizes the need of manually weights setup under dynamic environments but also ameliorates the accuracy of lidar odometry.

Multisensor Fusion SLAM Research Based on Improved RBPF‐SLAM Algorithm

Simultaneous localization and map construction (SLAM) technology provides the foundation for indoor robots to realize autonomous path planning. The Rao‐Blackwellized particle filtering (RBPF) algorithm is widely used to obtain information and perform map construction in unknown environments. This paper proposes a multisensor fusion algorithm that improves the RBPF‐SLAM algorithm by addressing the issues of particle distribution errors and degradation. To achieve this, the extended Kalman filter (EKF) is first adopted to effectively fuse odometry and inertial navigation (inertial measurement unit (IMU)) data as the initial positional information. Then, a high‐precision light detection and ranging (LIDAR) observation model is integrated to calculate the proposed distribution, and a threshold is introduced to determine the number of valid particles to simplify the resampling step. Finally, raster map construction experiments were carried out on both the Gazebo simulation environment and Robot Operating System (ROS)‐based mobile robot Keepbot platform in different scenarios. The experimental results demonstrate that the optimized RBPF‐SLAM algorithm generates a clearer and more complete map outline, which can effectively improve the accuracy of robot pose estimation, acquire a reliable 2D raster map with a reduced number of particles, and greatly reduces the computational effort.

Invariant Cubature Kalman Filtering-Based Visual-Inertial Odometry for Robot Pose Estimation

To maintain mechanistic stability while tracking the designated walking route, a robot must be cognizant of employed posture. Generally, visual-inertial odometer (VIO) is utilized for robot state estimation; however, the traditional cubature Kalman filter VIO (CKF-VIO) cannot transfer rotational uncertainty and compensate for the system’s processing error. To effectively improve the accuracy and stability of robot rigid body pose estimation, this paper proposes a matrix Lie group representation-based CKF framework which characterizes the uncertainty prompting in robotic motion while eliminating the VIO system internalization errors. The robot state, consisting of inertial measurement unit (IMU) pose, velocity, and 3-D landmarks’ positions, is deemed to be a single element of a high-dimensional Lie group SE2+p <xref ref-type="disp-formula" rid="deqn3" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(3)</xref> , while the accelerometers’ and gyrometers’ biases are appended to the state and estimated as well. The algorithm is validated by simulations with Monte Carlo and experiment. Results show that the CKF-VIO with a high-dimensional Lie group can improve the accuracy and consistency of robot pose estimation.

Comparison of Graph Fitting and Sparse Deep Learning Model for Robot Pose Estimation.

The paper presents a simple, yet robust computer vision system for robot arm tracking with the use of RGB-D cameras. Tracking means to measure in real time the robot state given by three angles and with known restrictions about the robot geometry. The tracking system consists of two parts: image preprocessing and machine learning. In the machine learning part, we compare two approaches: fitting the robot pose to the point cloud and fitting the convolutional neural network model to the sparse 3D depth images. The advantage of the presented approach is direct use of the point cloud transformed to the sparse image in the network input and use of sparse convolutional and pooling layers (sparse CNN). The experiments confirm that the robot tracking is performed in real time and with an accuracy comparable to the accuracy of the depth sensor.

Variational Bayesian Estimator for Mobile Robot Localization With Unknown Noise Covariance

This article studies mobile robot localization with unknown noise covariance. The AprilTag is used as landmarks and observed using the onboard camera. The system model is created based on the mobile robot motion and AprilTag measurements. The unknown measurement noise covariance is considered as a random matrix satisfying an inverse Wishart distribution. A variational Bayesian estimator is proposed to estimate the robot pose, AprilTag locations, and measurement noise covariance, where the Rao–Blackwellized estimator is developed for robot pose and AprilTag location estimation, and variational Bayesian approximation is adopted for measurement noise covariance estimation. Simulations and experiments are conducted to validate the effectiveness of the proposed method.

Marked-LIEO: Visual Marker-Aided LiDAR/IMU/Encoder Integrated Odometry.

In this paper, we propose a visual marker-aided LiDAR/IMU/encoder integrated odometry, Marked-LIEO, to achieve pose estimation of mobile robots in an indoor long corridor environment. In the first stage, we design the pre-integration model of encoder and IMU respectively to realize the pose estimation combined with the pose estimation from the second stage providing prediction for the LiDAR odometry. In the second stage, we design low-frequency visual marker odometry, which is optimized jointly with LiDAR odometry to obtain the final pose estimation. In view of the wheel slipping and LiDAR degradation problems, we design an algorithm that can make the optimization weight of encoder odometry and LiDAR odometry adjust adaptively according to yaw angle and LiDAR degradation distance respectively. Finally, we realize the multi-sensor fusion localization through joint optimization of an encoder, IMU, LiDAR, and camera measurement information. Aiming at the problems of GNSS information loss and LiDAR degradation in indoor corridor environment, this method introduces the state prediction information of encoder and IMU and the absolute observation information of visual marker to achieve the accurate pose of indoor corridor environment, which has been verified by experiments in Gazebo simulation environment and real environment.

Mobile Robot Control Based on 3D Visual Servoing: A New Approach Combining Pose Estimation by Neural Network and Differential Flatness

This paper deals with 3D visual servoing applied to mobile robots in the presence of measurement disturbances, caused in particular by target occlusion. We propose a new approach based on the flatness concept. In 3D visual servoing, the task is performed out of image coordinate space and targets may leave the camera field of view during navigation (servoing). Forced to navigate blindly during one or more periods of time, the robot will use our new open-loop control algorithm inspired by the flatness concept. The 3D visual servoing method is performed using robot pose estimation. This estimation generally contains some errors. The exact position of the robot is therefore not guaranteed, and robust feedback control is necessary to reject these errors in the input. To solve this problem, we propose a new pose estimation method that uses neural networks. We reduce the complexity of the architecture of the neural networks used (the number of variables to estimate) by proving that the location and the orientation of the robot can be ensured by using a single point in the image coordinate space for mobile robots with two degrees of freedom. To show the efficiency of the proposed algorithm, we use the RVCTOOLS MATLAB toolbox.