Target Pose Estimation Research Articles

Automatic loading method for robot guided by 3D vision for scattered and stacked shoe soles

This paper primarily introduces a method for the automated feeding of scattered stacked shoe soles using a 3D visual-guided robot. Initially, addressing issues like slow speed in pose estimation and poor robustness during the robot sorting and feeding process, we introduce an enhanced pose estimation algorithm. This algorithm combines the improved Super 4-Point Congruent Sets (Super 4PCS) with the Truncated Least Squares Semidefinite Relaxation (TEASER) algorithm, significantly boosting the speed and robustness of pose estimation during sole sorting, and achieving precise target pose estimation. Building upon this foundation, we present a sorting strategy for disordered stacked shoe soles. This strategy integrates spatial positional information of each sole, employing multi-objective decision-making and recognition algorithms to determine optimal grasping targets. Finally, an experimental platform for automated sole feeding is designed to validate the proposed method. The experimental results indicate that the pose estimation method proposed in this paper achieves an average distance error of 2.04 mm and an average angular error of 2.72°, with the robot’s average success rate in grasping reaching 97.08%. Moreover, the average processing time of the vision algorithm is only 1.34 s, demonstrating good efficiency, precision, and robustness. This method effectively meets the automated feeding needs of scattered and stacked shoe soles in the actual production processes of shoemaking enterprises.

Vulnerable Road User Skeletal Pose Estimation Using mmWave Radars

A skeletal pose estimation method, named RVRU-Pose, is proposed to estimate the skeletal pose of vulnerable road users based on distributed non-coherent mmWave radar. In view of the limitation that existing methods for skeletal pose estimation are only applicable to small scenes, this paper proposes a strategy that combines radar intensity heatmaps and coordinate heatmaps as input to a deep learning network. In addition, we design a multi-resolution data augmentation and training method suitable for radar to achieve target pose estimation for remote and multi-target application scenarios. Experimental results show that RVRU-Pose can achieve better than 2 cm average localization accuracy for different subjects in different scenarios, which is superior in terms of accuracy and time compared to existing state-of-the-art methods for human skeletal pose estimation with radar. As an essential performance parameter of radar, the impact of angular resolution on the estimation accuracy of a skeletal pose is quantitatively analyzed and evaluated in this paper. Finally, RVRU-Pose has also been extended to the task of estimating the skeletal pose of a cyclist, reflecting the strong scalability of the proposed method.

Keypoints Filtrating Nonlinear Refinement in Spatial Target Pose Estimation with Deep Learning

Keypoints Filtrating Nonlinear Refinement in Spatial Target Pose Estimation with Deep Learning

Enhancing 6-DoF Object Pose Estimation through Multiple Modality Fusion: A Hybrid CNN Architecture with Cross-Layer and Cross-Modal Integration

Recently, applying the utilization of RGB-D data for robot perception tasks has garnered significant attention in domains like robotics and autonomous driving. However, a prominent challenge in this field lies in the substantial impact of feature robustness on both segmentation and pose estimation tasks. To tackle this challenge, we proposed a pioneering two-stage hybrid Convolutional Neural Network (CNN) architecture, which connects segmentation and pose estimation in tandem. Specifically, we developed Cross-Modal (CM) and Cross-Layer (CL) modules to exploit the complementary information from RGB and depth modalities, as well as the hierarchical features from diverse layers of the network. The CM and CL integration strategy significantly enhanced the segmentation accuracy by effectively capturing spatial and contextual information. Furthermore, we introduced the Convolutional Block Attention Module (CBAM), which dynamically recalibrated the feature maps, enabling the network to focus on informative regions and channels, thereby enhancing the overall performance of the pose estimation task. We conducted extensive experiments on benchmark datasets to evaluate the proposed method and achieved exceptional target pose estimation results, with an average accuracy of 94.5% using the ADD-S AUC metric and 97.6% of ADD-S smaller than 2 cm. These results demonstrate the superior performance of our proposed method.

Motion Capture for Sporting Events Based on Graph Convolutional Neural Networks and Single Target Pose Estimation Algorithms

Human pose estimation refers to accurately estimating the position of the human body from a single RGB image and detecting the location of the body. It serves as the basis for several computer vision tasks, such as human tracking, 3D reconstruction, and autonomous driving. Improving the accuracy of pose estimation has significant implications for the advancement of computer vision. This paper addresses the limitations of single-branch networks in pose estimation. It presents a top-down single-target pose estimation approach based on multi-branch self-calibrating networks combined with graph convolutional neural networks. The study focuses on two aspects: human body detection and human body pose estimation. The human body detection is for athletes appearing in sports competitions, followed by human body pose estimation, which is divided into two methods: coordinate regression-based and heatmap test-based. To improve the accuracy of the heatmap test, the high-resolution feature map output from HRNet is used for deconvolution to improve the accuracy of single-target pose estimation recognition.

Аналіз стану дослідження проблеми визначення параметрів відносного положення об'єктів орбітального сервісу

Recently considerable attention has been paid to the problem of estimating the pose of an on-orbit service object. Determining the pose at a close distance still remains an open line of research, especially for non-cooperative objects (targets) of on-orbit service. The goal of this work is to overview the state of the art in the problem of determining the relative motion parameters of on-orbit service objects with emphasis on close proximity operations with non-cooperative and unknown targets. The method employed is the analysis of publications devoted to this problem over the last decade. The analysis showed the following. Determining the pose of a non-cooperative orbital object using video systems is a classical approach due to the advantages of light weight and low power consumption. Video camera based pose estimation algorithms usually require prior knowledge of the target features. The main methods of pose estimation still involve approaches based on the recognition and correspondence of image features for consecutive frames or with a target model. Another major approach to pose determination is lidar navigation, where the recognition and correspondence of features based on lidar-derived target surface point clouds are also common methods. Recently, a trend has emerged towards the development of non-feature methods for target pose determination, including unknown targets. The three-dimensional nature of lidar point cloud data is favorable for target pose estimation without any target model. As to the applicability of target pose estimation methods to an unknown target, the implementation of the obvious approach based on constructing a three-dimensional model of the target by processing a series of its images prior to estimating its spatial motion takes a lot of time, which is critical in close proximity operations. The trend in target pose estimation is the development of methods for simultaneous estimation of the pose and shape of an unknown object. In general, the case of an unknown object has not yet been fully investigated.

Six-Dimensional Target Pose Estimation for Robot Autonomous Manipulation: Methodology and Verification

The autonomous and precise grasping operation of robots is considered challenging in situations where there are different objects with different shapes and postures. In this study, we proposed a method of 6-D target pose estimation for robot autonomous manipulation. The proposed method is based on: 1) a fully convolutional neural network for scene semantic segmentation and 2) fast global registration to achieve target pose estimation. To verify the validity of the proposed algorithm, we built a robot grasping operation system and used the point cloud model of the target object and its pose estimation results to generate the robot grasping posture control strategy. Experimental results showed that the proposed method can achieve a six-degree-of-freedom pose estimation for arbitrarily placed target objects and complete the autonomous grasping of the target. Comparative experiments demonstrated that the proposed target pose estimation method achieved a significant improvement in average accuracy and real-time performance compared with traditional methods.

Fully Automatic Visual Servoing Control for Underwater Vehicle Manipulator Systems Based on a Heuristic Inverse Kinematics

The use of underwater vehicle manipulator systems (UVMS) equipped with cameras has gained significant attention due to their capacity to perform underwater tasks autonomously. However, controlling both the manipulator and the remotely operated vehicle (ROV) based on the vision system information is not an easy task, especially in situations where the vehicle cannot be parked/held stationary. Most of the existing approaches work based on complex matrix calculations for the inverse kinematics (IK), which can lead to high computational costs and the need to deal with singularity problems. A problem arises when the amount of time needed to calculate the UVMS configuration can result in reduced frequency of target pose estimation, beyond the point where the target has moved out of the camera field of view. Therefore, this paper proposes an autonomous visual servoing approach for UVMS, including an extension of a heuristic technique named M-FABRIK (Mobile - Forward and Backward Reaching IK) to calculate the UVMS inverse kinematics in a simple and fast way. This approach aims to control both the configuration of the manipulator and ROV position in order to allow underwater intervention in situations where the ROV cannot be parked/held stationary. This solution allows the vehicle to be positioned according to additional criteria, besides avoiding matrix inversion and being robust to singularities. Trials have been performed with a manipulator mounted on a work-class ROV for an autonomous underwater monitoring task and results demonstrate a simple and fast approach, which is able to set the configuration of the manipulator as well as the ROV for visual servoing applications in real-time, such as for monitoring, tracking and intervention tasks underwater.

Deep Learning Based Target Pose Estimation Using LiDAR Measurements in Active Debris Removal Operations

Deep Learning Based Target Pose Estimation Using LiDAR Measurements in Active Debris Removal Operations

MF-GCN: Motion Flow-Based Graph Network Learning Dynamics for Aerial IR Target Recognition

The primary technology of the infrared imaging seeker optoelectronic countermeasure system includes the automatic recognition and tracking of air targets. New complex infrared interference factors pose severe challenges to the ability of infrared thermal imaging seekers to identify and lock targets accurately. Despite significant recent advances, machine vision systems still lag substantially behind biological vision systems in terms of performance and robustness. The distinguishing feature of the brain is its ability to detect learned objects under various non-ideal situations. Inspired by the sensitivity of the dorsal visual pathway to object motion information, a graph convolutional network learning model is developed for aerial target recognition, called the adaptive graph reasoning method driven by motion flow model, which employs the predicted object optical flow information to promote the keypoint estimation performance. First, an adaptive graph fusion module is designed to establish an adaptation mechanism to generalize keypoint features from the regular network to the target skeleton graph model. Then, the optical flow estimation features are utilized to understand the graph reasoning of the target keypoint motion. Finally, we propose a aerial target pose estimation framework which effectively integrates the temporal cues across frames by combining the motion features of optical flow estimation, and effectively promotes the estimation accuracy of keypoints. Finally, our extensive experiments in the established aerial target flight infrared dataset show the efficiency of the presented model.

Relative Pose Estimation of Non-Cooperative Space Targets Using a TOF Camera

It is difficult to determine the accurate pose of non-cooperative space targets in on-orbit servicing (OOS). The visual camera is easily affected by the extreme light environment in space, and the scanning lidar will have motion distortion when the target moves at high speed. Therefore, we proposed a non-cooperative target pose-estimation system combining a registration and a mapping algorithm using a TOF camera. We first introduce the projection model of the TOF camera and proposed a new calibration method. Then, we introduce the three modules of the proposed method: the TOF data preprocessing module, the registration module and the model mapping module. We assembled the experimental platform to conduct semi-physical experiments; the results showed that the proposed method has the smallest translation error 8 mm and Euler angle error 1° compared with other classical methods. The total time consumption is about 100 ms, and the pose tracking frequency can reach 10 Hz. We can conclude that the proposed pose-estimation scheme can achieve the high-precision pose estimation of non-cooperative targets and meet the requirements necessary for aerospace applications.

An Uncertainty Weighted Non-Cooperative Target Pose Estimation Algorithm, Based on Intersecting Vectors

Aiming at the relative pose estimation of non-cooperative targets in space traffic management tasks, a two-step pose estimation method, based on spatially intersecting straight lines, is proposed, which mainly includes three aspects: (1) Use binocular vision to reconstruct the straight space line, and based on the direction vector of the straight line and the intersection of the straight line, solve the pose of the measured target in the measurement coordinate system, and obtain the initial value of the pose estimation. (2) Analyze the uncertainty of the spatial straight-line imaging, construct the uncertainty description matrix of the line, and filter the line features, accordingly. (3) Analyze the problems existing in the current linear distance measurement, construct the spatial linear back-projection error in the parametric coordinate space, and use the linear imaging uncertainty to weigh the projection error term to establish the optimization objective function of the pose estimation. Finally, the nonlinear optimization algorithm is used to iteratively solve the above optimization problem, to obtain high-precision pose estimation results. The experimental results show that the two-step pose estimation algorithm, proposed in this paper, can effectively achieve a high-precision and robust pose estimation for non-cooperative spatial targets. When the measurement distance is 10 m, the position accuracy can reach 10 mm, and the attitude measurement accuracy can reach 1°, which meets the pose estimation accuracy requirements of space traffic management.

Non-Cooperative Target Pose Estimation based on Improved Iterative Closest Point Algorithm

For localisation of unknown non-cooperative targets in space, the existence of interference points causes inaccuracy of pose estimation while utilizing point cloud registration. To address this issue, this paper proposes a new iterative closest point (ICP) algorithm combined with distributed weights to intensify the dependability and robustness of the non-cooperative target localisation. As interference points in space have not yet been extensively studied, we classify them into two broad categories, far interference points and near interference points. For the former, the statistical outlier elimination algorithm is employed. For the latter, the Gaussian distributed weights, simultaneously valuing with the variation of the Euclidean distance from each point to the centroid, are commingled to the traditional ICP algorithm. In each iteration, the weight matrix <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$W$</tex> in connection with the overall localisation is obtained, and the singular value decomposition is adopted to accomplish high-precision estimation of the target pose. Finally, the experiments are implemented by shooting the satellite model and setting the position of interference points. The outcomes suggest that the proposed algorithm can effectively suppress interference points and enhance the accuracy of non-cooperative target pose estimation. When the interference point number reaches about 700, the average error of angle is superior to 0.88°.

Robust Orthogonal-View 2-D/3-D Rigid Registration for Minimally Invasive Surgery.

Intra-operative target pose estimation is fundamental in minimally invasive surgery (MIS) to guiding surgical robots. This task can be fulfilled by the 2-D/3-D rigid registration, which aligns the anatomical structures between intra-operative 2-D fluoroscopy and the pre-operative 3-D computed tomography (CT) with annotated target information. Although this technique has been researched for decades, it is still challenging to achieve accuracy, robustness and efficiency simultaneously. In this paper, a novel orthogonal-view 2-D/3-D rigid registration framework is proposed which combines the dense reconstruction based on deep learning and the GPU-accelerated 3-D/3-D rigid registration. First, we employ the X2CT-GAN to reconstruct a target CT from two orthogonal fluoroscopy images. After that, the generated target CT and pre-operative CT are input into the 3-D/3-D rigid registration part, which potentially needs a few iterations to converge the global optima. For further efficiency improvement, we make the 3-D/3-D registration algorithm parallel and apply a GPU to accelerate this part. For evaluation, a novel tool is employed to preprocess the public head CT dataset CQ500 and a CT-DRR dataset is presented as the benchmark. The proposed method achieves 1.65 ± 1.41 mm in mean target registration error(mTRE), 20% in the gross failure rate(GFR) and 1.8 s in running time. Our method outperforms the state-of-the-art methods in most test cases. It is promising to apply the proposed method in localization and nano manipulation of micro surgical robot for highly precise MIS.

A Deep Learning Method for Target Pose Estimation of Planetary Surface

The key technical challenge of 6D target attitude estimation from RGB-D image is to make full use of 2d and 3D image point cloud data. Previous traditional target pose estimation methods either extract information from RGB images and depth, respectively, or use expensive post-processing steps that limit their performance in highly cluttered scenarios and real-time applications. Therefore, in this paper, a method is proposed to estimate the 6D attitude of a group of known objects from RGB-D images. The network used is a heterogeneous architecture, which processes two data from different dimensions separately, and uses a new dense fusion network to extract pixel-level dense feature embedding, and estimate the target position and attitude from it. In addition, an end-to-end iterative attitude optimization method is integrated to further improve attitude estimation, while real-time visualization of point clouds and rendering of bounding boxes are realized. Experimental results show that this method is effective on open source data sets.

Target pose estimation based on polarization imaging in low light and strong background noise

Target pose estimation based on polarization imaging in low light and strong background noise

Multi-body 3D–2D registration for image-guided reduction of pelvic dislocation in orthopaedic trauma surgery

Surgical reduction of pelvic dislocation is a challenging procedure with poor long-term prognosis if reduction does not accurately restore natural morphology. The procedure often requires long fluoroscopic exposure times and trial-and-error to achieve accurate reduction. We report a method to automatically compute the target pose of dislocated bones in preoperative CT and provide 3D guidance of reduction using routine 2D fluoroscopy. A pelvic statistical shape model (SSM) and a statistical pose model (SPM) were formed from an atlas of 40 pelvic CT images. Multi-body bone segmentation was achieved by mapping the SSM to a preoperative CT via an active shape model. The target reduction pose for the dislocated bone is estimated by fitting the poses of undislocated bones to the SPM. Intraoperatively, multiple bones are registered to fluoroscopy images via 3D–2D registration to obtain 3D pose estimates from 2D images. The method was examined in three studies: (1) a simulation study of 40 CT images simulating a range of dislocation patterns; (2) a pelvic phantom study with controlled dislocation of the left innominate bone; (3) a clinical case study investigating feasibility in images acquired during pelvic reduction surgery. Experiments investigated the accuracy of registration as a function of initialization error (capture range), image quality (radiation dose and image noise), and field of view (FOV) size. The simulation study achieved target pose estimation with translational error of median 2.3 mm (1.4 mm interquartile range, IQR) and rotational error of 2.1° (1.3° IQR). 3D–2D registration yielded 0.3 mm (0.2 mm IQR) in-plane and 0.3 mm (0.2 mm IQR) out-of-plane translational error, with in-plane capture range of ±50 mm and out-of-plane capture range of ±120 mm. The phantom study demonstrated 3D–2D target registration error of 2.5 mm (1.5 mm IQR), and the method was robust over a large dose range, down to 5 Gy/frame (an order of magnitude lower than the nominal fluoroscopic dose). The clinical feasibility study demonstrated accurate registration with both preoperative and intraoperative radiographs, yielding 3.1 mm (1.0 mm IQR) projection distance error with robust performance for FOV ranging from 340 × 340 mm2 to 170 × 170 mm2 (at the image plane). The method demonstrated accurate estimation of the target reduction pose in simulation, phantom, and a clinical feasibility study for a broad range of dislocation patterns, initialization error, dose levels, and FOV size. The system provides a novel means of guidance and assessment of pelvic reduction from routinely acquired preoperative CT and intraoperative fluoroscopy. The method has the potential to reduce radiation dose by minimizing trial-and-error and to improve outcomes by guiding more accurate reduction of joint dislocations.

RGB-D Image Processing Algorithm for Target Recognition and Pose Estimation of Visual Servo System.

This paper studies the control performance of visual servoing system under the planar camera and RGB-D cameras, the contribution of this paper is through rapid identification of target RGB-D images and precise measurement of depth direction to strengthen the performance indicators of visual servoing system such as real time and accuracy, etc. Firstly, color images acquired by the RGB-D camera are segmented based on optimized normalized cuts. Next, the gray scale is restored according to the histogram feature of the target image. Then, the obtained 2D graphics depth information and the enhanced gray image information are distort merged to complete the target pose estimation based on the Hausdorff distance, and the current image pose is matched with the target image pose. The end angle and the speed of the robot are calculated to complete a control cycle and the process is iterated until the servo task is completed. Finally, the performance index of this control system based on proposed algorithm is tested about accuracy, real-time under position-based visual servoing system. The results demonstrate and validate that the RGB-D image processing algorithm proposed in this paper has the performance in the above aspects of the visual servoing system.

Improvements to Target-Based 3D LiDAR to Camera Calibration

The homogeneous transformation between a LiDAR and monocular camera is required for sensor fusion tasks, such as SLAM. While determining such a transformation is not considered glamorous in any sense of the word, it is nonetheless crucial for many modern autonomous systems. Indeed, an error of a few degrees in rotation or a few percent in translation can lead to 20 cm translation errors at a distance of 5 m when overlaying a LiDAR image on a camera image. The biggest impediments to determining the transformation accurately are the relative sparsity of LiDAR point clouds and systematic errors in their distance measurements. This paper proposes (1) the use of targets of known dimension and geometry to ameliorate target pose estimation in face of the quantization and systematic errors inherent in a LiDAR image of a target, and (2) a fitting method for the LiDAR to monocular camera transformation that fundamentally assumes the camera image data is the most accurate information in one's possession.

Robot visual guide with Fourier-Mellin based visual tracking

Robot vision guide is an important research area in industrial automation, and image-based target pose estimation is one of the most challenging problems. We focus on target pose estimation and present a solution based on the binocular stereo vision in this paper. To improve the robustness and speed of pose estimation, we propose a novel visual tracking algorithm based on Fourier-Mellin transform to extract the target region. We evaluate the proposed tracking algorithm on online tracking benchmark-50 (OTB-50) and the results show that it outperforms other lightweight trackers, especially when the target is rotated or scaled. The final experiment proves that the improved pose estimation approach can achieve a position accuracy of 1.84 mm and a speed of 7 FPS (frames per second). Besides, this approach is robust to the variances of illumination and can work well in the range of 250–700 lux.