Object Pose Research Articles

PointSLOT: Real-Time Simultaneous Localization and Object Tracking for Dynamic Environment

The applicability of SLAM algorithms is largely limited to the assumption of scene rigidity. Moreover, dynamic object perception is an essential technique for many robotic applications such as autonomous driving and multi-robot collaboration. In this letter, we present PointSLOT, an online, real-time and general stereo system that simultaneously performs SLAM and dynamic object tracking without any artificial priors. Unlike previous SLOT systems, we explicitly identify moving and stationary objects by computing the object motion probabilities, such that features from static objects are utilized to improve the camera pose estimation instead of treating all object regions as outliers. Based on the camera-centric parameterization for the object pose, the trajectories and dynamic landmarks of multi-keyframe objects are efficiently computed through an object-based bundle adjustment approach. The evaluations on KITTI datasets and real-world experiments show that the proposed system achieves comparable results to state-of-the-art solutions on both camera motion estimation and dynamic object tracking. The open-source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/pkzhou/PointSLOT</uri> .

Distributed object pose estimation over strongly connected networks

Distributed object pose estimation over strongly connected networks

Imitrob: Imitation Learning Dataset for Training and Evaluating 6D Object Pose Estimators

This paper introduces a dataset for training and evaluating methods for 6D pose estimation of hand-held tools in task demonstrations captured by a standard RGB camera. Despite the significant progress of 6D pose estimation methods, their performance is usually limited for heavily occluded objects, which is a common case in imitation learning, where the object is typically partially occluded by the manipulating hand. Currently, there is a lack of datasets that would enable the development of robust 6D pose estimation methods for these conditions. To overcome this problem, we collect a new dataset (Imitrob) aimed at 6D pose estimation in imitation learning and other applications where a human holds a tool and performs a task. The dataset contains image sequences of nine different tools and twelve manipulation tasks with two camera viewpoints, four human subjects, and left/right hand. Each image is accompanied by an accurate ground truth measurement of the 6D object pose obtained by the HTC Vive motion tracking device. The use of the dataset is demonstrated by training and evaluating a recent 6D object pose estimation method (DOPE) in various setups.

Self-Supervised Category-Level 6D Object Pose Estimation With Optical Flow Consistency

Category-level 6D object pose estimation aims at determining the pose of an object of a given category. Most current state-of-the-art methods require a significant amount of real training data to supervise their models. Moreover, annotating the 6D pose is very time consuming, error-prone, and it does not scale well to a large amount of object classes. Therefore, a handful of methods have recently been proposed to use unlabelled data to establish weak supervision. In this letter we propose a self-supervised method that leverages the 2D optical flow as a proxy for supervising the 6D pose. To this purpose, we estimate the 2D optical flow between consecutive frames based on the pose estimation. Then, we harness an off-the-shelf optical flow method to enable weak supervision using a 2D-3D optical flow based consistency loss. Experiments show that our approach for self-supervised learning yields state-of-the-art performance on the NOCS benchmark, and it reaches comparable results with some fully-supervised approaches.

Evaluation of single-stage vision models for pose estimation of surgical instruments.

Multiple applications in open surgical environments may benefit from adoption of markerless computer vision depending on associated speed and accuracy requirements. The current work evaluates vision models for 6-degree of freedom pose estimation of surgical instruments in RGB scenes. Potential use cases are discussed based on observed performance. Convolutional neural nets were developed with simulated training data for 6-degree of freedom pose estimation of a representative surgical instrument in RGB scenes. Trained models were evaluated with simulated and real-world scenes. Real-world scenes were produced by using a robotic manipulator to procedurally generate a wide range of object poses. CNNs trained in simulation transferred to real-world evaluation scenes with a mild decrease in pose accuracy. Model performance was sensitive to input image resolution and orientation prediction format. The model with highest accuracy demonstrated mean in-plane translation error of 13mm and mean long axis orientation error of 5[Formula: see text] in simulated evaluation scenes. Similar errors of 29mm and 8[Formula: see text] were observed in real-world scenes. 6-DoF pose estimators can predict object pose in RGB scenes with real-time inference speed. Observed pose accuracy suggests that applications such as coarse-grained guidance, surgical skill evaluation, or instrument tracking for tray optimization may benefit from markerless pose estimation.

DOPESLAM: High-Precision ROS-Based Semantic 3D SLAM in a Dynamic Environment.

Recent advancements in deep learning techniques have accelerated the growth of robotic vision systems. One way this technology can be applied is to use a mobile robot to automatically generate a 3D map and identify objects within it. This paper addresses the important challenge of labeling objects and generating 3D maps in a dynamic environment. It explores a solution to this problem by combining Deep Object Pose Estimation (DOPE) with Real-Time Appearance-Based Mapping (RTAB-Map) through means of loose-coupled parallel fusion. DOPE's abilities are enhanced by leveraging its belief map system to filter uncertain key points, which increases precision to ensure that only the best object labels end up on the map. Additionally, DOPE's pipeline is modified to enable shape-based object recognition using depth maps, allowing it to identify objects in complete darkness. Three experiments are performed to find the ideal training dataset, quantify the increased precision, and evaluate the overall performance of the system. The results show that the proposed solution outperforms existing methods in most intended scenarios, such as in unilluminated scenes. The proposed key point filtering technique has demonstrated an improvement in the average inference speed, achieving a speedup of 2.6× and improving the average distance to the ground truth compared to the original DOPE algorithm.

Open-source software for automated rodent behavioral analysis.

Rodent behavioral analysis is a major specialization in experimental psychology and behavioral neuroscience. Rodents display a wide range of species-specific behaviors, not only in their natural habitats but also under behavioral testing in controlled laboratory conditions. Detecting and categorizing these different kinds of behavior in a consistent way is a challenging task. Observing and analyzing rodent behaviors manually limits the reproducibility and replicability of the analyses due to potentially low inter-rater reliability. The advancement and accessibility of object tracking and pose estimation technologies led to several open-source artificial intelligence (AI) tools that utilize various algorithms for rodent behavioral analysis. These software provide high consistency compared to manual methods, and offer more flexibility than commercial systems by allowing custom-purpose modifications for specific research needs. Open-source software reviewed in this paper offer automated or semi-automated methods for detecting and categorizing rodent behaviors by using hand-coded heuristics, machine learning, or neural networks. The underlying algorithms show key differences in their internal dynamics, interfaces, user-friendliness, and the variety of their outputs. This work reviews the algorithms, capability, functionality, features and software properties of open-source behavioral analysis tools, and discusses how this emergent technology facilitates behavioral quantification in rodent research.

Object pose estimation based on stereo vision with improvedK‐DtreeICPalgorithm

SummaryWith the wide application of stereovision in SLAM, object pose estimation has gradually become one of the research hotspots. This article proposes an object pose estimation for robotic grasping based on stereo vision with improved K‐D tree ICP algorithm. The feature points and feature descriptors of the point cloud of the object to be captured are extracted, and the feature template set is established. The SAC‐IA algorithm is used to carry out initial registration of the point cloud of the target, and the ICP algorithm based on K‐D tree is used for fine registration. The experimental results show that the average coincidence degree of the final registration of the proposed object pose estimation method reaches 94.1%, and the accurate 6D pose of the object to be grasped is obtained.

Аналіз стану дослідження проблеми визначення параметрів відносного положення об'єктів орбітального сервісу

Recently considerable attention has been paid to the problem of estimating the pose of an on-orbit service object. Determining the pose at a close distance still remains an open line of research, especially for non-cooperative objects (targets) of on-orbit service. The goal of this work is to overview the state of the art in the problem of determining the relative motion parameters of on-orbit service objects with emphasis on close proximity operations with non-cooperative and unknown targets. The method employed is the analysis of publications devoted to this problem over the last decade. The analysis showed the following. Determining the pose of a non-cooperative orbital object using video systems is a classical approach due to the advantages of light weight and low power consumption. Video camera based pose estimation algorithms usually require prior knowledge of the target features. The main methods of pose estimation still involve approaches based on the recognition and correspondence of image features for consecutive frames or with a target model. Another major approach to pose determination is lidar navigation, where the recognition and correspondence of features based on lidar-derived target surface point clouds are also common methods. Recently, a trend has emerged towards the development of non-feature methods for target pose determination, including unknown targets. The three-dimensional nature of lidar point cloud data is favorable for target pose estimation without any target model. As to the applicability of target pose estimation methods to an unknown target, the implementation of the obvious approach based on constructing a three-dimensional model of the target by processing a series of its images prior to estimating its spatial motion takes a lot of time, which is critical in close proximity operations. The trend in target pose estimation is the development of methods for simultaneous estimation of the pose and shape of an unknown object. In general, the case of an unknown object has not yet been fully investigated.

A model-free 6-DOF grasp detection method based on point clouds of local sphere area

ABSTRACT Grasping object is one of the basic tasks of robots in many scenarios. The main challenge is how to generate grasping poses for unknown objects in cluttered scenes. This paper proposes a model-free 6-DOF grasp detection framework based on single-view local point clouds. The whole process includes three stages: Candidate Generation Network(CGN), Reliable Adjustment Module(RAM), and Quality Assessment Network(QAN). CGN predicts the graspability and the initial grasp pose of the sampled points by the features extracted from the local sphere region based on the improved Pointnet. In order to better learn local area point clouds, we propose a progressive local region data learning mechanism, which can extract features from small to large scales efficiently. Candidate grasps are then consisting of graspable points and their grasping poses. RAM adjusts the position and width of the generated candidate grasps by using reliable heuristic rules. QAN uses a simpled-pointnet to evaluate the quality of the grasp candidates and filters out grasps with high confidence to execute. The proposed method not only achieves state-of-the-art results on GraspNet-1Billion but also shows high grasping success rates in real cluttered scenes.

Animal Movement Prediction based on the Corroboration of Object Detection and Pose Estimation

With the development of automated driving, driving safety has become a hot topic in the field of automated driving. However, the existing methods mainly define living objects as obstacle handling and mostly focus on the safety of pedestrians, thus ignoring the safety of animal movement in automatic driving. In this paper, we propose an animal movement prediction model with the corroboration of object detection and pose estimation, which aims, to help the autopilot perception system judge animal pose more accurately. First of all, we preprocess the datasets, use a Convolutional Neural Network based object detection method to detect animal images, and then use the current mainstream 2D animal pose estimation method to obtain the key points of animal images, and divide the resulting data set into three categories: running, walking and waiting. After preprocessing, we use Resnet and MLP models to classify the pose. For Resnet model, we will evaluate the accuracy of the model on the test set during training and compare the accuracy of the last model to determine the model with the highest training accuracy. The final model has achieved an accuracy of up to 75% for running evaluation. For MLP model, we tend to use the rank of 13 specific bone points’ angles as the standard of pose classification. Three types of test sets will be put into three trained MLPRegressor model for training to get the classification we want. Finally, the accuracy of all three poses is more than 80%. Finally, we choose the model with the highest accuracy as our classification method. This work is expected to apply animal pose estimation and animal and pose classification to automated driving to improve driving safety.

Certified grasping

This paper studies the robustness of grasping in the frictionless plane from a geometric perspective. By treating grasping as a process that shapes the free-space object over time, we define three types of certificates to guarantee success of a grasp: (a) invariance under an initial set, (b) convergence toward a goal grasp, and (c) observability over the final object pose. We develop convex-combinatorial models for each of these certificates, which can be expressed as simple semi-algebraic relations under mild-modeling assumptions, such as point-fingers and frictionless contact. By leveraging these models to synthesize certificates, we optimize certifiable grasps of planar objects composed as a union of convex polygons, using manipulators described as point-fingers. We validate this approach in simulations by grasping random polygons, and with real sensorless grasps of several objects.

A Virtual Multi-Ocular 3D Reconstruction System Using a Galvanometer Scanner and a Camera.

A novel visual 3D reconstruction system, composed of a two-axis galvanometer scanner, a camera with a lens, and a set of control units, is introduced in this paper. By changing the mirror angles of the galvanometer scanner fixed in front of the camera, the boresight of the camera can be quickly adjusted. With the variable boresight, the camera can serve as a virtual multi-ocular system (VMOS), which captures the object at different perspectives. The working mechanism with a definite physical meaning is presented. A simple and efficient method for calibrating the intrinsic and extrinsic parameters of the VMOS is presented. The applicability of the proposed system for 3D reconstruction is investigated. Owing to the multiple virtual poses of the camera, the VMOS can provide stronger constraints in the object pose estimation than an ordinary perspective camera does. The experimental results demonstrate that the proposed VMOS is able to achieve 3D reconstruction performance competitive with that of a conventional stereovision system with a much more concise hardware configuration.

Improved First-Order Motion Model of Image Animation with Enhanced Dense Motion and Repair Ability

Image animation aims to transfer the posture change of a driving video to the static object of the source image, and has potential applications in various domains, such as film and game industries. The essential part in this task is to generate a video by learning the motion from the driving video while preserving the appearance from the source image. As a result, a new object with the same motion will be generated in the animated video. However, it is a significant challenge if the object pose shows large-scale change. Even the most recent method failed to achieve this correctly with good visual effects. In order to solve the problem of poor visual effects in the videos with the large-scale pose change, a novel method based on an improved first-order motion model (FOMM) with enhanced dense motion and repair ability was proposed in this paper. Firstly, when generating optical flow, we propose an attention mechanism that optimizes the feature representation of the image in both channel and spatial domains through maximum pooling. This enables better distortion of the source image into the feature domain of the driving image. Secondly, we further propose a multi-scale occlusion restoration module that generates a multi-resolution occlusion map by upsampling the low-resolution occlusion map. Following this, the generator redraws the occluded part of the reconstruction result across multiple scales through the multi-resolution occlusion map to achieve more accurate and vivid visual effects. In addition, the proposed model can be trained effectively in an unsupervised manner. We evaluated the proposed model on three benchmark datasets. The experimental results showed that multiple evaluation indicators were improved by our proposed method, and the visual effect of the animated videos obviously outperformed the FOMM. On the Voxceleb1 dataset, the pixel error, average keypoints distance and average Euclidean distance by our proposed method were reduced by 6.5%, 5.1% and 0.7%, respectively. On the TaiChiHD dataset, the pixel error, average keypoints distance and missing keypoints rate measured by our proposed method were reduced by 4.9%, 13.5% and 25.8%, respectively.

Capsule Network Extension Based on Metric Learning

A capsule network (CapsNet) is a deep learning model for image classification that provides robustness to changes in the poses of objects in the images. A capsule is a vector whose direction represents the presence, position, size, and pose of an object. However, with CapsNet, the distribution of capsules is concentrated in a class, and the number of capsules increases with the number of classes. In addition, learning is computationally expensive for a CapsNet. We proposed a method to increase the diversity of capsule directions and decrease the computational cost of CapsNet training by allowing a single capsule to represent multiple object classes. To determine the distance between classes, we used an additive angular margin loss called ArcFace. To validate the proposed method, the distribution of the capsules was determined using principal component analysis to validate the proposed method. In addition, using the MNIST, fashion-MNIST, EMNIST, SVHN, and CIFAR-10 datasets, as well as the corresponding affine-transformed datasets, we determined the accuracy and training time of the proposed method and original CapsNet. The accuracy of the proposed method improved by 8.91% on the CIFAR-10 dataset, and the training time reduced by more than 19% for each dataset compared with those of the original CapsNets.

6D Object Localization in Car-Assembly Industrial Environment

In this work, a visual object detection and localization workflow integrated into a robotic platform is presented for the 6D pose estimation of objects with challenging characteristics in terms of weak texture, surface properties and symmetries. The workflow is used as part of a module for object pose estimation deployed to a mobile robotic platform that exploits the Robot Operating System (ROS) as middleware. The objects of interest aim to support robot grasping in the context of human-robot collaboration during car door assembly in industrial manufacturing environments. In addition to the special object properties, these environments are inherently characterised by cluttered background and unfavorable illumination conditions. For the purpose of this specific application, two different datasets were collected and annotated for training a learning-based method that extracts the object pose from a single frame. The first dataset was acquired in controlled laboratory conditions and the second in the actual indoor industrial environment. Different models were trained based on the individual datasets and a combination of them were further evaluated in a number of test sequences from the actual industrial environment. The qualitative and quantitative results demonstrate the potential of the presented method in relevant industrial applications.

Towards 3D Scene Understanding Using Differentiable Rendering

Deep learning methods have achieved significant results in many 2D computer vision tasks. To realize similar results in 3D tasks, equipping deep learning pipelines with components that incorporate knowledge about 2D image generation from the 3D scene description is a promising research direction. Rasterization, the standard formulation of the image generation process is not differentiable, and thus not compatible with the deep learning models trained using gradient-based optimization schemes. In recent years, many new approximate differentiable renderers have been proposed to enable compatibility between deep learning methods and image rendering techniques. Differentiable renderers fit naturally into the render-and-compare framework where the 3D scene parameters are estimated iteratively by minimizing the error between the observed image and the image rendered according to the current scene parameter estimate. In this article, we present StilllebenDR, a light-weight, scalable differentiable renderer built as an extension to the openly available Stillleben library. We demonstrate the usability of the proposed differentiable renderer for the task of iterative 3D deformable registration using a latent shape-space model and occluded object pose refinement using order-independent transparency based on analytical gradients and learned scene aggregation.

Weak6D: Weakly Supervised 6D Pose Estimation With Iterative Annotation Resolver

6D object pose estimation is an essential task in vision-based robotic grasping and manipulation. Prior works always train models with a large number of pose annotated images, limiting the efficiency of model transfer between different scenarios. This letter presents an end-to-end model named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Weak6D</i> , which could be learned with unannotated RGB-D data. The core of the proposed approach is the novel optimizing method Iterative Annotation Resolver, which has the ability to directly utilize the captured RGB-D data through the training process. Furthermore, we employ a weak refinement loss to optimize the pose estimation network with refined object poses. We evaluated the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Weak6D</i> in the YCB-Video dataset, and experimental results show our model achieved practical results without annotated data.

I2c-net: Using Instance-Level Neural Networks for Monocular Category-Level 6D Pose Estimation

Object detection and pose estimation are strict requirements for many robotic grasping and manipulation applications to endow robots with the ability to grasp objects with different properties in cluttered scenes and with various lighting conditions. This work proposes the framework <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i2c-net</i> to extract the 6D pose of multiple objects belonging to different categories, starting from an instance-level pose estimation network and relying only on RGB images. The network is trained on a custom-made synthetic photo-realistic dataset, generated from some base CAD models, opportunely deformed, and enriched with real textures for domain randomization purposes. At inference time, the instance-level network is employed in combination with a 3D mesh reconstruction module, achieving category-level capabilities. Depth information is used for post-processing as a correction. Tests conducted on real objects of the YCB-V and NOCS-REAL datasets outline the high accuracy of the proposed approach.

A Comprehensive Study of 3-D Vision-Based Robot Manipulation.

Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.