LINEMOD Dataset Research Articles

Efficient Encoding and Aligning Viewpoints for 6D Pose Estimation of Unseen Industrial Parts

Abstract Estimating the 6D poses of industrial parts is a fundamental task in automated industries. However, the scarcity of industrial part datasets and the effort involved to retrain networks present challenges when estimating poses for unseen parts. Although a few pre-trained networks demonstrate effectiveness on unseen objects, they often struggle to encode correct viewpoint for unseen industrial parts, which have significant geometrical differences compared to the pre-trained objects. Additionally, they overlook the viewpoint non-uniformity that frequently occurs in industrial settings, resulting in significant 3D rotation errors. To address these issues, a novel 6D pose estimator for unseen industrial parts is proposed. First, a Self-to-Inter (S2I) viewpoint encoder is introduced to efficiently generate discriminative descriptors that capture the viewpoint information of the observed image. The S2I viewpoint encoder utilizes an Inter-viewpoint attention module to facilitate prior viewpoint communication and leverages a saliency descriptor selection strategy to boost inference speed. Second, a Viewpoint Alignment module (VAM) is established and integrated with the ICP refiner. The VAM aligns non-uniform viewpoints in an analytical paradigm, leading to enhanced efficiency of the refinement process and more accurate final predictions. Experimental results on the LINEMOD dataset demonstrate competitive performance compared to state-of-the-art methods. Furthermore, the experiments conducted on eight unseen industrial parts validate the exceptional generalizability of our method, highlighting its potential in industrial applications.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

FusionNetV2: Explicit Enhancement of Edge Features for 6D Object Pose Estimation

FusionNet is a hybrid model that incorporates convolutional neural networks and Transformers, achieving state-of-the-art performance in 6D object pose estimation while significantly reducing the number of model parameters. Our study reveals that FusionNet has local and global attention mechanisms for enhancing deep features in two paths and the attention mechanisms play a role in implicitly enhancing features around object edges. We found that enhancing the features around object edges was the main reason for the performance improvement in 6D object pose estimation. Therefore, in this study, we attempt to enhance the features around object edges explicitly and intuitively. To this end, an edge boosting block (EBB) is introduced that replaces the attention blocks responsible for local attention in FusionNet. EBB is lightweight and can be directly applied to FusionNet with minimal modifications. EBB significantly improved the performance of FusionNet in 6D object pose estimation in experiments on the LINEMOD dataset.

Six-Degree-of-Freedom Pose Estimation Method for Multi-Source Feature Points Based on Fully Convolutional Neural Network

An object’s six-degree-of-freedom (6DoF) pose information has great importance in various fields. Existing methods of pose estimation usually detect two-dimensional (2D)-three-dimensional (3D) feature point pairs, and directly estimates the pose information through Perspective-n-Point (PnP) algorithms. However, this approach ignores the spatial association between pixels, making it difficult to obtain high-precision results. In order to apply pose estimation based on deep learning methods to real-world scenarios, we hope to design a method that is robust enough in more complex scenarios. Therefore, we introduce a method for 3D object pose estimation from color images based on farthest point sampling (FPS) and object 3D bounding box. This method detects the 2D projection of 3D feature points through a convolutional neural network, matches it with the 3D model of the object, and then uses the PnP algorithm to restore the feature point pair to the object pose. Due to the global nature of the bounding box, this approach can be considered effective even in partially occluded or complex environments. In addition, we propose a heatmap suppression method based on weighted coordinates to further improve the prediction accuracy of feature points and the accuracy of the PnP algorithm in solving the pose position. Compared with other algorithms, this method has higher accuracy and better robustness. Our method yielded 93.8% of the ADD(-s) metrics on the Linemod dataset and 47.7% of the ADD(-s) metrics on the Occlusion Linemod dataset. These results show that our method is more effective than existing methods in pose estimation of large objects.

Pose estimation algorithm based on point pair features using PointNet + +

This study proposes an innovative deep learning algorithm for pose estimation based on point clouds, aimed at addressing the challenges of pose estimation for objects affected by the environment. Previous research on using deep learning for pose estimation has primarily been conducted using RGB-D data. This paper introduces an algorithm that utilizes point cloud data for deep learning-based pose computation. The algorithm builds upon previous work by integrating PointNet + + technology and the classical Point Pair Features algorithm, achieving accurate pose estimation for objects across different scene scales. Additionally, an adaptive parameter-density clustering method suitable for point clouds is introduced, effectively segmenting clusters in varying point cloud density environments. This resolves the complex issue of parameter determination for density clustering in different point cloud environments and enhances the robustness of clustering. Furthermore, the LineMod dataset is transformed into a point cloud dataset, and experiments are conducted on the transformed dataset to achieve promising results with our algorithm. Finally, experiments under both strong and weak lighting conditions demonstrate the algorithm's robustness.

Zero‐Shot 3D Pose Estimation of Unseen Object by Two‐step RGB-D Fusion

3D Object pose estimation is a critical task in many real-world applications, e.g., robotic manipulation and augmented reality. Most existing methods focus on estimating the object instances or categories which have been seen in the training phase. However, it is imperative to estimate the pose of unseen objects without re-training the network in real world. Therefore, we proposed a 3D pose estimation method for unseen objects without re-training. Specifically, given the CAD model of the unseen object, a set of template RGB-D images (RGB images and depth images) is rendered at different viewpoints. Then a feature embedding network, named PoseFusion, is designed to extract the scene feature. In this network, RGB-D images are utilized to extract the texture feature and geometric feature, respectively. Afterwards, a cross-modality alignment module is proposed to eliminate the noise in single modality. The aligned texture feature and aligned geometric feature are fused through a geometry guided fusion module. Thus, by PoseFusion, the template RGB-D images generated from the CAD model are abstracted into a set of template scene features, and the query scene features are also embedded from the captured RGB-D images from the unseen object. Finally, the query scene features are matched with the template scene features by calculating the masked local similarity. Then the identity and pose of unseen object are determined by the most similar template. Experiments on LINEMOD and T-LESS datasets demonstrate that our method outperforms other methods and generalizes better to unseen objects. Extensive ablation studies are performed to verify the effectiveness of the PoseFusion.

A Robust CoS-PVNet Pose Estimation Network in Complex Scenarios

Object 6D pose estimation, as a key technology in applications such as augmented reality (AR), virtual reality (VR), robotics, and autonomous driving, requires the prediction of the 3D position and 3D pose of objects robustly from complex scene images. However, complex environmental factors such as occlusion, noise, weak texture, and lighting changes may affect the accuracy and robustness of object 6D pose estimation. We propose a robust CoS-PVNet (complex scenarios pixel-wise voting network) pose estimation network for complex scenes. By adding a pixel-weight layer based on the PVNet network, more accurate pixel point vectors are selected, and dilated convolution and adaptive weighting strategies are used to capture local and global contextual information of the input feature map. At the same time, the perspective-n-point localization algorithm is used to accurately locate 2D key points to solve the pose of 6D objects, and then, the transformation relationship matrix of 6D pose projection is solved. The research results indicate that on the LineMod and Occlusion LineMod datasets, CoS-PVNet has high accuracy and can achieve stable and robust 6D pose estimation even in complex scenes.

DON6D: a decoupled one-stage network for 6D pose estimation

The six-dimensional (6D) pose object estimation is a key task in robotic manipulation and grasping scenes. Many existing two-stage solutions with a slow inference speed require extra refinement to handle the challenges of variations in lighting, sensor noise, object occlusion, and truncation. To address these challenges, this work proposes a decoupled one-stage network (DON6D) model for 6D pose estimation that improves inference speed on the premise of maintaining accuracy. Particularly, since the RGB images are aligned with the RGB-D images, the proposed DON6D first uses a two-dimensional detection network to locate the interested objects in RGB-D images. Then, a module of feature extraction and fusion is used to extract color and geometric features fully. Further, dual data augmentation is performed to enhance the generalization ability of the proposed model. Finally, the features are fused, and an attention residual encoder–decoder, which can improve the pose estimation performance to obtain an accurate 6D pose, is introduced. The proposed DON6D model is evaluated on the LINEMOD and YCB-Video datasets. The results demonstrate that the proposed DON6D is superior to several state-of-the-art methods regarding the ADD(-S) and ADD(-S) AUC metrics.

Instance-level 6D pose estimation based on multi-task parameter sharing for robotic grasping

Six-dimensional pose estimation task predicts its 3D rotation matrix and 3D translation matrix in the world coordinate system by inputting the color image or depth image of the target object. Existing methods usually use deep neural networks to directly predict or regress object poses based on keypoint methods. The prediction results usually have deviations depending on whether the surface shape of the object is prominent or not and the size of the object. To solve this problem, we propose the six-dimensional pose estimation based on multi-task parameter sharing (PMP) framework to incorporate object category information into the pose estimation network through the form of an object classification auxiliary task. First, we extract the image features and point cloud features of the target object separately, and fuse them point by point; then, we share the confidence of each keypoint in pose estimation task and the knowledge of the classification task, get the key points with higher confidence, and predict the object pose; finally, the obtained object pose is passed through an iterative optimization network to obtain the final pose. The experimental results on the LineMOD dataset show that the proposed method can improve the accuracy of pose estimation and narrow the gap in the prediction accuracy of objects with different shapes. We also tested on a new dataset of small-scale objects, which contains object RGBD images and accurate 3D point cloud information. The proposed method is applied to the grasping experiment on the UR5 robotic arm, which satisfies the real-time pose estimation results during the grasping process.

CatFormer: Category-Level 6D Object Pose Estimation with Transformer

Although there has been significant progress in category-level object pose estimation in recent years, there is still considerable room for improvement. In this paper, we propose a novel transformer-based category-level 6D pose estimation method called CatFormer to enhance the accuracy pose estimation. CatFormer comprises three main parts: a coarse deformation part, a fine deformation part, and a recurrent refinement part. In the coarse and fine deformation sections, we introduce a transformer-based deformation module that performs point cloud deformation and completion in the feature space. Additionally, after each deformation, we incorporate a transformer-based graph module to adjust fused features and establish geometric and topological relationships between points based on these features. Furthermore, we present an end-to-end recurrent refinement module that enables the prior point cloud to deform multiple times according to real scene features. We evaluate CatFormer's performance by training and testing it on CAMERA25 and REAL275 datasets. Experimental results demonstrate that CatFormer surpasses state-of-the-art methods. Moreover, we extend the usage of CatFormer to instance-level object pose estimation on the LINEMOD dataset, as well as object pose estimation in real-world scenarios. The experimental results validate the effectiveness and generalization capabilities of CatFormer. Our code and the supplemental materials are avaliable at https://github.com/BIT-robot-group/CatFormer.

Deep Learning-Based 6-DoF Object Pose Estimation Considering Synthetic Dataset.

Due to the difficulty in generating a 6-Degree-of-Freedom (6-DoF) object pose estimation dataset, and the existence of domain gaps between synthetic and real data, existing pose estimation methods face challenges in improving accuracy and generalization. This paper proposes a methodology that employs higher quality datasets and deep learning-based methods to reduce the problem of domain gaps between synthetic and real data and enhance the accuracy of pose estimation. The high-quality dataset is obtained from Blenderproc and it is innovatively processed using bilateral filtering to reduce the gap. A novel attention-based mask region-based convolutional neural network (R-CNN) is proposed to reduce the computation cost and improve the model detection accuracy. Meanwhile, an improved feature pyramidal network (iFPN) is achieved by adding a layer of bottom-up paths to extract the internalization of features of the underlying layer. Consequently, a novel convolutional block attention module-convolutional denoising autoencoder (CBAM-CDAE) network is proposed by presenting channel attention and spatial attention mechanisms to improve the ability of AE to extract images' features. Finally, an accurate 6-DoF object pose is obtained through pose refinement. The proposed approach is compared to other models using the T-LESS and LineMOD datasets. Comparison results demonstrate the proposed approach outperforms the other estimation models.

A lightweight method of pose estimation for indoor object

Due to the multiple types of objects and the uncertainty of their geometric structures and scales in indoor scenes, the position and pose estimation of point clouds of indoor objects by mobile robots has the problems of domain gap, high learning cost, and high computing cost. In this paper, a lightweight 6D pose estimation method is proposed, which decomposes the pose estimation into a viewpoint and the in-plane rotation around the optical axis of the viewpoint, and the improved PointNet+⁣+ network structure and two lightweight modules are used to construct a codebook, and the 6d pose estimation of the point cloud of the indoor objects is completed by building and querying the codebook. The model was trained on the ShapeNetV2 dataset, and reports the ADD-S metric validation on the YCB-Video and LineMOD datasets, reaching 97.0% and 94.6% respectively. The experiment shows that the model can be trained to estimate the 6d position and pose of the unknown object point cloud with lower computation and storage cost, and the model with fewer parameters and better real-time performance is superior to other high-recision methods.

FusionNet: An End-to-End Hybrid Model for 6D Object Pose Estimation

In this study, we propose a hybrid model for Perspective-n-Point (PnP)-based 6D object pose estimation called FusionNet that takes advantage of convolutional neural networks (CNN) and Transformers. CNN is an effective and potential tool for feature extraction, which is considered the most popular architecture. However, CNN has difficulty in capturing long-range dependencies between features, and most CNN-based models for 6D object pose estimation are bulky and heavy. To address these problems, we propose a lighter-weight CNN building block with attention, design a Transformer-based global dependency encoder, and integrate them into a single model. Our model is able to extract dense 2D–3D point correspondences more accurately while significantly reducing the number of model parameters. Followed with a PnP header that replaces the PnP algorithm for general end-to-end pose estimation, our model showed better or highly competitive performance in pose estimation compared with other state-of-the-art models in experiments on the LINEMOD dataset.

6D Object Pose Estimation Based on Cross-Modality Feature Fusion.

The 6D pose estimation using RGBD images plays a pivotal role in robotics applications. At present, after obtaining the RGB and depth modality information, most methods directly concatenate them without considering information interactions. This leads to the low accuracy of 6D pose estimation in occlusion and illumination changes. To solve this problem, we propose a new method to fuse RGB and depth modality features. Our method effectively uses individual information contained within each RGBD image modality and fully integrates cross-modality interactive information. Specifically, we transform depth images into point clouds, applying the PointNet++ network to extract point cloud features; RGB image features are extracted by CNNs and attention mechanisms are added to obtain context information within the single modality; then, we propose a cross-modality feature fusion module (CFFM) to obtain the cross-modality information, and introduce a feature contribution weight training module (CWTM) to allocate the different contributions of the two modalities to the target task. Finally, the result of 6D object pose estimation is obtained by the final cross-modality fusion feature. By enabling information interactions within and between modalities, the integration of the two modalities is maximized. Furthermore, considering the contribution of each modality enhances the overall robustness of the model. Our experiments indicate that the accuracy rate of our method on the LineMOD dataset can reach 96.9%, on average, using the ADD (-S) metric, while on the YCB-Video dataset, it can reach 94.7% using the ADD-S AUC metric and 96.5% using the ADD-S score (<2 cm) metric.

High-accuracy pose estimation algorithm with monocular camera based on residual network

Pose estimation is a typical problem encountered in computer vision. Therefore, it is important to improve the accuracy of this method. Aiming at the accuracy of pose estimation, we propose a high-accuracy pose estimation algorithm based on the you only look once network and residual network with a monocular camera as the sensor for visual acquisition. This algorithm uses ArUco markers as a reference for object localization and uses the red, green, and blue (RGB) image as the input. The input image is sampled 16 and 32 times to extract the feature image, and the feature image extracted by 16 times sampling is passed through the pass-through layer and then combined with the feature image extracted by 32 times sampling to accomplish the dimension expansion. The feature image is identified by the convolutional layer. The EPnP algorithm is used to solve the camera poses. The pose information of the target object in the RGB image is used as the output. By comparing the pose estimation accuracy for the LINEMOD dataset with three evaluation metrics—the 2D projection metric, ADD metric, and 5 cm to 5 deg metric—it can be observed that the pose estimation algorithm proposed has advantages in terms of accuracy compared with traditional pose estimation algorithms. When the target is very similar to the background objects, the algorithm also achieves good performance.

6D pose estimation of object based on fused region-level feature in cluttered scenes

RGB-based methods have certain disadvantages because they lack 3D information and cannot cope with the pose estimation problem in cluttered scenes. Therefore, we propose a region-level pose estimation network that uses RGB-D images. We first extracted the objects’ color features and geometric features by convolutional neural networks (CNN) and PointNet, respectively, and then performed feature fusion. The fused features were fed into a region-level feature extraction network to obtain the region-level features, which extracted the local geometry features from the point cloud and learned the point set’s semantic information. We used the output of the region-level feature extraction network to perform region-level pose estimation, then selected the pose with the highest confidence level as the output and iteratively optimized the pose to obtain the final results. The experimental results showed that the proposed solution performed well on the LINEMOD data set, which verified the effectiveness of the proposed method in the pose estimation problem and the algorithm’s robustness in severely cluttered scenes.

Fusing Dense Features and Pose Consistency: A Regression Method for Attitude Measurement of Aircraft Landing

The aircraft pose is the important measurement indicator related to the safe landing. The existing pose estimation methods are limited by the extraction accuracy of salient sparse points or line features, and it is hard to obtain higher accuracy and robustness, especially at long distance. In this paper, we implement attitude measurement utilizing a pose estimation. The aircraft pose estimation method based on dense features and pose consistency is proposed, which can infer the aircraft pose relative to the camera from the RGB image directly. This method uses the encoder-decoder network to predict the dense 3D coordinates, the surface regions and visible segmentation. Guided by the dense 2D-3D correspondences and surface regions, the aircraft pose is estimated using the pose regression module. To improve the robustness of decoupled pose estimation relative to object detection, this paper proposes a dual-channel regression framework connected with the pose consistency constraint, which enables mutual supervision between different dynamic zoom in (DZI). The experimental results on the aircraft dataset show the superiority of our aircraft pose estimation method significantly. The pose consistency constraints further improve the prediction performance on the aircraft dataset, LINEMOD dataset, and YCB-V dataset. The pose estimation method can be used for real-time measurement of aircraft attitude directly with the mean error of 0.377°, the RMS error of 0.491°, and the rate of 59FPS.

C.DOT - Convolutional Deep Object Tracker for Augmented Reality Based Purely on Synthetic Data.

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

Robust 6-DoF Pose Estimation under Hybrid Constraints.

To solve the problem of the insufficient accuracy and stability of the two-stage pose estimation algorithm using heatmap in the problem of occluded object pose estimation, a new robust 6-DoF pose estimation algorithm under hybrid constraints is proposed in this paper. First, a new loss function suitable for heatmap regression is formulated to improve the quality of the predicted heatmaps and increase keypoint accuracy in complex scenes. Second, the heatmap regression network is expanded and a translation regression branch is added to constrain the pose further. Finally, a robust pose optimization module is used to fuse the heatmap and translation estimates and improve the pose estimation accuracy. The proposed algorithm achieves ADD(-S) accuracy rates of 93.5% and 46.2% on the LINEMOD dataset and the Occlusion LINEMOD dataset, which are better than other state-of-the-art algorithms. Compared with the conventional two-stage heatmap-based pose estimation algorithms, the mean estimation error is greatly reduced, and the stability of pose estimation is improved. The proposed algorithm can run at a maximum speed of 22 FPS, thus constituting both a performant and efficient method.

VP-KLNet: efficient 6D object pose estimation with an enhanced vector-field prediction network and a keypoint localization network

6D object pose estimation is essential for many applications with high demands in accuracy and speed. Compared with end-to-end approaches, pixel-wise voting network (PVNet), a vector-field based two-stage approach, has shown the superiority in accuracy but the inferiority in speed because of the time-consuming RANSAC-based voting strategy. To resolve this problem, we propose an efficient deep architecture that consists of an enhanced vector-field prediction network (VPNet) and a keypoint localization network (KLNet) and call it VP-KLNet. Specifically, the KLNet replaces PVNet’s time-consuming voting scheme by directly regressing 2D keypoints from the vector fields, which significantly improves the inference speed. Furthermore, to capture multiscale contextual information, we embed the pyramid pooling module between the encoder and decoder in VPNet to obtain more accurate object segmentation and vector-field prediction, with negligible speed loss. Experiments demonstrate that our method has more than 50% improved in the running speed to the baseline method PVNet and achieves comparable accuracy with the state-of-the-art methods on the LINEMOD and occlusion LINEMOD datasets.