Textureless Objects Research Articles

MLFNet: Monocular lifting fusion network for 6DoF texture-less object pose estimation

This paper addresses the challenge of 6DoF texture-less object pose estimation from a single RGB image. Many recent works have shown that two-stage deep learning approaches based on the fusion of 2D geometric intermediate representations achieve remarkable results. These methods implicitly explore the mapping from the 2D appearance domain to the 3D structure domain. However, due to the lack of 3D geometric constraints from depth maps, it is difficult to extract enough clues based on appearance features to master the geometric relation of projection from 3D viewpoints to 2D planes, and this estimation process is extremely sensitive to occlusion. We propose a novel network called MLFNet that lifts the feature space from 2D to 3D based on hybrid 3D geometric intermediate representations. For the first time, we propose the surface normals in the object coordinate system as an intermediate representation of pose; its violent change provides strong clues for the keypoints usually located at the abrupt change of object surface. Dense 3D surfaces can enhance the geometric consistency of multi-representation constraints and retain more information in occluded scenes. With the proposed multi-modality dual attention mechanism and the embedding of standard 3D shape knowledge, the 2D geometric representation learning process explicitly depends on the fusion of 2D appearance features and 3D geometric features. This standardized information fusion pattern among 2D intermediate representations, 3D intermediate representations, and CAD models prior significantly reduces the network learning space. The proposed method achieves competitive performance on the Linemod dataset and outperforms the state-of-the-art methods on the Occlusion Linemod and T-Less datasets, which demonstrates the feasibility of the pose multi-representation fusion technique. The project site is at https://github.com/JJJano/MLFNet.

Pseudo-Siamese Graph Matching Network for Textureless Objects’ 6-D Pose Estimation

Pose estimation is an essential technology for product grasping and assembly in intelligent manufacturing. Finding local correspondences between the 2-D image and the 3-D model is the key step to estimate the 6-D pose of an object. However, when the objects are textureless, it is difficult to identify distinguishable point features. In this article, we propose a novel deep learning framework called the pseudo-Siamese graph matching network to tackle the problem of feature matching of textureless objects and estimate accurate object poses with a single RGB-only image. We utilize a pseudo-Siamese network structure to learn the similarity between the 2-D image features and the 3-D mesh model of the object. A fully convolutional network and a graph convolutional network are used to extract high-dimensional deep features of the 2-D image and the 3-D model, respectively. Dense 2-D–3-D correspondences are inferred using the pseudo-Siamese matching network. Then, the pose of the object is calculated by the Perspective-n-Point and random sample consensus (RANSAC) methods. Experiments on the LINEMOD dataset and a grasping task for metal part show the accuracy and robustness of our proposed method. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World

Region-based methods have become increasingly popular for model-based, monocular 3D tracking of texture-less objects in cluttered scenes. However, while they achieve state-of-the-art results, most methods are computationally expensive, requiring significant resources to run in real-time. In the following, we build on our previous work and develop SRT3D, a sparse region-based approach to 3D object tracking that bridges this gap in efficiency. Our method considers image information sparsely along so-called correspondence lines that model the probability of the object’s contour location. We thereby improve on the current state of the art and introduce smoothed step functions that consider a defined global and local uncertainty. For the resulting probabilistic formulation, a thorough analysis is provided. Finally, we use a pre-rendered sparse viewpoint model to create a joint posterior probability for the object pose. The function is maximized using second-order Newton optimization with Tikhonov regularization. During the pose estimation, we differentiate between global and local optimization, using a novel approximation for the first-order derivative employed in the Newton method. In multiple experiments, we demonstrate that the resulting algorithm improves the current state of the art both in terms of runtime and quality, performing particularly well for noisy and cluttered images encountered in the real world.

EANet: Edge-Attention 6D Pose Estimation Network for Texture-Less Objects

The 6-D pose estimation is a crucial task in vision-based measurement for robotic manipulation. It becomes a challenging task because of the variety of lighting conditions, cluttered background, occlusion, and texture-less objects. The various lighting conditions and texture-less objects lead to dramatic changes for imaging. In this article, we propose an edge-attention 6-D pose estimation network (EANet) for texture-less objects that achieve the autonomous perception of the edge, which is invariant when the lighting conditions change and objects are texture-less. To achieve this, EANet adopts a multitask learning strategy that introduces edge cues into the pixelwise dense fusion framework. We design a shared-weight edge extractor serving for edge reconstruction and pose estimation simultaneously. The purpose of edge reconstruction is to guide the network to pay more attention to edges, so as to enhance the performance of pose estimation implicitly. The edge cues can further improve the performance of pose refinement. Furthermore, we apply a skip-connection scheme to the edge extractor, merging feature maps of the deep and shallow layers. Due to the lightweight design, EANet obtains an inference time of 20 frames per second (FPS) on a desktop PC with an RTX 2070 GPU, which satisfies the requirement of most real-time applications. Experiments show that our network has a significant effect on texture-less objects and approaches the state-of-the-art performance on both the LINEMOD and the YCB-Video datasets. Finally, we present a PARTS dataset to test real-world performance and apply EANet to bin-picking vision measurement, which demonstrates that our method can provide a complete and robust solution for vision-based pose measurement.

Multi-sensor aided deep pose tracking

Accurate pose estimation of nearby objects is critical for robots to dynamically interact with their surroundings. The complexity of this task has led researchers to explore deep learning methods. Nowadays, many works have solely focused on developing complicated neural network architectures to estimate pose from a simple monocular camera. However, most of these methods struggle with inherent limitations of a single sensor system, like occlusion, which are commonly encountered in mobile robotics applications. Online, occlusion-robust pose estimation is extremely important in such cases, as mobility of a robot introduces major uncertainty that complicates manipulation. Hence, we present Multi-Sensor aided Deep Pose Tracking (MS-DPT), a framework for online object pose estimation to enable robust mobile manipulation. A Convolutional Neural Network (CNN) identifies key objects in an RGB-D image, from which object pose estimates are generated using a variant of the Iterative Closest Point (ICP) algorithm. An extended Kalman filter is used to fuse this pose estimate with onboard motion sensors to compensate for occlusion and robot motion during manipulation. This three stage method focuses on cohering different modalities and algorithms to improve the pose tracking stability and continuity in cases where the target object becomes heavily occluded by an obstacle or a mobile robot itself. The proposed approach accurately tracks textureless objects with high symmetries while operating at 10 FPS during experiments.

GeoPose: Dense Reconstruction Guided 6D Object Pose Estimation With Geometric Consistency

6D object pose estimation for texture-less objects from RGB images remains challenging, especially in occlusion scenarios. Instead of localizing sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion, we introduce GeoPose, a novel reconstruction guided pose estimation pipeline that predicts dense correspondences and leverages geometric consistency effectively. We first design a dense reconstruction network (ReconNet) to reconstruct pixel-wise object coordinates in normalization space. Dense 2D-3D correspondences are generated intuitively by our explicit parameterization for 3D object models, which dismisses keypoint selection efforts. These 2D-3D correspondences are then utilized to estimate 6D poses by the PnP algorithm with RANSAC iterations. Furthermore, a novel Cycle Loss is proposed to provide 3D prior supervision, which significantly correlates with the pose estimation task by guiding geometric consistency between reconstruction (pixel to 3D) and reprojection (3D to pixel). In addition, a training data augmentation method is proposed to handle the insufficiency of 6D datasets, the acquisition of which is error-prone and time-consuming. Extensive experiments demonstrate that, compared with existing RGB-based methods, our GeoPose can achieve state-of-the-art (SOTA) 6D pose estimation performance on the LINEMOD, Occlusion LINEMOD and T-LESS datasets.

Texture-less object detection method for industrial components picking system

Texture-less object detection method for industrial components picking system

A Novel Metric-Learning-Based Method for Multi-Instance Textureless Objects’ 6D Pose Estimation

6D pose estimation of objects is essential for intelligent manufacturing. Current methods mainly place emphasis on the single object’s pose estimation, which limit its use in real-world applications. In this paper, we propose a multi-instance framework of 6D pose estimation for textureless objects in an industrial environment. We use a two-stage pipeline for this purpose. In the detection stage, EfficientDet is used to detect target instances from the image. In the pose estimation stage, the cropped images are first interpolated into a fixed size, then fed into a pseudo-siamese graph matching network to calculate dense point correspondences. A modified circle loss is defined to measure the differences of positive and negative correspondences. Experiments on the antenna support demonstrate the effectiveness and advantages of our proposed method.

Object Pose Estimation via Pruned Hough Forest With Combined Split Schemes for Robotic Grasp

Robotic grasp in complex open-world scenarios requires an effective and generalizable perception. Estimating object’s pose is needed in a variety of practical grasping scenarios. Here we present a novel approach of pose estimation of textureless and textured objects. The algorithm utilizes a single RGB-D image to exploit depth invariant, oriented point pair feature as well as local contextual sensitivity in cluttered environments. To enhance the performance of the voting process and improve learning efficiency, we employ a global pruning algorithm that reduces the risk of overfitting and simplifies the structure of decision trees after compensating for the complementary information among multiple trees by optimizing a designed global objective function. Finally, we also refine the pose obtained from the above stage. The proposed approach of estimating 6-D (degree of freedom) poses of textured and textureless objects is evaluated on publicly available data sets against the recent works under various conditions. It illustrates that our framework is superior to these recent works. Further, we perform extensive qualitative experiments of robotic grasp to illustrate the proposed approach can be applied to practical scenarios. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article is motivated by the problem of the pose estimation of textured and textureless objects in clutter environments. It is difficult for conventional works to address the issue of estimating textured or textureless objects’ poses in such scenarios. We considered that a novel system should be able to obtain the 6-D poses of objects. Therefore, we investigate the combined use of multiple split functions with different characteristics. Learning the model based on Hough forests always cost much computational resource; therefore, we construct a novel pruned Hough forest for solving this issue. Through the comparison and robotic grasp verifications, the behavior of our system can be used in practical applications. In future, we will deploy the proposed system in robotic assembling tasks.

An Improved Algorithm for Detection and Pose Estimation of Texture-Less Objects

This paper introduces an improved algorithm for texture-less object detection and pose estimation in industrial scenes. In the template training stage, a multi-scale template training method is proposed to improve the sensitivity of LineMOD to template depth. When this method performs template matching, the test image is first divided into several regions, and then training templates with similar depth are selected according to the depth of each test image region. In this way, without traversing all the templates, the depth of the template used by the algorithm during template matching is kept close to the depth of the target object, which improves the speed of the algorithm while ensuring that the accuracy of recognition will not decrease. In addition, this paper also proposes a method called coarse positioning of objects. The method avoids a lot of useless matching operations, and further improves the speed of the algorithm. The experimental results show that the improved LineMOD algorithm in this paper can effectively solve the algorithm’s template depth sensitivity problem.

A training free technique for 3D object recognition using the concept of vibration, energy and frequency

This paper presents a local surface feature based 3D object recognition technique that is free from any training and handles texture-less objects. Our technique is proposed based on building a strong relationship among the different regions of an object using the combination of Vibration, Energy and Frequency of points in a point cloud. The robustness of the proposed technique has been validated by comparing with top-rated training free recognition techniques on the Bologna dataset. Results show that the proposed technique has performed well and efficiently as top-rated techniques on this dataset. In real time scenario, captured scenes by an RGBD camera are cluttered with many unwanted objects and background. Most of the state-of-the-art techniques (techniques that are training free and recognize texture-less objects) have not experimented on such scenes in the literature. To observe the performance, we propose to present a 3D dataset of 10 texture-less objects (including industrial and household objects). Our experimental results demonstrate that the proposed technique has outperformed other state-of-the-art techniques on the proposed dataset. We also experiment on three very cluttered and occluded RGBD datasets (Challenge, Clutter and Willow). The poor performance of all techniques on these datasets has revealed the need for more robust techniques in the future.

Evaluating Feature Extraction Methods with Synthetic Noise Patterns for Image-Based Modelling of Texture-Less Objects

Image-based three-dimensional (3D) reconstruction is a process of extracting 3D information from an object or entire scene while using low-cost vision sensors. A structure-from-motion coupled with multi-view stereo (SFM-MVS) pipeline is a widely used technique that allows 3D reconstruction from a collection of unordered images. The SFM-MVS pipeline typically comprises different processing steps, including feature extraction and feature matching, which provide the basis for automatic 3D reconstruction. However, surfaces with poor visual texture (repetitive, monotone, etc.) challenge the feature extraction and matching stage and affect the quality of reconstruction. The projection of image patterns while using a video projector during the image acquisition process is a well-known technique that has been shown to be successful for such surfaces. In this study, we evaluate the performance of different feature extraction methods on texture-less surfaces with the application of synthetically generated noise patterns (images). Seven state-of-the-art feature extraction methods (HARRIS, Shi-Tomasi, MSER, SIFT, SURF, KAZE, and BRISK) are evaluated on problematic surfaces in two experimental phases. In the first phase, the 3D reconstruction of real and virtual planar surfaces evaluates image patterns while using all feature extraction methods, where the patterns with uniform histograms have the most suitable morphological features. The best performing pattern from Phase One is used in Phase Two experiments in order to recreate a polygonal model of a 3D printed object using all of the feature extraction methods. The KAZE algorithm achieved the lowest standard deviation and mean distance values of 0.0635 mm and −0.00921 mm, respectively.

Accurate 6DOF Pose Tracking for Texture-Less Objects

A reliable and accurate visual object 6DoF pose tracking system for texture-less objects plays an important role in various fields of modern industry hence it has been a hot research topic for decades. Traditional feature-based pose tracking methods require rich feature points on objects, and it cannot handle texture-less objects. To tackle this problem, the paper proposes a novel edge-based method for continuous 6DOF pose tracking of texture-less objects. The pose of the object is estimated by minimizing the matching error between extracted image edges and re-projected CAD model edges. The matching error is represented using the Directional Chamfer Matching (DCM) Tensor. Compared with previous methods, the proposed system improves the overall performance of pose tracking system in two ways. Firstly, the method proposes an analytical mathematic model in the optimization process; Secondly, the method proposes an adaptive edge point weighting algorithm to tackle the occlusion or edge weakness problem. Both methods help improve the accuracy and robustness of the pose estimation system. With the benefit of GPU acceleration on DCM Tensor calculation the proposed method could run in real time. Extensive experiments are conducted both on public datasets and CG-rendered synthetic datasets to validate the accuracy and robustness of the proposed algorithm against other state-of-the-art methods.

Edge Enhanced Implicit Orientation Learning With Geometric Prior for 6D Pose Estimation

Estimating 6D poses of rigid objects from RGB images is an important but challenging task. This is especially true for textureless objects with strong symmetry, since they have only sparse visual features to be leveraged for the task and their symmetry leads to pose ambiguity. The implicit encoding of orientations learned by autoencoders [31], [32] has demonstrated its effectiveness in handling such objects without requiring explicit pose labeling. In this letter, we further improve this methodology with two key technical contributions. First, we use edge cues to complement the color images with more discriminative features and reduce the domain gap between the real images for testing and the synthetic ones for training. Second, we enhance the regularity of the implicitly learned pose representations by a self-supervision scheme to enforce the geometric prior that the latent representations of two images presenting nearby rotations should be close too. Our approach achieves the state-of-the-art performance on the T-LESS benchmark in the RGB domain; its evaluation on the LINEMOD dataset also outperforms other synthetically trained approaches. Extensive ablation tests demonstrate the improvements enabled by our technical designs. Our code is publicly available for research use. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">**</sup> The code is available at https://github.com/fylwen/EEGP-AAE.

Depth Image-Based Deep Learning of Grasp Planning for Textureless Planar-Faced Objects in Vision-Guided Robotic Bin-Picking.

Bin-picking of small parcels and other textureless planar-faced objects is a common task at warehouses. A general color image–based vision-guided robot picking system requires feature extraction and goal image preparation of various objects. However, feature extraction for goal image matching is difficult for textureless objects. Further, prior preparation of huge numbers of goal images is impractical at a warehouse. In this paper, we propose a novel depth image–based vision-guided robot bin-picking system for textureless planar-faced objects. Our method uses a deep convolutional neural network (DCNN) model that is trained on 15,000 annotated depth images synthetically generated in a physics simulator to directly predict grasp points without object segmentation. Unlike previous studies that predicted grasp points for a robot suction hand with only one vacuum cup, our DCNN also predicts optimal grasp patterns for a hand with two vacuum cups (left cup on, right cup on, or both cups on). Further, we propose a surface feature descriptor to extract surface features (center position and normal) and refine the predicted grasp point position, removing the need for texture features for vision-guided robot control and sim-to-real modification for DCNN model training. Experimental results demonstrate the efficiency of our system, namely that a robot with 7 degrees of freedom can pick randomly posed textureless boxes in a cluttered environment with a 97.5% success rate at speeds exceeding 1000 pieces per hour.

DETECTION OF TEXTURE-LESS OBJECTS BY LINE-BASED APPROACH

This paper proposes a method for tackling the problem of scalable object instance detection in the presence of clutter and occlusions. It gathers together advantages in respect of the state-of-the-art object detection approaches, being at the same time able to scale favorably with the number of models, computationally efficient and suited to texture-less objects as well. The proposed method has the following advantages: a) generality – it works for both texture-less and textured objects, b) scalability – it scales sub-linearly with the number of objects stored in the object database, and c) computational efficiency – it runs in near real-time. In contrast to the traditional affine-invariant detectors/descriptors which are local and not discriminative for texture-less objects, our method is based on line segments around which it computes semi-global descriptor by encoding gradient information in scale and rotation invariant manner. It relies on both texture and shape information and is, therefore, suited for both textured and texture-less objects. The descriptor is integrated into efficient object detection procedure which exploits the fact that the line segment determines scale, orientation and position of an object, by its two endpoints. This is used to construct several effective techniques for object hypotheses generation, scoring and multiple object reasoning; which are integrated in the proposed object detection procedure. Thanks to its ability to detect objects even if only one correct line match is found, our method allows detection of the objects under heavy clutter and occlusions. Extensive evaluation on several public benchmark datasets for texture-less and textured object detection, demonstrates its scalability and high effectiveness.

RGB-D Tracking and Optimal Perception of Deformable Objects

Addressing the perception problem of texture-less objects that undergo large deformations and movements, this article presents a novel RGB-D learning-free deformable object tracker in combination with a camera position optimisation system for optimal deformable object perception. The approach is based on the discretisation of the object’s visible area through the generation of a supervoxel graph that allows weighting new supervoxel candidates between object states over time. Once a deformation state of the object is determined, supervoxels of its associated graph serve as input for the camera position optimisation problem. Satisfactory results have been obtained in real time with a variety of objects that present different deformation characteristics.

A Hybrid Framework Based on Warped Hierarchical Tree for Pose Estimation of Texture-Less Objects

The pose of texture-less objects is very important for intelligent manufacturing and intelligent assembly. Existing methods cannot accurately estimate pose when partial features are missing or cluttered due to shading, reflection, and occlusion. We propose a hybrid framework based on the warped hierarchical tree for pose estimation which integrates template matching and sparse representation classification in this paper. Firstly, the template is formed by the prospectively cumulated orientation feature (PCOF), which is a probabilistic representation of orientations extracted from template images. And the warped hierarchical tree can be built offline according to the parameters for projecting the 3D object and the similarity between templates. Then the online searching can be repeated through the warped hierarchical tree until the template candidates have been found. Finally, the pose corresponding to the best-fitting template can be obtained by sparse representation classification based on the dictionaries consisted of the spreading orientations of template candidate images. The experiment results show the effectiveness of our method when partial features are missing or cluttered.

A Novel Texture-Less Object Oriented Visual SLAM System

Traditional mapping modules in Visual Simultaneous Localization and Mapping (i.e. Visual SLAM) systems can only estimate 3D information of isolated sparse or semi-dense feature points. But there are lots of object instances in the environments which geometric information can be utilized to enhance the quality of mapping and localization. Hence, it is required for the Visual SLAM system to utilize high-dimensional features like object instances or structural lines in mapping and localization. To meet the gap between the above requirements and the traditional implementation of Visual SLAM systems, we present in this paper a novel Visual SLAM method that can effectively utilize texture-less object instances for mapping and localization. The proposed Visual SLAM method includes newly designed feature extraction, matching, localization and mapping modules, which jointly use object features and point features to estimate camera 6-DOF poses and do richer map construction. A group of organized raster points is used to represent objects during feature matching and pose estimation process in the proposed Visual SLAM pipeline. Owing to the object feature fusion in the co-visibility graph it could conduct scale aware bundle adjustments to reduce accumulated error. The advantages of proposed Visual SLAM method are demonstrated through experiments conducted both on synthetic datasets and real-world datasets.

A fast object registration method for augmented reality assembly with simultaneous determination of multiple 2D-3D correspondences

Abstract A fast and robust object registration is one of the fundamental steps of an augmented reality (AR) assembly guidance system. To manage the registration of texture-less objects from monocular images, this paper presents a three-dimensional (3D) object registration solution in the view-based framework, which incorporates a new image feature named as chain-of-lines feature (COLF). The COLF, constructed by several directed line segments with geometric constraints, enhances the robustness of the registration by establishing multiple correspondences between the two-dimensional (2D) image and 3D model simultaneously. In order to save on computational cost and improve the efficiency of the COLF-based registration, a compression algorithm based on the local breadth-first search (LBFS) strategy is also proposed; this adaptively reduces the number of keyframes according to the target model. In the experimental evaluation, the combination of COLF and LBFS-based keyframe compression shows higher registration success rate and faster speed in comparison with conventional registration methods. Moreover, an AR assembly guidance prototype system for reductors is introduced to illustrate the application of the proposed registration method.