Single RGB-D Image Research Articles

MaskRecon: High-quality human reconstruction via masked autoencoders using a single RGB-D image

In this paper, we explore reconstructing high-quality clothed 3D humans from a single RGB-D image, assuming that virtual humans can be represented by front-view and back-view depths. Due to the scarcity of captured real RGB-D human images, we employ rendered images to train our method. However, rendered images lack background with significant depth variation in silhouettes, leading to shape prediction inaccuracies and noise. To mitigate this issue, we introduce a pseudo-multi-task framework, which incorporates a Conditional Generative Adversarial Network (CGAN) to infer back-view RGB-D images and a self-supervised Masked Autoencoder (MAE) to capture latent structural information of the human body. Additionally, we propose a Multi-scale Feature Fusion (MFF) module to effectively merge structural information and conditional features at various scales. Our method surpasses many existing techniques, as demonstrated through evaluations on the Thuman, RenderPeople, and BUFF datasets. Notably, our approach excels in reconstructing high-quality human models, even under challenging conditions such as complex poses and loose clothing, both on rendered and real-world images. Codes are available at https://github.com/Archaic-Atom/MaskRecon.

Head model reconstruction and animation method using color image with depth information

The article presents a method for reconstructing and animating a digital model of a human head from a single RGBD image, a color RGB image with depth information. An approach is proposed for optimizing the parametric FLAME model using a point cloud of a face corresponding to a single RGBD image. The results of experimental studies have shown that the proposed optimization approach makes it possible to obtain a head model with more prominent features of the original face compared to optimization approaches using RGB images or the same approaches generalized to RGBD images.

Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image.

We present a deep reinforcement learning method of progressive view inpainting for colored semantic point cloud scene completion under volume guidance, achieving high-quality scene reconstruction from only a single RGB-D image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D RGB-D and segmentation image inpainting, and multi-view selection for completion. Given a single RGB-D image, our method first predicts its semantic segmentation map and goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view RGB-D and segmentation map, and integrating all RGB-D and segmentation maps into the point cloud. Since the occluded areas are unavailable, we resort to a A3C network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the 3D-FUTURE data, obtaining better results than state-of-the-arts.

Neural-Based Detection and Segmentation of Articulated Objects for Robotic Interaction in a Household Environment

Robots operating in household environments should detect and estimate the properties of articulated objects to efficiently perform tasks given by human operators. This paper presents the design and implementation of a system for estimating a point-cloud-based model of a scene, enhanced with information about articulated objects, based on a single RGB-D image. This article describes the neural method used to detect handles, detect and extract fronts, detect rotational joints, and build a point cloud model. It compares various architectures of neural networks to detect handles, the fronts of drawers and cabinets, and estimate rotational joints. In the end, the results are merged to build a 3D model of articulated objects in the environment.

Learning to Detect 3D Symmetry From Single-View RGB-D Images With Weak Supervision.

3D symmetry detection is a fundamental problem in computer vision and graphics. Most prior works detect symmetry when the object model is fully known, few studies symmetry detection on objects with partial observation, such as single RGB-D images. Recent work addresses the problem of detecting symmetries from incomplete data with a deep neural network by leveraging the dense and accurate symmetry annotations. However, due to the tedious labeling process, full symmetry annotations are not always practically available. In this work, we present a 3D symmetry detection approach to detect symmetry from single-view RGB-D images without using symmetry supervision. The key idea is to train the network in a weakly-supervised learning manner to complete the shape based on the predicted symmetry such that the completed shape be similar to existing plausible shapes. To achieve this, we first propose a discriminative variational autoencoder to learn the shape prior in order to determine whether a 3D shape is plausible or not. Based on the learned shape prior, a symmetry detection network is present to predict symmetries that produce shapes with high shape plausibility when completed based on those symmetries. Moreover, to facilitate end-to-end network training and multiple symmetry detection, we introduce a new symmetry parametrization for the learning-based symmetry estimation of both reflectional and rotational symmetry. The proposed approach, coupled symmetry detection with shape completion, essentially learns the symmetry-aware shape prior, facilitating more accurate and robust symmetry detection. Experiments demonstrate that the proposed method is capable of detecting reflectional and rotational symmetries accurately, and shows good generality in challenging scenarios, such as objects with heavy occlusion and scanning noise. Moreover, it achieves state-of-the-art performance, improving the F1-score over the existing supervised learning method by 2%-11% on the ShapeNet and ScanNet datasets.

Robust Symmetry Prediction with Multi-Modal Feature Fusion for Partial Shapes

In geometry processing, symmetry research benefits from global geometric features of complete shapes, but the shape of an object captured in real-world applications is often incomplete due to the limited sensor resolution, single viewpoint, and occlusion. Different from the existing works predicting symmetry from the complete shape, we propose a learning approach for symmetry prediction based on a single RGB-D image. Instead of directly predicting the symmetry from incomplete shapes, our method consists of two modules, i.e., the multi-modal feature fusion module and the detection-by-reconstruction module. Firstly, we build a channel-transformer network (CTN) to extract cross-fusion features from the RGB-D as the multi-modal feature fusion module, which helps us aggregate features from the color and the depth separately. Then, our self-reconstruction network based on a 3D variational auto-encoder (3D-VAE) takes the global geometric features as input, followed by a prediction symmetry network to detect the symmetry. Our experiments are conducted on three public datasets: ShapeNet, YCB, and ScanNet, we demonstrate that our method can produce reliable and accurate results.

Multiple geometry representations for 6D object pose estimation in occluded or truncated scenes

Deep learning-based 6D object pose estimation methods from a single RGBD image have recently received increasing attention because of their powerful representation learning capabilities. These methods, however, cannot handle severe occlusion and truncation. In this paper, we present a novel 6D object pose estimation method based on multiple geometry representations. Specifically, we introduce a network to fuse the appearance and geometry features extracted from input color and depth images. Then, we utilize these per-point fusion features to estimate keypoint offsets, edge vectors, and dense symmetry correspondences in the canonical coordinate system. Finally, a two-stage pose regression module is applied to compute the 6D pose of an object. Relative to the unitary 3D keypoint-based strategy, such combination of multiple geometry representations provides sufficient and diverse information, especially for occluded or truncated scenes. To show the robustness to occlusion and truncation of the proposed method, we conduct comparative experiments on the Occlusion LineMOD, Truncation LineMOD, and T-LESS datasets. Results reveal that the proposed method outperforms state-of-the-art techniques by a large margin.

PANet: A Pixel-Level Attention Network for 6D Pose Estimation With Embedding Vector Features

In this work, we present PANet, a pixel-level attention network with embedding vector features, which addresses the challenge of 6D pose estimation from a single RGBD image under severe occlusion. PANet produces pixel-wise attention for strong representation learning and leverages a novel selection scheme for robust pose estimation. Specifically, at the representation learning stage, we devise Pyramid Pixel-level Attention Module that unites attention mechanism with spatial pyramid to learn a discriminative representation, and Attention Upsample Module that utilizes arbitrary combinations of the CNN encoders’ feature maps to recover precise pixel-wise prediction, after which we embed the two modules into CNN to gain rich appearance features from RGB images. For depth images, we apply the current advanced point cloud network adopting attention mechanism to earn geometry features, which are further fused with the appearance features to obtain point-wise dense feature embedding. In the pose estimation stage, we define point-wise embedding vector features which can provide rich viewpoint information to better cope with the case of occluded objects. Further, a novel and effective RANSAC-based Selection Scheme is also founded to select vector features with high scores for pose estimation. Extensive experimental results manifest that our method outperforms the state-of-the-art by large margins on several benchmarks.

A dynamic keypoint selection network for 6DoF pose estimation

6 DoF pose estimation problem aims to estimate the rotation and translation parameters between two coordinates, such as object world coordinate and camera world coordinate. Although some advances are made with the help of deep learning, how to full use scene information is still a problem. Prior works tackle the problem by pixel-wise feature fusion but need to randomly select numerous points from images, which can not satisfy the demands of fast inference simultaneously and accurate pose estimation. In this work, we present a novel deep neural network based on dynamic keypoint selection designed for 6DoF pose estimation from a single RGBD image. Our network includes three parts, instance semantic segmentation, edge points detection and 6DoF pose estimation. Given an RGBD image, our network is trained to predict pixel category and the translation to edge points and center points. Then, a least-square fitting manner is applied to estimate the 6DoF pose parameters. Specifically, we propose a dynamic keypoint selection algorithm to choose keypoints from the foreground feature map. It allows us to leverage geometric and appearance information. During 6DoF pose estimation, we utilize the instance semantic segmentation result to filter out background points and only use foreground points to finish edge points detection and 6DoF pose estimation. Experiments on two commonly used 6DoF estimation benchmark datasets, YCB-Video and LineMoD, demonstrate that our method outperforms the state-of-the-art methods and achieves significant improvements over other same category methods time efficiency.

A 3D Keypoints Voting Network for 6DoF Pose Estimation in Indoor Scene

This paper addresses the problem of instance-level 6DoF pose estimation from a single RGBD image in an indoor scene. Many recent works have shown that a two-stage network, which first detects the keypoints and then regresses the keypoints for 6d pose estimation, achieves remarkable performance. However, the previous methods concern little about channel-wise attention and the keypoints are not selected by comprehensive use of RGBD information, which limits the performance of the network. To enhance RGB feature representation ability, a modular Split-Attention block that enables attention across feature-map groups is proposed. In addition, by combining the Oriented FAST and Rotated BRIEF (ORB) keypoints and the Farthest Point Sample (FPS) algorithm, a simple but effective keypoint selection method named ORB-FPS is presented to avoid the keypoints appear on the non-salient regions. The proposed algorithm is tested on the Linemod and the YCB-Video dataset, the experimental results demonstrate that our method outperforms the current approaches, achieves ADD(S) accuracy of 94.5% on the Linemod dataset and 91.4% on the YCB-Video dataset.

DDGC: Generative Deep Dexterous Grasping in Clutter

Recent advances in multi-fingered robotic grasping have enabled fast 6-Degrees-of-Freedom (DOF) single object grasping. Multi-finger grasping in cluttered scenes, on the other hand, remains mostly unexplored due to the added difficulty of reasoning over obstacles which greatly increases the computational time to generate high-quality collision-free grasps. In this work, we address such limitations by introducing DDGC, a fast generative multi-finger grasp sampling method that can generate high quality grasps in cluttered scenes from a single RGB-D image. DDGC is built as a network that encodes scene information to produce coarse-to-fine collision-free grasp poses and configurations. We experimentally benchmark DDGC against two state-of-the-art methods on 1200 simulated cluttered scenes and 7 real-world scenes. The results show that DDGC outperforms the baselines in synthesizing high-quality grasps and removing clutter. DDGC is also 4-5 times faster than GraspIt!. This, in turn, opens the door for using multi-finger grasps in practical applications which has so far been limited due to the excessive computation time needed by other methods. Code and videos are available at https://irobotics.aalto.fi/ddgc/.

Real-Time Block-Based Embedded CNN for Gesture Classification on an FPGA

This paper presents a block-based embedded convolutional neural network (CNN) for gesture classification on field-programmable gate array (FPGA) in real time. Gesture recognition is an important tool to spontaneous interact with human machine interface. Many CNN architectures using RGB images have been proposed for gesture classification. RGB based gesture classification may cause incorrect results under insufficient light or similar gestures. In addition, most of the CNN architectures cannot run in real time on edge devices due to their large number of parameters and DRAM data access. In this paper, a block-based CNN using RGB-D data is proposed for gesture classification. Adding depth images to RGB images boots the classification accuracy. A CNN architecture with block-based feature maps is built for embedded FPGA implementations. The total number of parameters of the proposed RGB-D embedded CNN (eCNN) model is only 0.17M and it achieves 99.96% and 99.88% accuracy with 32-bit floating point and 8-bit fixed point implementation for America Sign Language (ASL) data set. The RTL simulation of the proposed eCNN model has the average inference speed of 0.171 milliseconds at frequency of 250MHz for a single pair RGB-D image. Implemented on a FPGA integrated with Microsoft Kinect v2 achieve an inference time in 19.42 ms which achieves high accuracy and real-time performance.

Object Pose Estimation via Pruned Hough Forest With Combined Split Schemes for Robotic Grasp

Robotic grasp in complex open-world scenarios requires an effective and generalizable perception. Estimating object’s pose is needed in a variety of practical grasping scenarios. Here we present a novel approach of pose estimation of textureless and textured objects. The algorithm utilizes a single RGB-D image to exploit depth invariant, oriented point pair feature as well as local contextual sensitivity in cluttered environments. To enhance the performance of the voting process and improve learning efficiency, we employ a global pruning algorithm that reduces the risk of overfitting and simplifies the structure of decision trees after compensating for the complementary information among multiple trees by optimizing a designed global objective function. Finally, we also refine the pose obtained from the above stage. The proposed approach of estimating 6-D (degree of freedom) poses of textured and textureless objects is evaluated on publicly available data sets against the recent works under various conditions. It illustrates that our framework is superior to these recent works. Further, we perform extensive qualitative experiments of robotic grasp to illustrate the proposed approach can be applied to practical scenarios. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article is motivated by the problem of the pose estimation of textured and textureless objects in clutter environments. It is difficult for conventional works to address the issue of estimating textured or textureless objects’ poses in such scenarios. We considered that a novel system should be able to obtain the 6-D poses of objects. Therefore, we investigate the combined use of multiple split functions with different characteristics. Learning the model based on Hough forests always cost much computational resource; therefore, we construct a novel pruned Hough forest for solving this issue. Through the comparison and robotic grasp verifications, the behavior of our system can be used in practical applications. In future, we will deploy the proposed system in robotic assembling tasks.

Sparse intrinsic decomposition and applications

This paper proposes an intrinsic decomposition method from a single RGB-D image. To remedy the highly ill-conditioned problem, the reflectance component is regularized by a sparsity term, which is weighted by a bilateral kernel to exploit non-local structural correlation. As shading images are piece-wise smooth and have sparse gradient fields, the sparse-induced ℓ1-norm is used to regularize the finite difference of the direct irradiance component, which is the most dominant sub-component of shading and describes the light directly received by the surfaces of the objects from the light source. To derive an efficient algorithm, the proposed model is transformed into an unconstrained minimization of the augmented Lagrangian function, which is then optimized via the alternating direction method. The stability of the proposed method with respect to parameter perturbation and its robustness to noise are investigated by experiments. Quantitative and qualitative evaluation demonstrates that our method has better performance than state-of-the-art methods. Our method can also achieve intrinsic decomposition from a single color image by integrating existed depth estimation methods. We also present a depth refinement method based on our intrinsic decomposition method, which obtains more geometry details without texture artifacts. Other application, e.g., texture editing, also demonstrates the effectiveness of our method.

Unreal mask: one-shot multi-object class-based pose estimation for robotic manipulation using keypoints with a synthetic dataset

Object pose estimation is a prerequisite for many robotic applications. Preparing dataset for network training is a challenging part of the pose estimation approaches, and in most of them, the network can detect just the trained objects. Synthetic data are used to train deep neural networks in robotic manipulation as a promising method for obtaining a huge amount of prelabeled training data, which are generated safely. We are to investigate the reality gap in the pose estimation of intra-category objects from a single RGB-D image using keypoints. The proposed approach in this paper provides a fast and simple procedure for training a deep neural network to identify the object and its keypoints based on synthetic dataset and autolabeling program. To our knowledge, this is the first deep network trained only on synthetic data that can find keypoints of intra-category objects for pose estimation purposes. The speed of training and the simplicity of this method make it very easy to add a new class of objects to the system which is the main advantage of this approach. Using this approach, we demonstrate a near-real-time system estimating object poses with sufficient accuracy for real-world semantic grasping and manipulating of intra-category objects in clutter by a real robot.

L6DNet: Light 6 DoF Network for Robust and Precise Object Pose Estimation With Small Datasets

Estimating the 3D pose of an object is a challenging task that can be considered within augmented reality or robotic applications. In this paper, we propose a novel approach to perform 6 DoF object pose estimation from a single RGB-D image. We adopt a hybrid pipeline in two stages: data-driven and geometric respectively. The data-driven step consists of a classification CNN to estimate the object 2D location in the image from local patches, followed by a regression CNN trained to predict the 3D location of a set of keypoints in the camera coordinate system. To extract the pose information, the geometric step consists in aligning the 3D points in the camera coordinate system with the corresponding 3D points in world coordinate system by minimizing a registration error, thus computing the pose. Our experiments on the standard dataset LineMod show that our approach is more robust and accurate than state-of-the-art methods. The approach is also validated to achieve a 6 DoF positioning task by visual servoing.

Graph neural network for 6D object pose estimation

6D object pose estimation plays an important role in various applications such as robot manipulation and virtual reality. In this paper, we introduce a graph convolution neural network based method to addresses the problem of estimating the 6D pose of objects from a single RGB-D image. The proposed method fuses the appearance feature of the RGB image with the geometry feature of point clouds to predict pixel-level pose and the network also predicts pixel-level confidences to prune outlier predictions. The inner structure information of point cloud is learned by a graph convolution neural network. Specially, we adopt a residual graph convolution module to learn a discriminative feature. Our network enables end-to-end training and fast inference. The extensive experiments verify the method and the model achieves state-of-the-art for the LINEMOD and LINEMOD-OCCLUSION dataset (ADD-S: 88.68 and 65.38 respectively).

The design of tourism product CAD three-dimensional modeling system using VR technology.

In view of the high homogeneity of tourism products all over the country, an attempt is made to design virtual visit tourism products with cultural experience background, which can reflect the characteristics of culture + tourism in different scenic spots, so that tourists can deeply experience the local culture. Combined with computer aided design (CAD), the virtual three-dimensional (3D) modeling system of scenic spots is designed, and VR real scene visit interactive tourism products suitable for different scenic spots are designed. 360° VR panoramic display technology is used for 360° VR panoramic video shooting and visiting system display production of Elephant Trunk Hill park scenery. A total of 157 images are collected and 720 cloud panoramic interactive H5 tool is selected to produce a display system suitable for 360° VR panoramic display of scenic spots. Meanwhile, based on single view RGB-D image, the latest convolutional neural network (CNN) algorithm and point cloud processing algorithm are used to design the indoor 3D scene reconstruction algorithm based on semantic understanding. Experiments show that the pixel accuracy and mean intersection over union of the indoor scene layout segmentation network segmentation results are 89.5% and 60.9%, respectively, that is, it has high accuracy. The VR real scene visit interactive tourism product can make tourists have a more immersive sense of interaction and experience before, during and after the tour.

6DoF Pose Estimation of Transparent Object from a Single RGB-D Image.

6DoF object pose estimation is a foundation for many important applications, such as robotic grasping, automatic driving, and so on. However, it is very challenging to estimate 6DoF pose of transparent object which is commonly seen in our daily life, because the optical characteristics of transparent material lead to significant depth error which results in false estimation. To solve this problem, a two-stage approach is proposed to estimate 6DoF pose of transparent object from a single RGB-D image. In the first stage, the influence of the depth error is eliminated by transparent segmentation, surface normal recovering, and RANSAC plane estimation. In the second stage, an extended point-cloud representation is presented to accurately and efficiently estimate object pose. As far as we know, it is the first deep learning based approach which focuses on 6DoF pose estimation of transparent objects from a single RGB-D image. Experimental results show that the proposed approach can effectively estimate 6DoF pose of transparent object, and it out-performs the state-of-the-art baselines by a large margin.

Robust 3D Hand Detection from a Single RGB-D Image in Unconstrained Environments

Three-dimensional hand detection from a single RGB-D image is an important technology which supports many useful applications. Practically, it is challenging to robustly detect human hands in unconstrained environments because the RGB-D channels can be affected by many uncontrollable factors, such as light changes. To tackle this problem, we propose a 3D hand detection approach which improves the robustness and accuracy by adaptively fusing the complementary features extracted from the RGB-D channels. Using the fused RGB-D feature, the 2D bounding boxes of hands are detected first, and then the 3D locations along the z-axis are estimated through a cascaded network. Furthermore, we represent a challenging RGB-D hand detection dataset collected in unconstrained environments. Different from previous works which primarily rely on either the RGB or D channel, we adaptively fuse the RGB-D channels for hand detection. Specifically, evaluation results show that the D-channel is crucial for hand detection in unconstrained environments. Our RGB-D fusion-based approach significantly improves the hand detection accuracy from 69.1 to 74.1 comparing to one of the most state-of-the-art RGB-based hand detectors. The existing RGB- or D-based methods are unstable in unseen lighting conditions: in dark conditions, the accuracy of the RGB-based method significantly drops to 48.9, and in back-light conditions, the accuracy of the D-based method dramatically drops to 28.3. Compared with these methods, our RGB-D fusion based approach is much more robust without accuracy degrading, and our detection results are 62.5 and 65.9, respectively, in these two extreme lighting conditions for accuracy.