RGB-D Images Research Articles

Dynamic Object Removal and Spatio-Temporal RGB-D Inpainting via Geometry-Aware Adversarial Learning

Dynamic objects have a significant impact on the robot’s perception of the environment which degrades the performance of essential tasks such as localization and mapping. In this work, we address this problem by synthesizing plausible color, texture and geometry in regions occluded by dynamic objects. We propose the novel geometry-aware DynaFill architecture that follows a coarse-to-fine topology and incorporates our gated reccurrent feedback mechanism to adaptively fuse information from previous timesteps. We optimize our architecture using adversarial training to synthesize fine realistic textures which enables it to hallucinate color and depth structure in occluded regions online in a spatially and temporally coherent manner, without relying on future frame information. Casting our inpainting problem as an image-to-image translation task, our model also corrects regions correlated with the presence of dynamic objects in the scene, such as shadows or reflections. We introduce a large-scale hyperrealistic dataset with RGB-D images, semantic segmentation labels, camera poses as well as groundtruth RGB-D information of occluded regions. Extensive quantitative and qualitative evaluations show that our approach achieves state-of-the-art performance, even in challenging weather conditions. Furthermore, we present results for retrieval-based visual localization with the synthesized images that demonstrate the utility of our approach.

TMFNet: Three-Input Multilevel Fusion Network for Detecting Salient Objects in RGB-D Images

The use of depth information, acquired by depth sensors, for salient object detection (SOD) is being explored. Despite the remarkable results from recent deep learning approaches for RGB-D SOD, they fail to fully incorporate original and accurate information to express details of RGB-D images in salient objects. Here, we propose an RGB-D SOD model using a three-input multilevel fusion network (TMFNet), which differs from existing methods based on double-stream networks. In addition to RGB input (first input) and depth input (second input), the RGB image and depth map are combined into a four-channel representation (RGBD input) that constitutes the third input to the TMFNet. The RGBD input generates multilevel features that reflect details of the RGB-D image. In addition, the proposed TMFNet aggregates diverse region-based contextual information without discarding RGB and depth features. Thus, we introduce a cross-fusion module, and benefiting from rich low- and high-level information from salient features, feature fusion enables the improvement of localization of salient objects. The proposed TMFNet achieves state-of-the-art performance on six benchmark datasets for SOD.

MONOCULAR DEPTH ESTIMATION IN FOREST ENVIRONMENTS

Abstract. Depth estimation from a single image is a challenging task, especially inside the highly structured forest environment. In this paper, we propose a supervised deep learning model for monocular depth estimation based on forest imagery. We train our model on a new data set of forest RGB-D images that we collected using a terrestrial laser scanner. Alongside the input RGB image, our model uses a sparse depth channel as input to recover the dense depth information. The prediction accuracy of our model is significantly higher than that of state-of-the-art methods when applied in the context of forest depth estimation. Our model brings the RMSE down to 2.1 m, compared to 4 m and above for reference methods.

Automatic Weight Prediction System for Korean Cattle Using Bayesian Ridge Algorithm on RGB-D Image

Weighting the Hanwoo (Korean cattle) is very important for Korean beef producers when selling the Hanwoo at the right time. Recently, research is being conducted on the automatic prediction of the weight of Hanwoo only through images with the achievement of research using deep learning and image recognition. In this paper, we propose a method for the automatic weight prediction of Hanwoo using the Bayesian ridge algorithm on RGB-D images. The proposed system consists of three parts: segmentation, extraction of features, and estimation of the weight of Korean cattle from a given RGB-D image. The first step is to segment the Hanwoo area from a given RGB-D image using depth information and color information, respectively, and then combine them to perform optimal segmentation. Additionally, we correct the posture using ellipse fitting on segmented body image. The second step is to extract features for weight prediction from the segmented Hanwoo image. We extracted three features: size, shape, and gradients. The third step is to find the optimal machine learning model by comparing eight types of well-known machine learning models. In this step, we compared each model with the aim of finding an efficient model that is lightweight and can be used in an embedded system in the real field. To evaluate the performance of the proposed weight prediction system, we collected 353 RGB-D images from livestock farms in Wonju, Gangwon-do in Korea. In the experimental results, random forest showed the best performance, and the Bayesian ridge model is the second best in MSE or the coefficient of determination. However, we suggest that the Bayesian ridge model is the most optimal model in the aspect of time complexity and space complexity. Finally, it is expected that the proposed system will be casually used to determine the shipping time of Hanwoo in wild farms for a portable commercial device.

Noise-tolerant RGB-D feature fusion network for outdoor fruit detection

In the process of farm automation, fruit detection is the basis and guarantee for yield prediction, automatic picking, and other orchard operations. RGB images can only obtain the two-dimensional information of the scene, which is not sufficient to effectively distinguish fruits that are dense growth and occlusion by branches and leaves. With the development of depth sensors, using RGB-D images with more complementary information can boost the performance of fruit detection. However, due to the nature of sensors and scene configurations, the quality of outdoor depth images is poor, posing a challenge when fusing RGB-D features. Therefore, this paper proposes an end-to-end RGB-D object detection network, termed as noise-tolerant feature fusion network (NT-FFN), to utilize the outdoor multi-modal data properly and improve the detection accuracy. Specifically, the NT-FFN first uses two structurally identical feature extractors to extract single-modal (color and depth) features, which is the base of the subsequent feature fusion. Then, to avoid introducing too much depth noise and focus the perception on the important part of the features, an attention-based fusion module is designed to adaptively fuse the multi-modal features. Finally, multi-scale features from the color images and the fusion modules are used to predict object position, which not only improves the network's ability to detect multi-scale objects but also further enhances the noise immunity of the network. In addition, this paper constructs an RGB-D citrus fruit dataset, which contributes to comprehensively evaluating the proposed network. Evaluation metrics on the dataset show that the NT-FFN achieves an AP50 of 95.4% with a real-time speed, which outperforms single-modal methods, common multi-modal fusion strategies, and advanced multi-modal detection methods. The proposed NT-FFN also achieves excellent detection results in other fruit detection tasks, which verifies its generalization ability. This study provides the possibility and foundation for performing multi-modal information fusion in outdoor fruit detection.

Multi-View Visual Relationship Detection with Estimated Depth Map

The abundant visual information contained in multi-view images is widely used in computer vision tasks. Existing visual relationship detection frameworks have extended the feature vector to improve model performance. However, single view information can not fully reveal the visual relationships in complex visual scenes. To solve this problem and explore the multi-view information in a visual relationship detection (VRD) model, a novel multi-view VRD framework based on a monocular RGB image and an estimated depth map is proposed. The contributions of this paper are threefold. First, we construct a novel multi-view framework which fuses information of different views extracted from estimated RGB-D images. Second, a multi-view image generation method is proposed to transfer flat visual space to 3D multi-view space. Third, we redesign the visual relationship balanced classifier which can process multi-view feature vectors simultaneously. Detailed experiments were conducted on two datasets to demonstrate the effectiveness of the multi-view VRD framework. The experimental results showed that the multi-view VRD framework resulted in state-of-the-art zero-shot learning performance in specific depth conditions.

Robust 3D face modeling and tracking from RGB-D images

Robust 3D face modeling and tracking from RGB-D images

Weakly supervised learning of multi-object 3D scene decompositions using deep shape priors

Representing scenes at the granularity of objects is a prerequisite for scene understanding and decision making. We propose PriSMONet, a novel approach based on Prior Shape knowledge for learning Multi-Object 3D scene decomposition and representations from single images. Our approach learns to decompose images of synthetic scenes with multiple objects on a planar surface into its constituent scene objects and to infer their 3D properties from a single view. A recurrent encoder regresses a latent representation of 3D shape, pose and texture of each object from an input RGB image. By differentiable rendering, we train our model to decompose scenes from RGB-D images in a self-supervised way. The 3D shapes are represented continuously in function-space as signed distance functions which we pre-train from example shapes in a supervised way. These shape priors provide weak supervision signals to better condition the challenging overall learning task. We evaluate the accuracy of our model in inferring 3D scene layout, demonstrate its generative capabilities, assess its generalization to real images, and point out benefits of the learned representation.

RLLNet: a lightweight remaking learning network for saliency redetection on RGB-D images

RLLNet: a lightweight remaking learning network for saliency redetection on RGB-D images

Bifurcation Fusion Network for RGB-D Salient Object Detection

With the rapid development of sensor technology, multi-modal data fusion methods based on deep neural networks provide a reliable guarantee for object recognition and detection in complex scenarios. Most of the existing RGB-D image salient object detection methods improve the salient object detection methods in 2D scenes, which have many problems, such as ineffective fusion of feature information, incomplete image feature extraction and so on. Aiming at these problems, we proposed a salient object detection method based on bifurcated fusion network. First, we use the high- level features of the global context to locate the salient object, then we use the low-level features of the local details to extract the edge information. Second, we model the RGB information and depth information through the gate channel transformation control mechanism, and construct the bifurcation backbone network model and generate the initial salient map. Finally, based on the initial salient map and low-level features, the final salient map is generated. Experimental results show that the proposed method can fully utilize the multi-layer feature information of the image and effectively detect the salient object.

Scale-aware network with modality-awareness for RGB-D indoor semantic segmentation

This paper focuses on indoor semantic segmentation based on RGB-D images. Semantic segmentation is a pixel-level classification task that has made steady progress based on fully convolutional networks (FCNs). However, we find there is still room for improvements in the following three aspects. The first is related to multi-scale feature extraction. Recent state-of-the-art works forcibly concatenate multi-scale feature representations extracted by spatial pyramid pooling, dilated convolution or other architectures, regardless of the spatial extent for each pixel. The second is regarding RGB-D modal fusion. Most successful methods treat RGB and depth as two separate modalities and force them to be joined together regardless of their different contributions to the final prediction. The final aspect is about the modeling ability of extracted features. Due to the “local grid” defined by the receptive field, the learned feature representation lacks the ability to model spatial dependencies. In addition to these modules, we design a depth estimation module to encourage the RGB network to extract more effective features. To solve the above challenges, we propose four modules to address them: scale-aware module, modality-aware module, attention module and depth estimation module. Extensive experiments on the NYU-Depth v2 and SUN RGB-D datasets demonstrate that our method is effective against RGB-D indoor semantic segmentation.

Efficient 6D object pose estimation based on attentive multi‐scale contextual information

6D pose estimation has been pervasively applied to various robotic applications, such as service robots, collaborative robots, and unmanned warehouses. However, accurate 6D pose estimation is still a challenge problem due to the complexity of application scenarios caused by illumination changes, occlusion and even truncation between objects, and additional refinement is required for accurate 6D object pose estimation in prior work. Aiming at the efficiency and accuracy of 6D object pose estimation in these complex scenes, this paper presents a novel end-to-end network, which effectively utilises the contextual information within a neighbourhood region of each pixel to estimate the 6D object pose from RGB-D images. Specifically, our network first applies the attention mechanism to extract effective pixel-wise dense multimodal features, which are then expanded to multi-scale dense features by integrating pixel-wise features at different scales for pose estimation. The proposed method is evaluated extensively on the LineMOD and YCB-Video datasets, and the experimental results show that the proposed method is superior to several state-of-the-art baselines in terms of average point distance and average closest point distance.

Three-dimensional pose detection method based on keypoints detection network for tomato bunch

Non-destructive picking of fresh tomatoes is a delicate agronomical operation, based on comprehensive information about the plant organ, such as the location of stem, peduncle, and fruits. The matching between visual information supply and information demand from the agronomical technic is the key power to promote the picking robot from the laboratory to the field. The three-dimensional pose information, containing the location of each organ of the plant, can meet the demand of agronomical technic. It is the premise of precisely handling the cluster of fruits. In order to realize the fine tomato bunch harvesting operation in a bunch, this paper proposed a three-dimensional pose detection method for tomato bunch. The method, named Tomato Pose Method (TPM), is composed of a priori geometric model, a cascaded multi-task network, and a three-dimensional reconstruction process. Based on prior knowledge and agronomic technology, this prior geometric model comprehensively and flexibly describes the spatial location information of tomato bunch. The cascaded multi-task network is designed based on hourglass structure and transfer learning, which is suitable for bounding box and key point prediction of tomato bunches in complex environments. Finally, combining the prior geometric model and the spatial position information of each key point, the tomato bunch is reconstructed. Only a medium training dataset, containing 1800 RGBD images covering changing lighting, occlusion, and various poses, is needed for training. Its success rate of TPM on two-dimensional keypoint detection is 94.02%, the accuracy of 85.77% predicted points are at medium level. And 70.05% tomato bunch with multi-pose can be constructed. More importantly, this method only needs one RGBD image taken by a commercial camera to realize the three-dimensional reconstruction of a single-bunch scenario in 1.0 s, and a multi-bunch scenario in 2.0 s. It provides comprehensive information, and provides data basis for target positioning and path planning of picking robot, which makes the non-destructive harvesting possible.

SE-ResUNet: A Novel Robotic Grasp Detection Method

In this letter, a novel grasp detection neural network Squeeze-and-Excitation ResUNet (SE-ResUNet) is developed, where the residual block with the channel attention is integrated. The proposed framework can not only generate the grasp pose from the RGB-D images, but also predict the quality score of each grasp pose. The experimental results show that the accuracy on the Cornell dataset and the Jacquard dataset is 98.2% and 95.7%, respectively. And the processing speed for the RGB-D images can reach 30fps, which shows the good real-time performance. In the comparison study, better performance is also obtained by the proposed method, which improves the accuracy and time efficiency. Finally, it is also demonstrated by physical grasping on the Baxter robot, where the average grasp success rate is 96.3%.

Robotic Cable Routing with Spatial Representation

Cable routing is a challenging task for robotic automation. To accomplish the task, it requires a high-level path planner to generate a sequence of cable configurations from the initial state to the target state and a low-level manipulation planner to plan the robot motion commands to transit between adjacent states. However, there are yet no proper representations to model the cable with the environment objects, impeding the design of both high-level path planning and low-level manipulation planning. In this letter, we propose a framework for cable routing with spatial representation. For high-level planning, by considering the spatial relations between the cable and the environment objects such as fixtures, the proposed method is able to plan a path from the initial state to the goal state in a graph. For low-level manipulation, multiple manipulation primitives are efficiently learned from human demonstration, to configure the cable to planned intermediate states leveraging the same spatial representation. We also implement a cable state estimator that robustly extracts the spatial representation from raw RGB-D images, thus completing the cable routing framework. We evaluate the proposed framework with various cables and fixture settings, and demonstrate that it outperforms some baselines in terms of reliability and generalizability.

PANet: A Pixel-Level Attention Network for 6D Pose Estimation With Embedding Vector Features

In this work, we present PANet, a pixel-level attention network with embedding vector features, which addresses the challenge of 6D pose estimation from a single RGBD image under severe occlusion. PANet produces pixel-wise attention for strong representation learning and leverages a novel selection scheme for robust pose estimation. Specifically, at the representation learning stage, we devise Pyramid Pixel-level Attention Module that unites attention mechanism with spatial pyramid to learn a discriminative representation, and Attention Upsample Module that utilizes arbitrary combinations of the CNN encoders’ feature maps to recover precise pixel-wise prediction, after which we embed the two modules into CNN to gain rich appearance features from RGB images. For depth images, we apply the current advanced point cloud network adopting attention mechanism to earn geometry features, which are further fused with the appearance features to obtain point-wise dense feature embedding. In the pose estimation stage, we define point-wise embedding vector features which can provide rich viewpoint information to better cope with the case of occluded objects. Further, a novel and effective RANSAC-based Selection Scheme is also founded to select vector features with high scores for pose estimation. Extensive experimental results manifest that our method outperforms the state-of-the-art by large margins on several benchmarks.

Learning geodesic-aware local features from RGB-D images

Most of the existing handcrafted and learning-based local descriptors are still at best approximately invariant to affine image transformations, often disregarding deformable surfaces. In this paper, we take one step further by proposing a new approach to compute descriptors from RGB-D images (where RGB refers to the pixel color brightness and D stands for depth information) that are invariant to isometric non-rigid deformations, as well as to scale changes and rotation. Our proposed description strategies are grounded on the key idea of learning feature representations on undistorted local image patches using surface geodesics. We design two complementary local descriptors strategies to compute geodesic-aware features efficiently: one efficient binary descriptor based on handcrafted binary tests (named GeoBit), and one learning-based descriptor (GeoPatch) with convolutional neural networks (CNNs) to compute features. In different experiments using real and publicly available RGB-D data benchmarks, they consistently outperforms state-of-the-art handcrafted and learning-based image and RGB-D descriptors in matching scores, as well as in object retrieval and non-rigid surface tracking experiments, with comparable processing times. We also provide to the community a new dataset with accurate matching annotations of RGB-D images of different objects (shirts, cloths, paintings, bags), subjected to strong non-rigid deformations, for evaluation benchmark of deformable surface correspondence algorithms.

Autonomous, Mobile Manipulation in a Wall-building Scenario: Team LARICS at MBZIRC 2020

In this paper we present our hardware design and control approaches for a mobile manipulation platform used in Challenge 2 of the MBZIRC 2020 competition. In this challenge, a team of UAVs and a single UGV collaborate in an autonomous, wall-building scenario, motivated by construction automation and large-scale robotic 3D printing. The robots must be able, autonomously, to detect, manipulate, and transport bricks in an unstructured, outdoor environment. Our control approach is based on a state machine that dictates which controllers are active at each stage of the Challenge. In the first stage our UGV uses visual servoing and local controllers to approach the target object without considering its orientation. The second stage consists of detecting the object’s global pose using OpenCV-based processing of RGB-D image and point-cloud data, and calculating an alignment goal within a global map. The map is built with Google Cartographer and is based on onboard LIDAR, IMU, and GPS data. Motion control in the second stage is realized using the ROS Move Base package with Time-Elastic Band trajectory optimization. Visual servo algorithms guide the vehicle in local object-approach movement and the arm in manipulating bricks. To ensure a stable grasp of the brick’s magnetic patch, we developed a passively-compliant, electromagnetic gripper with tactile feedback. Our fully-autonomous UGV performed well in Challenge 2 and in post-competition evaluations of its brick pick-and-place algorithms.

An Image Synthesis Method Generating Underwater Images

The objective of this study is to convert normal aerial images into underwater images based on attenuation values for different water types by utilizing the image formation model (IFM) with Jerlov water types. Firstly, the depth values are derived from RGB-D images. If the depth information is not available, the values between 0.5 m to 10 m are chosen, and the transmission map is estimated by these values. Secondly, the statistical average background light values of Br = 0.6240, Bg = 0.805, and Bb = 0.7651 have been derived by analyzing 890 images using two methods, namely quad-tree decomposition and four-block division. Finally, the conversion of aerial-to-underwater images is done using the derived values, and the images are verified by computer simulation using MATLAB software. The result indicates that this method can easily generate underwater images from aerial images and makes it easier for the availability of ground truth.

A novel fuzzy clustering based method for image segmentation in RGB-D images

Automatic image segmentation is a challenging task in computer vision applications, especially in the presence of occluded objects, varying color, and different lighting conditions. The advancement of depth-sensing technologies has introduced RGB-Depth cameras which are capable to generate RGB-Depth images and brought significant changes in computer vision applications. However, the segmentation of RGB-Depth images is a difficult task. Therefore, in this paper, a new segmentation method for RGB-Depth images has been introduced and named as random Henry gas solubility optimization-fuzzy clustering method. Firstly, a random Henry gas solubility optimization algorithm has been developed. Next, the proposed optimization algorithm has been employed to obtain optimal fuzzy clusters which are finally merged through segmentation by aggregating superpixels. The standard NYU depth V2 RGB-Depth indoor image dataset is used for performance evaluation. The proposed segmentation approach has been compared with five different methods namely, kmeans, fuzzy c-means, Henry gas solubility optimization algorithm, chaotic gravitational search algorithm, and J-Segmentation in terms of qualitative and quantitative measures. The result analysis shows that the proposed RGB-D segmentation method outperforms the other considered methods.