RGB-D Images Research Articles

Nondestructive perch target detection and size measurement from RGB-D images in recirculating aquaculture system

Nondestructive perch target detection and size measurement from RGB-D images in recirculating aquaculture system

Estimation of Lower Limb Joint Angles Using sEMG Signals and RGB-D Camera.

Estimating human joint angles is a crucial task in motion analysis, gesture recognition, and motion intention prediction. This paper presents a novel model-based approach for generating reliable and accurate human joint angle estimation using a dual-branch network. The proposed network leverages combined features derived from encoded sEMG signals and RGB-D image data. To ensure the accuracy and reliability of the estimation algorithm, the proposed network employs a convolutional autoencoder to generate a high-level compression of sEMG features aimed at motion prediction. Considering the variability in the distribution of sEMG signals, the proposed network introduces a vision-based joint regression network to maintain the stability of combined features. Taking into account latency, occlusion, and shading issues with vision data acquisition, the feature fusion network utilizes high-frequency sEMG features as weights for specific features extracted from image data. The proposed method achieves effective human body joint angle estimation for motion analysis and motion intention prediction by mitigating the effects of non-stationary sEMG signals.

Rebar-tying Robot based on machine vision and coverage path planning

Automation technology can replace manual work in the traditional construction industry, improve quality and efficiency, and reduce costs. This paper developed a rebar-tying robot for tying the intersection of rebars. It proposed a Hough transform multi-segment fitting method to detect the intersections of rebars in real-time acquired RGB-D images. To cover the surface of the rebar net as much as possible under the condition of limited camera FOV, this paper designed a coverage path planning method to plan the path of the photo positions and the detected intersections of rebars efficiently and orderly. The experimental results show that the robot achieved an accuracy rate of 99.4 % in intersection detection, the detection error is within 2.8 mm, the single tying time is 1.85 s, and the average tying time is 5.5 s, which is faster than most robots. The robot realizes the task of automatically tying the intersection of rebars, which is robust and efficient, without duplication or omission.

Cherry Tomato Detection for Harvesting Using Multimodal Perception and an Improved YOLOv7-Tiny Neural Network

Robotic fruit harvesting has great potential to revolutionize agriculture, but detecting cherry tomatoes in farming environments still faces challenges in accuracy and efficiency. To overcome the shortcomings of existing cherry tomato detection methods for harvesting, this study introduces a deep-learning-based cherry tomato detection scheme for robotic harvesting in greenhouses using multimodal RGB-D perception and an improved YOLOv7-tiny Cherry Tomato Detection (YOLOv7-tiny-CTD) network, which has been modified from the original YOLOv7-tiny by eliminating the “Objectness” output layer, introducing a new “Classness” method for the prediction box, and incorporating a new hybrid non-maximum suppression. Acquired RGB-D images undergo preprocessing such as color space transformation, point cloud normal vector angle computation, and multimodal regions of interest segmentation before being fed into the YOLOv7-tiny-CTD. The proposed method was tested using an AGV-based robot in a greenhouse cherry tomato farming facility. The results indicate that the multimodal perception and deep learning method improves detection precision and accuracy over existing methods while running in real time, and the robot achieved over 80% successful picking rates in two-trial mode in the greenhouse farm, showing promising potential for practical harvesting applications.

A vision-based dietary survey and assessment system for college students in China

Understanding the current dietary habits of college students is essential due to the pressing issues of unbalanced diets and insufficient nutrition. However, traditional approaches frequently depend on recollection, which can introduce unconscious bias and make them difficult to implement. Herein, we introduce a new computer vision system to evaluate the dietary habits of college students in China. A specialized food dataset comprising RGB-D images, recipes with ingredient masses, and nutrient composition was created using data collected from college canteens. First, object detection models were utilized to identify food categories and locations. Subsequently, we introduced a method for automatically estimating the food volume of nonstandard portions using position and depth information. The final nutrients were derived directly or indirectly through the database. Experimental results demonstrate the applicability of the YOLOv8 object detection model and volume estimation method to our tasks. To support the development of devices for detecting food intake and diet management, we have made our dataset publicly available. The College Canteen Food-100 dataset is publicly available at https://github.com/zichengzichengzi/CC-FOOD-100.

Tree detection and in-row localization for autonomous precision orchard management

This work presents a framework for localizing ground robots within fruit tree orchards. The standard practice of managing orchards at the large block-level does not maximize the potential of farms — individual plants have different needs due to variations in soil, pests, disease, irrigation, etc. In order to make selective management decisions for individual trees, such as precision fertilization, a robot must be able to accurately localize itself within the row. This is a challenge since in high density, modern orchard systems it is often difficult to obtain accurate GNSS measurements. Our algorithm begins by using deep learning to segment a tree trunk in an RGB-D image and then estimate its width. We then use the trunk segmentations and widths to calculate particle weights in a particle filter-based localization system. We show that integrating trunk width into the particle update step led to a 45% decrease in the distance traveled before convergence, and a 31% decrease in convergence time, alongside a marginal increase in the rate of correct convergence. We also demonstrate autonomous tree-level localization with a large ground robot in realistic field experiments in a commercial apple orchard.

Ground4Act: Leveraging visual-language model for collaborative pushing and grasping in clutter

The challenge in robotics is to enable robots to transition from visual perception and language understanding to performing tasks such as grasp and assembling objects, bridging the gap between “seeing” and “hearing” to “doing”. In this work, we propose Ground4Act, a two-stage approach for collaborative pushing and grasping in clutter using a visual-language model. In the grounding stage, Ground4Act extracts target features from multi-modal data via visual grounding. In the action stage, it embeds a collaborative pushing and grasping framework to generate the action's position and direction. Specifically, we propose a DQN-based reinforcement learning pushing policy that uses RGBD images as the state space to determine the push action's pixel-level coordinates and direction. Additionally, a least squares-based linear fitting grasping policy takes the target mask from the grounding stage as input to achieve efficient grasp. Simulations and real-world experiments demonstrate Ground4Act's superior performance. The simulation suite, source code, and trained models will be made publicly available.

A pixel-level grasp detection method based on Efficient Grasp Aware Network

Abstract This work proposes a novel grasp detection method, the Efficient Grasp Aware Network (EGA-Net), for robotic visual grasp detection. Our method obtains semantic information for grasping through feature extraction. It efficiently obtains feature channel weights related to grasping tasks through the constructed ECA-ResNet module, which can smooth the network’s learning. Meanwhile, we use concatenation to obtain low-level features with rich spatial information. Our method inputs an RGB-D image and outputs the grasp poses and their quality score. The EGA-Net is trained and tested on the Cornell and Jacquard datasets, and we achieve 98.9% and 95.8% accuracy, respectively. The proposed method only takes 24 ms for real-time performance to process an RGB-D image. Moreover, our method achieved better results in the comparison experiment. In the real-world grasp experiments, we use a 6-degree of freedom (DOF) UR-5 robotic arm to demonstrate its robust grasping of unseen objects in various scenes. We also demonstrate that our model can successfully grasp different types of objects without any processing in advance. The experiment results validate our model’s exceptional robustness and generalization.

Category-level Pose Estimation and Iterative Refinement for Monocular RGB-D Image

Category-level Pose Estimation and Iterative Refinement for Monocular RGB-D Image

RGBTSDF: An Efficient and Simple Method for Color Truncated Signed Distance Field (TSDF) Volume Fusion Based on RGB-D Images

RGB-D image mapping is an important tool in applications such as robotics, 3D reconstruction, autonomous navigation, and augmented reality (AR). Efficient and reliable mapping methods can improve the accuracy, real-time performance, and flexibility of sensors in various fields. However, the currently widely used Truncated Signed Distance Field (TSDF) still suffers from the problem of inefficient memory management, making it difficult to directly use it for large-scale 3D reconstruction. In order to address this problem, this paper proposes a highly efficient and accurate TSDF voxel fusion method, RGBTSDF. First, based on the sparse characteristics of the volume, an improved grid octree is used to manage the whole scene, and a hard coding method is proposed for indexing. Second, during the depth map fusion process, the depth map is interpolated to achieve a more accurate voxel fusion effect. Finally, a mesh extraction method with texture constraints is proposed to overcome the effects of noise and holes and improve the smoothness and refinement of the extracted surface. We comprehensively evaluate RGBTSDF and similar methods through experiments on public datasets and the datasets collected by commercial scanning devices. Experimental results show that RGBTSDF requires less memory and can achieve real-time performance experience using only the CPU. It also improves fusion accuracy and achieves finer grid details.

Digital image correlation calculation method for RGB-D camera multi-view matching using variable template

Multi-view matching is an indispensable method for digital image correlation 3D reconstruction. However, as the camera viewing angle changes, traditional DIC methods can lead to the accumulation of matching errors and an increase in mismatches, thereby affecting the 3D reconstruction quality. This paper proposes a digital image correlation method for multi-view RGB-D images based on variable template matching. Initial positioning is achieved by dynamically adjusting the size and sampling direction of the variable template. Sub-pixel registration is then performed using the inverse compositional algorithm, followed by multi-view point cloud fusion using the Iterative Closest Point (ICP) registration method. Experimental results show that the iteration speed of point cloud registration is increased by 30%, and the cumulative error is reduced by 62.5%. These improvements demonstrate that the proposed method is suitable for multi-viewpoint matching scenarios, achieving excellent results in multi-view feature point matching and multi-view point cloud fusion, significantly enhancing 3D reconstruction accuracy and efficiency.

MaskRecon: High-quality human reconstruction via masked autoencoders using a single RGB-D image

In this paper, we explore reconstructing high-quality clothed 3D humans from a single RGB-D image, assuming that virtual humans can be represented by front-view and back-view depths. Due to the scarcity of captured real RGB-D human images, we employ rendered images to train our method. However, rendered images lack background with significant depth variation in silhouettes, leading to shape prediction inaccuracies and noise. To mitigate this issue, we introduce a pseudo-multi-task framework, which incorporates a Conditional Generative Adversarial Network (CGAN) to infer back-view RGB-D images and a self-supervised Masked Autoencoder (MAE) to capture latent structural information of the human body. Additionally, we propose a Multi-scale Feature Fusion (MFF) module to effectively merge structural information and conditional features at various scales. Our method surpasses many existing techniques, as demonstrated through evaluations on the Thuman, RenderPeople, and BUFF datasets. Notably, our approach excels in reconstructing high-quality human models, even under challenging conditions such as complex poses and loose clothing, both on rendered and real-world images. Codes are available at https://github.com/Archaic-Atom/MaskRecon.

A rich RGBD images captioning for scene understanding

A rich RGBD images captioning for scene understanding

Learning accurate and efficient three-finger grasp generation in clutters with an auto-annotated large-scale dataset

With the development of intelligent manufacturing and robotic technologies, the capability of grasping unknown objects in unstructured environments is becoming more prominent for robots with extensive applications. However, current robotic three-finger grasping studies only focus on grasp generation for single objects or scattered scenes, and suffer from high time expenditure to label grasp ground truth, making them incapable of predicting grasp poses for cluttered objects or generating large-scale datasets. To address such limitations, we first introduce a novel three-finger grasp representation with fewer prediction dimensions, which balances the training difficulty and representation accuracy to obtain efficient grasp performance. Based on this representation, we develop an auto-annotation pipeline and contribute a large-scale three-finger grasp dataset (TF-Grasp Dataset). Our dataset contains 222,720 RGB-D images with over 2 billion grasp annotations in cluttered scenes. In addition, we also propose a three-finger grasp pose detection network (TF-GPD), which detects globally while fine-tuning locally to predict high-quality collision-free grasps from a single-view point cloud. In sum, our work addresses the issue of high-quality collision-free three-finger grasp generation in cluttered scenes based on the proposed pipeline. Extensive comparative experiments show that our proposed methodology outperforms previous methods and improves the grasp quality and efficiency in clutters. The superior results in real-world robot grasping experiments not only prove the reliability of our grasp model but also pave the way for practical applications of three-finger grasping. Our dataset and source code will be released.

Enhancing Visual Odometry with Estimated Scene Depth: Leveraging RGB-D Data with Deep Learning

Advances in visual odometry (VO) systems have benefited from the widespread use of affordable RGB-D cameras, improving indoor localization and mapping accuracy. However, older sensors like the Kinect v1 face challenges due to depth inaccuracies and incomplete data. This study compares indoor VO systems that use RGB-D images, exploring methods to enhance depth information. We examine conventional image inpainting techniques and a deep learning approach, utilizing newer depth data from devices like the Kinect v2. Our research highlights the importance of refining data from lower-quality sensors, which is crucial for cost-effective VO applications. By integrating deep learning models with richer context from RGB images and more comprehensive depth references, we demonstrate improved trajectory estimation compared to standard methods. This work advances budget-friendly RGB-D VO systems for indoor mobile robots, emphasizing deep learning’s role in leveraging connections between image appearance and depth data.

Detection Method of Manipulator Grasp Pose Based on RGB-D Image

In order to better solve the visual detection problem of manipulator grasping non-cooperative targets, we propose a method of grasp pose detection based on pixel point and feature fusion. By using the improved U2net network as the backbone for feature extraction and feature fusion of the input image, and the grasp prediction layer detects the grasp pose on each pixel. In order to adapt the U2net to grasp pose detection and improve its detection performance, we improve detection speed and control sampling depth by simplifying its network structure, while retaining some shallow features in feature fusion to enhance its feature extraction capability. We introduce depthwise separable convolution in the grasp prediction layer, further fusing the features extracted from the backbone to obtain predictive feature maps with stronger feature expressiveness. FocalLoss is selected as the loss function to solve the problem of unbalanced positive and negative samples in network training. We use the Cornell dataset for training and testing, perform pixel-level labeling on the image, and replace the labels that are not conducive to the actual grasping. This adaptation helps the dataset better suit the network training and testing while meeting the real-world grasping requirements of the manipulator. The evaluation results on image-wise and object-wise are 95.65% and 91.20% respectively, and the detection speed is 0.007 s/frame. We also used the method for actual manipulator grasping experiments. The results show that our method has improved accuracy and speed compared with previous methods, and has strong generalization ability and portability.

Pyramid Deep Fusion Network for Two-Hand Reconstruction From RGB-D Images

Pyramid Deep Fusion Network for Two-Hand Reconstruction From RGB-D Images

Deep learning model for automatic limb detection and gripping in a novel meat factory cell

The goal of this work has been the development of a deep-learning model which may be used in conjunction with a novel so-called “Meat Factory Cell” platform, namely to enable an industrial robot within the system to identify and successfully grip the limbs of an entire pig carcass. The model consists of three main components: (1) a U-Net-based deep learning model that predicts heatmaps with a probability distribution of gripping and key point locations on the limbs, within RGB-D images; (2) a post-processing element for the extraction of keypoints from heatmaps, and transferral of these points into 3D space using a pinhole camera model; and (3) gripper orientation estimation, which uses the predicted limb key points to define gripper orientation in 3D space. The proposed system demonstrates high precision and robustness in estimating gripping points on pig limbs based on a data test set, which includes two gripping definitions: Norwegian and Danish. These gripping definitions account for variation in the slaughter process in two different European countries. The Norwegian definition gives mAP(0.5…0.95) = 0.971, mAR(0.5…0.95) = 0.982, and distance error 13 mm, while the Danish definition gives mAP(0.5…0.95) = 0.985, mAR(0.5…0.95) = 0.995, and distance error 14 mm. The model was validated in practice during experimental trials at the Meat Factory Cell test facility at the Norwegian University of Life Science (Ås, Norway), with whole pig carcasses (n=25).

Zero‐Shot 3D Pose Estimation of Unseen Object by Two‐step RGB-D Fusion

3D Object pose estimation is a critical task in many real-world applications, e.g., robotic manipulation and augmented reality. Most existing methods focus on estimating the object instances or categories which have been seen in the training phase. However, it is imperative to estimate the pose of unseen objects without re-training the network in real world. Therefore, we proposed a 3D pose estimation method for unseen objects without re-training. Specifically, given the CAD model of the unseen object, a set of template RGB-D images (RGB images and depth images) is rendered at different viewpoints. Then a feature embedding network, named PoseFusion, is designed to extract the scene feature. In this network, RGB-D images are utilized to extract the texture feature and geometric feature, respectively. Afterwards, a cross-modality alignment module is proposed to eliminate the noise in single modality. The aligned texture feature and aligned geometric feature are fused through a geometry guided fusion module. Thus, by PoseFusion, the template RGB-D images generated from the CAD model are abstracted into a set of template scene features, and the query scene features are also embedded from the captured RGB-D images from the unseen object. Finally, the query scene features are matched with the template scene features by calculating the masked local similarity. Then the identity and pose of unseen object are determined by the most similar template. Experiments on LINEMOD and T-LESS datasets demonstrate that our method outperforms other methods and generalizes better to unseen objects. Extensive ablation studies are performed to verify the effectiveness of the PoseFusion.

An end-to-end learning framework for visual camera relocalization using RGB and RGB-D images

Camera relocalization plays a vital role in the realms of machine perception, robotics, and augmented reality. Direct learning methods based on structures can have a learning-based approach that can learn scene coordinates and use them for camera position estimation. However, the two-stage learning of scene coordinate regression and camera position estimation can result in some of the scene coordinate regression knowledge being lost throughout the learning process of the final pose estimation system, thereby reducing the accuracy of the pose estimation. This paper introduces an innovative end-to-end learning framework tailored for visual camera relocalization by employing both RGB and RGB-D images. Distinguished by its integration of scene coordinate regression with pose estimation into a concurrent inner and outer loop during a singular training phase, this framework notably enhances pose estimation accuracy. Engineered for flexibility, it accommodates training with or without depth cues and necessitates merely a single RGB image during testing. Empirical evaluation substantiates the proposed method’s state-of-the-art precision, attaining an average pose accuracy of 0.019 m and 0.74° on the indoor 7Scenes dataset, together with 0.162 m and 0.30° on the outdoor Cambridge Landmarks dataset.