Red, Green, Blue And Depth Research Articles

PGGNet: Pyramid gradual-guidance network for RGB-D indoor scene semantic segmentation

In RGB-D (red–green–blue and depth) scene semantic segmentation, depth maps provide rich spatial information to RGB images to achieve high performance. However, properly aggregating depth information and reducing noise and information loss during feature encoding after fusion are challenging aspects in scene semantic segmentation. To overcome these problems, we propose a pyramid gradual-guidance network for RGB-D indoor scene semantic segmentation. First, the quality of depth information is improved by a modality-enhancement fusion module and RGB image fusion. Then, the representation of semantic information is improved by multiscale operations. The two resulting adjacent features are used in a feature refinement module with an attention mechanism to extract semantic information. The features from adjacent modules are successively used to form an encoding pyramid, which can substantially reduce information loss and thereby ensure information integrity. Finally, we gradually integrate features at the same scale obtained from the encoding pyramid during decoding to obtain high-quality semantic segmentation. Experimental results obtained from two commonly used indoor scene datasets demonstrate that the proposed pyramid gradual-guidance network attains the highest level of performance in semantic segmentation, as compared to other existing methods.

Sweet Pepper Leaf Area Estimation Using Semantic 3D Point Clouds Based on Semantic Segmentation Neural Network

In the field of agriculture, measuring the leaf area is crucial for the management of crops. Various techniques exist for this measurement, ranging from direct to indirect approaches and destructive to non-destructive techniques. The non-destructive approach is favored because it preserves the plant's integrity. Among these, several methods utilize leaf dimensions, such as width and length, to estimate leaf areas based on specific models that consider the unique shapes of leaves. Although this approach does not damage plants, it is labor-intensive, requiring manual measurements of leaf dimensions. In contrast, some indirect non-destructive techniques leveraging convolutional neural networks can predict leaf areas more swiftly and autonomously. In this paper, we propose a new direct method using 3D point clouds constructed by semantic RGB-D (Red Green Blue and Depth) images generated by a semantic segmentation neural network and RGB-D images. The key idea is that the leaf area is quantified by the count of points depicting the leaves. This method demonstrates high accuracy, with an R2 value of 0.98 and a RMSE (Root Mean Square Error) value of 3.05 cm2. Here, the neural network's role is to segregate leaves from other plant parts to accurately measure the leaf area represented by the point clouds, rather than predicting the total leaf area of the plant. This method is direct, precise, and non-invasive to sweet pepper plants, offering easy leaf area calculation. It can be implemented on laptops for manual use or integrated into robots for automated periodic leaf area assessments. This innovative method holds promise for advancing our understanding of plant responses to environmental changes. We verified the method's reliability and superior performance through experiments on individual leaves and whole plants.

AMCFNet: Asymmetric multiscale and crossmodal fusion network for RGB-D semantic segmentation in indoor service robots

Red-green-blue and depth (RGB-D) semantic segmentation is essential for indoor service robots to achieve accurate artificial intelligence. Various RGB-D indoor semantic segmentation methods have been proposed since the widespread adoption of depth maps. These methods have focused mainly on integrating the multiscale and crossmodal features extracted from RGB images and depth maps in the encoder and used unified strategies to recover the local details at the decoder progressively. However, these methods emphasized crossmodal fusion at the encoder, neglecting the distinguishability between RGB and depth features during decoding, thereby undermining the segmentation performance. To adequately exploit the features, we propose an efficient encoder-decoder architecture called asymmetric multiscale and crossmodal fusion network (AMCFNet) endowed with a differential feature integration strategy. Unlike existing methods, we use simple crossmodal fusion at the encoder and design an elaborate decoder to improve the semantic segmentation performance. Specifically, considering high- and low-level features, we propose a semantic aggregation module (SAM) to process the multiscale and crossmodal features in the last three network layers to aggregate high-level semantic information through a cascaded pyramid structure. Moreover, we design a spatial detail supplement module using low-level spatial details from depth maps to adaptively fuse these details and the information obtained from the SAM. Extensive experiments are conducted to demonstrate that the proposed AMCFNet outperforms state-of-the-art approaches.

Improving Lettuce Fresh Weight Estimation Accuracy through RGB-D Fusion

Computer vision provides a real-time, non-destructive, and indirect way of horticultural crop yield estimation. Deep learning helps improve horticultural crop yield estimation accuracy. However, the accuracy of current estimation models based on RGB (red, green, blue) images does not meet the standard of a soft sensor. Through enriching more data and improving the RGB estimation model structure of convolutional neural networks (CNNs), this paper increased the coefficient of determination (R2) by 0.0284 and decreased the normalized root mean squared error (NRMSE) by 0.0575. After introducing a novel loss function mean squared percentage error (MSPE) that emphasizes the mean absolute percentage error (MAPE), the MAPE decreased by 7.58%. This paper develops a lettuce fresh weight estimation method through the multi-modal fusion of RGB and depth (RGB-D) images. With the multimodal fusion based on calibrated RGB and depth images, R2 increased by 0.0221, NRMSE decreased by 0.0427, and MAPE decreased by 3.99%. With the novel loss function, MAPE further decreased by 1.27%. A MAPE of 8.47% helps to develop a soft sensor for lettuce fresh weight estimation.

THCANet: Two-layer hop cascaded asymptotic network for robot-driving road-scene semantic segmentation in RGB-D images

In several existing red–green–blue and depth (RGB-D) semantic segmentation algorithms, schemes are used to supplement contextual information through multilayer feature interactions. However, these approaches ignore the complementation of the contextual information and the introduction of noise interfering with the segmentation process. To minimize noise interference during this process, we introduce a two-layer hop cascaded asymptotic network (THCANet) for robot-driving road-scene semantic segmentation in RGB-D images. To exploit the depth map and supervision to strengthen semantic segmentation, we propose an attention cross-fusion module for the interactive combination of RGB-D features through multimodality weighting. Notably, the information of the two modalities reduces noise during fusion. After fusing features from the RGB-D modalities, we also use a novel multiscale context module to fuse features at multiple scales and employ a jump cascade architecture between the modules to recover lost context information and suppress irrelevant noise. Moreover, multiple supervision is performed at different segmentation stages to improve accuracy. The proposed THCANet system demonstrates the best performance on a robot-driving road dataset compared with similar methods, and its generalization ability is demonstrated using the NYU-Depth V2 dataset.

Development of 3D Scanner Application with Stereo Camera for 3D Object Reconstruction

Visual perception in RGBD (Red-Green-Blue and Depth) camera technology allows the camera to provide color image and depth structure information simultaneously from the surface of the scanned object. Based on this technology, data from an RGBD camera can be processed to reconstruct the scanned object in a three-dimensional (3D) model. However, devices that are prebuilt to perform this task with good accuracy are expensive and are only intended for 3D scanning tasks. On the other hand, some multi-purpose depth cameras can be used for 3D scanning but require some tweaks to ensure a good and accurate reconstructed model. A method has been proposed to optimize the camera and reconstruction system to compile RGBD information from recorded close-up objects and process them into a full-scale 3D digital model. This process involves developing the appropriate recording method along with settings and adjustments for scanning and model reconstruction of close-range objects. As a result, information from multi-purpose RGBD camera can be reconstructed in three-dimensional form that accurately represents real-life objects. The reconstructed model can be utilized for applications that rely on real-life objects as their main basis. The proposed method enables the use of inexpensive multi-purpose RGBD cameras for 3D scanning.

Point Cloud Completion of Plant Leaves under Occlusion Conditions Based on Deep Learning.

The utilization of 3-dimensional point cloud technology for non-invasive measurement of plant phenotypic parameters can furnish important data for plant breeding, agricultural production, and diverse research applications. Nevertheless, the utilization of depth sensors and other tools for capturing plant point clouds often results in missing and incomplete data due to the limitations of 2.5D imaging features and leaf occlusion. This drawback obstructed the accurate extraction of phenotypic parameters. Hence, this study presented a solution for incomplete flowering Chinese Cabbage point clouds using Point Fractal Network-based techniques. The study performed experiments on flowering Chinese Cabbage by constructing a point cloud dataset of their leaves and training the network. The findings demonstrated that our network is stable and robust, as it can effectively complete diverse leaf point cloud morphologies, missing ratios, and multi-missing scenarios. A novel framework is presented for 3D plant reconstruction using a single-view RGB-D (Red, Green, Blue and Depth) image. This method leveraged deep learning to complete localized incomplete leaf point clouds acquired by RGB-D cameras under occlusion conditions. Additionally, the extracted leaf area parameters, based on triangular mesh, were compared with the measured values. The outcomes revealed that prior to the point cloud completion, the R2 value of the flowering Chinese Cabbage's estimated leaf area (in comparison to the standard reference value) was 0.9162. The root mean square error (RMSE) was 15.88 cm2, and the average relative error was 22.11%. However, post-completion, the estimated value of leaf area witnessed a significant improvement, with an R2 of 0.9637, an RMSE of 6.79 cm2, and average relative error of 8.82%. The accuracy of estimating the phenotypic parameters has been enhanced significantly, enabling efficient retrieval of such parameters. This development offers a fresh perspective for non-destructive identification of plant phenotypes.

An improved cascade R-CNN and RGB-D camera-based method for dynamic cotton top bud recognition and localization in the field

Cotton top cutting is an indispensable part of cotton planting. Cotton top bud detection and localization are highly challenging tasks because of tiny targets, dense growth and varying illumination. To achieve the automatic cutting of cotton buds in the field, a cotton top bud recognition and location algorithm adapted to a moving platform in the field based on images acquired by a red, green, blue and depth (RGB-D) camera was developed. In this study, an improved Cascade R-CNN network was proposed to detect cotton top bud regions on RGB images, and three-dimensional (3D) coordinates of targets were obtained by combining color images and depth images from RGB-D cameras. The 3D spatial position of the target in the world coordinate system was affected by the time of the cotton top bud recognition algorithm, the forward speed of the mobile platform and the time consumption of the manipulator moving to the target position. A dynamic compensation method of target coordinates in the moving direction was proposed to ensure the identification and positioning accuracy of cotton top buds in the moving process. To verify the effectiveness of the proposed algorithm, cotton top bud recognition and localization experiments were conducted in the field. The average precision with the proposed improved Cascade R-CNN model was 97.5 %, and the FPS was 13.3 frames per second, which was suitable for different period of a day. The average error of the positioning accuracy in the platform forwarding direction was 4.2 mm, 6.6 mm and 9.6 mm at speeds of 0.1 m/s, 0.2 m/s and 0.3 m/s, respectively. All the results demonstrate that the proposed method could be used for robotic cotton top cutting.

An RGB‐D framework for capturing soft‐sediment microtopography

Abstract Capturing sediment topography is one component in understanding how large animals in soft sediment ecosystems affect ecosystem processing. Traditional approaches for sampling soft‐sediment systems are too invasive and destructive for the fragile sediment structures. In contrast, non‐invasive approaches, such as LIDAR and monocular structure‐from‐motion (SfM) systems, can be expensive and cumbersome for data processing. We developed a low cost and practical framework for measuring morphological characteristics of soft sediment topography at the millimetre scale. Using an RGB‐D (red, green, blue and depth) device and a semi‐opaque container, the sediment surface can be imaged rapidly while operating in outdoor environments. The RGB‐D device imaged 0.3 m2 intertidal sediment surfaces, creating depth imagery over seven sites and 82 surfaces. First, using simulated surface data, we formulated and tested four variations of a geometrical detrending model to extract and measure the undulating surface microtopography. We then computed nine morphological characteristics, including arithmetic mean roughness, mean sediment peaks and rugosity, for the detrending models. We then used the two best performing models, linear and quadratic detrending models, to compare extracted morphological characteristics from the field data. We found a strong correlation between the models for extracted surface measures and similar extracted sediment trends. The results showed that RGB‐D real‐time imaging is a promising rapid scanning tool for collecting field data in intertidal regions and can be expanded to other fragile sediment surfaces outside the marine environment. The low costs and real‐time feedback makes it an attractive data collection tool for environments where data collection is challenging.

Fruit recognition method for a harvesting robot with RGB-D cameras

In this study, we present a recognition method for a fruit-harvesting robot to automate the harvesting of pears and apples on joint V-shaped trellis. It is necessary to recognize the three-dimensional position of the harvesting target for harvesting by the fruit-harvesting robot to insert its end-effector. However, the RGB-D (red, green, blue and depth) camera on the harvesting robot has a problem in that the point cloud obtained in outdoor environments can be inaccurate. Therefore, in this study, we propose an effective method for the harvesting robot to recognize fruits using not only three-dimensional information obtained from the RGB-D camera but also two-dimensional images and information from the camera. Furthermore, we report a method for determining the ripeness of pears using the information on fruit detection. Through experiments, we confirmed that the proposed method satisfies the accuracy required for a harvesting robot to continuously harvest fruits.

Automating Aircraft Scanning for Inspection or 3D Model Creation with a UAV and Optimal Path Planning

Visual inspections of aircraft exterior surfaces are required in aircraft maintenance routines for identifying possible defects such as dents, cracks, leaking, broken or missing parts, etc. This process is time-consuming and is also prone to error if performed manually. Therefore, it has become a trend to use mobile robots equipped with visual sensors to perform automated inspections. For such a robotic inspection, a digital model of the aircraft is usually required for planning the robot’s path, but a CAD model of the entire aircraft is usually inaccessible to most maintenance shops. It is very labor-intensive and time-consuming to generate an accurate digital model of an aircraft, or even a large portion of it, because the scanning work still must be performed manually or by a manually controlled robotic system. This paper presents a two-stage approach of automating aircraft scanning with an unmanned aerial vehicle (UAV) or autonomous drone equipped with a red–green–blue and depth (RGB-D) camera for detailed inspection or for reconstructing a digital replica of the aircraft when its original CAD model is unavailable. In the first stage, the UAV–camera system follows a predefined path far from the aircraft surface (for safety) to quickly scan the aircraft and generate a coarse model of the aircraft. Then, an optimal scanning path (much closer to the surface) in the sense of the shortest flying distance for full coverage is computed based on the coarse model. In the second stage, the UAV–camera system follows the computed path to closely inspect the surface for possible defects or scan the surface for generating a dense and precise model of the aircraft. We solved the coverage path planning (CPP) problem for the aircraft inspection or scanning using a Monte Carlo tree search (MCTS) algorithm. We also implemented the max–min ant system (MMAS) strategy to demonstrate the effectiveness of our approach. We carried out a digital experiment and the results showed that our approach can scan 70% of the aircraft surface within one hour, which is much more efficient than manual scanning.

3D Instance Segmentation Using Deep Learning on RGB-D Indoor Data

3D object recognition is a challenging task for intelligent and robot systems in industrial and home indoor environments. It is critical for such systems to recognize and segment the 3D object instances that they encounter on a frequent basis. The computer vision, graphics, and machine learning fields have all given it a lot of attention. Traditionally, 3D segmentation was done with hand-crafted features and designed approaches that didn’t achieve acceptable performance and couldn’t be generalized to large-scale data. Deep learning approaches have lately become the preferred method for 3D segmentation challenges by their great success in 2D computer vision. However, the task of instance segmentation is currently less explored. In this paper, we propose a novel approach for efficient 3D instance segmentation using red green blue and depth (RGB-D) data based on deep learning. The 2D region based convolutional neural networks (Mask R-CNN) deep learning model with point based rending module is adapted to integrate with depth information to recognize and segment 3D instances of objects. In order to generate 3D point cloud coordinates (x, y, z), segmented 2D pixels (u, v) of recognized object regions in the RGB image are merged into (u, v) points of the depth image. Moreover, we conducted an experiment and analysis to compare our proposed method from various points of view and distances. The experimentation shows the proposed 3D object recognition and instance segmentation are sufficiently beneficial to support object handling in robotic and intelligent systems.

Fast detection and location of longan fruits using UAV images

In agriculture, fruit picking robots on the ground have difficulty adapting to the terrain conditions of mountain orchards and cannot pick longan fruit from tall longan trees. In this paper, aiming to allow picking of longan fruit by unmanned aerial vehicles (UAVs), a deep learning-based scheme to quickly and accurately detect and locate suitable picking points on fruit branches is proposed. The scheme includes a UAV fuzzy image preprocessing method, longan detection based on a convolutional neural network (CNN), red, green, blue and depth (RGB-D) information fusion and an accurate target location strategy. First, the UAV is equipped with an Intel Realsense D455 camera, which collects longan images from the front for training and testing the model. Second, the lightweight MobileNet backbone network is used to improve the performance of the You Only Look Once version 4 (YOLOv4) model in feature extraction. The results for the test set show that compared with the classical feature pyramid network (FPN), YOLOv3 and YOLOv4 models, this model reduces the computation, parameters and detection time of the model. Compared with MobileNet single-shot multibox detector (MobileNet-SSD) and YOLOv4-tiny, this model exhibits improved detection accuracy. Third, according to the target detection result map, a strategy is formulated to accurately determine the suitable picking point on the main branch of the result. Finally, the performance of the improved model and picking platform in the harvest scene is evaluated by performing picking experiments in a longan orchard. In summary, we fully exploit the advantages of the combination of UAVs, RGB-D cameras and CNNs to improve the speed and accuracy of target detection and location for longan picking by UAVs based on vision.

Suction-based Grasp Point Estimation in Cluttered Environment for Robotic Manipulator Using Deep Learning-based Affordance Map

Perception and manipulation tasks for robotic manipulators involving highly-cluttered objects have become increasingly indemand for achieving a more efficient problem solving method in modern industrial environments. But, most of the available methods for performing such cluttered tasks failed in terms of performance, mainly due to inability to adapt to the change of the environment and the handled objects. Here, we propose a new, near real-time approach to suction-based grasp point estimation in a highly cluttered environment by employing an affordance-based approach. Compared to the state-of-the-art, our proposed method offers two distinctive contributions. First, we use a modified deep neural network backbone for the input of the semantic segmentation, to classify pixel elements of the input red, green, blue and depth (RGBD) channel image which is then used to produce an affordance map, a pixel-wise probability map representing the probability of a successful grasping action in those particular pixel regions. Later, we incorporate a high speed semantic segmentation to the system, which makes our solution have a lower computational time. This approach does not need to have any prior knowledge or models of the objects since it removes the step of pose estimation and object recognition entirely compared to most of the current approaches and uses an assumption to grasp first then recognize later, which makes it possible to have an object-agnostic property. The system was designed to be used for household objects, but it can be easily extended to any kind of objects provided that the right dataset is used for training the models. Experimental results show the benefit of our approach which achieves a precision of 88.83%, compared to the 83.4% precision of the current state-of-the-art.

Multiscale multilevel context and multimodal fusion for RGB-D salient object detection

Red–green–blue and depth (RGB-D) saliency detection has recently attracted much research attention; however, the effective use of depth information remains challenging. This paper proposes a method that leverages depth information in clear shapes to detect the boundary of salient objects. As context plays an important role in saliency detection, the method incorporates a proposed end-to-end multiscale multilevel context and multimodal fusion network (MCMFNet) to aggregate multiscale multilevel context feature maps for accurate saliency detection from objects of varying sizes. Finally, a coarse-to-fine approach is applied to an attention module retrieving multilevel and multimodal feature maps to produce the final saliency map. A comprehensive loss function is also incorporated in MCMFNet to optimize the network parameters. Extensive experiments demonstrate the effectiveness of the proposed method and its substantial improvement over state-of-the-art methods for RGB-D salient object detection on four representative datasets.

A New Edge Patch with Rotation Invariance for Object Detection and Pose Estimation

Local patch-based methods of object detection and pose estimation are promising. However, to the best of the authors’ knowledge, traditional red-green-blue and depth (RGB-D) patches contain scene interference (foreground occlusion and background clutter) and have little rotation invariance. To solve these problems, a new edge patch is proposed and experimented with in this study. The edge patch is a local sampling RGB-D patch centered at the edge pixel of the depth image. According to the normal direction of the depth edge, the edge patch is sampled along a canonical orientation, making it rotation invariant. Through a process of depth detection, scene interference is eliminated from the edge patch, which improves the robustness. The framework of the edge patch-based method is described, and the method was evaluated on three public datasets. Compared with existing methods, the proposed method achieved a higher average F1-score (0.956) on the Tejani dataset and a better average detection rate (62%) on the Occlusion dataset, even in situations of serious scene interference. These results showed that the proposed method has higher detection accuracy and stronger robustness.

A High-Accuracy Indoor-Positioning Method with Automated RGB-D Image Database Construction

High-accuracy indoor positioning is a prerequisite to satisfy the increasing demands of position-based services in complex indoor scenes. Current indoor visual-positioning methods mainly include image retrieval-based methods, visual landmarks-based methods, and learning-based methods. To better overcome the limitations of traditional methods such as them being labor-intensive, of poor accuracy, and time-consuming, this paper proposes a novel indoor-positioning method with automated red, green, blue and depth (RGB-D) image database construction. First, strategies for automated database construction are developed to reduce the workload of manually selecting database images and ensure the requirements of high-accuracy indoor positioning. The database is automatically constructed according to the rules, which is more objective and improves the efficiency of the image-retrieval process. Second, by combining the automated database construction module, convolutional neural network (CNN)-based image-retrieval module, and strict geometric relations-based pose estimation module, we obtain a high-accuracy indoor-positioning system. Furthermore, in order to verify the proposed method, we conducted extensive experiments on the public indoor environment dataset. The detailed experimental results demonstrated the effectiveness and efficiency of our indoor-positioning method.

Automatic recognition of juicy peaches on trees based on 3D contour features and colour data

In recent years, fruit harvesting robots have become a research hotspot for agricultural machinery. To be effective, robots should quickly and accurately detect fruits on trees in the complex environments. In this study, a method using an RGB-D (Red, green, blue, and depth space) camera combining colour data and 3D contour features for the detection of juicy peach was developed based on clustering and model segmentation. The point cloud of fruits was acquired in realistic field conditions. The pre-processed point cloud was divided into clusters by the conditional Euclidean clustering algorithm, and the 3D contour features were then used to extract fruits. The random sample consensus (RANSAC) spherical segmentation method was applied to clusters considered to contain multiple fruits that have close distance with one another. The fruit point cloud obtained is incomplete due to the limited range of camera view and actual occlusions in orchard, these clusters are considered to contain only one fruit. The progressive sample consensus (PROSAC) circular segmentation method was exploited to improve accuracy. Experiments were conducted under three situations with different occlusions. The total recognition accuracy reached 88.68%, and the average recognition processing time for a single fruit was about 292.4 ms, it can meet the requirements for fruit detection considering that the harvesting robot mainly takes the ripe peaches as object.

Wearable Travel Aid for Environment Perception and Navigation of Visually Impaired People

Assistive devices for visually impaired people (VIP) which support daily traveling and improve social inclusion are developing fast. Most of them try to solve the problem of navigation or obstacle avoidance, and other works focus on helping VIP to recognize their surrounding objects. However, very few of them couple both capabilities (i.e., navigation and recognition). Aiming at the above needs, this paper presents a wearable assistive device that allows VIP to (i) navigate safely and quickly in unfamiliar environment, and (ii) to recognize the objects in both indoor and outdoor environments. The device consists of a consumer Red, Green, Blue and Depth (RGB-D) camera and an Inertial Measurement Unit (IMU), which are mounted on a pair of eyeglasses, and a smartphone. The device leverages the ground height continuity among adjacent image frames to segment the ground accurately and rapidly, and then search the moving direction according to the ground. A lightweight Convolutional Neural Network (CNN)-based object recognition system is developed and deployed on the smartphone to increase the perception ability of VIP and promote the navigation system. It can provide the semantic information of surroundings, such as the categories, locations, and orientations of objects. Human–machine interaction is performed through audio module (a beeping sound for obstacle alert, speech recognition for understanding the user commands, and speech synthesis for expressing semantic information of surroundings). We evaluated the performance of the proposed system through many experiments conducted in both indoor and outdoor scenarios, demonstrating the efficiency and safety of the proposed assistive system.

Real-World Multiobject, Multigrasp Detection

A deep learning architecture is proposed to predict graspable locations for robotic manipulation. It considers situations where no, one, or multiple object(s) are seen. By defining the learning problem to be classified with null hypothesis competition instead of regression, the deep neural network with red, green, blue and depth (RGB-D) image input predicts multiple grasp candidates for a single object or multiple objects, in a single shot. The method outperforms state-of-the-art approaches on the Cornell dataset with 96.0% and 96.1% accuracy on imagewise and object-wise splits, respectively. Evaluation on a multiobject dataset illustrates the generalization capability of the architecture. Grasping experiments achieve 96.0% grasp localization and 89.0% grasping success rates on a test set of household objects. The real-time process takes less than 0.25 s from image to plan.