Consumer-grade RGB-D Cameras Research Articles

Automated microgreen phenotyping for yield estimation using a consumer-grade depth camera

Microgreens are the first leafy seedlings of edible plants. Microgreen farming is yet to be automated; the main challenge for automation is the lack of a sensory mechanism to detect and quantify microgreen phenotypes. This paper presents a novel automated microgreen phenotyping method targeting yield estimation. The paper demonstrates that phenotyping can be effectively performed using a consumer-grade RGB-D camera. First, the depth and RGB images are captured. Thereafter, the plant segments are filtered and the canopy is identified. Using image processing, the canopy height and density are calculated. Both yield prediction regression analysis and a TensorFlow learning algorithm are evaluated to estimate the yield as a function of height and canopy density. The authors believe the algorithm discussed in this paper is the first phenotyping algorithm combining RGB and depth data for microgreen yield estimation.

Dynamic detection of three-dimensional crop phenotypes based on a consumer-grade RGB-D camera.

Nondestructive detection of crop phenotypic traits in the field is very important for crop breeding. Ground-based mobile platforms equipped with sensors can efficiently and accurately obtain crop phenotypic traits. In this study, we propose a dynamic 3D data acquisition method in the field suitable for various crops by using a consumer-grade RGB-D camera installed on a ground-based movable platform, which can collect RGB images as well as depth images of crop canopy sequences dynamically. A scale-invariant feature transform (SIFT) operator was used to detect adjacent date frames acquired by the RGB-D camera to calculate the point cloud alignment coarse matching matrix and the displacement distance of adjacent images. The data frames used for point cloud matching were selected according to the calculated displacement distance. Then, the colored ICP (iterative closest point) algorithm was used to determine the fine matching matrix and generate point clouds of the crop row. The clustering method was applied to segment the point cloud of each plant from the crop row point cloud, and 3D phenotypic traits, including plant height, leaf area and projected area of individual plants, were measured. We compared the effects of LIDAR and image-based 3D reconstruction methods, and experiments were carried out on corn, tobacco, cottons and Bletilla striata in the seedling stage. The results show that the measurements of the plant height (R²= 0.9~0.96, RSME = 0.015~0.023 m), leaf area (R²= 0.8~0.86, RSME = 0.0011~0.0041 m 2 ) and projected area (R² = 0.96~0.99) have strong correlations with the manual measurement results. Additionally, 3D reconstruction results with different moving speeds and times throughout the day and in different scenes were also verified. The results show that the method can be applied to dynamic detection with a moving speed up to 0.6m/s and can achieve acceptable detection results in the daytime, as well as at night. Thus, the proposed method can improve the efficiency of individual crop 3D point cloud data extraction with acceptable accuracy, which is a feasible solution for crop seedling 3D phenotyping outdoors.

3D real-time human reconstruction with a single RGBD camera.

3D human reconstruction is an important technology connecting the real world and the virtual world, but most of previous work needs expensive computing resources, making it difficult in real-time scenarios. We propose a lightweight human body reconstruction system based on parametric model, which employs only one RGBD camera as input. To generate a human model end to end, we build a fast and lightweight deep-learning network named Fast Body Net (FBN). The network pays more attention on the face and hands to enrich the local details. Additionally, we train a denoising auto-encoder to reduce unreasonable states of human model. Due to the lack of human dataset based on RGBD images, we propose an Indoor-Human dataset to train the network, which contains a total of 2500 frames of action data of five actors collected by Azure Kinect camera. Depth images avoid using RGB to extract depth features, which makes FBN lightweight and high-speed in reconstructing parametric human model. Qualitative and quantitative analysis on experimental results show that our method can improve at least 57% in efficiency with similar accuracy, as compared to state-of-the-art methods. Through our study, it is also demonstrated that consumer-grade RGBD cameras can provide great applications in real-time display and interaction for virtual reality.

Depth distortion correction for consumer-grade depth cameras in crop reconstruction

Modern consumer-grade RGB-D cameras provide intensive depth estimation and high frame rates. They have been widely used in agriculture. However, depth anamorphose occurs when using the RGB-D camera. In order to address this issue, this paper proposes a novel approach to correct the distorted depth images that fit the relationship between the true distances and the depth values from depth images. This study considers the structured light camera ASUS Xtion PRO LIVE as example to develop a system for obtaining a series of depth images at different distances from the target plane. A comparison analysis is conducted between the images before and after correction to evaluate the performance of the distortion correction. The point cloud image of corrected maize leaves is well coincident with the original plant. This approach improves the accuracy of depth measurement and optimizes the subsequent use of the depth camera in crop reconstruction and phenotyping studies.

High-Resolution Depth Maps Imaging via Attention-Based Hierarchical Multi-Modal Fusion.

Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging issue for guided DSR is how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhancement block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.

Deep neural networks for grape bunch segmentation in natural images from a consumer-grade camera

Precision agriculture relies on the availability of accurate knowledge of crop phenotypic traits at the sub-field level. While visual inspection by human experts has been traditionally adopted for phenotyping estimations, sensors mounted on field vehicles are becoming valuable tools to increase accuracy on a narrower scale and reduce execution time and labor costs, as well. In this respect, automated processing of sensor data for accurate and reliable fruit detection and characterization is a major research challenge, especially when data consist of low-quality natural images. This paper investigates the use of deep learning frameworks for automated segmentation of grape bunches in color images from a consumer-grade RGB-D camera, placed on-board an agricultural vehicle. A comparative study, based on the estimation of two image segmentation metrics, i.e. the segmentation accuracy and the well-known Intersection over Union (IoU), is presented to estimate the performance of four pre-trained network architectures, namely the AlexNet, the GoogLeNet, the VGG16, and the VGG19. Furthermore, a novel strategy aimed at improving the segmentation of bunch pixels is proposed. It is based on an optimal threshold selection of the bunch probability maps, as an alternative to the conventional minimization of cross-entropy loss of mutually exclusive classes. Results obtained in field tests show that the proposed strategy improves the mean segmentation accuracy of the four deep neural networks in a range between 2.10 and 8.04%. Besides, the comparative study of the four networks demonstrates that the best performance is achieved by the VGG19, which reaches a mean segmentation accuracy on the bunch class of 80.58%, with IoU values for the bunch class of 45.64%.

Volume estimation of cassava using consumer-grade RGB-D camera

Mismanagement during postharvest handling of cassava can degrade the quality of the product and depreciate its selling price considerably. This study proposed the feasibility of using RGB-depth camera to measure the quality of cassava roots in a non-destructive, fast and cost-effective manner. Methodology to estimate the volume of cassavas Kasetsart 50 variety was the focus of this study. Using RGB-D images collected from 60 cassava samples with each one being photographed from 6 different orientations. Image Processing model and Point Cloud image model were used to find the volume from depth images, and then disk method and box method were used to estimate the volume of cassava under ellipsoidal shape. Both estimation methods provided usable values for the volumes in the range of 100 - 500 ml with RMSE values of 5.91% and 4.02%, respectively. The estimated volume can be applied to find density to predict the rotten cassava for improving quality and efficiency of cassava industry.

基于彩色图像中感兴趣区域的苹果点云分割方法

Autonomous harvesting and evaluation of apples reduce the labour cost. Segmentation of apple point clouds from consumer-grade RGB-D camera is the most important and challenging step in the harvesting process due to the complex structure of apple trees. This paper put forward a segmentation method of apple point clouds based on regions of interest (ROI) in RGB images. Firstly, an annotated RGB dataset of apple trees was built and applied to train the optimized Faster R-CNN to locate ROI containing apples in RGB images. Secondly, the relationship between RGB images and depth images was built to roughly segment the apple point clouds by ROI. Finally, the quality control procedure (QCP) was proposed to improve the quality of segmented apple point clouds. Images for training mainly included two lighting condition, two colours and three apple varieties in orchard, making this method more suitable for practical applications. QCP performed well in filtering noise points and achieved Purity as 96.7% and 96.2% for red and green apples, respectively. Through the comparison method, experimental results indicated that the segmentation method based on ROI is more effective and accurate for red and green apples in orchard. The segmentation method of point clouds based on ROI has great potential for segmentation of point clouds in unstructured scenes.

A robust and efficient framework for fast cylinder detection

In this work, a complete solution is provided for detecting and identifying cylindrical shapes, which are commonly found in household and industrial environments, using consumer-grade RGB-D cameras. Most standard approaches to detect and identify cylinders are not robust to outliers (e.g. points on other objects in the scene), which limits their applicability in realistic scenes. In addition, these methods fail to benefit from environmental constraints, e.g. the fact that cylinders often lie or stand on flat surfaces. To tackle the aforementioned limitations, we introduce three main novelties: (i) a point cloud soft voting scheme with curvature information that reduces the influence of outliers and noise, (ii) a selective sampling of the orientation space that favorsorientations known a priori, and (iii) a deep-learning based classifier to filter out objects with non-cylindrical appearance in the 2D images, thus further improving robustness to outliers.A set of experiments with synthetically generated data are used to assess the robustness of our fitting method to different levels of outliers and noise. The results demonstrate that incorporating the principal curvature direction within the orientation voting process allows for large improvements on cylinders parameters estimation. Furthermore, we demonstrate that combining the 2D deep-learning cylinder classifier with the 3D orientation voting scheme allows for large speed-up and accuracy improvements on cylinder identification. The qualitative and quantitative results with real data acquired from a consumer RGB-D camera, confirm the advantages of the proposed framework.

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation.

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates.

Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras

Color-depth cameras (RGB-D cameras) have become the primary sensors in most robotics systems, from service robotics to industrial robotics applications. Typical consumer-grade RGB-D cameras are provided with a coarse intrinsic and extrinsic calibration that generally does not meet the accuracy requirements needed by many robotics applications (e.g., highly accurate 3D environment reconstruction and mapping, high precision object recognition and localization, ...). In this paper, we propose a human-friendly, reliable and accurate calibration framework that enables to easily estimate both the intrinsic and extrinsic parameters of a general color-depth sensor couple. Our approach is based on a novel two components error model. This model unifies the error sources of RGB-D pairs based on different technologies, such as structured-light 3D cameras and time-of-flight cameras. Our method provides some important advantages compared to other state-of-the-art systems: it is general (i.e., well suited for different types of sensors), based on an easy and stable calibration protocol, provides a greater calibration accuracy, and has been implemented within the ROS robotics framework. We report detailed experimental validations and performance comparisons to support our statements.

FusionMLS: Highly dynamic 3D reconstruction with consumer-grade RGB-D cameras

Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory, making it memory-efficient and scalable to large scenes. Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to state-of-the-art approaches.

Quantitative Analysis of Cotton Canopy Size in Field Conditions Using a Consumer-Grade RGB-D Camera.

Plant canopy structure can strongly affect crop functions such as yield and stress tolerance, and canopy size is an important aspect of canopy structure. Manual assessment of canopy size is laborious and imprecise, and cannot measure multi-dimensional traits such as projected leaf area and canopy volume. Field-based high throughput phenotyping systems with imaging capabilities can rapidly acquire data about plants in field conditions, making it possible to quantify and monitor plant canopy development. The goal of this study was to develop a 3D imaging approach to quantitatively analyze cotton canopy development in field conditions. A cotton field was planted with 128 plots, including four genotypes of 32 plots each. The field was scanned by GPhenoVision (a customized field-based high throughput phenotyping system) to acquire color and depth images with GPS information in 2016 covering two growth stages: canopy development, and flowering and boll development. A data processing pipeline was developed, consisting of three steps: plot point cloud reconstruction, plant canopy segmentation, and trait extraction. Plot point clouds were reconstructed using color and depth images with GPS information. In colorized point clouds, vegetation was segmented from the background using an excess-green (ExG) color filter, and cotton canopies were further separated from weeds based on height, size, and position information. Static morphological traits were extracted on each day, including univariate traits (maximum and mean canopy height and width, projected canopy area, and concave and convex volumes) and a multivariate trait (cumulative height profile). Growth rates were calculated for univariate static traits, quantifying canopy growth and development. Linear regressions were performed between the traits and fiber yield to identify the best traits and measurement time for yield prediction. The results showed that fiber yield was correlated with static traits after the canopy development stage (R2 = 0.35–0.71) and growth rates in early canopy development stages (R2 = 0.29–0.52). Multi-dimensional traits (e.g., projected canopy area and volume) outperformed one-dimensional traits, and the multivariate trait (cumulative height profile) outperformed univariate traits. The proposed approach would be useful for identification of quantitative trait loci (QTLs) controlling canopy size in genetics/genomics studies or for fiber yield prediction in breeding programs and production environments.

Depth Recovery for Kinect Sensor Using Contour-Guided Adaptive Morphology Filter

Consumer-grade RGB-D cameras, such as Kinect sensors, can provide support for much more real-time tasks of 3-D vision than game controllers. However, the inherent depth degradations caused by their infrared ranging will constrain their application potential, but can hardly be avoided through the improvement of the sensor design. Therefore, in this paper, we proposed a contours-guided shape-adaptive morphology filter to efficiently recover the depth of Kinect sensors. First, we put forward a statistical concept to quantitatively evaluate the texture richness of imaging sensors’ data and verify the applicability of morphology filtering on both Kinect 1 and 2 depth data. Then, considering the significance of the semantic contours, a multiresolution RGB-D contour extraction method is introduced to suppress the texture inside objects. Therewith, shape-adaptive structuring element (SASE) for each missing or untrusted depth pixel is created in terms of the contour guidance and the feature of human visual system. Efficient and accurate depth recovery can be finally achieved by combining morphology filtering and the obtained SASEs. Experiments on simulated data set, real Kinect 1, and Kinect 2 data show that our method performs better than many competing state-of-the-art approaches, and avoids the blurring around depth discontinuities.

Color map optimization for 3D reconstruction with consumer depth cameras

We present a global optimization approach for mapping color images onto geometric reconstructions. Range and color videos produced by consumer-grade RGB-D cameras suffer from noise and optical distortions, which impede accurate mapping of the acquired color data to the reconstructed geometry. Our approach addresses these sources of error by optimizing camera poses in tandem with non-rigid correction functions for all images. All parameters are optimized jointly to maximize the photometric consistency of the reconstructed mapping. We show that this optimization can be performed efficiently by an alternating optimization algorithm that interleaves analytical updates of the color map with decoupled parameter updates for all images. Experimental results demonstrate that our approach substantially improves color mapping fidelity.