Red-Green-Blue-Depth Sensor Research Articles

Validation of the shoulder range of motion software for measurement of shoulder ranges of motion in consultation: coupling a red/green/blue-depth video camera to artificial intelligence.

Clinical evaluation of the shoulder range of motion (RoM) may vary significantly depending on the surgeon. We aim to validate an automatic shoulder RoM measurement system associating image acquisition by an RGB-D (red/green/blue-depth) video camera to an artificial intelligence (AI) algorithm. Thirty healthy volunteers were included. A 3D RGB-D sensor that simultaneously generated a colour image and a depth map was used. Then, an open-access convolutional neural network algorithm that was programmed for shoulder recognition provided a 3D motion measure. Each volunteer adopted a randomized position successively. For each position, two observers made a visual (EyeREF) and goniometric measurement (GonioREF), blind to the automated software which was implemented by an orthopaedic surgeon. We evaluated the inter-tester intra-class correlation (ICC) between observers and the concordance correlation coefficient (CCC) between the three methods. For manual evaluations EyeREF and GonioREF, ICC remained constantly excellent for the widest motions in the vertical plane (i.e., abduction and flexion). It was very good for ER1 and IR2 and fairly good for adduction, extension, and ER2. Differences between the measurements' means of EyeREF and shoulder RoM was significant for all motions. Compared to GonioREF, shoulder RoM provided similar results for abduction, adduction, and flexion and EyeREF provided similar results for adduction, ER1, and ER2. The three methods showed an overall good to excellent CCC. The mean bias between the three methods remained under 10° and clinically acceptable. RGB-D/AI combination is reliable in measuring shoulder RoM in consultation, compared to classic goniometry and visual observation.

Detection Method of Straw Mulching Unevenness with RGB-D Sensors

Returning straw to the field is very important of for the conservation tillage to increase land fertility. It is vital to detect the unevenness of the straw covering to evaluate the performance of no-tillage planter, especially for the ones with returning full amount of straw. In this study, two kinds of RGB-D(Red, Green, Blue-Depth) sensors (RealSense D435i and Kinect v2) were applied to estimate the straw mulching unevenness by detecting the depth of straw coverage. Firstly, the overall structure and working principle of no-tillage planter with returning the full amount of straw was introduced. Secondly, field images were captured with the two kinds of RGB-D sensors after no tillage planter operation. Thirdly, straw covering unevenness computing was carried on a system developed by Matlab. Finally, the correlation analysis was conducted to test for the relationship between the straw covering unevenness by manual and deep sensors, with R (correlation coefficient) of 0.93, RMSE(Root Mean Square Error) of 4.59% and MAPE(Mean of Absolute Percentage Error) of 3.86% with D435i sensor, and with R of 0.915, RMSE of 6.53% and MAPE of 13.85% with Kinect V2, which showed both kinds of RGB-D sensors can acquire the unevenness of straw covering efficiently. The finding can provide a potential way to detect the unevenness of straw coverage and data support for operation evaluation and improvement of no-tillage planter.

Deep Convolutional Neural Networks for Pavement Crack Detection using an Inexpensive Global Shutter RGB-D Sensor and ARM-Based Single-Board Computer

Pavement distress assessment is a significant aspect of pavement management. Automated pavement crack detection is a challenging task that has been researched for decades in response to complicated pavement conditions. Current pavement condition assessment procedures are extensively time consuming, expensive, and labor-intensive. The primary goal of this paper is to develop a cost-effective and reliable platform using a red, green, blue, depth (RGB-D) sensor and deep learning detection models for automated pavement crack detection on a single-board ARM-based computer. To the best of our knowledge, for the first time, a pavement crack data set is prepared using a global shutter RGB-D sensor mounted on a car and annotated according to the Pascal visual object classes protocol, named PAVDIS2020. The proposed data set comprises 2,085 pavement crack images that are captured in a wide variety of weather and illuminance conditions with 5,587 instances of pavement cracks included in these images. A unified implementation of the Faster region-based convolutional neural networks and single shot multibox detector meta-architecture-based models is implemented to evaluate the accuracy, speed, and memory usage trade-off by using various convolutional neural networks-based backbones and various other training parameters on PAVDIS2020. The proposed pavement crack detection model was able to classify the cracks with 97.6% accuracy on PAVDIS2020 data set. The detection model is able to locate pavement crack patterns at the speed of 12 frames per second on a passively cooled Raspberry Pi 4 single-board computer.

Evaluating the severity of depressive symptoms using upper body motion captured by RGB-depth sensors and machine learning in a clinical interview setting: A preliminary study

BackgroundMood disorders have long been known to affect motor function. While methods to objectively assess such symptoms have been used in experiments, those same methods have not yet been applied in clinical practice because the methods are time-consuming, labor-intensive, or invasive. MethodsWe videotaped the upper body of each subject using a Red-Green-Blue-Depth (RGB-D) sensor during a clinical interview setting. We then examined the relationship between depressive symptoms and body motion by comparing the head motion of patients with major depressive disorders (MDD) and bipolar disorders (BD) to the motion of healthy controls (HC). Furthermore, we attempted to predict the severity of depressive symptoms by using machine learning. ResultsA total of 47 participants (HC, n = 16; MDD, n = 17; BD, n = 14) participated in the study, contributing to 144 data sets. It was found that patients with depression move significantly slower compared to HC in the 5th percentile and 50th percentile of motion speed. In addition, Hamilton Depression Rating Scale (HAMD)-17 scores correlated with 5th percentile, 50th percentile, and mean speed of motion. Moreover, using machine learning, the presence and/or severity of depressive symptoms based on HAMD-17 scores were distinguished by a kappa coefficient of 0.37 to 0.43. LimitationsLimitations include the small number of subjects, especially the number of severe cases and young people. ConclusionsThe RGB-D sensor captured some differences in upper body motion between depressed patients and controls. If much larger samples are accumulated, machine learning may be useful in identifying objective measures for depression in the future.

Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion

This study presents a novel method to apply the RGB-D (Red Green Blue-Depth) sensors and fuse aligned RGB and NIR images with deep convolutional neural networks (CNN) for fruit detection. It aims to build a more accurate, faster, and more reliable fruit detection system, which is a vital element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). A common Faster R-CNN network VGG16 was adopted through transfer learning, for the task of kiwifruit detection using imagery obtained from two modalities: RGB (red, green, blue) and Near-Infrared (NIR) images. Kinect v2 was used to take a bottom view of the kiwifruit canopy's NIR and RGB images. The NIR (1 channel) and RGB images (3 channels) were aligned and arranged side by side into a 6-channel image. The input layer of the VGG16 was modified to receive the 6-channel image. Two different fusion methods were used to extract features: Image-Fusion (fusion of the RGB and NIR images on input layer) and Feature-Fusion (fusion of feature maps of two VGG16 networks where the RGB and NIR images were input respectively). The improved networks were trained end-to-end using back-propagation and stochastic gradient descent techniques and compared to original VGG16 networks with RGB and NIR image input only. Results showed that the average precision (APs) of the original VGG16 with RGB and NIR image input only were 88.4% and 89.2% respectively, the 6-channel VGG16 using the Feature-Fusion method reached 90.5%, while that using the Image-Fusion method reached the highest AP of 90.7% and the fastest detection speed of 0.134 s/image. The results indicated that the proposed kiwifruit detection approach shows a potential for better fruit detection.

Development of a Human–Robot Hybrid Intelligent System Based on Brain Teleoperation and Deep Learning SLAM

To achieve the better navigation performance of a mobile robot in the unknown environments, a novel human–robot hybrid system incorporating a motor-imagery (MI)-based brain teleoperation control is presented in this paper, where a deep-learning-based active perception is developed in the simultaneous localization and mapping (SLAM) framework. Using the deep-learning-based object recognition in the red–green–blue-depth (RGB-D) data acquisition process, the designed SLAM approach can select the valid feature points effectively, and the speed of displacement tracking can be improved by combining the oriented FAST and rotated BRIEF (ORB) SLAM algorithm with the optical flow method. The global trajectory map can also be mended using graph-based nonlinear error optimization. In addition, to build the connection between human intentions and the robot control commands flexibly in the developed mobile robot, a common spatial pattern (CSP)-based support vector machine (SVM) classification algorithm is proposed so that the control commands can be obtained directly from the human electroencephalograph (EEG) signals, which are preanalyzed and classified using the phenomena of event-related synchronization/desynchronization (ERS/ERD). Experiments involving several operators have verified the effectiveness of the proposed framework in the actual unstructured environments. Note to Practitioners —This paper is motivated by the exploration of the mobile robots remotely controlled by the disables in an unstructured environment that includes some unknown moving objects in the background. In the conventional approaches, the image feedback and the brain–computer interface-based EEG signals invoked by the visual stimulus are used to guide the motion of the robots. In this paper, the MI of left or right hand is adopted as an input of the brain teleoperation system, and the EEG signals are classified into two categories representing two robot control commands through the designed algorithm. Besides, a deep-learning-based SLAM is designed to analyze the image and depth of the environmental information provided by the robot RGB-D sensor, and the results can be transferred into a 3-D map locating the robot and assisting the operator to understand the operating environment. It has been verified that the deep-learning-based SLAM presented is more efficient and robust than traditional SLAM. The feasibility of the system is demonstrated by a set of experiments in the corridor environment.

Enhanced 3D Mapping with an RGB-D Sensor via Integration of Depth Measurements and Image Sequences

State-of-the-art visual simultaneous localization and mapping (SLAM) techniques greatly facilitate three-dimensional (3D) mapping and modeling with the use of low-cost red-green-blue-depth (RGB-D) sensors. However, the effective range of such sensors is limited due to the working range of the infra-red (IR) camera, which provides depth information, and thus the practicability of such sensors in 3D mapping and modeling is limited. To address this limitation, we present a novel solution for enhanced 3D mapping using a low-cost RGB-D sensor. We carry out state-of-the-art visual SLAM to obtain 3D point clouds within the mapping range of the RGB-D sensor and implement an improved structure-from-motion (SfM) on the collected RGB image sequences with additional constraints from the depth information to produce image-based 3D point clouds. We then develop a feature-based scale-adaptive registration to merge the gained point clouds to further generate enhanced and extended 3D mapping results. We use two challenging test sites to examine the proposed method. At these two sites, the coverage of both generated 3D models increases by more than 50% with the proposed solution. Moreover, the proposed solution achieves a geometric accuracy of about 1% in a measurement range of about 20 m. These positive experimental results not only demonstrate the feasibility and practicality of the proposed solution but also its potential.

An Empirical Evaluation of Ten Depth Cameras: Bias, Precision, Lateral Noise, Different Lighting Conditions and Materials, and Multiple Sensor Setups in Indoor Environments

Over the last few years, novel color and depth sensors have pushed the boundaries of robot perception significantly. Today, several new depth-sensing products are replacing the earlier, well-examined red-green-blue-depth (RGBD) sensors, which have reached the end of their product life cycle and are no longer available. The properties of the new sensors have not yet been investigated, and it is unclear how they will compare to earlier RGBD sensors.

RGB-D Object Recognition Using Multi-Modal Deep Neural Network and DS Evidence Theory.

With the development of low-cost RGB-D (Red Green Blue-Depth) sensors, RGB-D object recognition has attracted more and more researchers’ attention in recent years. The deep learning technique has become popular in the field of image analysis and has achieved competitive results. To make full use of the effective identification information in the RGB and depth images, we propose a multi-modal deep neural network and a DS (Dempster Shafer) evidence theory based RGB-D object recognition method. First, the RGB and depth images are preprocessed and two convolutional neural networks are trained, respectively. Next, we perform multi-modal feature learning using the proposed quadruplet samples based objective function to fine-tune the network parameters. Then, two probability classification results are obtained using two sigmoid SVMs (Support Vector Machines) with the learned RGB and depth features. Finally, the DS evidence theory based decision fusion method is used for integrating the two classification results. Compared with other RGB-D object recognition methods, our proposed method adopts two fusion strategies: Multi-modal feature learning and DS decision fusion. Both the discriminative information of each modality and the correlation information between the two modalities are exploited. Extensive experimental results have validated the effectiveness of the proposed method.

Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field.

Fruit detection in real outdoor conditions is necessary for automatic guava harvesting, and the branch-dependent pose of fruits is also crucial to guide a robot to approach and detach the target fruit without colliding with its mother branch. To conduct automatic, collision-free picking, this study investigates a fruit detection and pose estimation method by using a low-cost red–green–blue–depth (RGB-D) sensor. A state-of-the-art fully convolutional network is first deployed to segment the RGB image to output a fruit and branch binary map. Based on the fruit binary map and RGB-D depth image, Euclidean clustering is then applied to group the point cloud into a set of individual fruits. Next, a multiple three-dimensional (3D) line-segments detection method is developed to reconstruct the segmented branches. Finally, the 3D pose of the fruit is estimated using its center position and nearest branch information. A dataset was acquired in an outdoor orchard to evaluate the performance of the proposed method. Quantitative experiments showed that the precision and recall of guava fruit detection were 0.983 and 0.948, respectively; the 3D pose error was 23.43° ± 14.18°; and the execution time per fruit was 0.565 s. The results demonstrate that the developed method can be applied to a guava-harvesting robot.

Enhanced Autonomous Exploration and Mapping of an Unknown Environment with the Fusion of Dual RGB-D Sensors

Abstract The autonomous exploration and mapping of an unknown environment is useful in a wide range of applications and thus holds great significance. Existing methods mostly use range sensors to generate two-dimensional (2D) grid maps. Red/green/blue-depth (RGB-D) sensors provide both color and depth information on the environment, thereby enabling the generation of a three-dimensional (3D) point cloud map that is intuitive for human perception. In this paper, we present a systematic approach with dual RGB-D sensors to achieve the autonomous exploration and mapping of an unknown indoor environment. With the synchronized and processed RGB-D data, location points were generated and a 3D point cloud map and 2D grid map were incrementally built. Next, the exploration was modeled as a partially observable Markov decision process. Partial map simulation and global frontier search methods were combined for autonomous exploration, and dynamic action constraints were utilized in motion control. In this way, the local optimum can be avoided and the exploration efficacy can be ensured. Experiments with single connected and multi-branched regions demonstrated the high robustness, efficiency, and superiority of the developed system and methods.

Automated Rutting Measurement Using an Inexpensive RGB-D Sensor Fusion Approach

Pavement management systems have become more crucial through the development of road networks. Data collection is at the core of any pavement management system. Rutting is a critical defect on pavements that can lead to hydroplaning and accidents. In this study, a system is proposed to quantify the rut depths autonomously by utilizing an inexpensive red green blue—depth (RGB-D) sensor. To this end, a cart is designed, built, and used to mount an array of RGB-D sensors for data collection purposes. Furthermore, a procedure is proposed for autonomous stitching of three-dimensional (3D) point clouds obtained from multiple RGB-D sensors to generate the pavement profile of a lane. Next, transverse profiles are extracted and the rut depths are estimated based on the simulated straight-edge method. Several field experiments are conducted to evaluate the capabilities as well as the limitations of the proposed system. Results show that the proposed approach is capable of estimating different severity levels of rutting accurately, particularly moderate and high severity levels with relative errors of 15% and 9%, respectively. Additionally, the repeatability of the proposed approach is evaluated and confirmed.

Encountered‐type haptic display for large VR environment using per‐plane reachability maps

AbstractWe show a novel encountered‐type haptic system, H‐Wall, to enable haptic feedback using a manipulator of seven degrees of freedom suitable for simulating indoor virtual reality environments, which are characterized and confined by a set of vertical walls and revolving doors. At runtime, our system tracks hand motion using a red–green–blue depth sensor and locates its configuration. Then, the robotic manipulator plans a trajectory for the end effector, attached to a rectangular rigid board, to make contact with the hand to deliver a sense of touch as long as the perceived hand contact force is substantial. The force feedback is generated in a passive sense for static walls that the rigid board, corresponding to a vertical wall, holds its position as long as the perceived hand contact force is substantial. For a revolving door, the force feedback is generated in an active sense based on impedance control. In order to address the issue of limited workspace, we also propose a new reachability map, called per‐plane reachability map, that is optimized to answer whether passive haptic feedback can be generated by a manipulator when the user touches a vertical wall at a given orientation. We successfully demonstrate our system to provide an illusion to the user in a virtual environment with touch sensation to the surrounding environment.

Automatic intraoperative stitching of nonoverlapping cone-beam CT acquisitions.

Cone-beam computed tomography (CBCT) is one of the primary imaging modalities in radiation therapy, dentistry, and orthopedic interventions. While CBCT provides crucial intraoperative information, it is bounded by a limited imaging volume, resulting in reduced effectiveness. This paper introduces an approach allowing real-time intraoperative stitching of overlapping and nonoverlapping CBCT volumes to enable 3D measurements on large anatomical structures. A CBCT-capable mobile C-arm is augmented with a red-green-blue-depth (RGBD) camera. An offline cocalibration of the two imaging modalities results in coregistered video, infrared, and x-ray views of the surgical scene. Then, automatic stitching of multiple small, nonoverlapping CBCT volumes is possible by recovering the relative motion of the C-arm with respect to the patient based on the camera observations. We propose three methods to recover the relative pose: RGB-based tracking of visual markers that are placed near the surgical site, RGBD-based simultaneous localization and mapping (SLAM) of the surgical scene which incorporates both color and depth information for pose estimation, and surface tracking of the patient using only depth data provided by the RGBD sensor. On an animal cadaver, we show stitching errors as low as 0.33, 0.91, and 1.72 mm when the visual marker, RGBD SLAM, and surface data are used for tracking, respectively. The proposed method overcomes one of the major limitations of CBCT C-arm systems by integrating vision-based tracking and expanding the imaging volume without any intraoperative use of calibration grids or external tracking systems. We believe this solution to be most appropriate for 3D intraoperative verification of several orthopedic procedures.

Stairs and Doors Recognition as Natural Landmarks Based on Clouds of 3D Edge-Points from RGB-D Sensors for Mobile Robot Localization.

Natural landmarks are the main features in the next step of the research in localization of mobile robot platforms. The identification and recognition of these landmarks are crucial to better localize a robot. To help solving this problem, this work proposes an approach for the identification and recognition of natural marks included in the environment using images from RGB-D (Red, Green, Blue, Depth) sensors. In the identification step, a structural analysis of the natural landmarks that are present in the environment is performed. The extraction of edge points of these landmarks is done using the 3D point cloud obtained from the RGB-D sensor. These edge points are smoothed through the algorithm, which minimizes the standard deviation of the normals at each point. Then, the second step of the proposed algorithm begins, which is the proper recognition of the natural landmarks. This recognition step is done as a real-time algorithm that extracts the points referring to the filtered edges and determines to which structure they belong to in the current scenario: stairs or doors. Finally, the geometrical characteristics that are intrinsic to the doors and stairs are identified. The approach proposed here has been validated with real robot experiments. The performed tests verify the efficacy of our proposed approach.

Can real-time RGBD enhance intraoperative Cone-Beam CT?

Cone-Beam Computed Tomography (CBCT) is an important 3D imaging technology for orthopedic, trauma, radiotherapy guidance, angiography, and dental applications. The major limitation of CBCT is the poor image quality due to scattered radiation, truncation, and patient movement. In this work, we propose to incorporate information from a co-registered Red-Green-Blue-Depth (RGBD) sensor attached near the detector plane of the C-arm to improve the reconstruction quality, as well as correcting for undesired rigid patient movement. Calibration of the RGBD and C-arm imaging devices is performed in two steps: (i) calibration of the RGBD sensor and the X-ray source using a multimodal checkerboard pattern, and (ii) calibration of the RGBD surface reconstruction to the CBCT volume. The patient surface is acquired during the CBCT scan and then used as prior information for the reconstruction using Maximum-Likelihood Expectation-Maximization. An RGBD-based simultaneous localization and mapping method is utilized to estimate the rigid patient movement during scanning. Performance is quantified and demonstrated using artificial data and bone phantoms with and without metal implants. Finally, we present movement-corrected CBCT reconstructions based on RGBD data on an animal specimen, where the average voxel intensity difference reduces from 0.157 without correction to 0.022 with correction. This work investigated the advantages of a C-arm X-ray imaging system used with an attached RGBD sensor. The experiments show the benefits of the opto/X-ray imaging system in: (i) improving the quality of reconstruction by incorporating the surface information of the patient, reducing the streak artifacts as well as the number of required projections, and (ii) recovering the scanning trajectory for the reconstruction in the presence of undesired patient rigid movement.

Real-time multiview data fusion for object tracking with RGBD sensors

SUMMARYThis paper presents a new approach to accurately track a moving vehicle with a multiview setup of red–green–blue depth (RGBD) cameras. We first propose a correction method to eliminate a shift, which occurs in depth sensors when they become worn. This issue could not be otherwise corrected with the ordinary calibration procedure. Next, we present a sensor-wise filtering system to correct for an unknown vehicle motion. A data fusion algorithm is then used to optimally merge the sensor-wise estimated trajectories. We implement most parts of our solution in the graphic processor. Hence, the whole system is able to operate at up to 25 frames per second with a configuration of five cameras. Test results show the accuracy we achieved and the robustness of our solution to overcome uncertainties in the measurements and the modelling.

Review of Local Descriptor in RGB-D Object Recognition

The emergence of an RGB-D (Red-Green-Blue-Depth) sensor which is capable of providing depth and RGB images gives hope to the computer vision community. Moreover, the use of local features began to increase over the last few years and has shown impressive results, especially in the field of object recognition. This article attempts to provide a survey of the recent technical achievements in this area of research. We review the use of local descriptors as the feature representation which is extracted from RGB-D images, in instances and category-level object recognition. We also highlight the involvement of depth images and how they can be combined with RGB images in constructing a local descriptor. Three different approaches are used in involving depth images into compact feature representation, that is classical approach using distribution based, kernel-trick, and feature learning. In this article, we show that the involvement of depth data successfully improves the accuracy of object recognition.

Solution to the SLAM Problem in Low Dynamic Environments Using a Pose Graph and an RGB-D Sensor

In this study, we propose a solution to the simultaneous localization and mapping (SLAM) problem in low dynamic environments by using a pose graph and an RGB-D (red-green-blue depth) sensor. The low dynamic environments refer to situations in which the positions of objects change over long intervals. Therefore, in the low dynamic environments, robots have difficulty recognizing the repositioning of objects unlike in highly dynamic environments in which relatively fast-moving objects can be detected using a variety of moving object detection algorithms. The changes in the environments then cause groups of false loop closing when the same moved objects are observed for a while, which means that conventional SLAM algorithms produce incorrect results. To address this problem, we propose a novel SLAM method that handles low dynamic environments. The proposed method uses a pose graph structure and an RGB-D sensor. First, to prune the falsely grouped constraints efficiently, nodes of the graph, that represent robot poses, are grouped according to the grouping rules with noise covariances. Next, false constraints of the pose graph are pruned according to an error metric based on the grouped nodes. The pose graph structure is reoptimized after eliminating the false information, and the corrected localization and mapping results are obtained. The performance of the method was validated in real experiments using a mobile robot system.