Low-cost Depth Camera Research Articles

Grape Maturity Estimation Using Time-of-Flight and LiDAR Depth Cameras.

This article investigates the potential for using low-cost depth cameras to estimate the maturity of green table grapes after they have been harvested. Time-of-flight (Kinect Azure) and LiDAR (Intel L515) depth cameras were used to capture depth scans of green table grape berries over time. The depth scans of the grapes are distorted due to the diffused scattering of the light emitted from the cameras within the berries. This causes a distance bias where a grape berry appears to be further from the camera than it is. As the grape aged, the shape of the peak corresponding to the grape became increasingly flattened in shape, resulting in an increased distance bias over time. The distance bias variation with time was able to be fitted with an R2 value of 0.969 for the Kinect Azure and an average of 0.904 for the Intel L515. This work shows that there is potential to use time-of-flight and LIDAR cameras for estimating grape maturity postharvest in a non-contact and nondestructive manner.

A computer vision system for apple fruit sizing by means of low-cost depth camera and neural network application

AbstractFruit size is crucial for growers as it influences consumer willingness to buy and the price of the fruit. Fruit size and growth along the seasons are two parameters that can lead to more precise orchard management favoring production sustainability. In this study, a Python-based computer vision system (CVS) for sizing apples directly on the tree was developed to ease fruit sizing tasks. The system is made of a consumer-grade depth camera and was tested at two distances among 17 timings throughout the season, in a Fuji apple orchard. The CVS exploited a specifically trained YOLOv5 detection algorithm, a circle detection algorithm, and a trigonometric approach based on depth information to size the fruits. Comparisons with standard-trained YOLOv5 models and with spherical objects were carried out. The algorithm showed good fruit detection and circle detection performance, with a sizing rate of 92%. Good correlations (r > 0.8) between estimated and actual fruit size were found. The sizing performance showed an overall mean error (mE) and RMSE of + 5.7 mm (9%) and 10 mm (15%). The best results of mE were always found at 1.0 m, compared to 1.5 m. Key factors for the presented methodology were: the fruit detectors customization; the HoughCircle parameters adaptability to object size, camera distance, and color; and the issue of field natural illumination. The study also highlighted the uncertainty of human operators in the reference data collection (5–6%) and the effect of random subsampling on the statistical analysis of fruit size estimation. Despite the high error values, the CVS shows potential for fruit sizing at the orchard scale. Future research will focus on improving and testing the CVS on a large scale, as well as investigating other image analysis methods and the ability to estimate fruit growth.

A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

Efficient Model-Based Anthropometry under Clothing Using Low-Cost Depth Sensors.

Measuring human body dimensions is critical for many engineering and product design domains. Nonetheless, acquiring body dimension data for populations using typical anthropometric methods poses challenges due to the time-consuming nature of manual methods. The measurement process for three-dimensional (3D) whole-body scanning can be much faster, but 3D scanning typically requires subjects to change into tight-fitting clothing, which increases time and cost and introduces privacy concerns. To address these and other issues in current anthropometry techniques, a measurement system was developed based on portable, low-cost depth cameras. Point-cloud data from the sensors are fit using a model-based method, Inscribed Fitting, which finds the most likely body shape in the statistical body shape space and providing accurate estimates of body characteristics. To evaluate the system, 144 young adults were measured manually and with two levels of military ensembles using the system. The results showed that the prediction accuracy for the clothed scans remained at a similar level to the accuracy for the minimally clad scans. This approach will enable rapid measurement of clothed populations with reduced time compared to manual and typical scan-based methods.

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image.

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.

Dynamic Parallel Mapping and Trajectory Planning of Robot Arm in Unknown Environment

A high real-time robot arm mapping and dynamic shortest path planning algorithm is proposed, which is used for low-cost depth cameras and robot arms. First of all, an inverse kinematics solution algorithm for a non-spherical wrist joint is proposed for the Kinova Mico2 manipulator. Based on the idea of position and attitude decoupling combined with the numerical solution, the solution of multiple sets of inverse kinematics is acquired. Secondly, a hand-eye system is established on the basis of the depth camera-RealSense. The point cloud is inversely solved through the depth map, which is pre-processed by the pass-through filtering and outlier removal algorithm. A scene parallel mapping method is designed to achieve the real-time reconstruction of the working scene for the robot arm, in which the maintenance of map and large matrix operations can be completed by GPU. Thirdly, a path planning algorithm with the shortest path for the manipulator end in Cartesian space is designed based on D* method and grid map. The attitude interpolation and smoothing are performed on the path points to obtain the executable joint trajectories on the time axis. Finally, the ROS system is used to build a simulation test verification environment, and the effectiveness of the related algorithms is verified through multiple simulation cases.

LOW-COST DEPTH-CAMERA: OPEN-SOURCE 3D DISPLACEMENT MEASUREMENTS FOR 4D PRINTED HYGROSCOPIC COMPOSITES

Abstract. 4D printing (4DP) is a growing branch of 3D printing technology that involves the design of composite material architectures capable of shape-change transformations, which occur post printing, in response to external stimulus. Among these, Wood Polymer Composites (WPCs) change their shape in reaction to changes of moisture content, shrinking or swelling like natural wood until the equilibrium with the environment is reached. Such intrinsic material behavior can be particularly useful in the development of passive moisture airflow controllers that can modulate humidity and airflow in indoor environments to improve air quality. Precise measurement of the time-based stimulus induced shape-change response of these composites is critical to assess the responsiveness, velocity of reaction and overall deformation of the designed 4DP composite mechanisms. Up until now, Digital Image Correlation (DIC) techniques have been widely used for such purpose. However, DIC methods require expensive equipment and costly commercial software. This paper presents a Low-Cost Depth-Camera (LCDC) method that uses a free custom algorithm that returns a 3D coloured displacement map with the corresponding meshes of the acquired object. The LCDC method does not require specialized equipment and allows for an overall understanding of the time-dependent deformation of 4DP actuators, this method also facilitates the comparison between composites with different properties under the same external conditions. This new LCDC method has the potential to further 4DP research by providing an open-source, accessible and reliable tool to assess 3D displacement measurements.

Planning algorithms for acquiring high fidelity pointclouds using a robot for accurate and fast 3D reconstruction

Sensors are widely used to construct a 3D model of parts by collecting data. The accuracy of the collected data depends on sensor placement with respect to the part and sensor operational range. The operational range and limitations of the sensor need to be applied as constraints while planning for robot motions that move the sensor around the part to collect data. Overly conservative constraints on sensor placement will lead to high execution times. We present an approach where we develop a robot motion planning algorithm that takes into account the camera performance constraints and produces output with low error. An RGB-D camera is used to obtain the pointcloud of the part. An offline planning method improves point density in the regions having zero or low density. Our method guarantees a high point density across the surface of the part. Results are presented on six geometries with different complexity and surface properties. We also present results on how camera parameters influence the output of our method. Our results show that algorithmic advances reported in this paper enable us to use low-cost depth cameras for producing high accuracy uniform density scans of physical objects.

Monocular vision with deep neural networks for autonomous mobile robots navigation

Enabling mobile robots to explore the formerly unidentified environment is a challenging task. The current paper describes the internal analysis algorithm for mobile robots that combines various convolutional neural network (CNN) layers with the decision-making process in a hierarchical way. The whole system is trained end-to-end on data captured by a low-cost depth camera (RGB-D). The output consists of the proposed expansion model of the robot's critical moving directions to achieve autonomous analysis ability. Training this model through the dataset is created using Hand-Controlled Mobile Robot (HCMR) built for this purpose. The experiments were conducted by moving this robot in natural and diverse environments. The robot was trained using this data and applied for environmental investigation decisions (the control labels) using CNN to enable the robot to automatically sense the navigation without a map in an unknown environment. Furthermore, extensive experiments were conducted indoors and attained an accuracy of 77%. Experiments showed that the proposed model was able to reach equivalent results that are generally obtained enormously from an expensive sensor. In addition, comprehensive comparisons were drawn between the human-controlled robot and a robot trained using a deep learning process to determine decisions to control the robot's movement. The reached results were identical and satisfactory.

Dynamic Hand Gesture Recognition Using Improved Spatio-Temporal Graph Convolutional Network

Hand gesture recognition is essential to human-computer interaction as the most natural way of communicating. Furthermore, with the development of 3D hand pose estimation technology and the performance improvement of low-cost depth cameras, skeleton-based dynamic hand gesture recognition has received much attention. This paper proposes a novel multi-stream improved spatio-temporal graph convolutional network (MS-ISTGCN) for skeleton-based dynamic hand gesture recognition. We adopt an adaptive spatial graph convolution that can learn the relationship between distant hand joints and propose an extended temporal graph convolution with multiple dilation rates that can extract informative temporal features from short to long periods. Furthermore, we add a new attention layer consisting of effective spatio-temporal attention and channel attention between the spatial and temporal graph convolution layers to find and focus on key features. Finally, we propose a multi-stream structure that feeds multiple data modalities (i.e., joints, bones, and motions) as inputs to improve performance using the ensemble technique. Each of the three-stream networks is independently trained and fused to predict the final hand gesture. The performance of the proposed method is verified through extensive experiments with two widely used public dynamic hand gesture datasets: SHREC’17 Track and DHG-14/28. Our proposed method achieves the highest recognition accuracy in various gesture categories for both datasets compared with state-of-the-art methods.

Algorithms for Improving Speed and Accuracy of Automated Three-Dimensional Reconstruction With a Depth Camera Mounted on An Industrial Robot

Abstract Three-dimensional reconstruction technology is used in a wide variety of applications. Automatically creating accurate pointclouds for large parts with complex geometries usually requires expensive metrology instruments. We are interested in using low-cost depth cameras mounted on commonly available industrial robots to create accurate pointclouds for large parts automatically. Manufacturing applications require fast cycle times. Therefore, we are interested in speeding up the 3D reconstruction process. We present algorithmic advances in 3D reconstruction that achieve a sub-millimeter accuracy using a low-cost depth camera. Our system can be used to determine a pointcloud model of large and complex parts. Advances in camera calibration, cycle time reduction for pointcloud capturing, and uncertainty estimation are made in this work. We continuously capture pointclouds at an optimal camera location with respect to part distance during robot motion execution. The redundancy in pointclouds achieved by the moving camera significantly reduces errors in measurements without increasing cycle time. Our system produces sub-millimeter accuracy.

A real-time algorithm for the detection of compensatory movements during reaching.

Introduction: Interactive game systems can motivate stroke survivors to engage with their rehabilitation exercises. However, it is crucial that systems are in place to detect if exercises are performed correctly as stroke survivors often perform compensatory movements which can be detrimental to recovery. Very few game systems integrate motion tracking algorithms to monitor performance and detect such movements. This paper describes the development of algorithms which monitor for compensatory movements during upper limb reaching movements in real-time and provides quantitative metrics for health professionals to monitor performance and progress over time.Methods: A real-time algorithm was developed to analyse reaching motions in real-time through a low-cost depth camera. The algorithm segments cyclical reaching motions into component parts, including compensatory movement, and provides a graphical representation of task performance. Healthy participants (n = 10) performed reaching motions facing the camera. The real-time accuracy of the algorithm was assessed by comparing offline analysis to real-time collection of data.Results: The algorithm’s ability to segment cyclical reaching motions and detect the component parts in real-time was assessed. Results show that movement types can be detected in real time with accuracy, showing a maximum error of 1.71%.Conclusions: Using the methods outlined, the real-time detection and quantification of compensatory movements is feasible for integration within home-based, repetitive task practice game systems for people with stroke.

Estimating pavement roughness using a low-cost depth camera

ABSTRACT In a continuation of the Wyoming Technology Transfer Center (WYT2/LTAP) efforts to minimise the costs associated with the pavement condition assessment process, this study examined the capability of a low-cost depth camera (D435i) in measuring the International Roughness Index (IRI) as an indicator of pavement roughness. This study was performed on 48 pavement segments with various lengths and geometric features extracted from the Wyoming local roads database. A computer program was written specifically to control the camera, process the acquired data, and to return the final IRI values. The data was collected at posted speed limits. Noise reduction and hole filling techniques were applied to rectify the acquired depth data. It was found that a D435i depth camera can estimate the IRI of paved roads with a reasonable certainty compared to a standard profiler. The D435i depth camera can explain 83% of the variations in the actual IRI values measured using a standard road profiler. Statistical analysis showed no significant difference between both measurement methods at a 95% confidence level. The proposed approach has the potential to be a baseline for an inexpensive data collection system suitable for local agencies with limited budgets.

Real-time posture feedback for effective motor learning in table tennis in virtual reality

Table tennis is a demanding sport in terms of motor skills, requiring precise time-critical postures and techniques for the player to excel. Therefore, it is a compelling case study for motor learning in Virtual Reality (VR). Though VR table tennis training applications have been used to study the learning process in VR, each has not fully explored the role of real-time feedback on motor learning, which is a fundamental component of real-life table tennis training. To address this, we have developed a training system that provides multi-modal training instructions and real-time posture guidance feedback to assist beginner players in improving their playing outcomes. This system synthesizes a table tennis training environment by providing realistic visual and audio stimuli that includes a custom highly-accurate physics implementation of ball flight. An experimental group (n = 9) trained to learn correct posture and paddle handling for forehand and backhand drives. The results show that the participants improved significantly in terms of technique and ball return quality. This work is the first VR table tennis system that provides real-time posture feedback using a low-cost depth camera. Additionally, the participants’ significant improvement in playing posture over a short period of time shows the integral role of feedback in learning and further validates VR as an effective motor learning tool.

3D Hand Pose Estimation via Graph-Based Reasoning

Hand pose estimation from a single depth image has recently received significant attention owing to its importance in many applications requiring human–computer interaction. The rapid progress of convolutional neural networks (CNNs) and technological advances in low-cost depth cameras have greatly improved the performance of the hand pose estimation method. Nevertheless, regressing joint coordinates is still a challenging task due to joint flexibility and self-occlusion. Previous hand pose estimation methods have limitations in relying on a deep and complex network structure without fully utilizing hand joint connections. A hand is an articulated object and consists of six parts that represent the palm and five fingers. The kinematic constraints can be obtained by modeling the dependency between adjacent joints. This paper proposes a novel CNN-based approach incorporating hand joint connections to features through both a global relation inference for the entire hand and local relation inference for each finger. Modeling the relations between the hand joints can alleviate critical problems for occlusion and self-similarity. We also present a hierarchical structure with six branches that independently estimate the position of the palm and five fingers by adding hand connections of each joint using graph reasoning based on graph convolutional networks. Experimental results on public hand pose datasets show that the proposed method outperforms previous state-of-the-art methods. Specifically, our method achieves the best accuracy compared to state-of-the-art methods on public datasets. In addition, the proposed method can be utilized for real-time applications with an execution speed of 103 fps in a single GPU environment.

Estimating body weight of pigs from posture analysis using a depth camera.

A noninvasive method for estimating the body weight (BW) of a pig considering its posture using a low-cost depth camera (Kinect v2) was proposed. A total of 150 pigs were used, and 738 depth images (point clouds) were obtained for them. The pig "volume" was calculated from the pig point cloud, and it was found to have a very high correlation to BW. To evaluate the posture of a pig quantitatively, seven posture angles were calculated based on the "spine" extracted from a pig point cloud. We found the posture angles representing the height of the head position correlated with the accuracy of BW estimation using the "volume." Based on this finding, we proposed an "adjusted volume," which was adjusted based on the relationship between the posture angles and the estimation error. The BW of pigs was estimated using the simple regression model with the "adjusted volume," and the MAPE and RMSPE were 4.87% and 6.13%, respectively. The accuracy of the suggested model was similar to that of the volume-based estimation models of other studies that used only data with an appropriate pig posture for BW estimation.

A depth camera-based full-body reconstruction method for body pose training in occupational safety

Occupational injuries have high incidence rates across various industries. Safety education is a key component to effectively reduce work-related injuries. Posture training for work safety is widely adopted to increase the awareness of unsafe movements at work and to evaluate workers to minimize work-related musculoskeletal stresses. However, existing one-size-fits-all pamphlet-based posture training is facing challenges in its effectiveness. In recent years, the substantial technological development in virtual reality (VR) and augmented reality (AR) has made immersive and personalized education possible. For VR/AR-assisted posture training, full-body reconstruction from multiple point clouds is the key step. In this study, we propose a fast and coarse method to reconstruct the full-body pose of safety instructors using multiple low-cost depth cameras. The reconstructed body images from depth cameras are registered through iterative closet point algorithm. The reconstructed full-body pose can be further rendered in VR/AR environments for next-generation safety education.

Texture Synthesis Repair of RealSense D435i Depth Images with Object-Oriented RGB Image Segmentation.

A depth camera is a kind of sensor that can directly collect distance information between an object and the camera. The RealSense D435i is a low-cost depth camera that is currently in widespread use. When collecting data, an RGB image and a depth image are acquired simultaneously. The quality of the RGB image is good, whereas the depth image typically has many holes. In a lot of applications using depth images, these holes can lead to serious problems. In this study, a repair method of depth images was proposed. The depth image is repaired using the texture synthesis algorithm with the RGB image, which is segmented through a multi-scale object-oriented method. The object difference parameter is added to the process of selecting the best sample block. In contrast with previous methods, the experimental results show that the proposed method avoids the error filling of holes, the edge of the filled holes is consistent with the edge of RGB images, and the repair accuracy is better. The root mean square error, peak signal-to-noise ratio, and structural similarity index measure from the repaired depth images and ground-truth image were better than those obtained by two other methods. We believe that the repair of the depth image can improve the effects of depth image applications.

Reachable Workspace and Proximal Function Measures for Quantifying Upper Limb Motion.

There are a lack of quantitative measures for clinically assessing upper limb function. Conventional biomechanical performance measures are restricted to specialist labs due to hardware cost and complexity, while the resulting measurements require specialists for analysis. Depth cameras are low cost and portable systems that can track surrogate joint positions. However, these motions may not be biologically consistent, which can result in noisy, inaccurate movements. This paper introduces a rigid body modelling method to enforce biological feasibility of the recovered motions. This method is evaluated on an existing depth camera assessment: the reachable workspace (RW) measure for assessing gross shoulder function. As a rigid body model is used, position estimates of new proximal targets can be added, resulting in a proximal function (PF) measure for assessing a subject's ability to touch specific body landmarks. The accuracy, and repeatability of these measures is assessed on ten asymptomatic subjects, with and without rigid body constraints. This analysis is performed both on a low-cost depth camera system and a gold-standard active motion capture system. The addition of rigid body constraints was found to improve accuracy and concordance of the depth camera system, particularly in lateral reaching movements. Both RW and PF measures were found to be feasible candidates for clinical assessment, with future analysis needed to determine their ability to detect changes within specific patient populations.

Evaluation of low-cost depth cameras for agricultural applications

Low-cost depth-cameras have been used in many agricultural applications with reported advantages of low cost, reliability and speed of measurement. However, some problems were also reported and seem to be technology-related, so understanding the limitations of each type of depth camera technology could provide a basis for technology selection and the development of research involving its use. The cameras use one or a combination of two of the three available technologies: structured light, time-of-flight (ToF), and stereoscopy. The objectives were to evaluate these different technologies for depth sensing, including measuring accuracy and repeatability of distance data and measurements at different positions within the image, and cameras usefulness in indoor and outdoor settings. Then, cameras were tested in a swine facility and in a corn field. Five different cameras were used: (1) Microsoft Kinect v.1, (2) Microsoft Kinect v.2, (3) Intel® RealSense™ Depth Camera D435, (4) ZED Stereo Camera (StereoLabs), and (5) CamBoard Pico Flexx (PMD Technologies). Results indicate that there were significant camera to camera differences for ZED Stereo Camera and Kinect v.1 camera (p < 0.05). All cameras showed an increase in the standard deviation as the distance between camera and object increased; however, the Intel RealSense camera had a larger increase. Time-of-flight cameras had the smallest error between different sizes of objects. Time-of-flight cameras had non-readable zones on the corners of the images. The results indicate that the ToF technology is the best to be used for indoor applications and stereoscopy is the best technology for outdoor applications.