Captured Depth Images Research Articles

Online chicken carcass volume estimation using depth imaging and 3-D reconstruction

Variability in the size of slaughtered chickens remains a longstanding challenge in the standardization of the poultry industry. To address this issue, we present a novel approach that uses volume as a grading metric for chicken carcasses. This innovative method, unexplored in existing studies, employs real-time data capture of moving chicken carcasses on a production line using Kinect v2 depth imaging and 3-D reconstruction technologies. The captured depth images are processed into point clouds followed by 3-D reconstruction. Volume is calculated from the reconstructed models using the surface integration method, and additional 2-D and 3-D features are extracted as input parameters for machine learning models. Multiple regression models were evaluated, with the bagged tree model demonstrating superior performance, achieving an R² value of 0.9988, RMSE of 5.335, and ARE of 2.125%. Furthermore, our method showed remarkable efficiency with an average processing time of less than 1.6 seconds per carcass. These results indicate that our novel approach fills a critical gap in existing automated grading methodologies by offering both accuracy and efficiency. This validates the applicability of depth imaging, 3-D reconstruction, and machine learning for estimating chicken carcass volume with high precision, thereby enabling a more comprehensive, efficient, and reliable chicken carcass grading system.

A Deep Learning-Based Chair System That Detects Sitting Posture.

Long-term poor sitting posture leads to physical injuries such as muscle soreness and waist and neck alignment problems. In this study, we proposed an intelligent sitting posture detection system that uses depth cameras fixed on a chair to capture depth images of the user's sitting posture, and then applies a trained artificial intelligence (AI) model on an embedded Raspberry Pi board to recognize the user's sitting posture from the image data. Finally, through Bluetooth on the Raspberry Pi, the results are sent to the user's smartphone application for display and recording to achieve rapid detection of sitting posture and warning of poor sitting posture. The contribution of this study is its use of two depth cameras mounted on a chair, thereby eliminating the problem of cumbersome sensors that compromise user comfort or are prone to damage. The detection of the user's entire sitting posture was completed on an edge computing platform, which leads to power savings and offers privacy protection. Furthermore, because of the low battery power usage, the system is portable. To perform quick AI calculations, we developed a lightweight EfficientNet model and programmed it for the Raspberry Pi. The system achieved an accuracy of 99.71% and an execution speed of almost one posture result per second.

Self-Supervised Learning for RGB-Guided Depth Enhancement by Exploiting the Dependency between RGB and Depth.

Due to the imaging mechanism of time-of-flight (ToF) sensors, the captured depth images usually suffer from severe noise and degradation. Though many RGB-guided methods have been proposed for depth image enhancement in the past few years, yet the enhancement performance on real-world depth images is still largely unsatisfactory. Two main reasons are the complexity of realistic noise and degradation in depth images, and the difficulty in collecting noise-clean pairs for supervised enhancement learning. This work aims to develop a self-supervised learning method for RGB-guided depth image enhancement, which does not require any noisy-clean pairs but can significantly boost the enhancement performance on real-world noisy depth images. To this end, we exploit the dependency between RGB and depth images to self-supervise the learning of the enhancement model. It is achieved by maximizing the cross-modal dependency between RGB and depth to promote the enhanced depth having dependency with the RGB of the same scene as much as possible. Furthermore, we augment the cross-modal dependency maximization formulation based on the optimal transport theory to achieve further performance improvement. Experimental results on both synthetic and real-world data demonstrate that our method can significantly outperform existing state-of-the-art methods on depth denoising, multi-path interference suppression, and hole filling. Particularly, our method shows remarkable superiority over existing ones on real-world data in handling various realistic complex degradation. Code is available at https://github.com/wjcyt/SRDE.

3DBodyNet: Fast Reconstruction of 3D Animatable Human Body Shape From a Single Commodity Depth Camera

Knowledge about individual body shape has numerous applications in various domains such as healthcare, fashion and personalized entertainment. Most of the depth based whole body scanners need multiple cameras surrounding the user and requiring the user to keep a canonical pose strictly during capturing depth images. These scanning devices are expensive and need professional knowledge for operation. In order to make 3D scanning as easy-to-use and fast as possible, there is a great demand to simplify the process and to reduce the hardware requirements. In this paper, we propose a deep learning algorithm, dubbed 3DBodyNet, to rapidly reconstruct the 3D shape of human bodies using a single commodity depth camera. As easy-to-use as taking a photo using a mobile phone, our algorithm only needs two depth images of the front-facing and back-facing bodies. The proposed algorithm has strong operability since it is insensitive to the pose and the pose variations between the two depth images. It can also reconstruct an accurate body shape for users under tight/loose clothing. Another advantage of our method is the ability to generate an animatable human body model. Extensive experimental results show that the proposed method enables robust and easy-to-use animatable human body reconstruction, and outperforms the state-of-the-art methods with respect to running time and accuracy.

RGB-D-based gaze point estimation via multi-column CNNs and facial landmarks global optimization

In this work, we utilize a multi-column CNNs framework to estimate the gaze point of a person sitting in front of a display from an RGB-D image of the person. Given that gaze points are determined by head poses, eyeball poses, and 3D eye positions, we propose to infer the three components separately and then integrate them for gaze point estimation. The captured depth images, however, usually contain noises and black holes which prevent us from acquiring reliable head pose and 3D eye position estimation. Therefore, we propose to refine the raw depth for 68 facial keypoints by first estimating their relative depths from RGB face images, which along with the captured raw depths are then used to solve the absolute depth for all facial keypoints through global optimization. The refined depths will provide us reliable estimation for both head pose and 3D eye position. Given that existing publicly available RGB-D gaze tracking datasets are small, we also build a new dataset for training and validating our method. To the best of our knowledge, it is the largest RGB-D gaze tracking dataset in terms of the number of participants. Comprehensive experiments demonstrate that our method outperforms existing methods by a large margin on both our dataset and the Eyediap dataset.

Edge-preserving Filtering and Fuzzy Image Enhancement in Depth Images Captured by Realsense Cameras in Robotic Applications

This paper presents both the use of depth cameras in robotic applications and effects of post-processing on the captured depth images. The performance of depth cameras and post-process ...

RGBD Based Gaze Estimation via Multi-Task CNN

This paper tackles RGBD based gaze estimation with Convolutional Neural Networks (CNNs). Specifically, we propose to decompose gaze point estimation into eyeball pose, head pose, and 3D eye position estimation. Compared with RGB image-based gaze tracking, having depth modality helps to facilitate head pose estimation and 3D eye position estimation. The captured depth image, however, usually contains noise and black holes which noticeably hamper gaze tracking. Thus we propose a CNN-based multi-task learning framework to simultaneously refine depth images and predict gaze points. We utilize a generator network for depth image generation with a Generative Neural Network (GAN), where the generator network is partially shared by both the gaze tracking network and GAN-based depth synthesizing. By optimizing the whole network simultaneously, depth image synthesis improves gaze point estimation and vice versa. Since the only existing RGBD dataset (EYEDIAP) is too small, we build a large-scale RGBD gaze tracking dataset for performance evaluation. As far as we know, it is the largest RGBD gaze dataset in terms of the number of participants. Comprehensive experiments demonstrate that our method outperforms existing methods by a large margin on both our dataset and the EYEDIAP dataset.

Color-Guided Depth Image Recovery With Adaptive Data Fidelity and Transferred Graph Laplacian Regularization

Depth images play an important role and are prevalently used in many computer vision and computational imaging tasks. However, due to the limitation of active sensing technology, the captured depth images in practice usually suffer from low resolution and noise, which prevents its further applications. To remedy this problem, in this paper, we first propose an adaptive data fidelity formulation to optimally generate each depth pixel from a mixture probability distribution, characterizing the similarity both in the depth map and the corresponding high-resolution guided color image. The proposed method is able to fit the distribution of the input depth signal as an optimization problem by maximizing the mixture probability. Furthermore, to promote the piecewise property that depth images exhibit, we propose a transferred graph Laplacian model as a regularization term, which is general and able to handle various depth recovery tasks such as super-resolution and denoising well. Specifically, each pixel within the recovered depth image is represented as a vertex in a graph with weights in connected edges representing the similarity between vertices. By minimizing the squared variations of the image signal, the task of depth image recovery can be converted to the problem of graph-based image filtering. Since the proposed graph Laplacian regularization model is able to fully exploit a priori information about the depth image, a much more accurate and robust estimation of the underlying depth can be obtained. Extensive experiment evaluations verify that the proposed method obtains recovered depth with higher quality in terms of both objective and subjective criteria, compared with most of the state-of-the-art methods.

Intensity Guided Depth Upsampling Using Edge Sparsity and Super-Weighted $L_0$ Gradient Minimization

Although depth cameras acquire depth in dynamic scenes, the captured depth images are often noisy and of low resolution. Depth images have the physical nature of being represented by smooth regions and edges in between them, i.e. depth images have high edge sparsity in the gradient domain. In this paper, we propose intensity guided depth upsampling using edge sparsity and super-weighted L 0 gradient minimization. First, we get mutual structure between intensity and depth using joint mutual structure filtering. Second, we generate an initial depth image using recursive interpolation. Next, we generate weights for L 0 gradient minimization based on gradient and entropy of the intensity image, and upsample the depth image using super-weighted L 0 gradient minimization. In super-weighted L 0 gradient minimization, we combine two main terms: Hybrid data fidelity and weighted L 0 gradient regularization. The hybrid data fidelity term combines both zero-order and first-order data differences to suppress staircase artifacts, while the weighted L 0 gradient regularization term preserves depth structures and removes noise. Finally, we further refine the depth image using adaptive fast weighted median filtering. Experiments on Middlebury and realworld scene datasets verify that the proposed method produces edge preserving depth upsampling results and outperforms state-of-the-arts in terms of both visual quality and quantitative measurements.

3-D Reconstruction of Human Body Shape From a Single Commodity Depth Camera

3-D human body reconstruction is an important research topic in computer vision. A 3-D human body model can be used in sports science, movie industry and personalized entertainment, especially virtual reality games. Most of depth-based 3-D reconstruction algorithms need multiple cameras surrounding the user and require the user to keep a specific pose strictly while capturing depth images. In this paper, we propose an algorithm to reconstruct the 3-D shape of human bodies using a single commodity depth camera. Our algorithm only needs two depth images of the front-facing and back-facing bodies. It also has strong operability since the proposed method is insensitive to the pose variations between the two depth images. We reconstruct 3-D shapes of front-facing and back-facing bodies from the two depth images, respectively, and stitch them together. We also propose a novel registration method, namely, “iterative mid-distance points,” which has fast convergence and robustness to the depth noise. The proposed method enables robust and easy-to-use human body reconstruction, and achieves higher accuracy than state-of-the-art methods.

3D Scanning of High Dynamic Scenes Using an RGB-D Sensor and an IMU on a Mobile Device

With the development of RGB-D sensors and mobile devices, 3D scanning has witnessed great progress in recent years. KinectFusion opens up an era of RGB-D 3D reconstruction, which integrates captured depth images into a voxel-based representation. A number of improvements have been applied to the KinectFusion to reduce large footprint and make the 3D reconstruction on mobile devices possible. However, these methods are designed to handle static scenes. In this paper, we propose a method which can perform the 3D scanning of high dynamic scenes using an RGB-D sensor and an inertial measurement unit (IMU) on a mobile device. We first introduce a novel method to segment the depth images into static and dynamic elements with the use of sparse optical flow and the rotational part of the IMU. Then, we determine the camera pose with pixels labeled as static. At last depth, images are integrated into a voxel-based representation. The truncated signed distance function values of static voxels are updated while dynamic voxels are set to free-space. The experiments show that compared with some state-of-the-art systems, our method has better results when scanning high dynamic scenes and has comparable results when scanning low dynamic and static scenes. Besides, our method processes eight frames/s on an Apple iPad Air 2.

Comparison of Four SVM Classifiers Used with Depth Sensors to Recognize Arabic Sign Language Words

The objective of this research was to recognize the hand gestures of Arabic Sign Language (ArSL) words using two depth sensors. The researchers developed a model to examine 143 signs gestured by 10 users for 5 ArSL words (the dataset). The sensors captured depth images of the upper human body, from which 235 angles (features) were extracted for each joint and between each pair of bones. The dataset was divided into a training set (109 observations) and a testing set (34 observations). The support vector machine (SVM) classifier was set using different parameters on the gestured words’ dataset to produce four SVM models, with linear kernel (SVMLD and SVMLT) and radial kernel (SVMRD and SVMRT) functions. The overall identification accuracy for the corresponding words in the training set for the SVMLD, SVMLT, SVMRD, and SVMRT models was 88.92%, 88.92%, 90.88%, and 90.884%, respectively. The accuracy from the testing set for SVMLD, SVMLT, SVMRD, and SVMRT was 97.059%, 97.059%, 94.118%, and 97.059%, respectively. Therefore, since the two kernels in the models were close in performance, it is far more efficient to use the less complex model (linear kernel) set with a default parameter.

Joint Denoising/Compression of Image Contours via Shape Prior and Context Tree.

The advent of depth sensing technologies means that the extraction of object contours in images-a common and important pre-processing step for later higher level computer vision tasks like object detection and human action recognition-has become easier. However, captured depth images contain acquisition noise and the detected contours suffer from errors as a result. In this paper, we propose to jointly denoise and compress detected contours in an image for bandwidth-constrained transmission to a client, who can then carry out aforementioned application-specific tasks using the decoded contours as input. First, we prove theoretically that in general a joint denoising/compression approach can outperform a separate two-stage approach that first denoises then encodes contours lossily. Adopting a joint approach, we propose a burst error model that models typical errors encountered in an observed string of directional edges. We then formulate a rate-constrained maximum a posteriori problem that trades off the posterior probability of an estimated string given with its code rate. We design a dynamic programming algorithm that solves the posed problem optimally, and propose a compact context representation called total suffix tree that can reduce complexity of the algorithm dramatically. To the best of our knowledge, we are the first in the literature to study the problem of joint denoising/compression of image contours and offer a computation-efficient optimization algorithm. Experimental results show that our joint denoising/compression scheme can reduce bitrate by up to 18% compared with a competing separate scheme at comparable visual quality.

Estimating Heart Rate and Rhythm via 3D Motion Tracking in Depth Video

Low-cost depth sensors, such as Microsoft Kinect, have potential for noncontact health monitoring that is robust to ambient lighting conditions. However, captured depth images typically suffer from high acquisition noise, and hence, processing them to estimate biometrics is difficult. In this paper, we propose to capture depth video of a human subject using Kinect 2.0 to estimate his/her heart rate and rhythm; as blood is pumped from the heart to circulate through the head, tiny oscillatory head motion due to Newtonian mechanics can be detected for periodicity analysis. Specifically, we first restore a captured depth video via a joint bit-depth enhancement/denoising procedure, using a graph-signal smoothness prior for regularization. Second, we track an automatically detected head region throughout the depth video to deduce 3D motion vectors. The detected vectors are fed back to the depth restoration module in a loop to ensure that the motion information in two modules is consistent, improving performance of both restoration and motion tracking. Third, the computed 3D motion vectors are projected onto its principal component for 1D signal analysis, composed of trend removal, bandpass filtering, and wavelet-based motion denoising. Finally, the heart rate is estimated via Welch power spectrum analysis, and the heart rhythm is computed via peak detection. Experimental results show accurate estimation of the heart rate and rhythm using our proposed algorithm as compared to rate and rhythm estimated by a portable oximeter.

Surface Reconstruction via Fusing Sparse-Sequence of Depth Images.

Handheld scanning using commodity depth cameras provides a flexible and low-cost manner to get 3D models. The existing methods scan a target by densely fusing all the captured depth images, yet most frames are redundant. The jittering frames inevitably embedded in handheld scanning process will cause feature blurring on the reconstructed model and even trigger the scan failure (i.e., camera tracking losing). To address these problems, in this paper, we propose a novel sparse-sequence fusion (SSF) algorithm for handheld scanning using commodity depth cameras. It first extracts related measurements for analyzing camera motion. Then based on these measurements, we progressively construct a supporting subset for the captured depth image sequence to decrease the data redundancy and the interference from jittering frames. Since SSF will reveal the intrinsic heavy noise of the original depth images, our method introduces a refinement process to eliminate the raw noise and recover geometric features for the depth images selected into the supporting subset. We finally obtain the fused result by integrating the refined depth images into the truncated signed distance field (TSDF) of the target. Multiple comparison experiments are conducted and the results verify the feasibility and validity of SSF for handheld scanning with a commodity depth camera.

Depth Map Super-Resolution Considering View Synthesis Quality.

Accurate and high-quality depth maps are required in lots of 3D applications, such as multi-view rendering, 3D reconstruction and 3DTV. However, the resolution of captured depth image is much lower than that of its corresponding color image, which affects its application performance. In this paper, we propose a novel depth map super-resolution (SR) method by taking view synthesis quality into account. The proposed approach mainly includes two technical contributions. First, since the captured low-resolution (LR) depth map may be corrupted by noise and occlusion, we propose a credibility based multi-view depth maps fusion strategy, which considers the view synthesis quality and interview correlation, to refine the LR depth map. Second, we propose a view synthesis quality based trilateral depth-map up-sampling method, which considers depth smoothness, texture similarity and view synthesis quality in the up-sampling filter. Experimental results demonstrate that the proposed method outperforms state-of-the-art depth SR methods for both super-resolved depth maps and synthesized views. Furthermore, the proposed method is robust to noise and achieves promising results under noise-corruption conditions.

Intensity-guided edge-preserving depth upsampling through weighted L0 gradient minimization

Depth is an important visual cue to perceive real-world scenes. Although a time-of-flight (ToF) depth camera can provide depth information in dynamic scenes, captured depth images are often noisy and of low resolution. In this paper, we propose an intensity-guided edge-preserving depth upsampling method through weighted L0 gradient minimization to enhance both resolution and visual quality of depth images. Guided by the high-resolution intensity image, we perform optimization to preserve boundaries of objects. We apply L0 gradient to the regularization term, and compute its weight from both intensity and depth images. We optimize the objective function using alternating minimization and half-quadratic splitting. Experimental results on Middlebury 2005, 2014, and real-world scene datasets demonstrate that the proposed method produces boundary-preserving depth upsampling results and outperforms state-of-the-art ones in terms of accuracy.

Fast Depth Video Compression for Mobile RGB-D Sensors

We propose a new method, called 3-D image warping-based depth video compression (IW-DVC), for fast and efficient compression of depth images captured by mobile RGB-D sensors. The emergence of low-cost RGB-D sensors has created opportunities to find new solutions for a number of computer vision and networked robotics problems, such as 3-D map building, immersive telepresence, or remote sensing. However, efficient transmission and storage of depth data still presents a challenging task to the research community in these applications. Image/video compression has been comprehensively studied and several methods have already been developed. However, these methods result in unacceptably suboptimal outcomes when applied to the depth images. We have designed the IW-DVC method to exploit the special properties of the depth data to achieve a high compression ratio while preserving the quality of the captured depth images. Our solution combines the egomotion estimation and 3-D image warping techniques and includes a lossless coding scheme that is capable of adapting to depth data with a high dynamic range. IW-DVC operates at a high speed, suitable for real-time applications, and is able to attain an enhanced motion compensation accuracy compared with the conventional approaches. In addition, it removes the existing redundant information between the depth frames to further increase compression efficiency. Our experiments show that IW-DVC attains a very high performance yielding significant compression ratios without sacrificing image quality.

Touch Pen Using Depth Information

Current touch pen requires the special equipments to detect a touch and its price increases in proportion to the screen size. In this paper, we propose a method for detecting a touch and implementing a pen using the depth information. The proposed method obtains a background depth image using a depth camera and extracts an object by comparing a captured depth image with the background depth image. Also, we determine a touch if the depth value of the object is the same as the background and then provide the pen event. Using this method, we can implement a cheaper and more convenient touch pen.

Rotated top-bottom dual-kinect for improved field of view

Existing commodity depth sensors have limited the field of view (FOV) of depth scanning. Our solution for extending the FOV is to use multiple depth sensors and stitch the captured depth images to a depth panorama. In our case study, we use two Kinects to address the following two questions: what is the best layout of the two Kinects to maximize the FOV and, second, how to combine the depth images together to form the depth panorama. We answer these questions by proposing a rotated top-bottom (RTB) arrangement of the two Kinects to maximize the FOV. Since the two Kinects capture the depth images from their own views, the depth values are not necessarily identical for the same object. To solve this problem, the depth adjustments are made for a frontal reference coordinate. Moreover, the perspective distortions of the two Kinects with respect to the frontal reference coordinate are corrected by perspective transformations. Experimental results show that our RTB sensor can generate panorama depth images with an almost doubled FOV.