Structured Light Depth Cameras Research Articles

An Efficient High-Throughput Structured-Light Depth Engine

In this article, an efficient high-throughput depth engine is proposed to generate high-quality 3-D depth maps for speckle-pattern structured-light depth cameras. A dynamic-binarization (DB) method is introduced with a significant reduction of computational complexity in contrast to the sum-of-absolute-distance (SAD) method. The depth map evaluation shows good robustness compared with other window-based correlation methods. Parallel architecture and reuse of intermediate results are employed for efficient hardware implementation. Our design is verified on a field-programmable gate array (FPGA) and implemented in the SMIC 55-nm CMOS technology, achieving a frame rate of 1731.77 fps ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times480$ </tex-math></inline-formula> ) with an area efficiency of 3.75 fps/KGE. The proposed engine shows a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.71\times $ </tex-math></inline-formula> promotion of area efficiency in contrast to the SAD-based implementation. In addition, the subpixel estimation algorithm deployed in postprocessing is optimized for efficient hardware implementation, reducing the gate count by 69.2% without significant performance loss.

BiLuNetICP: A Deep Neural Network for Object Semantic Segmentation and 6D Pose Recognition

The ability of understanding a scene and predicting the pose of objects has attracted significant interests in recent years. Specifically, it is used with visual sensors to provide the information for a robotic manipulator to interact with the target. Thus, 6D pose estimation and object recognition from point clouds or RGB-D images are important tasks for visual servoing. In this article, we propose a learning based approach to perform 6D pose estimation for robotic manipulation using a BiLuNetICP pipeline. It consists of a multi-path convolutional neural network (CNN) for semantic segmentation on RGB images. The network extracts the object mask and uses it to merge with the depth information to perform 6D pose estimation by the Iterative Closest Point (ICP) algorithm. We collected our own dataset for training and evaluate with Intersection over Union (IoU). The proposed method is able to provide better results compared with Unet++ using a small amount of training data. For the robotic grasping application, we test and evaluate our approach using a HIWIN 6-axis robot with Asus Xtion Live 3D camera and our structured-light depth camera. The experimental results demonstrate its efficiency in computation and the high success rate in grasping.

Co-axial depth sensor with an extended depth range for AR/VR applications

BackgroundDepth sensor is an essential element in virtual and augmented reality devices to digitalize users' environment in real time. The current popular technologies include the stereo, structured light, and Time-of-Flight (ToF). The stereo and structured light method require a baseline separation between multiple sensors for depth sensing, and both suffer from a limited measurement range. The ToF depth sensors have the largest depth range but the lowest depth map resolution. To overcome these problems, we propose a co-axial depth map sensor which is potentially more compact and cost-effective than conventional structured light depth cameras. Meanwhile, it can extend the depth range while maintaining a high depth map resolution. Also, it provides a high-resolution 2D image along with the 3D depth map. MethodsThis depth sensor is constructed with a projection path and an imaging path. Those two paths are combined by a beamsplitter for a co-axial design. In the projection path, a cylindrical lens is inserted to add extra power in one direction which creates an astigmatic pattern. For depth measurement, the astigmatic pattern is projected onto the test scene, and then the depth information can be calculated from the contrast change of the reflected pattern image in two orthogonal directions. To extend the depth measurement range, we use an electronically focus tunable lens at the system stop and tune the power to implement an extended depth range without compromising depth resolution. ResultsIn the depth measurement simulation, we project a resolution target onto a white screen which is moving along the optical axis and then tune the focus tunable lens power for three depth measurement subranges, namely, near, middle and far. In each sub-range, as the test screen moves away from the depth sensor, the horizontal contrast keeps increasing while the vertical contrast keeps decreasing in the reflected image. Therefore, the depth information can be obtained by computing the contrast ratio between features in orthogonal directions. ConclusionsThe proposed depth map sensor could implement depth measurement for an extended depth range with a co-axial design.

Robust structured-light depth mapping via recursive decomposition of binary codes.

Structured-light depth cameras rely on projecting and resolving coded patterns on a three-dimensional scene with high contrast. The front-end optics of such depth cameras impose a fundamental restriction on the depth-sensing range and accuracy: the patterns only remain sharp within the depth of field jointly determined by the camera and projector. We present here a robust method to improve the depth-sensing range and accuracy for a structured-light depth camera without changing the underlying optical design. Moreover, it shows the unique advantage in macrophotography of highly light-scattering objects. We analyze the proposed method theoretically and validate it in experiments.

A Feasibility Study on the Use of a Structured Light Depth-Camera for Three-Dimensional Body Measurements of Dairy Cows in Free-Stall Barns.

Frequent checks on livestock’s body growth can help reducing problems related to cow infertility or other welfare implications, and recognizing health’s anomalies. In the last ten years, optical methods have been proposed to extract information on various parameters while avoiding direct contact with animals’ body, generally causes stress. This research aims to evaluate a new monitoring system, which is suitable to frequently check calves and cow’s growth through a three-dimensional analysis of their bodies’ portions. The innovative system is based on multiple acquisitions from a low cost Structured Light Depth-Camera (Microsoft Kinect™ v1). The metrological performance of the instrument is proved through an uncertainty analysis and a proper calibration procedure. The paper reports application of the depth camera for extraction of different body parameters. Expanded uncertainty ranging between 3 and 15 mm is reported in the case of ten repeated measurements. Coefficients of determination R² > 0.84 and deviations lower than 6% from manual measurements where in general detected in the case of head size, hips distance, withers to tail length, chest girth, hips, and withers height. Conversely, lower performances where recognized in the case of animal depth (R² = 0.74) and back slope (R² = 0.12).

Differentiating facial incongruity and flatness in schizophrenia, using structured light camera data.

Incongruity between emotional experience and its outwardly expression is one of the prominent symptoms in schizophrenia. Though widely reported and used in clinical evaluation, this symptom is inadequately defined in the literature and may be confused with mere affect flattening. In this study we used structured-light depth camera and dedicated software to automatically measure facial activity of schizophrenia patients and healthy individuals during an emotionally evocative task. We defined novel measures for the congruence of emotional experience and emotional expression and for Flat Affect, compared them between patients and controls, and examined their consistency with clinical evaluation. We found incongruity in schizophrenia to be manifested in a less specific range of facial expressions in response to similar emotional stimuli, while the emotional experience remains intact. Our study also suggests that when taking into consideration affect flatness, no contextually inappropriate facial expressions are evident.

Interfered depth map recovery with texture guidance for multiple structured light depth cameras

In depth acquisition systems with multiple structured light depth cameras (SLDCs), interference between the devices causes interfered regions, which degrade depth quality and impair applications. This paper proposes a novel approach to recover the depth maps. Under the guidance of texture segments, interfered regions are categorized into flat regions and boundary regions. After that, different strategies are applied to the two kinds of regions because of their property differences. For flat regions, a Markov random field (MRF) model is utilized to get the optimal gradient solution. With the gradients, discrete Poisson equation (DPE) is applied to calculate the final depth solution. In boundary regions, another texture-guided MRF is utilized to peruse depth directly. Experiment results demonstrate that the proposed method works well on both synthetic data and captured data. In flat regions, we get smooth-varied depth, and boundaries between interfered and non-interfered regions are seamless. In boundary regions, sharp depth edges between objects are well preserved. Moreover, our method improves the PSNR of synthetic data by 4–14dB.

Towards precise real-time 3D difference detection for industrial applications

3D difference detection is the task to verify whether the 3D geometry of a real object exactly corresponds to a 3D model of this object. We present an approach for 3D difference detection with a hand-held depth camera. In contrast to previous approaches, with the presented approach geometric differences can be detected in real-time and from arbitrary viewpoints. The 3D difference detection accuracy is improved by two approaches: first, the precision of the depth camera's pose estimation is improved by coupling the depth camera with a high precision industrial measurement arm. Second, the influence of the depth measurement noise is reduced by integrating a 3D surface reconstruction algorithm. The effects of both enhancements are quantified by a ground-truth based quantitative evaluation, both for a time-of-flight (SwissRanger 4000) and a structured light depth camera (Kinect). With the proposed enhancements, differences of few millimeters can be detected from 1m measurement distance.