RGB-Depth Camera Research Articles

A study on improving the calibration of body scanner built on multiple RGB-Depth cameras

PurposeThe calibration is a key but cumbersome process for 3D body scanning using multiple depth cameras. The purpose of this paper is to simplify the calibration process by introducing a new method to calibrate the extrinsic parameters of multiple depth cameras simultaneously.Design/methodology/approachAn improved method is introduced to enhance the accuracy based on the virtual checkerboards. Laplace coordinates are employed for a point-to-point adjustment to increase the accuracy of scanned data. A system with eight depth cameras is developed for full-body scanning, and the performance of this system is verified by actual results.FindingsThe agreement of measurements between scanned human bodies and the real subjects demonstrates the accuracy of the proposed method. The entire calibration process is automatic.Originality/valueA complete algorithm for a full human body scanning system is introduced in this paper. This is the first publically study on the refinement and the point-by-point adjustment based on the virtual checkerboards toward the scanning accuracy enhancement.

Automatic foot scanning and measurement based on multiple RGB-depth cameras

In this paper, a system with six depth cameras was built to scan both feet simultaneously. An improved calibration method based on a T-shaped checkerboard was used to calculate the extrinsic parameters of the cameras. T-shaped virtual checkerboards were introduced to further fine-tune the accuracy of calibration based on the iterative closest point algorithm. Based on the proposed foot scanner, a complete procedure was introduced to measure the foot automatically by locating the anatomical landmarks without manual intervention. Various experiments were presented to validate the performance of the scanner and the measurements. The results verified that the proposed methods were efficient and versatile for three-dimensional foot scanning and measurement.

Hierarchical Activity Recognition Using Smart Watches and RGB-Depth Cameras.

Human activity recognition is important for healthcare and lifestyle evaluation. In this paper, a novel method for activity recognition by jointly considering motion sensor data recorded by wearable smart watches and image data captured by RGB-Depth (RGB-D) cameras is presented. A normalized cross correlation based mapping method is implemented to establish association between motion sensor data with corresponding image data from the same person in multi-person situations. Further, to improve the performance and accuracy of recognition, a hierarchical structure embedded with an automatic group selection method is proposed. Through this method, if the number of activities to be classified is changed, the structure will be changed correspondingly without interaction. Our comparative experiments against the single data source and single layer methods have shown that our method is more accurate and robust.

Enhanced RGB-D Mapping Method for Detailed 3D Indoor and Outdoor Modeling

RGB-D sensors (sensors with RGB camera and Depth camera) are novel sensing systems that capture RGB images along with pixel-wise depth information. Although they are widely used in various applications, RGB-D sensors have significant drawbacks including limited measurement ranges (e.g., within 3 m) and errors in depth measurement increase with distance from the sensor with respect to 3D dense mapping. In this paper, we present a novel approach to geometrically integrate the depth scene and RGB scene to enlarge the measurement distance of RGB-D sensors and enrich the details of model generated from depth images. First, precise calibration for RGB-D Sensors is introduced. In addition to the calibration of internal and external parameters for both, IR camera and RGB camera, the relative pose between RGB camera and IR camera is also calibrated. Second, to ensure poses accuracy of RGB images, a refined false features matches rejection method is introduced by combining the depth information and initial camera poses between frames of the RGB-D sensor. Then, a global optimization model is used to improve the accuracy of the camera pose, decreasing the inconsistencies between the depth frames in advance. In order to eliminate the geometric inconsistencies between RGB scene and depth scene, the scale ambiguity problem encountered during the pose estimation with RGB image sequences can be resolved by integrating the depth and visual information and a robust rigid-transformation recovery method is developed to register RGB scene to depth scene. The benefit of the proposed joint optimization method is firstly evaluated with the publicly available benchmark datasets collected with Kinect. Then, the proposed method is examined by tests with two sets of datasets collected in both outside and inside environments. The experimental results demonstrate the feasibility and robustness of the proposed method.

People detection and tracking using RGB-D cameras for mobile robots

People detection and tracking is an essential capability for mobile robots in order to achieve natural human–robot interaction. In this article, a human detection and tracking system is designed and validated for mobile robots using color data with depth information RGB-depth (RGB-D) cameras. The whole framework is composed of human detection, tracking and re-identification. Firstly, ground points and ceiling planes are removed to reduce computation effort. A prior-knowledge guided random sample consensus fitting algorithm is used to detect the ground plane and ceiling points. All left points are projected onto the ground plane and subclusters are segmented for candidate detection. Meanshift clustering with an Epanechnikov kernel is conducted to partition different points into subclusters. We propose the new idea of spatial region of interest plan view maps which are employed to identify human candidates from point cloud subclusters. Here, a depth-weighted histogram is extracted online to feature a human candidate. Then, a particle filter algorithm is adopted to track the human’s motion. The integration of the depth-weighted histogram and particle filter provides a precise tool to track the motion of human objects. Finally, data association is set up to re-identify humans who are tracked. Extensive experiments are conducted to demonstrate the effectiveness and robustness of our human detection and tracking system.

Robust Vision-Based Relative-Localization Approach Using an RGB-Depth Camera and LiDAR Sensor Fusion

This paper describes a robust vision-based relative-localization approach for a moving target based on an RGB-depth (RGB-D) camera and sensor measurements from two-dimensional (2-D) light detection and ranging (LiDAR). With the proposed approach, a target’s three-dimensional (3-D) and 2-D position information is measured with an RGB-D camera and LiDAR sensor, respectively, to find the location of a target by incorporating visual-tracking algorithms, depth information of the structured light sensor, and a low-level vision-LiDAR fusion algorithm, e.g., extrinsic calibration. To produce 2-D location measurements, both visual- and depth-tracking approaches are introduced, utilizing an adaptive color-based particle filter (ACPF) (for visual tracking) and an interacting multiple-model (IMM) estimator with intermittent observations from depth-image segmentation (for depth image tracking). The 2-D LiDAR data enhance location measurements by replacing results from both visual and depth tracking; through this procedure, multiple LiDAR location measurements for a target are generated. To deal with these multiple-location measurements, we propose a modified track-to-track fusion scheme. The proposed approach shows robust localization results, even when one of the trackers fails. The proposed approach was compared to position data from a Vicon motion-capture system as the ground truth. The results of this evaluation demonstrate the superiority and robustness of the proposed approach.

Color-aware surface registration

Shape registration is fundamental to 3D object acquisition; it is used to fuse scans from multiple views. Existing algorithms mainly utilize geometric information to determine alignment, but this typically results in noticeable misalignment of textures (i.e. surface colors) when using RGB-depth cameras. We address this problem using a novel approach to color-aware registration, which takes both color and geometry into consideration simultaneously. Color information is exploited throughout the pipeline to provide more effective sampling, correspondence and alignment, in particular for surfaces with detailed textures. Our method can furthermore tackle both rigid and non-rigid registration problems (arising, for example, due to small changes in the object during scanning, or camera distortions). We demonstrate that our approach produces significantly better results than previous methods.

Fast capture of textured full-body avatar with RGB-D cameras

We present a practical system which can provide a textured full-body avatar within 3 s. It uses sixteen RGB-depth (RGB-D) cameras, ten of which are arranged to capture the body, while six target the important head region. The configuration of the multiple cameras is formulated as a constraint-based minimum set space-covering problem, which is approximately solved by a heuristic algorithm. The camera layout determined can cover the full-body surface of an adult, with geometric errors of less than 5 mm. After arranging the cameras, they are calibrated using a mannequin before scanning real humans. The 16 RGB-D images are all captured within 1 s, which both avoids the need for the subject to attempt to remain still for an uncomfortable period, and helps to keep pose changes between different cameras small. All scans are combined and processed to reconstruct the photorealistic textured mesh in 2 s. During both system calibration and working capture of a real subject, the high-quality RGB information is exploited to assist geometric reconstruction and texture stitching optimization.

A REVIEW OF 3D GESTURE INTERACTION FOR HANDHELD AUGMENTED REALITY

Interaction for Handheld Augmented Reality (HAR) is a challenging research topic because of the small screen display and limited input options. Although 2D touch screen input is widely used, 3D gesture interaction is a suggested alternative input method. Recent 3D gesture interaction research mainly focuses on using RGB-Depth cameras to detect the spatial position and pose of fingers, using this data for virtual object manipulations in the AR scene. In this paper we review previous 3D gesture research on handheld interaction metaphors for HAR. We present their novelties as well as limitations, and discuss future research directions of 3D gesture interaction for HAR. Our results indicate that 3D gesture input on HAR is a potential interaction method for assisting a user in many tasks such as in education, urban simulation and 3D games.

Real-Time Head Pose Tracking with Online Face Template Reconstruction.

We propose a real-time method to accurately track the human head pose in the 3-dimensional (3D) world. Using a RGB-Depth camera, a face template is reconstructed by fitting a 3D morphable face model, and the head pose is determined by registering this user-specific face template to the input depth video.

A Survey of Human Action Recognition Approaches that use an RGB-D Sensor

Human action recognition from a video scene has remained a challenging problem in the area of computer vision and pattern recognition. The development of the low-cost RGB depth camera (RGB-D) allows new opportunities to solve the problem of human action recognition. In this paper, we present a comprehensive review of recent approaches to human action recognition based on depth maps, skeleton joints, and other hybrid approaches. In particular, we focus on the advantages and limitations of the existing approaches and on future directions.

Real-time Full-view 3D Human Reconstruction using Multiple RGB-D Cameras

This manuscript presents a real-time solution for 3D human body reconstruction with multiple RGB-D cameras. The proposed system uses four consumer RGB/Depth (RGB-D) cameras, each located at approximately 90° from the next camera around a freely moving human body. A single mesh is constructed from the captured point clouds by iteratively removing the estimated overlapping regions from the boundary. A cell-based mesh construction algorithm is developed, recovering the 3D shape from various conditions, considering the direction of the camera and the mesh boundary. The proposed algorithm also allows problematic holes and/or occluded regions to be recovered from another view. Finally, calibrated RGB data is merged with the constructed mesh so it can be viewed from an arbitrary direction. The proposed algorithm is implemented with general-purpose computation on graphics processing unit (GPGPU) for real-time processing owing to its suitability for parallel processing.

Using latent features for short-term person re-identification with RGB-D cameras

This paper presents a system for people re-identification in uncontrolled scenarios using RGB-depth cameras. Compared to conventional RGB cameras, the use of depth information greatly simplifies the tasks of segmentation and tracking. In a previous work, we proposed a similar architecture where people were characterized using color-based descriptors that we named bodyprints. In this work, we propose the use of latent feature models to extract more relevant information from the bodyprint descriptors by reducing their dimensionality. Latent features can also cope with missing data in case of occlusions. Different probabilistic latent feature models, such as probabilistic principal component analysis and factor analysis, are compared in the paper. The main difference between the models is how the observation noise is handled in each case. Re-identification experiments have been conducted in a real store where people behaved naturally. The results show that the use of the latent features significantly improves the re-identification rates compared to state-of-the-art works.

Application of the Kinect sensor for dynamic soil surface characterization

Agricultural soil roughness is pertinent to important agricultural phenomena, such as evaporation, infiltration or compression. Monitoring roughness variations would make possible the improvement of tillage operations. In the present work, implementation of the Microsoft Kinect™ RGB-depth camera for dynamic characterization of soil micro-relief is proposed and discussed. The metrological performance and the effect of the operating conditions on three-dimensional reconstruction was analyzed considering both laboratory tests on calibrated reference surfaces and field tests on different agricultural soil surfaces. Data set analysis was made on the basis of surface roughness parameters, as defined by ISO 25178 (2012) series: average roughness, root mean square roughness, skewness and kurtosis. Correlation between different tillage conditions and roughness parameters describing soil morphology was finally discussed.

Practical and accurate calibration of RGB-D cameras using spheres

RGB-Depth (or RGB-D) cameras are increasingly being adopted in robotic and vision applications, including mobile robot localization and mapping, gesture recognition, and at-home healthcare monitoring. As with any other sensor, calibrating RGB-D cameras is needed to increase their sensing accuracy, especially since the manufacturer’s calibration parameters might change between models. In this paper, we present a novel RGB-D camera-calibration algorithm for the estimation of the full set of intrinsic and extrinsic parameters. Our method is easy to use, can be utilized with any arrangement of RGB and depth sensors, and only requires that a spherical object (e.g., a basketball) is moved in front of the camera for a few seconds. Our image-processing pipeline automatically and robustly detects the moving calibration object while rejecting noise and outliers in the image data. Our calibration method uses all the frames of the detected sphere and leverages novel analytical results on the multi-view projection of spheres to accurately estimate all the calibration parameters. Extensive numerical simulations and real-world experiments have been conducted to validate our algorithm and compare its performance with that of other state-of-the-art calibration methods. An RGB-D Calibration Toolbox for MATLAB is also made freely available for the scientific community.

RGB-Depth 카메라를 이용한 현실-가상 융합 홀로그램 생성 시스템

Generating of digital hologram of video contents with computer graphics(CG) requires natural fusion of 3D information between real and virtual. In this paper, we propose the system which can fuse real-virtual 3D information naturally and fast generate the digital hologram of fused results using multiple-GPUs based computer-generated-hologram(CGH) computing part. The system calculates camera projection matrix of RGB-Depth camera, and estimates the 3D information of virtual object. The 3D information of virtual object from projection matrix and real space are transmitted to Z buffer, which can fuse the 3D information, naturally. The fused result in Z buffer is transmitted to multiple-GPUs based CGH computing part. In this part, the digital hologram of fused result can be calculated fast. In experiment, the 3D information of virtual object from proposed system has the mean relative error(MRE) about 0.5138% in relation to real 3D information. In other words, it has the about 99% high-accuracy. In addition, we verify that proposed system can fast generate the digital hologram of fused result by using multiple GPUs based CGH calculation.

Lightweight wrinkle synthesis for 3D facial modeling and animation

We present a lightweight non-parametric method to generate wrinkles for 3D facial modeling and animation. The key lightweight feature of the method is that it can generate plausible wrinkles using a single low-cost Kinect camera and one high quality 3D face model with details as the example. Our method works in two stages: (1) offline personalized wrinkled blendshape construction. User-specific expressions are recorded using the RGB-Depth camera, and the wrinkles are generated through example-based synthesis of geometric details. (2) Online 3D facial performance capturing. These reconstructed expressions are used as blendshapes to capture facial animations in real-time. Experiments on a variety of facial performance videos show that our method can produce plausible results, approximating the wrinkles in an accurate way. Furthermore, our technique is low-cost and convenient for common users.

Three-Dimensional Indoor Mobile Mapping With Fusion of Two-Dimensional Laser Scanner and RGB-D Camera Data

Three-dimensional mobile mapping in indoor environment, mostly global navigation satellite system-denied space, is to consecutively align the frames to build a global 3-D map of an indoor environment. One of the major difficulties of the current solutions is the failure at the insufficient overlapping between the frames, which is the reality of a lack of correspondences between the frames. To overcome this problem, a 3-D indoor mobile mapping system that integrates a 2-D laser scanner, and an RGB-Depth camera is presented in this letter. In this system, a fusion-iterative closest point (ICP) method, which combines the 2-D mobile platform pose from a Rao-Blackwellized particle filter estimation, an ICP, and a generalized-ICP method, is proposed for the consecutive frame alignment. Fusion-ICP achieves effective frame alignment, particularly in solving the insufficient overlapping frame alignment problem. Comparative experiments were conducted to evaluate the mapping system. The experimental results demonstrate the effectiveness and efficiency of our system for 3-D indoor mobile mapping.

키넥트를 이용한 실내에서의 키 추정 방법

객체 인식은 지능적이고 다양화된 범죄 예방을 위해 감시 시스템에서 중요한 기술이다. 사람의 신체 정보인 키는 대상이 가지고 있는 신체적인 특징으로 신원을 확인하는데 중요한 정보가 될 수 있다. 본 논문에서는 RGB-Depth 카메라, 키넥트를 활용한 새로운 키 추정 방법을 제안한다. 사람의 키를 측정하기 위해 키넥트의 높이를 알고 있는 것으로 가정하고, 키넥트에서 머리와 발까지의 거리를 키넥트의 깊이 정보를 이용하여 사람의 키를 추정한다. 실험을 통하여 제안하는 방법이 실내에서 사람의 키를 추정하는데 효과적임을 확인한다. Object recognition is one of the key technologies of the monitoring system for the prevention of crimes diversified the intelligent. The height is one of the physical information of the person, it may be important information to confirm the identity with physical characteristics of the subject has. In this paper, we provide a method of measuring the height that utilize RGB-Depth camera, the Kinect. Given that in order to measure the height of a person, and know the height of Kinect, by using the depth information of Kinect the distance to the head and foot of Kinect, estimating the height of a person. The proposed method throughout the experiment confirms that it is effective to estimate the height of a person in the room.

RGB-Depth 카메라 기반의 실내 연기검출

본 논문에서 RGB-Depth 카메라를 이용하여 실내에서의 연기를 검출하는 알고리즘을 제안한다. RGB-Depth 카메라는 RGB 색영상과 깊이 정보를 제공한다. 키넥트(Kinect)는 특정한 패턴의 적외선을 출력하고 이를 적외선 카메라로 수집하고 분석하여 깊이 정보를 획득한다. 특정한 패턴을 구성하는 점들 각각에 대하여 거리를 측정하고 객체면의 깊이를 추정한다. 따라서, 이웃하는 점들의 깊이 변화가 많은 객체인 경우에는 객체면의 깊이를 결정하지 못한다. 연기의 농도가 일정 주파수로 변화하고, 적외선 영상의 이웃하는 화소간의 변화가 많기 때문에 키넥트가 깊이를 결정하지 못한다. 본 논문에서는 연기에 대한 키넥트의 특성을 이용하여 연기를 검출한다. 키넥트가 깊이를 결정하지 못한 영역을 후보영역으로 설정하고, 색영상의 밝기가 임계값보다 큰 경우 연기영역으로 결정한다. 본 논문에서는 시뮬레이션을 통하여 실내에서의 연기 검출에 RGB-Depth 카메라가 효과적임을 확인할 수 있다. In this paper, an algorithm using RGB-depth camera is proposed to detect smoke in interrior. RGB-depth camera, the Kinect provides RGB color image and depth information. The Kinect sensor consists of an infra-red laser emitter, infra-red camera and an RGB camera. A specific pattern of speckles radiated from the laser source is projected onto the scene. This pattern is captured by the infra-red camera and is analyzed to get depth information. The distance of each speckle of the specific pattern is measured and the depth of object is estimated. As the depth of object is highly changed, the depth of object plain can not be determined by the Kinect. The depth of smoke can not be determined too because the density of smoke is changed with constant frequency and intensity of infra-red image is varied between each pixels. In this paper, a smoke detection algorithm using characteristics of the Kinect is proposed. The region that the depth information is not determined sets the candidate region of smoke. If the intensity of the candidate region of color image is larger than a threshold, the region is confirmed as smoke region. As results of simulations, it is shown that the proposed method is effective to detect smoke in interior.