Raw Depth Research Articles

Human Depth Sensors-Based Activity Recognition Using Spatiotemporal Features and Hidden Markov Model for Smart Environments

Nowadays, advancements in depth imaging technologies have made human activity recognition (HAR) reliable without attaching optical markers or any other motion sensors to human body parts. This study presents a depth imaging-based HAR system to monitor and recognize human activities. In this work, we proposed spatiotemporal features approach to detect, track, and recognize human silhouettes using a sequence of RGB-D images. Under our proposed HAR framework, the required procedure includes detection of human depth silhouettes from the raw depth image sequence, removing background noise, and tracking of human silhouettes using frame differentiation constraints of human motion information. These depth silhouettes extract the spatiotemporal features based on depth sequential history, motion identification, optical flow, and joints information. Then, these features are processed by principal component analysis for dimension reduction and better feature representation. Finally, these optimal features are trained and they recognized activity using hidden Markov model. During experimental results, we demonstrate our proposed approach on three challenging depth videos datasets including IM-DailyDepthActivity, MSRAction3D, and MSRDailyActivity3D. All experimental results show the superiority of the proposed approach over the state-of-the-art methods.

Simulating Kinect Infrared and Depth Images.

With the emergence of the Microsoft Kinect sensor, many developer communities and research groups have found countless uses and have already published a wide variety of papers that utilize the raw depth images for their specific goals. New methods and applications that use the device generally require an appropriately large ensemble of data sets with accompanying ground truth for testing purposes, as well as accurate models that account for the various systematic and stochastic contributors to Kinect errors. Current error models, however, overlook the intermediate infrared (IR) images that directly contribute to noisy depth estimates. We, therefore, propose a high fidelity Kinect IR and depth image predictor and simulator that models the physics of the transmitter/receiver system, unique IR dot pattern, disparity/depth processing technology, and random intensity speckle and IR noise in the detectors. The model accounts for important characteristics of Kinect's stereo triangulation system, including depth shadowing, IR dot splitting, spreading, and occlusions, correlation-based disparity estimation between windows of measured and reference IR images, and subpixel refinement. Results show that the simulator accurately produces axial depth error from imaged flat surfaces with various tilt angles, as well as the bias and standard lateral error of an object's horizontal and vertical edge.

High-coverage sequencing and annotated assembly of the genome of the Australian dragon lizard Pogona vitticeps.

BackgroundThe lizards of the family Agamidae are one of the most prominent elements of the Australian reptile fauna. Here, we present a genomic resource built on the basis of a wild-caught male ZZ central bearded dragon Pogona vitticeps.FindingsThe genomic sequence for P. vitticeps, generated on the Illumina HiSeq 2000 platform, comprised 317 Gbp (179X raw read depth) from 13 insert libraries ranging from 250 bp to 40 kbp. After filtering for low-quality and duplicated reads, 146 Gbp of data (83X) was available for assembly. Exceptionally high levels of heterozygosity (0.85 % of single nucleotide polymorphisms plus sequence insertions or deletions) complicated assembly; nevertheless, 96.4 % of reads mapped back to the assembled scaffolds, indicating that the assembly included most of the sequenced genome. Length of the assembly was 1.8 Gbp in 545,310 scaffolds (69,852 longer than 300 bp), the longest being 14.68 Mbp. N50 was 2.29 Mbp. Genes were annotated on the basis of de novo prediction, similarity to the green anole Anolis carolinensis, Gallus gallus and Homo sapiens proteins, and P. vitticeps transcriptome sequence assemblies, to yield 19,406 protein-coding genes in the assembly, 63 % of which had intact open reading frames. Our assembly captured 99 % (246 of 248) of core CEGMA genes, with 93 % (231) being complete.ConclusionsThe quality of the P. vitticeps assembly is comparable or superior to that of other published squamate genomes, and the annotated P. vitticeps genome can be accessed through a genome browser available at https://genomics.canberra.edu.au.Electronic supplementary materialThe online version of this article (doi:10.1186/s13742-015-0085-2) contains supplementary material, which is available to authorized users.

An RGB–D based image set classification for robust face recognition from Kinect data

The paper proposes a method for robust face recognition from low quality Kinect acquired images which have a wide range of variations in head pose, illumination, facial expressions, sunglass disguise and occlusions by hand. Multiple Kinect images of a person are considered as an image set and face recognition from these images is formulated as an RGB–D image set classification problem. The Kinect acquired raw depth data is used for pose estimation and an automatic cropping of the face region. Based upon the estimated poses, the face images of a set are divided into multiple image subsets. An efficient block based covariance matrix representation is proposed to model images in an image subset on Riemannian manifold (Lie group). For classification, SVM models are separately learnt for each image subset on the Lie group of Riemannian manifold and a fusion strategy is introduced to combine results from all image subsets. The proposed technique has been evaluated on a combination of three large data sets containing over 35,000 RGB–D images under challenging conditions. The proposed RGB–D based image set classification incurs low computational cost and achieves an identification rate as high as 99.5%.

Unified multi-lateral filter for real-time depth map enhancement

This paper proposes a unified multi-lateral filter to efficiently increase the spatial resolution of low-resolution and noisy depth maps in real-time. Time-of-Flight (ToF) cameras have become a very promising alternative to stereo-based range sensing systems as they provide depth measurements at a high frame rate. However, there are actually two main drawbacks that restrict their use in a wide range of applications; namely, their fairly low spatial resolution as well as the amount of noise within the depth estimation. In order to address these drawbacks, we propose a new approach based on sensor fusion. That is, we couple a ToF camera of low-resolution with a 2-D camera of higher resolution to which the low-resolution depth map will be efficiently upsampled. In this paper, we first review the existing depth map enhancement approaches based on sensor fusion and discuss their limitations. We then propose a unified multi-lateral filter that accounts for the inaccuracy of depth edges position due to the low-resolution ToF depth maps. By doing so, unwanted artefacts such as texture copying and edge blurring are almost entirely eliminated. Moreover, the proposed filter is configurable to behave as most of the alternative depth enhancement approaches. Using a convolution-based formulation and data quantization and downsampling, the described filter has been effectively and efficiently implemented for dynamic scenes in real-time applications. The experimental results show a sensitive qualitative as well as quantitative improvement on raw depth maps, outperforming state-of-the-art multi-lateral filters.

Subset based deep learning for RGB-D object recognition

RGB-D camera can easily record both color and depth images and previous works have proved that combining them together could dramatically improve the RGB-D based object recognition accuracy. In this paper, a new method based on a subset approach was introduced to learn higher level features from the raw data. The raw RGB and depth images were divided into several subsets according to their shapes and colors, guaranteeing that any two different objects in each subset are nearly not similar. Then a RGB-Subset-Sparse auto-encoder was trained to extract features from RGB images and a Depth-Subset-Sparse auto-encoder was trained to extract features from depth images for each subset. Then the learned features were transmitted to recursive neural networks (RNNs) to reduce the dimensionality of the features and learn robust hierarchical feature representations. The feature representations learned from RGB images and depth images were concatenated as the final features and then sent to a softmax classifier for classification. The proposed method is evaluated on three benchmark RGB-D datasets, RGB-D dataset of Lai et al., 2D3D dataset of Browatzki et al. and Aharon dataset of Aharon et al. Compared with other methods, ours achieves state-of-the-art performance on the first two datasets. Furthermore, to validate the generalization of our subset approach, we also do some extra experiments of applying the subsets approach to several previous works, these accuracies improved significantly.

Real-time shading-based refinement for consumer depth cameras

We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. Per frame, our real-time algorithm takes raw noisy depth data and an aligned RGB image as input, and approximates the time-varying incident lighting, which is then used for geometry refinement. This leads to dramatically enhanced depth maps at 30Hz. Our algorithm makes few scene assumptions, handling arbitrary scene objects even under motion. To enable this type of real-time depth map enhancement, we contribute a new highly parallel algorithm that reformulates the inverse rendering optimization problem in prior work, allowing us to estimate lighting and shape in a temporally coherent way at video frame-rates. Our optimization problem is minimized using a new regular grid Gauss-Newton solver implemented fully on the GPU. We demonstrate results showing enhanced depth maps, which are comparable to offline methods but are computed orders of magnitude faster, as well as baseline comparisons with online filtering-based methods. We conclude with applications of our higher quality depth maps for improved real-time surface reconstruction and performance capture.

Depth map super-resolution using stereo-vision-assisted model

Abstract In this paper, we propose a novel Stereo-Vision-Assisted (SVA) model for depth map super-resolution. Given a low-resolution depth map as input, we recover a high-resolution depth map using the registered high-resolution color stereo image pair. First, based on the mutual benefits between raw depth map and features of high resolution color image, we model the relationship with two constraint terms of local and non-local priors and sufficiently explore their complementary nature. Moreover, by considering reliable disparity pixels calculated from stereo matching algorithm, we formulate a stereo disparity regularization term to further reinforce the preservation of fine depth detail. Hence, our SVA objective function includes a non-local prior constraint, a local prior constraint and a stereo disparity prior constraint. In addition, we employ an efficient algorithm to optimize the objective function. Experimental results demonstrate that our approach can obtain superior performance in comparison with other methods.

INITIAL RESULTS FROMNuSTAROBSERVATIONS OF THE NORMA ARM

Results are presented for an initial survey of the Norma Arm gathered with the focusing hard X-ray telescope NuSTAR. The survey covers 0.2 deg$^2$ of sky area in the 3-79 keV range with a minimum and maximum raw depth of 15 ks and 135 ks, respectively. Besides a bright black-hole X-ray binary in outburst (4U 1630-47) and a new X-ray transient (NuSTAR J163433-473841), NuSTAR locates three sources from the Chandra survey of this region whose spectra are extended above 10 keV for the first time: CXOU J163329.5-473332, CXOU J163350.9-474638, and CXOU J163355.1-473804. Imaging, timing, and spectral data from a broad X-ray range (0.3-79 keV) are analyzed and interpreted with the aim of classifying these objects. CXOU J163329.5-473332 is either a cataclysmic variable or a faint low-mass X-ray binary. CXOU J163350.9-474638 varies in intensity on year-long timescales, and with no multi-wavelength counterpart, it could be a distant X-ray binary or possibly a magnetar. CXOU J163355.1-473804 features a helium-like iron line at 6.7 keV and is classified as a nearby cataclysmic variable. Additional surveys are planned for the Norma Arm and Galactic Center, and those NuSTAR observations will benefit from the lessons learned during this pilot study.

Depth map super-resolution via local and nonlocal priors

We propose a method for solving the depth map super-resolution problem. Given a low-resolution depth map as input, we recover a high-resolution depth map using a registered and potentially high-resolution color image. We formulate the super-resolution process as a regularization and optimization framework. Specifically, taking into account that discontinuities in depth and color tend to coalign, we obtain the local and nonlocal regularization terms based on the raw depth map and also the features of high-resolution color images. Different from conventional methods, we model the relationship with two constraint terms of local and nonlocal priors to sufficiently explore the complementary nature. Experimental results demonstrate the effectiveness of our approach, which produces sharper edges and more faithful details compared with other state-of-the-art strategies.

Determination of interface locations and layer thicknesses in SIMS and AES depth profiling of Si/Ti multilayer films by 50 at% definition

Determination of the interface locations and the layer thicknesses in SIMS depth profiling of Si/Ti multilayer films was investigated by cesium ion beam. The raw SIMS depth profiles were converted to the compositional depth profiles by the relative sensitivity factors of Si and Ti derived from the atomic fractions of a reference Si‐Ti alloy film determined by Rutherford backscattering spectroscopy. The locations of the interfaces were determined by 50 at% definition in the compositional SIMS depth profiles, and the thicknesses of the Si and Ti layers were measured from the interfaces. The layer thicknesses of the Si/Ti multilayer film were also investigated by Auger electron spectroscopy depth profiling using the same process and same definition. Copyright © 2014 John Wiley & Sons, Ltd.

A Depth-Based Fall Detection System Using a Kinect® Sensor

We propose an automatic, privacy-preserving, fall detection method for indoor environments, based on the usage of the Microsoft Kinect® depth sensor, in an “on-ceiling” configuration, and on the analysis of depth frames. All the elements captured in the depth scene are recognized by means of an Ad-Hoc segmentation algorithm, which analyzes the raw depth data directly provided by the sensor. The system extracts the elements, and implements a solution to classify all the blobs in the scene. Anthropometric relationships and features are exploited to recognize one or more human subjects among the blobs. Once a person is detected, he is followed by a tracking algorithm between different frames. The use of a reference depth frame, containing the set-up of the scene, allows one to extract a human subject, even when he/she is interacting with other objects, such as chairs or desks. In addition, the problem of blob fusion is taken into account and efficiently solved through an inter-frame processing algorithm. A fall is detected if the depth blob associated to a person is near to the floor. Experimental tests show the effectiveness of the proposed solution, even in complex scenarios.

3P1-J02 Kinectを用いたラジオ体操自動評価システムにおける手の追跡(3次元計測/センサフュージョン)

In Japan, radio calisthenics is popular for a long time since we can do it anywhere. However, the effect of it depends on how seriously he or she plays it. Thus we have been developing an automatized evaluation system of a play by using Kinect. Our system utilizes the depth data acquired from it without using RGB data in order to avoid the effects of the environment and the clothes. We mainly use the joint detecting function of Kinect for Windows SDK, but it yields false detection signals in some special poses of the calisthenics. In this paper, we develop joint detection function algorithms of two chapters of the calisthenics by using the raw depth data.

Automatic alignment and reconstruction of facial depth images

Face, gender, ethnic and age group classification systems often work through an alignment, feature extraction, and identification pipeline. The quality of the alignment process is thus central to the performance of the identification process. Furthermore, missing portions of depth information can greatly affect results. Appropriate image reconstruction is therefore crucial for the correct operation of those systems. This paper presents a simple and effective approach for the automatic alignment and reconstruction of damaged facial depth images. By using only four facial landmarks and the raw depth data, our approach converts a given damaged depth image into a smooth depth function, performs the 3D alignment of the underlying face with the face of an average person, and produces an aligned depth image having arbitrary resolution. Our experiments show that the proposed approach outperforms commonly used methods. For instance, we show that it improves the quality of a state-of-art gender classification technique.

Depth map enhancement based on color and depth consistency

Current low-cost depth sensing techniques, such as Microsoft Kinect, still can achieve only limited precision. The resultant depth maps are often found to be noisy, misaligned with the color images, and even contain many large holes. These limitations make it difficult to be adopted by many graphics applications. In this paper, we propose a computational approach to address the problem. By fusing raw depth values with image color, edges and smooth priors in a Markov random field optimization framework, both misalignment and large holes can be eliminated effectively, our method thus can produce high-quality depth maps that are consistent with the color image. To achieve this, a confidence map is estimated for adaptive weighting of different cues, an image inpainting technique is introduced to handle large holes, and contrasts in the color image are also considered for an accurate alignment. Experimental results demonstrate the effectiveness of our method.

An improved edge detection algorithm for depth map inpainting

Three-dimensional (3D) measurement technology has been widely used in many scientific and engineering areas. The emergence of Kinect sensor makes 3D measurement much easier. However the depth map captured by Kinect sensor has some invalid regions, especially at object boundaries. These missing regions should be filled firstly. This paper proposes a depth-assisted edge detection algorithm and improves existing depth map inpainting algorithm using extracted edges. In the proposed algorithm, both color image and raw depth data are used to extract initial edges. Then the edges are optimized and are utilized to assist depth map inpainting. Comparative experiments demonstrate that the proposed edge detection algorithm can extract object boundaries and inhibit non-boundary edges caused by textures on object surfaces. The proposed depth inpainting algorithm can predict missing depth values successfully and has better performance than existing algorithm around object boundaries.

The Impact of the Search Depth on Chess Playing Strength

How deep does a chess Grandmaster think? This question has been asked many times, and yet there is hardly a definite answer. Raw depth and pure calculation are certainly not the only factors in the thinking process of a chess player, but it would be interesting to know more about the relationship between search depth and playing strength, so that the strength of a given player (which is usually expressed in terms of an Elo rating) can be said to correspond to a certain equivalent depth (of some given engine). Since the thinking depth of a human player is difficult to determine, we carry out an experiment with a chess engine running at different search depths in order to obtain an average score that can be translated into a rating difference in the Elo scale. However, knowing the rating difference is not sufficient; we need have at least one value of engine depth for which the corresponding Elo rating has been estimated, so that the Elo ratings for other values of search depth can also be determined. In order to obtain the Elo rating that corresponds to HOUDINI 1.5a 64-bit running at a fixed iteration depth of 20 plies, we carry out an analysis of the quality of play at the Candidates Tournament 2013. From these results, we show how the search depth of that particular engine correlates with the Elo ratings of human players. The paper also discusses related work on self-play experiments and the effect of diminishing returns, which becomes apparent in our experiment.

Laboratory demonstration of a mid-infrared AGPM vector vortex coronagraph

Coronagraphy is a powerful technique to achieve high contrast imaging and hence to image faint companions around bright targets. Various concepts have been used in the visible and near-infrared regimes, while coronagraphic applications in the mid-infrared remain nowadays largely unexplored. Vector vortex phase masks based on concentric subwavelength gratings show great promise for such applications. We aim at producing and validating the first high-performance broadband focal plane phase mask coronagraphs for applications in the mid-infrared regime, and in particular the L band with a fractional bandwidth of ~16% (3.5-4.1 \mu m). Based on rigorous coupled wave analysis, we designed an annular groove phase mask (AGPM) producing a vortex effect in the L band, and etched it onto a series of diamond substrates. The grating parameters were measured by means of scanning electron microscopy. The resulting components were then tested on a mid-infrared coronagraphic test bench. A broadband raw null depth of 2 x 10^{-3} was obtained for our best L-band AGPM after only a few iterations between design and manufacturing. This corresponds to a raw contrast of about 6 x 10^{-5} (10.5 mag) at 2\lambda/D. This result is fully in line with our projections based on rigorous coupled wave analysis modeling, using the measured grating parameters. The sensitivity to tilt and focus has also been evaluated. After years of technological developments, mid-infrared vector vortex coronagraphs finally become a reality and live up to our expectations. Based on their measured performance, our L-band AGPMs are now ready to open a new parameter space in exoplanet imaging at major ground-based observatories.

Kinect 깊이 카메라를 이용한 실감 원격 영상회의의 시선 맞춤 시스템

본 논문에서는 실감 원격 영상회의를 위한 시선 맞춤 시스템을 제안한다. 제안하는 방법은 적외선 구조광을 사용하는 Kinect 깊이 카메라를 이용해서 색상 영상과 깊이 영상을 획득하고, 깊이 영상을 이용해서 사용자를 배경으로부터 분리한다. 깊이 카메라로부터 획득한 가공되지 않은 깊이 영상은 다양한 형태의 잡음을 가지고 있기 때문에, 첫번째 전처리 과정으로 결합형 양방향 필터를 사용해서 잡음을 제거한다. 그 다음, 깊이값의 불연속성에 적응적인 저역 필터를 적용한다. 색상 영상과 전처리 과정을 거친 깊이 영상을 이용해서 우리는 가상시점에서의 화자를 3차원 모델로 복원한다. 전체 시스템은 GPU 기반의 병렬 프로그래밍을 통해 실시간 처리가 가능하도록 했다. 최종적으로, 우리는 시선이 조정된 원격의 화자 영상을 얻을 수 있게 된다. 실험 결과를 통해 제안하는 시스템이 자연스러운 화자간 시선 맞춤을 실시간으로 가능하게 하는 것을 확인했다. In this paper, we present a real-time eye contact system for realistic telepresence using a Kinect depth camera. In order to generate the eye contact image, we capture a pair of color and depth video. Then, the foreground single user is separated from the background. Since the raw depth data includes several types of noises, we perform a joint bilateral filtering method. We apply the discontinuity-adaptive depth filter to the filtered depth map to reduce the disocclusion area. From the color image and the preprocessed depth map, we construct a user mesh model at the virtual viewpoint. The entire system is implemented through GPU-based parallel programming for real-time processing. Experimental results have shown that the proposed eye contact system is efficient in realizing eye contact, providing the realistic telepresence.

D.E. Anderson, D.M. Rings, Current Veterinary Therapy Food Animal Practice, 5th ed., Saunders Elsevier, 2008, ISBN9781416035916, 736 pp.; £110.00 (hard)

In this paper, we propose a novel Stereo-Vision-Assisted (SVA) model for depth map super-resolution. Given a low-resolution depth map as input, we recover a high-resolution depth map using the registered high-resolution color stereo image pair. First, based on the mutual benefits between raw depth map and features of high resolution color image, we model the relationship with two constraint terms of local and non-local priors and sufficiently explore their complementary nature. Moreover, by considering reliable disparity pixels calculated from stereo matching algorithm, we formulate a stereo disparity regularization term to further reinforce the preservation of fine depth detail. Hence, our SVA objective function includes a non-local prior constraint, a local prior constraint and a stereo disparity prior constraint. In addition, we employ an efficient algorithm to optimize the objective function. Experimental results demonstrate that our approach can obtain superior performance in comparison with other methods.