Depth Image Sequences Research Articles

Insights on Using Time-of-Flight Camera for Recovering Cardiac Pulse From Chest Motion in Depth Videos.

In this article, we introduce a novel use of depth camera to extract cardiac pulse signal from human chest area, in which the depth information is obtained from a near infrared sensor using time-of-flight technology. We successfully isolate weak chest motion due to heartbeat by processing a sequence of depth images without raising privacy concern. We discuss motion sensitivity in depth video with examples from actuator simulation and human chest motion. Compared to other imaging modalities, the depth image intensity can be directly used for micromotion reconstruction. To deal with the challenges of recovering heartbeat from the chest area, we develop a set of coherent processing techniques to suppress the unwanted motion interference from breathing motion and involuntary body motion and eventually obtain clean cardiac pulse signal. We, thus, derive inter-beat-interval, showing high consistency to the contact photoplethysmography. Additionally, we develop a graphical interpretation of the most and the less pulsatile principal components in eigen space. For validation, we test our method on ten healthy human subjects with different resting heart rates. More importantly, we conduct a set of experiments to study the robustness and weakness of our methods, including extended range, multi-subject, thickness of clothes and generation to other measurement site.

Spike representation of depth image sequences and its application to hand gesture recognition with spiking neural network

Spike representation of depth image sequences and its application to hand gesture recognition with spiking neural network

Computer-Aided Depth Video Stream Masking Framework for Human Body Segmentation in Depth Sensor Images.

The identification of human activities from videos is important for many applications. For such a task, three-dimensional (3D) depth images or image sequences (videos) can be used, which represent the positioning information of the objects in a 3D scene obtained from depth sensors. This paper presents a framework to create foreground–background masks from depth images for human body segmentation. The framework can be used to speed up the manual depth image annotation process with no semantics known beforehand and can apply segmentation using a performant algorithm while the user only adjusts the parameters, or corrects the automatic segmentation results, or gives it hints by drawing a boundary of the desired object. The approach has been tested using two different datasets with a human in a real-world closed environment. The solution has provided promising results in terms of reducing the manual segmentation time from the perspective of the processing time as well as the human input time.

3DFCNN: real-time action recognition using 3D deep neural networks with raw depth information

This work describes an end-to-end approach for real-time human action recognition from raw depth image-sequences. The proposal is based on a 3D fully convolutional neural network, named 3DFCNN, which automatically encodes spatio-temporal patterns from raw depth sequences. The described 3D-CNN allows actions classification from the spatial and temporal encoded information of depth sequences. The use of depth data ensures that action recognition is carried out protecting people’s privacy, since their identities can not be recognized from these data. The proposed 3DFCNN has been optimized to reach a good performance in terms of accuracy while working in real-time. Then, it has been evaluated and compared with other state-of-the-art systems in three widely used public datasets with different characteristics, demonstrating that 3DFCNN outperforms all the non-DNN-based state-of-the-art methods with a maximum accuracy of 83.6% and obtains results that are comparable to the DNN-based approaches, while maintaining a much lower computational cost of 1.09 seconds, what significantly increases its applicability in real-world environments.

Baggage Weight Classification by Extracting Temporal Features of Body Sway Acquired from Depth Image Sequences

We propose a method for classifying the weight of baggage carried by a person in an upright posture by finding temporal cues of body sway from depth image sequences. When a standing person is viewed from an overhead depth camera, body sway, which is a slight movement that naturally occurred in the human body, is observed. We consider body sway as discriminative cues for baggage weight classification because it varies depending on the weight of baggage carried by a standing person. To find the cues of body sway from depth image sequences, we can use the existing feature extraction1). However, the accuracy of baggage weight classification is reduced if the existing feature extraction is simply performed. The existing feature extraction causes this problem by seeing both the motion and the shape representing each person's cue to identify people. We consider that the shape of a person does not change even if the weight of the baggage changes. To this end, we design a novel feature extraction that suppresses spatial cues of the shape of a person and emphasizes temporal cues of the motion using the head region's center position. The experimental results show that our feature extraction improves baggage weight classification accuracy compared to the existing feature extraction.

Dynamic Time Warping-Based Features With Class-Specific Joint Importance Maps for Action Recognition Using Kinect Depth Sensor

This paper proposes a novel 3D action recognition technique that uses time-series information extracted from depth image sequences for use in systems of human daily activity monitoring. To this end, each action is represented as a multi-dimensional time series, where each dimension represents the position variation of one skeleton joint over time. The time series is then mapped onto a vector space using Dynamic Time Warping (DTW) distance. Furthermore, to employ the correlation-distinctiveness relationship of the sequences in recognition, this vector space is remapped onto a discriminative space using the regularized Fisher method, where final decisions about the actions are made. Unlike other available methods, the time-warping used in the mapping strategy makes the feature space robust to temporal variations of the motion sequences. Moreover, our method eliminates the need for a complicated design method for extracting the static and dynamic information of a motion sequence. Furthermore, most existing methods treat all skeletal joints identically for different actions, while some joints are more discriminative to distinguish a specific action. Thanks to the nature of the proposed features, we propose to use a separate set of discriminative joints, called joint importance map for each class of action. Evaluation results on four well-known datasets, TST, UTKinect, UCFKinect, and NTU RGB+D show competitive performance with the state-of-the-art methods in human action recognition.

Sequence Image Registration for Large Depth of Microscopic Focus Stacking

The large depth of images for microscopic measurement can be achieved by using focus stacking techniques with a small depth of field of objective lens. It is implemented by fusing the image sequences of short depth images. However, the non-linear movement of the objective imaging system or the measured object caused by the moving stage straightness error brings the misalignment of the image sequences, such as transversal translation, rotation, and tilting. All of these interferences, as well as the image brightness variation must be corrected by image registration before fusing the image sequences. In this paper, a fast-automatic registration method based on the scale invariant feature transform (SIFT) is proposed. It is achieved by firstly segmenting the focal regions of the image sequences through fast edge detection. Then the image features are extracted within the small segmented focal areas. It greatly reduces the computational cost of feature extraction and the following steps of image correction, and alignment. In the process, the random sampling consistency (RANSAC) algorithm is also used to remove the mistake features. The Laplacian pyramid method is adopted for the large depth of image fusion. The experimental results show that the proposed method is more efficient than the traditional SIFT algorithm. Its registration efficiency is improved by about 60%. This method facilitates the high-precision and real-time imaging of a monocular three-dimensional focus stacking.

Behavioral Feature Description Method Based on the Vector Module Ratio and Vector Angle of Human Body Structure

In the field of computer vision, the depth image sequence collected by depth camera is not sensitive to the interference of light, occlusion and background environment. Therefore, in recent years, it is often used to collect behavior data, from which the characteristics of bone joint points are extracted as behavior information. However, it is found that the direct use of joint coordinate information collected by depth camera for behavior recognition is easily affected by individual differences in behavior and changes in shooting distance. Considering the position information of human joints as well as the angle and length information of hidden limbs, based on skeleton data, a behavior feature description method based on the ratio of vector angle and vector mode of human structure is proposed. This method solves the above problems perfectly and obtains ideal results on the self built data set, which is suitable for simple daily behavior recognition.

Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model.

With the recent growth of Smart TV technology, the demand for unique and beneficial applications motivates the study of a unique gesture-based system for a smart TV-like environment. Combining movie recommendation, social media platform, call a friend application, weather updates, chatting app, and tourism platform into a single system regulated by natural-like gesture controller is proposed to allow the ease of use and natural interaction. Gesture recognition problem solving was designed through 24 gestures of 13 static and 11 dynamic gestures that suit to the environment. Dataset of a sequence of RGB and depth images were collected, preprocessed, and trained in the proposed deep learning architecture. Combination of three-dimensional Convolutional Neural Network (3DCNN) followed by Long Short-Term Memory (LSTM) model was used to extract the spatio-temporal features. At the end of the classification, Finite State Machine (FSM) communicates the model to control the class decision results based on application context. The result suggested the combination data of depth and RGB to hold 97.8% of accuracy rate on eight selected gestures, while the FSM has improved the recognition rate from 89% to 91% in a real-time performance.

Enhanced 3D Mapping with an RGB-D Sensor via Integration of Depth Measurements and Image Sequences

State-of-the-art visual simultaneous localization and mapping (SLAM) techniques greatly facilitate three-dimensional (3D) mapping and modeling with the use of low-cost red-green-blue-depth (RGB-D) sensors. However, the effective range of such sensors is limited due to the working range of the infra-red (IR) camera, which provides depth information, and thus the practicability of such sensors in 3D mapping and modeling is limited. To address this limitation, we present a novel solution for enhanced 3D mapping using a low-cost RGB-D sensor. We carry out state-of-the-art visual SLAM to obtain 3D point clouds within the mapping range of the RGB-D sensor and implement an improved structure-from-motion (SfM) on the collected RGB image sequences with additional constraints from the depth information to produce image-based 3D point clouds. We then develop a feature-based scale-adaptive registration to merge the gained point clouds to further generate enhanced and extended 3D mapping results. We use two challenging test sites to examine the proposed method. At these two sites, the coverage of both generated 3D models increases by more than 50% with the proposed solution. Moreover, the proposed solution achieves a geometric accuracy of about 1% in a measurement range of about 20 m. These positive experimental results not only demonstrate the feasibility and practicality of the proposed solution but also its potential.

Geometry-Aware ICP for Scene Reconstruction from RGB-D Camera

The Iterative Closest Point (ICP) scheme has been widely used for the registration of surfaces and point clouds. However, when working on depth image sequences where there are large geometric planes with small (or even without) details, existing ICP algorithms are prone to tangential drifting and erroneous rotational estimations due to input device errors. In this paper, we propose a novel ICP algorithm that aims to overcome such drawbacks, and provides significantly stabler registration estimation for simultaneous localization and mapping (SLAM) tasks on RGB-D camera inputs. In our approach, the tangential drifting and the rotational estimation error are reduced by: 1) updating the conventional Euclidean distance term with the local geometry information, and 2) introducing a new camera stabilization term that prevents improper camera movement in the calculation. Our approach is simple, fast, effective, and is readily integratable with previous ICP algorithms. We test our new method with the TUM RGB-D SLAM dataset on state-of-the-art real-time 3D dense reconstruction platforms, i.e., ElasticFusion and Kintinuous. Experiments show that our new strategy outperforms all previous ones on various RGB-D data sequences under different combinations of registration systems and solutions.

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation.

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates.

Human activity recognition from histogram of spatiotemporal depth features

The recent evolution in sensor-based depth information has been developed a sounding scope to work for human activity recognition using depth image sequences. Activity recognition is having a key importance due to its advantages in several domain like surveillance systems at airport, patient monitoring system, care of elderly people, etc. This work proposed the objective and evaluations to recognise daily life human activities by spatiotemporal depth information. For this purpose, Kinect sensor is used to collect the data pertaining to a single activity performed by multiple person at a time. The spatiotemporal depth features are computed for activity recognition and support vector machine is used in classification phase. We have nine class of human actions in the database for RGB-D human activity recognition. This dataset is reconfigured from Cornell human activity and Berkeley multimodal human action databases. For multiple human action recognition, 91.38% accuracy is achieved on the synthetic dataset.

Human activity recognition from histogram of spatiotemporal depth features

The recent evolution in sensor-based depth information has been developed a sounding scope to work for human activity recognition using depth image sequences. Activity recognition is having a key importance due to its advantages in several domain like surveillance systems at airport, patient monitoring system, care of elderly people, etc. This work proposed the objective and evaluations to recognise daily life human activities by spatiotemporal depth information. For this purpose, Kinect sensor is used to collect the data pertaining to a single activity performed by multiple person at a time. The spatiotemporal depth features are computed for activity recognition and support vector machine is used in classification phase. We have nine class of human actions in the database for RGB-D human activity recognition. This dataset is reconfigured from Cornell human activity and Berkeley multimodal human action databases. For multiple human action recognition, 91.38% accuracy is achieved on the synthetic dataset.

Top-down model fitting for hand pose recovery in sequences of depth images

State-of-the-art approaches on hand pose estimation from depth images have reported promising results under quite controlled considerations. In this paper we propose a two-step pipeline for recovering the hand pose from a sequence of depth images. The pipeline has been designed to deal with images taken from any viewpoint and exhibiting a high degree of finger occlusion. In a first step we initialize the hand pose using a part-based model, fitting a set of hand components in the depth images. In a second step we consider temporal data and estimate the parameters of a trained bilinear model consisting of shape and trajectory bases. We evaluate our approach on a new created synthetic hand dataset along with NYU and MSRA real datasets. Results demonstrate that the proposed method outperforms the most recent pose recovering approaches, including those based on CNNs.

Split‐Depth Image Generation and Optimization

AbstractSplit‐depth images use an optical illusion, which can enhance the 3D impression of a 2D animation. In split‐depth images (also often called split‐depth GIFs due to the commonly used file format), static virtual occluders inform of vertical or horizontal bars are added to a video clip, which leads to occlusions that are interpreted by the observer as a depth cue. In this paper, we study different factors that contribute to the illusion and propose a solution to generate split‐depth images for a given RGB + depth image sequence. The presented solution builds upon a motion summarization of the object of interest (OOI) through space and time. It allows us to formulate the bar positioning as an energy‐minimization problem, which we solve efficiently. We take a variety of important features into account, such as the changes of the 3D effect due to changes in the motion topology, occlusion, the proximity of bars or the OOI, and scene saliency. We conducted a number of psycho‐visual experiments to derive an appropriate energy formulation. Our method helps in finding optimal positions for the bars and, thus, improves the 3D perception of the original animation. We demonstrate the effectiveness of our approach on a variety of examples. Our study with novice users shows that our approach allows them to quickly create satisfying results even for complex animations.

A High Spatial Resolution Depth Sensing Method Based on Binocular Structured Light.

Depth information has been used in many fields because of its low cost and easy availability, since the Microsoft Kinect was released. However, the Kinect and Kinect-like RGB-D sensors show limited performance in certain applications and place high demands on accuracy and robustness of depth information. In this paper, we propose a depth sensing system that contains a laser projector similar to that used in the Kinect, and two infrared cameras located on both sides of the laser projector, to obtain higher spatial resolution depth information. We apply the block-matching algorithm to estimate the disparity. To improve the spatial resolution, we reduce the size of matching blocks, but smaller matching blocks generate lower matching precision. To address this problem, we combine two matching modes (binocular mode and monocular mode) in the disparity estimation process. Experimental results show that our method can obtain higher spatial resolution depth without loss of the quality of the range image, compared with the Kinect. Furthermore, our algorithm is implemented on a low-cost hardware platform, and the system can support the resolution of 1280 × 960, and up to a speed of 60 frames per second, for depth image sequences.

Surface Reconstruction via Fusing Sparse-Sequence of Depth Images.

Handheld scanning using commodity depth cameras provides a flexible and low-cost manner to get 3D models. The existing methods scan a target by densely fusing all the captured depth images, yet most frames are redundant. The jittering frames inevitably embedded in handheld scanning process will cause feature blurring on the reconstructed model and even trigger the scan failure (i.e., camera tracking losing). To address these problems, in this paper, we propose a novel sparse-sequence fusion (SSF) algorithm for handheld scanning using commodity depth cameras. It first extracts related measurements for analyzing camera motion. Then based on these measurements, we progressively construct a supporting subset for the captured depth image sequence to decrease the data redundancy and the interference from jittering frames. Since SSF will reveal the intrinsic heavy noise of the original depth images, our method introduces a refinement process to eliminate the raw noise and recover geometric features for the depth images selected into the supporting subset. We finally obtain the fused result by integrating the refined depth images into the truncated signed distance field (TSDF) of the target. Multiple comparison experiments are conducted and the results verify the feasibility and validity of SSF for handheld scanning with a commodity depth camera.

Improved Collaborative Representation Classifier Based on l2-Regularized for Human Action Recognition

Human action recognition is an important recent challenging task. Projecting depth images onto three depth motion maps (DMMs) and extracting deep convolutional neural network (DCNN) features are discriminant descriptor features to characterize the spatiotemporal information of a specific action from a sequence of depth images. In this paper, a unified improved collaborative representation framework is proposed in which the probability that a test sample belongs to the collaborative subspace of all classes can be well defined and calculated. The improved collaborative representation classifier (ICRC) based on l2-regularized for human action recognition is presented to maximize the likelihood that a test sample belongs to each class, then theoretical investigation into ICRC shows that it obtains a final classification by computing the likelihood for each class. Coupled with the DMMs and DCNN features, experiments on depth image-based action recognition, including MSRAction3D and MSRGesture3D datasets, demonstrate that the proposed approach successfully using a distance-based representation classifier achieves superior performance over the state-of-the-art methods, including SRC, CRC, and SVM.

Dynamic Gesture Classification for Vietnamese Sign Language Recognition

This paper presents an approach of feature extraction and classification for recognizing continuous dynamic gestures corresponding to Vietnamese Sign Language (VSL). Input data are captured by the depth sensor of a Microsoft Kinect, which is almost not affected by the light of environment. In detail, each gesture is represented by a volume corresponding to a sequence of depth images. The feature extraction stage is performed by dividing such volume into a 3D grid of same-size blocks in which each one is then converted into a scalar value. This step is followed by the process of classification. The wellknown method Support Vector Machine (SVM) is employed in this work, and the Hidden Markov Model (HMM) technique is also applied in order to provide a comparison on recognition accuracy. Besides, a dataset of 3000 samples corresponding to 30 dynamic gestures in VSL was created by 5 volunteers. The experiments on this dataset to validate the approach and that shows the promising results with average accuracy up to 95%.