Depth Video Research Articles

FreeCast: Graceful Free-Viewpoint Video Delivery

Wireless multi-view plus depth (MVD) video streaming enables free viewpoint video playback on wireless devices, where a viewer can freely synthesize any preferred virtual viewpoint from the received MVD frames. Existing schemes of wireless MVD streaming use digital-based compression to achieve better coding efficiency. However, the digital-based schemes have an issue called the cliff effect, where the video quality is a step function in terms of wireless channel quality. In addition, parameter optimization to assign quantization levels and transmission power across MVD frames are cumbersome. To realize high-quality wireless MVD video streaming, we propose a novel graceful video delivery scheme, called FreeCast . FreeCast directly transmits linear-transformed signals based on 5-D discrete cosine transform, without digital quantization and entropy coding operations. In addition, we exploit a fitting function based on a multidimensional Gaussian Markov random field model for overhead reduction to mitigate rate and power loss due to large overhead. The proposed FreeCast achieves graceful video quality with the improvement of wireless channel quality under a low overhead requirement. In addition, the parameter optimization to achieve highest video quality can be simplified by only controlling a transmission power assignment. Performance results with several test MVD video sequences show that FreeCast yields better video quality in band-limited environments by significantly decreasing the amount of overhead. For instance, structural similarity (SSIM) performance of FreeCast is approximately 0.127 higher than the existing graceful video delivery schemes across wireless channel quality, i.e., signal-to-noise ratio, of 0–25 dB at a transmission symbol rate of 37.5 Msymbols/s.

A Two-Stage Clustering Based 3D Visual Saliency Model for Dynamic Scenarios

Three-dimensional (3D) visual saliency is fundamental for vision-guided applications such as human–computer interaction in virtual reality, image quality assessment, object tracking, and event retrieval. Classical models for 3D visual saliency can draw an appropriate saliency map when the quality of the required depth maps or auxiliary cues is high enough. However, the depth map is usually impaired with artifacts (such as holes or noise) from faults in stereo matching or multipaths in range sensors. In these cases, challenges arise in those 3D visual saliency models because the core preliminary processes, such as the detection of low-level visual features, may fail. To solve this problem, we proposed a two-stage clustering-based 3D visual saliency model for human visual fixation prediction in dynamic scenarios. In this model, a two-stage clustering scheme is designed to handle the negative influence of impaired depth videos. With the help of this scheme, representative cues are selected for saliency modeling. After that, multimodal saliency maps are obtained from depth, color, and 3D motion cues. Finally, a cross-Bayesian model is designed for the pooling of multimodal saliency maps. The experimental results demonstrate that the proposed 3D saliency model based on two-stage clustering outperforms other state-of-the-art models on a variety of metrics. Furthermore, the consistency and robustness of our model are also verified.

Global Auto-Regressive Depth Recovery via Iterative Non-Local Filtering

Existing depth sensing techniques have many shortcomings in terms of resolution, completeness, and accuracy. The performance of 3-D broadcasting systems is therefore limited by the challenges of capturing high-resolution depth data. In this paper, we present a novel framework for obtaining high-quality depth images and multi-view depth videos from simple acquisition systems. We first propose a single depth image recovery algorithm based on auto-regressive (AR) correlations. A fixed-point iteration algorithm under the global AR modeling is derived to efficiently solve the large-scale quadratic programming. Each iteration is equivalent to a nonlocal filtering process with a residue feedback. Then, we extend our framework to an AR-based multi-view depth video recovery framework, where each depth map is recovered from low-quality measurements with the help of the corresponding color image, depth maps from neighboring views, and depth maps of temporally adjacent frames. AR coefficients on nonlocal spatiotemporal neighborhoods in the algorithm are designed to improve the recovery performance. We further discuss the connections between our model and other methods like graph-based tools, and demonstrate that our algorithms enjoy the advantages of both global and local methods. Experimental results on both the Middleburry datasets and other captured datasets finally show that our method is able to improve the performances of depth images and multi-view depth videos recovery compared with state-of-the-art approaches.

Fast and high-quality virtual view synthesis from multi-view plus depth videos

Depth image based rendering (DIBR) is an effective method for virtual view synthesis from Multi-view Plus Depth(MVD) video. Synthetic images, however, often contain ghost effect and some holes of varying sizes. This paper uses color correction of reference views, and combines depth-based image fusion with direct color image fusion to decrease the ghost effect. Meanwhile, the cracks are filled using depth filtering and inverse warping. What’s more, the image depth-aided inpainting with GPU acceleration is used to fill the remaining big disocclusions. Experimental results show that our proposed method improved the quality of virtual view synthetic images and reduced the processing time sharply.

Understanding the Dynamics of Social Interactions

In this article, we deal with the problem of understanding human-to-human interactions as a fundamental component of social events analysis. Inspired by the recent success of multi-modal visual data in many recognition tasks, we propose a novel approach to model dyadic interaction by means of features extracted from synchronized 3D skeleton coordinates, depth, and Red Green Blue (RGB) sequences. From skeleton data, we extract new view-invariant proxemic features, named Unified Proxemic Descriptor (UProD), which is able to incorporate intrinsic and extrinsic distances between two interacting subjects. A novel key frame selection method is introduced to identify salient instants of the interaction sequence based on the joints’ energy. From Red Green Blue Depth (RGBD) videos, more holistic CNN features are extracted by applying an adaptive pre-trained Convolutional Neural Networks (CNNs) on optical flow frames. For better understanding the dynamics of interactions, we expand the boundaries of dyadic interactions analysis by proposing a fundamentally new modeling for non-treated problem aiming to discern the active from the passive interactor. Extensive experiments have been carried out on four multi-modal and multi-view interactions datasets. The experimental results demonstrate the superiority of our proposed techniques against the state-of-the-art approaches.

Hierarchical dynamic depth projected difference images–based action recognition in videos with convolutional neural networks

Temporal information plays a significant role in video-based human action recognition. How to effectively extract the spatial–temporal characteristics of actions in videos has always been a challenging problem. Most existing methods acquire spatial and temporal cues in videos individually. In this article, we propose a new effective representation for depth video sequences, called hierarchical dynamic depth projected difference images that can aggregate the action spatial and temporal information simultaneously at different temporal scales. We firstly project depth video sequences onto three orthogonal Cartesian views to capture the 3D shape and motion information of human actions. Hierarchical dynamic depth projected difference images are constructed with the rank pooling in each projected view to hierarchically encode the spatial–temporal motion dynamics in depth videos. Convolutional neural networks can automatically learn discriminative features from images and have been extended to video classification because of their superior performance. To verify the effectiveness of hierarchical dynamic depth projected difference images representation, we construct a hierarchical dynamic depth projected difference images–based action recognition framework where hierarchical dynamic depth projected difference images in three views are fed into three identical pretrained convolutional neural networks independently for finely retuning. We design three classification schemes in the framework and different schemes utilize different convolutional neural network layers to compare their effects on action recognition. Three views are combined to describe the actions more comprehensively in each classification scheme. The proposed framework is evaluated on three challenging public human action data sets. Experiments indicate that our method has better performance and can provide discriminative spatial–temporal information for human action recognition in depth videos.

Multimodal Spatiotemporal Networks for Sign Language Recognition

Different from other human behaviors, sign language has the characteristics of limited local motion of upper limb and meticulous hand action. Some sign language gestures are ambiguous in RGB video due to the influence of lighting and background color, which affects the recognition accuracy. We propose a multimodal deep learning architecture for sign language recognition which effectively combines RGB-D input and two-stream spatiotemporal networks. Depth videos, as an effective compensation of RGB input, can supply additional distance information about the signer's hands. A novel sampling method called ARSS (Aligned Random Sampling in Segments) is put forward to select and align optimal RGB-D video frames, which improves the capacity utilization of multimodal data and reduces the redundancy. We get the hand ROI by joints information of RGB data for local focus in spatial stream. D-shift Net is proposed as depth motion feature extraction in temporal stream, which fully utilizes three dimensional motion information of the sign language. Both streams are fused by convolutional fusion layer to get complementary features. Our approach explored the multimodal information and enhanced the recognition precision. It obtains the state-the-of-art performance on the datasets of CSL (96.7%) and IsoGD (63.78%).

Multimodal grasp data set: A novel visual–tactile data set for robotic manipulation

This article introduces a visual–tactile multimodal grasp data set, aiming to further the research on robotic manipulation. The data set was built by the novel designed dexterous robot hand, the Intel’s Eagle Shoal robot hand (Intel Labs China, Beijing, China). The data set contains 2550 sets data, including tactile, joint, time label, image, and RGB and depth video. With the integration of visual and tactile data, researchers could be able to better understand the grasping process and analyze the deeper grasping issues. In this article, the building process of the data set was introduced, as well as the data set composition. In order to evaluate the quality of data set, the tactile data were analyzed by short-time Fourier transform. The tactile data–based slip detection was realized by long short-term memory and contrasted with visual data. The experiments compared the long short-term memory with the traditional classifiers, and generalization ability on different grasp directions and different objects is implemented. The results have proved that the data set’s value in promoting research on robotic manipulation area showed the effective slip detection and generalization ability of long short-term memory. Further work on visual and tactile data will be devoted to in the future.

A Soft Proposal Segmentation Network (SPS-Net) for Hand Segmentation on Depth Videos

Hand segmentation is an important prerequisite for acquiring accurate 3D hand poses on depth images, as it can significantly reduce the complicity of hand pose estimation. However, it has been ignored or treated as a trivial problem for a long time, since most of the hand pose estimation works suppose the hand part is given or can be easily segmented by a depth threshold, which is not practical in many realistic scenarios (e.g., side/egocentric view). In order to perform the robust hand segmentation in various scenarios, we propose a Soft Proposal Segmentation Network (SPS-Net). A key difference from existing hand segmentation methods is that the SPS-Net can utilize the temporal information on depth videos. As a benefit, significant performance gains over the existing methods can be obtained, as demonstrated on two popular public datasets with diverse camera viewpoints and background. Moreover, integrating SPS-Net with a simple 3D hand pose estimator, we achieved the leading result on the 3D Hand Pose Tracking Task of the Hand2017 Challenge - the only world-wide open challenge of its kind.

Region-based trilateral filter for depth video coding

3D video systems using depth-image-based rendering (DIBR) are attracting significant interests because of their low processing cost. In order to enable these systems, the depth video must be coded in addition to the texture video. During the coding process, coding artefacts in the depth map could cause video quality degradation of virtual view. In this paper, a region-based trilateral filter for depth video coding is proposed to improve the coding quality. First, the depth map is split into multiple objects with the label of segmentation. Then, in order to remove the artefacts from different objects and preserve the sharp edge, each pixel is filtered by using neighbouring pixels belonging to the same object. The proposed method has been evaluated with several 3D video sequences in HEVC platform. Experimental results demonstrate that the rendering quality can achieve a considerable gain under the same bitrate.

Study of 3D Video Compression Using Nonlinear Depth Representation

The extensions of Advanced Video Coding (AVC) and High-Efficiency Video Coding (HEVC) for multiview video plus depth allow for the coding of 3D video scenes. During the standardization of these extensions, it was demonstrated that compression performance can often be improved by the application of nonlinear depth transformation prior to the compression and applying the respective backward nonlinear depth transformation after decompression. Such an approach is referred to as the compression of nonlinear depth representation (NDR). In this paper, a survey of the NDR technology is provided, but a large part of this survey comprises yet unpublished results. First, the motivation and rationale behind the idea of NDR are discussed. The options for the choice of nonlinear transformation are provided with the respective formulations. Also, the standardization of 3D video coding with NDR within ISO/IEC and ITU is discussed along with the experimental results that demonstrate benefits related to the application of NDR in 3D-AVC and 3D-HEVC. Furthermore, the experimental comparisons are provided for simulcast coding of multiview plus depth video using AVC and HEVC technology with and without the NDR. The similar experimental results are also provided for simulcast coding using the current version of Versatile Video Coding technology that is under development for the forthcoming standard of ISO/IEC (MPEG-I) and ITU. All of the experimental results have been obtained for the standard JVC-3V video test sequences under common test conditions for 3D video coding.

SynBF: A New Bilateral Filter for Postremoval of Noise From Synthesis Views in 3-D Video

In 3-D video systems, noise in the texture and depth videos of reference views may not be removed (Scenario 1) or not fully removed (Scenario 2) by prefiltering methods before the view synthesis procedure. In these scenarios, the noise is transferred to the generated synthesis view. After investigating the noise model of the synthesis view, we conclude that the noise in the synthesis view not only causes fluctuation in the photometric values of pixels in the range domain but also additionally shifts the positions of neighbor pixels in the spatial domain compared to that in natural images. It consequently damages the textural content near edges in the synthesis view, for which the popular local filters of natural images, that is, the bilateral filter (BF) and the guided filter, do not work well. In this paper, we develop a new local filter for the synthesis view (named SynBF) after it has been generated, which has a similar expression as that of the BF but not the exact same weight terms. On one hand, the spatial term in the classical BF is directly reused due to its robustness to noise, which gives high weights to spatially closed pixels of to-be-filtered pixels. On the other hand, a reliability term is designed that gives high weights to pixels that are unlikely to be affected by noise. It is inspired by the finding that not all pixels are significantly affected by noise in the synthesis view. In this way, true edge profiles are protected in the filtering process. Experiments are conducted on a set of synthesis views for both scenarios above and compared to the two local filters, which verifies its effectiveness in removing noise and protecting edge profiles. The proposed method can be considered as a supplement to prefiltering methods of texture/depth videos in 3-D video systems.

Region-based trilateral filter for depth video coding

Region-based trilateral filter for depth video coding

Action recognition for depth video using multi-view dynamic images

Dynamic imaging is a recently proposed action description paradigm for simultaneously capturing motion and temporal evolution information, particularly in the context of deep convolutional neural networks (CNNs). Compared with optical flow for motion characterization, dynamic imaging exhibits superior efficiency and compactness. Inspired by the success of dynamic imaging in RGB video, this study extends it to the depth domain. To better exploit three-dimensional (3D) characteristics, multi-view dynamic images are proposed. In particular, the raw depth video is densely projected with respect to different virtual imaging viewpoints by rotating the virtual camera within the 3D space. Subsequently, dynamic images are extracted from the obtained multi-view depth videos and multi-view dynamic images are thus constructed from these images. Accordingly, more view-tolerant visual cues can be involved. A novel CNN model is then proposed to perform feature learning on multi-view dynamic images. Particularly, the dynamic images from different views share the same convolutional layers but correspond to different fully connected layers. This is aimed at enhancing the tuning effectiveness on shallow convolutional layers by alleviating the gradient vanishing problem. Moreover, as the spatial occurrence variation of the actions may impair the CNN, an action proposal approach is also put forth. In experiments, the proposed approach can achieve state-of-the-art performance on three challenging datasets.

Group Sparse Regression-Based Learning Model for Real-Time Depth-Based Human Action Prediction

This paper addresses the problem of predicting human actions in depth videos. Due to the complex spatiotemporal structure of human actions, it is difficult to infer ongoing human actions before they are fully executed. To handle this challenging issue, we first propose two new depth-based features called pairwise relative joint orientations (PRJOs) and depth patch motion maps (DPMMs) to represent the relative movements between each pair of joints and human-object interactions, respectively. The two proposed depth-based features are suitable for recognizing and predicting human actions in real-time fashion. Then, we propose a regression-based learning approach with a group sparsity inducing regularizer to learn action predictor based on the combination of PRJOs and DPMMs for a sparse set of joints. Experimental results on benchmark datasets have demonstrated that our proposed approach significantly outperforms existing methods for real-time human action recognition and prediction from depth data.

Multi-sensor data fusion for sign language recognition based on dynamic Bayesian network and convolutional neural network

A new multi-sensor fusion framework is proposed, which is based on the Convolutional Neural Network (CNN) and the Dynamic Bayesian Network (DBN) for Sign Language Recognition (SLR). In this framework, a Microsoft Kinect, which is a low-cost RGB-D sensor, is used as tools of the Human-Computer-Interaction (HCI). In our method, at first, the color and depth videos are collected using the Kinect, the next, all image sequences features are extracted out using the CNN. The color and depth feature sequences are input into the DBN as observation data. Based on graph model fusion, the maximum recognition rate of dynamic isolated sign language is calculated. The proposed the DBN + CNN SLR framework is tested in our dataset, the highest recognition rate can up to 99.40%. The test results show that our approach is effective.

Action recognition using multi-directional projected depth motion maps

Camera-based action recognition plays a key role in diverse computer vision applications such as human computer interaction. This paper proposes a new action recognition approach using multi-directional projected depth motion map based motion descriptors. First, for the input depth video sequence, all the depth frames in the video are projected onto multiple planes to form the projected images. The absolute difference between two consecutive projected images is accumulated through the entire depth video for establishing maps from multiple views. Then, the local motion consistency of the map is examined to form a histogram of local binary patterns, which are then concatenated and further incorporated into a kernel-based extreme learning machine for action recognition. In contrast to that only three directions are used to calculated the projected depth images for motion feature extraction in the conventional approaches, the proposed approach is able to provide an effective and flexible framework to examine the depth motion maps in multiple projected directions. The proposed approach is evaluated in the well-known MSRA action and gesture video benchmark datasets to demonstrate its superior performance.

Action Recognition Based on Depth Motion Map and Hybrid Classifier

In order to efficiently extract and encode 3D information of human action from depth images, we present a feature extraction and recognition method based on depth video sequences. First, depth images are projected continuously onto three planes of Cartesian coordinate system, and differential images of the respective projection surfaces are accumulated to obtain the complete 3D information of the depth motion maps (DMMs). Then, discriminative completed LBP (disCLBP) encodes depth motion maps to extract effective human action information. A hybrid classifier combined with Extreme Learning Machine (ELM) and collaborative representation classification (CRC) is employed to reduce the computational complexity while reducing the impact of noise. The proposed method is tested on the MSR-Action3D database; the experimental results show that it achieves 96.0% accuracy and well performs better robustness comparing to other popular approaches.

Fast intra mode decision and fast CU size decision for depth video coding in 3D-HEVC

3D-HEVC is the new coding standard for both texture and associated depth video. In addition to several new coding tools that are designed for texture video with improved coding efficiency, some tools are specified for depth video, such as depth modeling modes (DMMs), segment-wise DC (SDC) mode and depth intra skip mode. This paper proposes techniques that accelerate the encoding of depth video, including a fast intra mode decision and a fast CU size decision. A hierarchical mode decision is proposed for the fast intra mode decision, based on a statistical analysis of mode distribution. The intra mode can be predicted at a very early stage if the hierarchical comparison of the rate–distortion (R–D) cost for selected candidate modes is satisfied for a given condition. For the fast CU size decision, smaller CU sizes are not evaluated if the current CU has the desired properties. The CU size of the collocated texture is also considered to avoid over-splitting of the current depth CU. The experimental results show that the proposed techniques achieve an average 62.5% time saving with 0.81% increase in the bitrate for the synthesized views under the all intra configuration, compared with the reference software HTM 15.0.

Searching Human Action Recognition Accuracy from Depth Video Sequences Using HOG and PHOG Shape Features

Research on human action recognition from depth video sequences are increasing day by day due to its vast application in automatic surveillance systems, entertainment environments, and healthcare systems etc. In our project, we improve human action recognition accuracy using shape features. We use Histogram of oriented gradients (HOG) and Pyramid Histogram of oriented gradients ( PHOG) to extract shape features. The feature extraction algorithms are used to extract shape feature from dataset of different action videos. At first, depth motion maps (DMMs) are constructed from every action video. Then, the HOG and PHOG features are extracted from each DMMs. Using these features, actions are recognized by the