RGB-D Data Research Articles

Three-dimensional convolutional restricted Boltzmann machine for human behavior recognition from RGB-D video

This paper provides a novel approach for recognizing human behavior from RGB-D video data. The three-dimensional convolutional restricted Boltzmann machine (3DCRBM) is proposed which can extract features from the raw RGB-D data. In a physical model, the 3DCRBM differs from the restricted Boltzmann machine (RBM) as its weights are shared among all locations in the input and preserving spatial locality. Adjacent frames of the RGB image and the corresponding adjacent frames of the depth image are set as the input of 3DCRBM. Then, multiple 3D convolutional kernels can be applied to these four frames to extract spatio-temporal features. In the experiment of human behavior recognition, the deep belief network (DBN) is established by a layer of 3DCRBM network, convolutional neural network (CNN), and back propagation (BP) network. 3DCRBM is adapted for unsupervised training and getting a feature, while CNN and BP are used for supervised training and classifying the human behavior. The experiment results demonstrate that the correct differentiation rate of 95.7% is achieved, so the effectiveness of our approach could be validated.

Face RGB-D Data Acquisition System Architecture for 3D Face Identification Technology

Face RGB-D Data Acquisition System Architecture for 3D Face Identification Technology

Face RGB-D Data Acquisition System Architecture for 3D Face Identification Technology

Face RGB-D Data Acquisition System Architecture for 3D Face Identification Technology

Octree-based localization using RGB-D data for indoor robots

Mobile robots need to know their position in the environment to perform complex tasks. Self-localization methods for indoor environments usually reduce sensory information to two dimensions in order to accelerate calculations, instead of taking advantage of the three-dimensional (3D) features. This paper proposes a computationally affordable probabilistic self-localization method that takes full advantage of the capacities of RGB-D sensors, particularly, using 3D dimensions and color. We use an a priori known map, previously built, that keeps the information of the environment’s structure and color. Storing this data allows to reduce symmetries caused by 2D and colorless simplification. Our approach is based on Monte Carlo algorithm, where hypothesis population varies dynamically to adapt itself to the existing conditions at any given time, which makes it more efficient in 3D information management.

Multiple Modal Features and Multiple Kernel Learning for Human Daily Activity Recognition

Introduction: Recognizing human activity in a daily environment has attracted much research in computer vision and recognition in recent years. It is a difficult and challenging topic not only inasmuch as the variations of background clutter, occlusion or intra-class variation in image sequences but also inasmuch as complex patterns of activity are created by interactions among people-people or people-objects. In addition, it also is very valuable for many practical applications, such as smart home, gaming, health care, human-computer interaction and robotics. Now, we are living in the beginning age of the industrial revolution 4.0 where intelligent systems have become the most important subject, as reflected in the research and industrial communities. There has been emerging advances in 3D cameras, such as Microsoft's Kinect and Intel's RealSense, which can capture RGB, depth and skeleton in real time. This creates a new opportunity to increase the capabilities of recognizing the human activity in the daily environment. In this research, we propose a novel approach of daily activity recognition and hypothesize that the performance of the system can be promoted by combining multimodal features.
 Methods: We extract spatial-temporal feature for the human body with representation of parts based on skeleton data from RGB-D data. Then, we combine multiple features from the two sources to yield the robust features for activity representation. Finally, we use the Multiple Kernel Learning algorithm to fuse multiple features to identify the activity label for each video. To show generalizability, the proposed framework has been tested on two challenging datasets by cross-validation scheme.
 Results: The experimental results show a good outcome on both CAD120 and MSR-Daily Activity 3D datasets with 94.16% and 95.31% in accuracy, respectively.
 Conclusion: These results prove our proposed methods are effective and feasible for activity recognition system in the daily environment.

Large-Scale Gesture Recognition With a Fusion of RGB-D Data Based on Saliency Theory and C3D Model

Gesture recognition has raised wide attention in computer vision owing to its many applications. However, the task of video-based large-scale gesture recognition yet faces many challenges, since many gesture-irrelevant factors like the background may disturb the recognition accuracy. To better recognize gestures with large-scale videos, we propose a method based on RGB-D data in this paper, where the “RGB-D” means RGB and depth data captured simultaneously by specific devices like Kinect. To learn gesture details better, we first use an adaptive frame unification strategy to unify the frame number of inputs, and then the RGB and depth data are sent to the C3D model to extract spatiotemporal features, respectively. In order to alleviate the interference of gesture-irrelevant factors, the saliency theory is also employed to generate auxiliary data. Next the features of these data are combined to boost the performance, which can also avoid unreasonable synthetic data, since the dimension of C3D features is uniform. Finally the performances of several classifiers are tested and the best one of SVM classifier is selected to output the ultimate accuracy. Our approach achieves 52.04% and 59.43% accuracy on the validation and testing subset of the Chalearn LAP IsoGD, respectively, both of which outperform our results in the chalearn LAP Large-scale Gesture Recognition Challenge as reported in ICPR 2016.

Learning to Segment Generic Handheld Objects Using Class-Agnostic Deep Comparison and Segmentation Network

Learning unknown objects in the environment is important for detection and manipulation tasks. Prior to learning the unknown objects the ground-truth labels have to be provided. The data annotation or labeling can be achieved in a number of ways but the most widely used method is still manual annotation. Although manual annotation has shown superior performance, it limits robots' capabilities to known object instances and is also a time consuming task. This letter considers the aforementioned limitations and presents a method that allows robots to autonomously annotate objects from observations of human-object interactions. Specifically, we present a novel method that segments handheld objects in real-time using the class-agnostic deep comparison and segmentation network. The inputs to the network are the RGB-D data of known object template and a search space, and it outputs a pixel-wise label of the object and an objectness score. The score indicates the likelihood that the same object is present in both the inputs. The object template is manually initialized in the first frame and thereafter, the object is segmented and the template is updated online. The template is strategically updated using the likelihood score. The segmented object regions are accumulated as pseudo-ground-truth labels, which are used for object learning. The approach efficiently handles both rigid and highly deformable objects.

RGB-D object recognition based on the joint deep random kernel convolution and ELM

Nowadays RGB-D object recognition has been a challenging and important task in computer vision field. Convolutional Neural Network is a current popular algorithm for feature extraction from RGB and Depth modality separately, which cannot fully exploit some potential and complementary information between different modalities. The conventional training methods designed for CNN involve many gradient-descent searching, and usually face some troubles such as time-consuming convergence, local minima. In order to solve these problems, we propose a Joint Deep Radom Kernel Convolution and ELM (JDRKC-ELM) method for object recognition, which integrating the power of CNN feature extraction and fast training of ELM-AE. Our JDRKC-ELM can learn feature representations from raw RGB-D data directly. In this structure, Radom Kernel Convolutional neural network (RKCNN) is used for lower-level feature extraction from RGB and Depth modality separately. And then, combining these features from different modality by a feature fusion layer and feeding these fusion features to a Double-layer ELM-AE (DLELM-AE) for higher-level features. At last, the final feature representations are sent to a standard ELM for the object classification. We evaluate the quality of the JDRKC-ELM method on the RGB-D Object Dataset. The results show that the proposed method achieves high recognition accuracy and good generalization performance in comparison with deep learning methods and other ELM methods.

Depth Restoration From RGB-D Data via Joint Adaptive Regularization and Thresholding on Manifolds.

In this paper, we propose a novel depth restoration algorithm from RGB-D data through combining characteristics of local and non-local manifolds, which provide low-dimensional parameterizations of the local and non-local geometry of depth maps. Specifically, on the one hand, a local manifold model is defined to favor local neighboring relationship of pixels in depth, according to which, manifold regularization is introduced to promote smoothing along the manifold structure. On the other hand, the non-local characteristics of the patch-based manifold can be used to build highly data-adaptive orthogonal bases to extract elongated image patterns, accounting for self-similar structures in the manifold. We further define a manifold thresholding operator in 3D adaptive orthogonal spectral bases-eigenvectors of the discrete Laplacian of local and non-local manifolds-to retain only low graph frequencies for depth maps restoration. Finally, we propose a unified alternating direction method of multipliers optimization framework, which elegantly casts the adaptive manifold regularization and thresholding jointly to regularize the inverse problem of depth maps recovery. Experimental results demonstrate that our method achieves superior performance compared with the state-of-the-art works with respect to both objective and subjective quality evaluations.

Hierarchical multi-modal fusion FCN with attention model for RGB-D tracking

In this paper, we propose a RGB-D tracking algorithm, built upon hierarchical multi-modal fusion fully convolutional neural network (FCN) with attention model. First, we encode the depth images into three channels using the HHA representation to obtain the similar structure to the RGB images. Second, a multi-modal fusing features learning FCN with attention model is constructed, which can extract hierarchical multi-modal fusing features of the samples in RGB-D data. The attention model is adopted to exploit the importance weight of RGB and depth to fuse the features of the two modalities at multiple layers effectively, rather than concatenating the feature vectors of the two channels simply. Finally, the hierarchical multi-modal fusing features of the samples are input to the Efficient Convolution Operators (ECO) tracker, and the update strategy is improved by the occlusion detection in the depth images. Experimental results datasets demonstrate that the proposed RGB-D tracker achieves new state-of-art performance on the large-scale Princeton RGB-D Tracking Benchmark (PTB) dataset and the University of Birmingham RGB-D Tracking Benchmark (BTB).

INDOOR MESH CLASSIFICATION FOR BIM

Abstract. This work addresses the automatic reconstruction of objects useful for BIM, like walls, floors and ceilings, from meshed and textured mapped 3D point clouds of indoor scenes. For this reason, we focus on the semantic segmentation of 3D indoor meshes as the initial step for the automatic generation of BIM models. Our investigations are based on the benchmark dataset ScanNet, which aims at the interpretation of 3D indoor scenes. For this purpose it provides 3D meshed representations as collected from low cost range cameras. In our opinion such RGB-D data has a great potential for the automated reconstruction of BIM objects.

Learning Effective RGB-D Representations for Scene Recognition.

Deep convolutional networks (CNN) can achieve impressive results on RGB scene recognition thanks to large datasets such as Places. In contrast, RGB-D scene recognition is still underdeveloped in comparison, due to two limitations of RGB-D data we address in this paper. The first limitation is the lack of depth data for training deep learning models. Rather than fine tuning or transferring RGB-specific features, we address this limitation by proposing an architecture and a twostep training approach that directly learns effective depth-specific features using weak supervision via patches. The resulting RGBD model also benefits from more complementary multimodal features. Another limitation is the short range of depth sensors (typically 0.5m to 5.5m), resulting in depth images not capturing distant objects in the scenes that RGB images can. We show that this limitation can be addressed by using RGB-D videos, where more comprehensive depth information is accumulated as the camera travels across the scenes. Focusing on this scenario, we introduce the ISIA RGB-D video dataset to evaluate RGB-D scene recognition with videos. Our video recognition architecture combines convolutional and recurrent neural networks (RNNs) that are trained in three steps with increasingly complex data to learn effective features (i.e. patches, frames and sequences). Our approach obtains state-of-the-art performances on RGB-D image (NYUD2 and SUN RGB-D) and video (ISIA RGB-D) scene recognition.

Robust RGB-D Hand Tracking Using Deep Learning Priors

With the irruption of inexpensive depth sensor devices, hand gesture tracking has become a topic of great interest. Two main problems to face respect other tracking algorithms are the high complexity of the hand structure, which translate in a very large amount of possible gestures, and the rapidness of the movements we are able to make when moving the hand or just the fingers. Recent approaches try to fit a 3D hand model to the observed RGB-D data by an optimization function that minimizes the error between the model and the data. However, these algorithms are very dependent on the initialization point, which are impractical to run in a natural environment. To solve these kinds of problems, it is common to use an offline data set with prelearned gestures that will serve as a first rough estimate. In concrete, we present an algorithm that uses an articulated ICP minimization function that is initialized by the parameters obtained from a data set of hand gestures trained through a deep learning framework. This setup has two strong points. First, deep learning provides a very fast and accurate estimate of performed hand gestures. Second, the articulated ICP algorithm allows capturing the possible variability of a gesture performed by different persons or slightly different gestures. Our proposed algorithm is evaluated and validated in several ways. Independent evaluations for the deep learning framework and articulated ICP are performed. Moreover, different real sequences are recorded to validate our approach and, finally, quantitative and qualitative comparisons are conducted with state-of-the-art algorithms.

A Method for 6D Pose Estimation of Free-Form Rigid Objects Using Point Pair Features on Range Data.

Pose estimation of free-form objects is a crucial task towards flexible and reliable highly complex autonomous systems. Recently, methods based on range and RGB-D data have shown promising results with relatively high recognition rates and fast running times. On this line, this paper presents a feature-based method for 6D pose estimation of rigid objects based on the Point Pair Features voting approach. The presented solution combines a novel preprocessing step, which takes into consideration the discriminative value of surface information, with an improved matching method for Point Pair Features. In addition, an improved clustering step and a novel view-dependent re-scoring process are proposed alongside two scene consistency verification steps. The proposed method performance is evaluated against 15 state-of-the-art solutions on a set of extensive and variate publicly available datasets with real-world scenarios under clutter and occlusion. The presented results show that the proposed method outperforms all tested state-of-the-art methods for all datasets with an overall 6.6% relative improvement compared to the second best method.

A Review on Visual Recognition of RGB Images and Videos by Learning from RGB-D Data

Domain adaptation provides the possibility of research that allows changes in data distribution across training and testing datasets. Recognizing the RGB images by learning RGB-D data contains the additional depth information. The unsupervised domain adaptation (UDA) take advantage of the additional depth features. UDA deals with domain mismatch between the source and the target. The various adaptation techniques deals with the source and target domain. The domain mismatch is minimized by describing a projection matrix that is optimized by reducing the Maximum Mean Discrepancy (MMD) and aligning the source and target domains. To optimize the depth information the correlation between different types of features are to be maximized. Inorder to simultaneously cope with the domain mismatch issues, a unified framework called domain adaptation from multi-view to single-view (DAM2S) is learned. The effectiveness of the proposed methods for recognizing RGB images and videos by learning from RGB-D data is demonstrated by comprehensive experiments for object recognition, cross dataset and cross-view action recognition.

A JOINT EFFORT OF SPEEDED-UP ROBUST FEATURES ALGORITHM AND A DISPARITY-BASED MODEL FOR 3D INDOOR MAPPING USING RGB-D DATA

Abstract In this paper, we present a method for 3D mapping of indoor environments using RGB-D data. The contribution of our proposed method is two-fold. First, our method exploits a joint effort of the speed-up robust features (SURF) algorithm and a disparity-to-plane model for a coarse-to-fine registration procedure. Once the coarse-to-fine registration task accumulates errors, the same features can appear in two different locations of the map. This is known as the loop closure problem. Then, the variance-covariance matrix that describes the uncertainty of transformation parameters (3D rotation and 3D translation) for view-based loop closure detection followed by a graph-based optimization are proposed to achieve a 3D consistent indoor map. To demonstrate and evaluate the effectiveness of the proposed method, experimental datasets obtained in three indoor environments with different levels of details are used. The experimental results shown that the proposed framework can create 3D indoor maps with an error of 11,97 cm into object space that corresponds to a positional imprecision around 1,5% at the distance of 9 m travelled by sensor.

Relative Pose Based Redundancy Removal: Collaborative RGB-D Data Transmission in Mobile Visual Sensor Networks.

In this paper, the Relative Pose based Redundancy Removal (RPRR) scheme is presented, which has been designed for mobile RGB-D sensor networks operating under bandwidth-constrained operational scenarios. The scheme considers a multiview scenario in which pairs of sensors observe the same scene from different viewpoints, and detect the redundant visual and depth information to prevent their transmission leading to a significant improvement in wireless channel usage efficiency and power savings. We envisage applications in which the environment is static, and rapid 3D mapping of an enclosed area of interest is required, such as disaster recovery and support operations after earthquakes or industrial accidents. Experimental results show that wireless channel utilization is improved by 250% and battery consumption is halved when the RPRR scheme is used instead of sending the sensor images independently.

Depth compression via planar segmentation

Augmented Reality applications are set to revolutionize the smartphone industry due to the integration of RGB-D sensors into mobile devices. Given the large number of smartphone users, efficient storage and transmission of RGB-D data is of paramount interest to the research community. While there exist Video Coding Standards such as HEVC and H.264/AVC for compression of RGB/texture component, the coding of depth data is still an area of active research. This paper presents a method for coding depth videos, captured from mobile RGB-D sensors, by planar segmentation. The segmentation algorithm is based on Markov Random Field assumptions on depth data and solved using Graph Cuts. While all prior works based on this approach remain restricted to images only and under noise-free conditions, this paper presents an efficient solution to planar segmentation in noisy depth videos. Also presented is a unique method to encode depth based on its segmented planar representation. Experiments on depth captured from a noisy sensor (Microsoft Kinect) shows superior Rate-Distortion performance over the 3D extension of HEVC codec.

(DE)$^2$CO: Deep Depth Colorization

The ability to classify objects is fundamental for robots. Besides knowledge about their visual appearance, captured by the RGB channel, robots heavily need also depth information to make sense of the world. While the use of deep networks on RGB robot images has benefited from the plethora of results obtained on databases like ImageNet, using convnets on depth images requires mapping them into three-dimensional channels. This transfer learning procedure makes them processable by pretrained deep architectures. Current mappings are based on heuristic assumptions over preprocessing steps and on what depth properties should be most preserved, resulting often in cumbersome data visualizations, and in suboptimal performance in terms of generality and recognition results. Here, we take an alternative route and we attempt instead to learn an optimal colorization mapping for any given pretrained architecture, using as training data a reference RGB-D database. We propose a deep network architecture, exploiting the residual paradigm, that learns how to map depth data to three channel images. A qualitative analysis of the images obtained with this approach clearly indicates that learning the optimal mapping preserves the richness of depth information better than current hand-crafted approaches. Experiments on the Washington, JHUIT-50 and BigBIRD public benchmark databases, using CaffeNet, VGG-16, GoogleNet, and ResNet50 clearly showcase the power of our approach, with gains in performance of up to 16% compared to state of the art competitors on the depth channel only, leading to top performances when dealing with RGB-D data.

Scale Estimation and Correction of the Monocular Simultaneous Localization and Mapping (SLAM) Based on Fusion of 1D Laser Range Finder and Vision Data.

This article presents a new sensor fusion method for visual simultaneous localization and mapping (SLAM) through integration of a monocular camera and a 1D-laser range finder. Such as a fusion method provides the scale estimation and drift correction and it is not limited by volume, e.g., the stereo camera is constrained by the baseline and overcomes the limited depth range problem associated with SLAM for RGBD cameras. We first present the analytical feasibility for estimating the absolute scale through the fusion of 1D distance information and image information. Next, the analytical derivation of the laser-vision fusion is described in detail based on the local dense reconstruction of the image sequences. We also correct the scale drift of the monocular SLAM using the laser distance information which is independent of the drift error. Finally, application of this approach to both indoor and outdoor scenes is verified by the Technical University of Munich dataset of RGBD and self-collected data. We compare the effects of the scale estimation and drift correction of the proposed method with the SLAM for a monocular camera and a RGBD camera.