RGB-D Video Streams Research Articles

Interactively learning behavior trees from imperfect human demonstrations.

Introduction: In Interactive Task Learning (ITL), an agent learns a new task through natural interaction with a human instructor. Behavior Trees (BTs) offer a reactive, modular, and interpretable way of encoding task descriptions but have not yet been applied a lot in robotic ITL settings. Most existing approaches that learn a BT from human demonstrations require the user to specify each action step-by-step or do not allow for adapting a learned BT without the need to repeat the entire teaching process from scratch. Method: We propose a new framework to directly learn a BT from only a few human task demonstrations recorded as RGB-D video streams. We automatically extract continuous pre- and post-conditions for BT action nodes from visual features and use a Backchaining approach to build a reactive BT. In a user study on how non-experts provide and vary demonstrations, we identify three common failure cases of an BT learned from potentially imperfect initial human demonstrations. We offer a way to interactively resolve these failure cases by refining the existing BT through interaction with a user over a web-interface. Specifically, failure cases or unknown states are detected automatically during the execution of a learned BT and the initial BT is adjusted or extended according to the provided user input. Evaluation and results: We evaluate our approach on a robotic trash disposal task with 20 human participants and demonstrate that our method is capable of learning reactive BTs from only a few human demonstrations and interactively resolving possible failure cases at runtime.

Efficient 3D Object Detection of Indoor Scenes Based on RGB-D Video Stream

<p indent=0mm>For indoor object detection, the input complex scenes often have some defects such as incomplete RGB-D scanning data or mutual occlusion of its objects. Meanwhile, due to the limitations of single RGB-D data or point cloud data input of indoor scenes, it is always difficult to detect all of 3D objects simultaneously. In order to overcome this issue and also alleviate its low efficiency for indoor object detection, an efficient 3D object detection method is proposed which takes RGB-D video streams as input. First, the RGB-D video stream of different indoor environments can be obtained using Kinect camera, and also captured its continuous RGB frames and corresponding point cloud data. Secondly, the Hash function is adopted to extract the content-sensitive key frames from the continuous RGB frames, and the objects semantic relationship can also be constructed according to the type/number of 3D objects contained in adjacent key frames for ensuring that different objects will appear in each key frame. Then, 3D objects of the extracted key frames can be detected by using VoteNet, and the detection results of other frames can be estimated owing to relative posture relationship between adjacent frames by using the quaternion spherical linear interpolation algorithm. Finally, it can achieve efficient 3D object detection for each frame in the RGB-D video stream. Using SUN RGB-D dataset to train the object detection network of key frame, the detection result of proposed method is accurate, and the overall detection time is greatly reduced if comparing with the VoteNet based frame-by-frame detection scheme. Experimental results demonstrate that proposed method is effective and efficient.

Comparing Seven Methodogies for Rigid Alignment of Point Clouds with Focus on Frame-to-Frame Registration in Depth Sequences

Pairwise rigid registration aims to find the rigid transformation that best registers two surfaces represented by point clouds. This work presents a comparison between seven algorithms, with different strategies to tackle rigid registration tasks. We focus on the frame-to-frame problem, in which the point clouds are extracted from a video sequence with depth information generating partial overlapping 3D data. We use both point clouds and RGB-D video streams in the experimental results. The former is considered under different viewpoints with the addition of a case-study simulating missing data. Since the ground truth rotation is provided, we discuss four different metrics to measure the rotation error in this case. Among the seven considered techniques, the Sparse ICP and Sparse ICP-CTSF outperform the other five ones in the point cloud registration experiments without considering incomplete data. However, the evaluation facing missing data indicates sensitivity for these methods against this problem and favors ICP-CTSF in such situations. In the tests with video sequences, the depth information is segmented in the first step, to get the target region. Next, the registration algorithms are applied and the average root mean squared error, rotation and translation errors are computed. Besides, we analyze the robustness of the algorithms against spatial and temporal sampling rates. We conclude from the experiments using a depth video sequences that ICP-CTSF is the best technique for frame-to-frame registration.

Real-time haptic interaction with RGBD video streams

Video interaction is a common way of communication in cyberspace. It can become more immersive by incorporating haptic modality. Using commonly available depth sensing controllers like Microsoft Kinect, information about the depth of a scene can be captured in real-time together with the video. In this paper, we present a method for real-time haptic interaction with videos containing depth data. Forces are computed based on the depth information. Spatial and temporal filtering of the depth stream is used to provide stability of force feedback delivered to the haptic device. Fast collision detection ensures the proposed approach to be used in real-time. We present an analysis of various factors that affect algorithm performance. The usefulness of the approach is illustrated by highlighting possible application scenarios.