Color-depth Camera Research Articles

Latent Space Representations for Marker-Less Realtime Hand-Eye Calibration.

Marker-less hand-eye calibration permits the acquisition of an accurate transformation between an optical sensor and a robot in unstructured environments. Single monocular cameras, despite their low cost and modest computation requirements, present difficulties for this purpose due to their incomplete correspondence of projected coordinates. In this work, we introduce a hand-eye calibration procedure based on the rotation representations inferred by an augmented autoencoder neural network. Learning-based models that attempt to directly regress the spatial transform of objects such as the links of robotic manipulators perform poorly in the orientation domain, but this can be overcome through the analysis of the latent space vectors constructed in the autoencoding process. This technique is computationally inexpensive and can be run in real time in markedly varied lighting and occlusion conditions. To evaluate the procedure, we use a color-depth camera and perform a registration step between the predicted and the captured point clouds to measure translation and orientation errors and compare the results to a baseline based on traditional checkerboard markers.

Estimation of human body 3D pose for parent-infant interaction settings using azure Kinect and OpenPose

Automatic pose estimation has become a valuable tool for the study of human behavior, including dyadic interactions. It allows researchers to analyze the nuanced dynamics of interactions more effectively, and facilitates the integration of behavioral data with other modalities (EEG, etc.). However, many technical difficulties remain. Particularly, for parent-infant interactions, automatic pose estimation for infants is unpredictable; the immature proportions and smaller bodies of children may cause misdetections. OpenPose is one tool that has shown high performance in pose tracking from video, even in infants. However, OpenPose is limited to 2D (i.e., coordinates relative to the image space). This may be undesirable in a multitude of paradigms (e.g., naturalistic settings). We developed a method for expanding the functionality of OpenPose to 3D, tailored to parent-infant interaction paradigms. This method merges the estimations from OpenPose with the depth information from a depth camera to obtain a 3D pose that works even for young infants.•Video recordings of interactions of parents and infants are taken using a dual color-depth camera.•2D-positions of parents and their infants are estimated from the color video.•Using the depth camera, we transform the 2D estimations into real-world 3D positions, allowing movement analysis in full-3D space.

Collision-Free Navigation in Human-Following Task Using a Cognitive Robotic System on Differential Drive Vehicles

As human–robot collaboration increases tremendously in real-world applications, a fully autonomous and reliable mobile robot for the collaboration has been a central research topic and investigated extensively in a large number of studies. One of the most pressing issues in such topic is the collision-free navigation that has a moving goal and unknown obstacles under the unstructured environment. In this article, a cognitive robotic system (CRS) is proposed for the robot to navigate itself to the moving target person without obstacle collision. This CRS consists of a cognitive agent, which is created based on the Soar cognitive architecture to reason its current situation and make action decision for the robot to avoid obstacles and reach the target position, and a speed planning module, which is based on dynamic window approach (DWA) to generate appropriate linear and angular velocities for driving the robot’s motors. For the implementation of the proposed system, we use a differential drive wheel robot equipped with two ultrawideband (UWB) sensors and a color depth camera as the experimental platform. Finally, to evaluate the performance of our system in actual operating conditions, we conduct experiments with a scenario that includes main tasks: avoiding consecutive unknown obstacles and turning at corner while the robot follows continuously human user along the corridor.

Approach for accurate calibration of RGB-D cameras using spheres.

RGB-D cameras (or color-depth cameras) play key roles in many vision applications. A typical RGB-D camera has only rough intrinsic and extrinsic calibrations that cannot provide the accuracy required in many vision applications. In this paper, we propose a novel and accurate sphere-based calibration framework for estimating the intrinsic and extrinsic parameters of color-depth sensor pair. Additionally, a method of depth error correction is suggested, and the principle of error correction is analyzed in detail. In our method, the feature extraction module can automatically and reliably detect the center and edges of the sphere projection, while excluding noise data and outliers, and the projection of the sphere center on RGB and depth images is used to obtain a closed solution of the initial parameters. Finally, all the parameters are accurately estimated within the framework of nonlinear global minimization. Compared to other state-of-the-art methods, our calibration method is easy to use and provides higher calibration accuracy. Detailed experimental analysis is performed to support our conclusions.

A Comparative Study of Markerless Systems Based on Color-Depth Cameras, Polymer Optical Fiber Curvature Sensors, and Inertial Measurement Units: Towards Increasing the Accuracy in Joint Angle Estimation

This paper presents a comparison between a multiple red green blue-depth (RGB-D) vision system, an intensity variation-based polymer optical fiber (POF) sensor, and inertial measurement units (IMUs) for human joint angle estimation and movement analysis. This systematic comparison aims to study the trade-off between the non-invasive feature of a vision system and its accuracy with wearable technologies for joint angle measurements. The multiple RGB-D vision system is composed of two camera-based sensors, in which a sensor fusion algorithm is employed to mitigate occlusion and out-range issues commonly reported in such systems. Two wearable sensors were employed for the comparison of angle estimation: (i) a POF curvature sensor to measure 1-DOF angle; and (ii) a commercially available IMUs MTw Awinda from Xsens. A protocol to evaluate elbow joints of 11 healthy volunteers was implemented and the comparison of the three systems was presented using the correlation coefficient and the root mean squared error (RMSE). Moreover, a novel approach for angle correction of markerless camera-based systems is proposed here to minimize the errors on the sagittal plane. Results show a correlation coefficient up to 0.99 between the sensors with a RMSE of 4.90 ∘ , which represents a two-fold reduction when compared with the uncompensated results (10.42 ∘ ). Thus, the RGB-D system with the proposed technique is an attractive non-invasive and low-cost option for joint angle assessment. The authors envisage the proposed vision system as a valuable tool for the development of game-based interactive environments and for assistance of healthcare professionals on the generation of functional parameters during motion analysis in physical training and therapy.

Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras

Color-depth cameras (RGB-D cameras) have become the primary sensors in most robotics systems, from service robotics to industrial robotics applications. Typical consumer-grade RGB-D cameras are provided with a coarse intrinsic and extrinsic calibration that generally does not meet the accuracy requirements needed by many robotics applications (e.g., highly accurate 3D environment reconstruction and mapping, high precision object recognition and localization, ...). In this paper, we propose a human-friendly, reliable and accurate calibration framework that enables to easily estimate both the intrinsic and extrinsic parameters of a general color-depth sensor couple. Our approach is based on a novel two components error model. This model unifies the error sources of RGB-D pairs based on different technologies, such as structured-light 3D cameras and time-of-flight cameras. Our method provides some important advantages compared to other state-of-the-art systems: it is general (i.e., well suited for different types of sensors), based on an easy and stable calibration protocol, provides a greater calibration accuracy, and has been implemented within the ROS robotics framework. We report detailed experimental validations and performance comparisons to support our statements.

Application of Motion Capture Attributes to Individual Identification under Corridor Surveillance

Abstract Accurate and fast identification of a person from a security point of view is a key procedure. The most common technique of person identification uses identity cards. In contrary to the common approach we focus our research on identification based on the body movement such as the gait in this paper. The gait and the posture belong to the unique characteristics of the person that helps us to facilitate the identification. The proposed methodology allows us to incorporate personal characteristics into the access control systems using the color depth camera (RGBD). For the sake of gait analysis, the important task is to recognize the figure and extract the skeleton data from a video recording. Besides the usage of the mathematical statistics methods, we are opting to use computer animation and computer vision methods, which makes the research interdisciplinary. The main novelty of the paper is the definition and extraction of the feature vector from motion capture data, the analysis methodology and finally the selection of few statistically dominant motion attributes for the identification purposes. Besides the development of new approaches in this field, we validate proposed approaches from the perspective of accuracy.

CoDe4D: Color-Depth Local Spatio-Temporal Features for Human Activity Recognition From RGB-D Videos

Human activity recognition has a variety of important real-world applications, such as video analysis, surveillance, and human-robot interaction. As a promising video representation method, local spatio-temporal (LST) features have received increasing attention from computer vision, machine learning, and robotics communities. However, approaches based on traditional LST features only use color information, which face several challenges, such as illumination changes and dynamic backgrounds. The recent availability of commercial color-depth cameras makes it much cheaper, faster, and easier to acquire depth information, which provides a potential to implement more discriminative and robust LST features. In this paper, we introduce the new 4-D color-depth (CoDe4D) LST feature that incorporates both intensity and depth information acquired from RGB-D cameras. Our feature detector constructs a saliency map through applying independent filters in $\boldsymbol {xyzt}$ dimension to represent texture, shape and pose variations, and selects its local maxima as interest points. Our multichannel orientation histogram descriptor applies a 4-D support region, which is adaptive to linear perspective view changes, on each interest point. Then, image gradients of color-depth patches within the support region are computed and quantized using a spherical coordinate-based method to form a final feature vector. We build a complete activity recognition system by combining our features with bag-of-features representations and support vector machines. To evaluate the performance of our CoDe4D LST features and the complete system, we conduct experiments using four benchmark color-depth human activity data sets, including UTK Action3-D, Berkeley MHAD, ACT $\boldsymbol {4^{2^{^{^{}}}}}$ , and MSR daily activity 3-D data sets. Experimental results demonstrate the promising representative power of our CoDe4D features, which obtain the state-of-the-art performance on activity recognition from RGB-D visual data.

An ultra-fast human detection method for color-depth camera

Real-time human detection is important for a wide range of applications. The task is highly challenging due to occlusions, complex backgrounds, and variation of human poses. We propose a cascade-structured approach to real-time human detection in cluttered and dynamic environments with both color and depth data seamlessly incorporated. The first stage efficiently exploits depth data which generates a set of physically plausible yet over-detected candidates. These candidates are then purified by another two filters: a knowledge based human upper portion locator and a data-driven learning based filter. Experimental results show high detection accuracy achieved by the proposed method at 80–140fps on a single CPU core (without GPU acceleration).

Real-Time Multiple Human Perception With Color-Depth Cameras on a Mobile Robot

The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.