Low-cost RGB-D Camera Research Articles

In Situ 3D Segmentation of Individual Plant Leaves Using a RGB-D Camera for Agricultural Automation.

In this paper, we present a challenging task of 3D segmentation of individual plant leaves from occlusions in the complicated natural scene. Depth data of plant leaves is introduced to improve the robustness of plant leaf segmentation. The low cost RGB-D camera is utilized to capture depth and color image in fields. Mean shift clustering is applied to segment plant leaves in depth image. Plant leaves are extracted from the natural background by examining vegetation of the candidate segments produced by mean shift. Subsequently, individual leaves are segmented from occlusions by active contour models. Automatic initialization of the active contour models is implemented by calculating the center of divergence from the gradient vector field of depth image. The proposed segmentation scheme is tested through experiments under greenhouse conditions. The overall segmentation rate is 87.97% while segmentation rates for single and occluded leaves are 92.10% and 86.67%, respectively. Approximately half of the experimental results show segmentation rates of individual leaves higher than 90%. Nevertheless, the proposed method is able to segment individual leaves from heavy occlusions.

Low-cost surface reconstruction for aesthetic results assessment and prediction in breast cancer surgery.

The high incidence and low mortality of breast cancer surgery has led to an increasing emphasis on the cosmetic outcome of surgical treatment. Advances in aesthetic evaluation, as well as surgical planning and outcome prediction, have been investigated by using geometrically precise 3D modelling of the breast surface prior to surgery and after the procedure. However, existing solutions are based on expensive site specific setups and remain weakly validated. In this paper, we explore the possibility of using low-cost RGBD cameras as an affordable and mobile system for breast surface reconstruction. The methodology relies on sensor calibration, uncertainty-driven point filtering, dense reconstruction and subsequent multi-view joint optimization to diffuse residual pose errors. Results from a phantom study, with ground truth obtained through commercially available scanners, indicate that the approach is promising with RMS errors in order of 2 mm. A clinical study shows the practical applicability of our method and compares favourably to high-end scanning solutions.

A proposal of a non-intrusive, global movement analysis of hemiparesis treatment

Hemiparesis is a loss of muscle strength on one side of the body, and the most disabling condition after a stroke. Hemiparetic individuals suffer from a loss of muscle strength on one side of the body, resulting in a decreased capacity of performing movements. To assess the quality of Physiotherapy treatment, rating scales are commonly used but with the shortcoming of being subjective. With the aim of developing a system that objectively outcomes how a hemiparetic individual is responding to a Physiotherapy treatment, this paper proposes a method to analyze human functional movement by means of an apparatus comprised of multiple low-cost RGB-D cameras. After extrinsically calibrating the cameras, the setup system should be able to compose a composite skeleton of the target patient, to globally analyze his/her movement according to a reachable workspace and specific energy. These latter both are proposed to be carried out by tracking the hand movements of the patient, and the movement volume produced. Here we present the concept of the proposed system, as well as, the idea of its parts.

A Single Frame Depth Visual Gyroscope and its Integration for Robot Navigation and Mapping in Structured Indoor Environments

An accurate navigation system is an essential and important part for the mobile robot. The recent appearance of low cost RGBD cameras has made 3D point clouds together with RGB information easy acc...

Handling occlusions in augmented reality based on 3D reconstruction method

The correct relationships between real and virtual objects are of utmost importance to a realistic augmented reality system, in which the occlusion handling method should be able to estimate the spatial relationships between real and virtual objects, as well as handle the mutual occlusion automatically in real-time. To accomplish the above tasks simultaneously, we propose a novel occlusion handling method based on 3D reconstruction, which consists of offline stage and online stage. In the offline stage, we get the depth map of the real scene using a low cost RGB-D camera. Then the 3D coordinate of each point in the global coordinate system are obtained and will be used in the online occlusion handling stage. In the online stage, we design a GPU based 3D point clouds alignment method by using point to tangent plane distance as error metric to accelerate the convergence speed and reduce the iterations. The correct relationships between real and virtual objects are then obtained automatically by comparing each pixel’s Z coordinate value of real objects with that of virtual objects in a smaller region to achieve real-time performance. More specifically, we can judge and handle the mutual occlusion without human interactivity in real time, and experimental results prove its effectiveness.

Robust 3D reconstruction with an RGB-D camera.

We present a novel 3D reconstruction approach using a low-cost RGB-D camera such as Microsoft Kinect. Compared with previous methods, our scanning system can work well in challenging cases where there are large repeated textures and significant depth missing problems. For robust registration, we propose to utilize both visual and geometry features and combine SFM technique to enhance the robustness of feature matching and camera pose estimation. In addition, a novel prior-based multicandidates RANSAC is introduced to efficiently estimate the model parameters and significantly speed up the camera pose estimation under multiple correspondence candidates. Even when serious depth missing occurs, our method still can successfully register all frames together. Loop closure also can be robustly detected and handled to eliminate the drift problem. The missing geometry can be completed by combining multiview stereo and mesh deformation techniques. A variety of challenging examples demonstrate the effectiveness of the proposed approach.

Real-time Visual Odometry Estimation Based on Principal Direction Detection on Ceiling Vision

In this paper, we present a novel algorithm for odometry estimation based on ceiling vision. The main contribution of this algorithm is the introduction of principal direction detection that can greatly reduce error accumulation problem in most visual odometry estimation approaches. The principal direction is defined based on the fact that our ceiling is filled with artificial vertical and horizontal lines which can be used as reference for the current robot's heading direction. The proposed approach can be operated in real-time and it performs well even with camera's disturbance. A moving low-cost RGB-D camera (Kinect), mounted on a robot, is used to continuously acquire point clouds. Iterative closest point (ICP) is the common way to estimate the current camera position by registering the currently captured point cloud to the previous one. However, its performance suffers from data association problem or it requires pre-alignment information. The performance of the proposed principal direction detection approach does not rely on data association knowledge. Using this method, two point clouds are properly pre-aligned. Hence, we can use ICP to fine-tune the transformation parameters and minimize registration error. Experimental results demonstrate the performance and stability of the proposed system under disturbance in real-time. Several indoor tests are carried out to show that the proposed visual odometry estimation method can help to significantly improve the accuracy of simultaneous localization and mapping (SLAM).

Background foreground segmentation with RGB-D Kinect data: An efficient combination of classifiers

Low cost RGB-D cameras such as the Microsoft’s Kinect or the Asus’s Xtion Pro are completely changing the computer vision world, as they are being successfully used in several applications and research areas. Depth data are particularly attractive and suitable for applications based on moving objects detection through foreground/background segmentation approaches; the RGB-D applications proposed in literature employ, in general, state of the art foreground/background segmentation techniques based on the depth information without taking into account the color information. The novel approach that we propose is based on a combination of classifiers that allows improving background subtraction accuracy with respect to state of the art algorithms by jointly considering color and depth data. In particular, the combination of classifiers is based on a weighted average that allows to adaptively modifying the support of each classifier in the ensemble by considering foreground detections in the previous frames and the depth and color edges. In this way, it is possible to reduce false detections due to critical issues that can not be tackled by the individual classifiers such as: shadows and illumination changes, color and depth camouflage, moved background objects and noisy depth measurements. Moreover, we propose, for the best of the author’s knowledge, the first publicly available RGB-D benchmark dataset with hand-labeled ground truth of several challenging scenarios to test background/foreground segmentation algorithms.

Employing a RGB-D sensor for real-time tracking of humans across multiple re-entries in a smart environment

The term smart environment refers to physical spaces equipped with sensors feeding into adaptive algorithms that enable the environment to become sensitive and responsive to the presence and needs of its occupants. People with special needs, such as the elderly or disabled people, stand to benefit most from such environments as they offer sophisticated assistive functionalities supporting independent living and improved safety. In a smart environment, the key issue is to sense the location and identity of its users. In this paper, we intend to tackle the problems of detecting and tracking humans in a realistic home environment by exploiting the complementary nature of (synchronized) color and depth images produced by a low-cost consumer-level RGB-D camera. Our system selectively feeds the complementary data emanating from the two vision sensors to different algorithmic modules which together implement three sequential components: (1) object labeling based on depth data clustering, (2) human re-entry identification based on comparing visual signatures extracted from the color (RGB) information, and (3) human tracking based on the fusion of both depth and RGB data. Experimental results show that this division of labor improves the system?s efficiency and classification performance.