RGB Depth Images Research Articles

A survey on indoor RGB-D semantic segmentation: from hand-crafted features to deep convolutional neural networks

Semantic segmentation is one of the most important tasks in the field of computer vision. It is the main step towards scene understanding. With the advent of RGB-Depth sensors, such as Microsoft Kinect, nowadays RGB-Depth images are easily available. This has changed the landscape of some tasks such as semantic segmentation. As the depth images are independent of illumination, the combination of depth and RGB images can improve the quality of semantic labeling. The related research has been divided into two main categories, based on the usage of hand-crafted features and deep learning. Although the state-of-the-art results are mainly achieved by deep learning methods, traditional methods have also been at the center of attention for some years and lots of valuable work have been done in that category. As the field of semantic segmentation is very broad, in this survey, a comprehensive analysis has been carried out on RGB-Depth semantic segmentation methods, their challenges and contributions, available RGB-Depth datasets, metrics of evaluation, state-of-the-art results, and promising directions of the field.

Superpixel Segmentation Using Weighted Coplanar Feature Clustering on RGBD Images

Superpixel segmentation is a widely used preprocessing method in computer vision, but its performance is unsatisfactory for color images in cluttered indoor environments. In this work, a superpixel method named weighted coplanar feature clustering (WCFC) is proposed, which produces full coverage of superpixels in RGB-depth (RGBD) images of indoor scenes. Basically, a linear iterative clustering is adopted based on a cluster criterion that measures the color similarity, space proximity and geometric resemblance between pixels. However, to avoid the adverse impact of RGBD image flaws and to make full use of the depth information, WCFC first preprocesses the raw depth maps with an inpainting algorithm called a Cross-Bilateral Filter. Second, a coplanar feature is extracted from the refined RGBD image to represent the geometric similarities between pixels. Third, combined with the colors and positions of the pixels, the coplanar feature constructs the feature vector of the clustering method; thus, the distance measure, as the cluster criterion, is computed by normalizing the feature vectors. Finally, in order to extract the features of the RGBD image dynamically, a content-adaptive weight is introduced as a coefficient of the coplanar feature, which strikes a balance between the coplanar feature and other features. Experiments performed on the New York University (NYU) Depth V2 dataset demonstrate that WCFC outperforms the available state-of-the-art methods in terms of accuracy of superpixel segmentation, while maintaining a high speed.

Robust approach to inverse lighting using RGB-D images

Proper lighting is a key element in developing a photorealistic computer-generated image. This paper introduces a novel approach for robustly extracting lighting conditions from an RGB-D (RGB + depth) image. Existing studies on lighting estimation have developed image analysis techniques by constraining the scope and condition of the target objects. For example, they have assumed that the objects have homogeneous surfaces, inter-reflections can be ignored, and their three-dimensional (3D) geometries consist of a noise-free mesh. These assumptions, however, are unrealistic; real objects often have complicated non-homogeneous surfaces, inter-reflections that affect a considerable portion of illumination, or unpredictable noise that can affect sensor measurements. To overcome these limitations, this study takes non-homogeneous surface objects into account in the inverse lighting framework via segment-based scene representation. Moreover, we employ outlier removal and appropriate region selection to achieve robust lighting estimation in the presence of inter-reflections and noise. We demonstrate the effectiveness of the proposed approach by conducting extensive experiments on synthetic and real RGB-D images.

Fast capture of textured full-body avatar with RGB-D cameras

We present a practical system which can provide a textured full-body avatar within 3 s. It uses sixteen RGB-depth (RGB-D) cameras, ten of which are arranged to capture the body, while six target the important head region. The configuration of the multiple cameras is formulated as a constraint-based minimum set space-covering problem, which is approximately solved by a heuristic algorithm. The camera layout determined can cover the full-body surface of an adult, with geometric errors of less than 5 mm. After arranging the cameras, they are calibrated using a mannequin before scanning real humans. The 16 RGB-D images are all captured within 1 s, which both avoids the need for the subject to attempt to remain still for an uncomfortable period, and helps to keep pose changes between different cameras small. All scans are combined and processed to reconstruct the photorealistic textured mesh in 2 s. During both system calibration and working capture of a real subject, the high-quality RGB information is exploited to assist geometric reconstruction and texture stitching optimization.

Extended quantum cuts for unsupervised salient object extraction

In this manuscript, an unsupervised salient object extraction algorithm is proposed for RGB and RGB-Depth images. Saliency estimation is formulated as a foreground detection problem. To this end, Q...

Detecting gait phases from RGB-D images based on hidden markov model

Gait contains important information about the status of the human body and physiological signs. In many medical applications, it is important to monitor and accurately analyze the gait of the patient. Since walking shows the reproducibility signs in several phases, separating these phases can be used for the gait analysis. In this study, a method based on image processing for extracting phases of human gait from RGB-Depth images is presented. The sequence of depth images from the front view has been processed to extract the lower body depth profile and distance features. Feature vector extracted from image is the same as observation vector of hidden Markov model, and the phases of gait are considered as hidden states of the model. After training the model using the images which are randomly selected as training samples, the phase estimation of gait becomes possible using the model. The results confirm the rate of 60-40% of two major phases of the gait and also the mid-stance phase is recognized with 85% precision.

Robust Multiview Feature Learning for RGB-D Image Understanding

The availability of massive RGB-depth (RGB-D) images poses a compelling need for effective RGB-D content understanding techniques. RGB-D images provide synchronized information from multiple views (e.g., color and depth) of real-world objects and scenes. This work proposes learning compact and discriminative features from the multiple views of RGB-D content toward effective feature representation for RGB-D image understanding. In particular, a robust multiview feature learning approach is developed, which exploits the intrinsic relations among multiple views. The feature learning in multiple views is jointly optimized in an integrated formulation. The joint optimization essentially exploits the intrinsic relations among the views, leading to effective features and making the learning process robust to noises. The feature learning function is formulated as a robust nonnegative graph embedding function over multiple graphs in various views. The graphs characterize the local geometric and discriminating structure of the multiview data. The joint sparsity in ℓ 1 -norm graph embedding and ℓ 21 -norm data factorization further enhances the robustness of feature learning. We derive an efficient computational solution for the proposed approach and provide rigorous theoretical proof with regard to its convergence. We apply the proposed approach to two RGB-D image understanding tasks: RGB-D object classification and RGB-D scene categorization. We conduct extensive experiments on two real-world RGB-D image datasets. The experimental results have demonstrated the effectiveness of the proposed approach.

Size estimation of sweet onions using consumer-grade RGB-depth sensor

Size estimation is an important aspect of the postharvest handling of onions. This study applied the RGB-depth (RGB-D) sensor to measure the maximum diameter and volume of sweet onions, and estimated the density of onions using measured parameters. RGB-D images were acquired when onions were placed at six different orientations. The maximum diameter was calculated using both the color and depth images. The volume was estimated using the depth images. The onion diameter estimated by depth images achieved a higher average accuracy and robustness (RMSE=2mm) than those calculated by color images (RMSE=3.4mm). The predicted volume of onions showed a RMSE of 18.5cm3 and an accuracy of 96.3%. Results also demonstrated that it is promising to nondestructively estimate the onion density based on its depth image. The proposed methods can be applied to improve the efficacy and efficiency of size estimation in onion phenotyping and postharvest sorting/grading.