RGB-D Segmentation Research Articles

A curved path extraction method using RGB-D multimodal data for single-edge guided navigation in irregularly shaped fields

Due to the undulating topography, the world contains numerous irregularly shaped small fields. Autonomous navigation along the curved edges of these fields is crucial for achieving complete unmanned operation in these small-field environments. Based on this, we proposed an accurate and robust curved path extraction method using RGB-D multimodal data for single-edge guided navigation in irregularly shaped fields. Leveraging the significant height difference between headland and farmland, we employed an efficient RGB-D multimodal semantic segmentation network to achieve high-precision segmentation of headland area. Meanwhile, a slope-based feature points adaptive selection algorithm and a feature points transformation algorithm fusing multimodal depth data were proposed innovatively for curved navigation path extraction. By utilizing the RGB-D segmentation model ESANet, we achieved a mean intersection over union (mIoU) of 96.94 %, surpassing other RGB segmentation models. The mean deviations between the ground truth path and the extracted path were less than 6.43 pixels on an image with a resolution of 640 × 480. The proposed method can finally achieve a processing efficiency of 23.80 FPS on images in 640 × 480 resolution, and the field experiments tested that the mean deviations of the agricultural vehicle navigating along the extracted curved navigation path were less than 0.188 m, which met the real-time performance requirement and accuracy requirement of autonomous navigation.

Weakly-Supervised RGBD Video Object Segmentation.

Depth information opens up new opportunities for video object segmentation (VOS) to be more accurate and robust in complex scenes. However, the RGBD VOS task is largely unexplored due to the expensive collection of RGBD data and time-consuming annotation of segmentation. In this work, we first introduce a new benchmark for RGBD VOS, named DepthVOS, which contains 350 videos (over 55k frames in total) annotated with masks and bounding boxes. We futher propose a novel, strong baseline model - Fused Color-Depth Network (FusedCDNet), which can be trained solely under the supervision of bounding boxes, while being used to generate masks with a bounding box guideline only in the first frame. Thereby, the model possesses three major advantages: a weakly-supervised training strategy to overcome the high-cost annotation, a cross-modal fusion module to handle complex scenes, and weakly-supervised inference to promote ease of use. Extensive experiments demonstrate that our proposed method performs on par with top fully-supervised algorithms. We will open-source our project on https://github.com/yjybuaa/depthvos/ to facilitate the development of RGBD VOS.

MMPL-Net: multi-modal prototype learning for one-shot RGB-D segmentation

MMPL-Net: multi-modal prototype learning for one-shot RGB-D segmentation

A novel fuzzy clustering based method for image segmentation in RGB-D images

Automatic image segmentation is a challenging task in computer vision applications, especially in the presence of occluded objects, varying color, and different lighting conditions. The advancement of depth-sensing technologies has introduced RGB-Depth cameras which are capable to generate RGB-Depth images and brought significant changes in computer vision applications. However, the segmentation of RGB-Depth images is a difficult task. Therefore, in this paper, a new segmentation method for RGB-Depth images has been introduced and named as random Henry gas solubility optimization-fuzzy clustering method. Firstly, a random Henry gas solubility optimization algorithm has been developed. Next, the proposed optimization algorithm has been employed to obtain optimal fuzzy clusters which are finally merged through segmentation by aggregating superpixels. The standard NYU depth V2 RGB-Depth indoor image dataset is used for performance evaluation. The proposed segmentation approach has been compared with five different methods namely, kmeans, fuzzy c-means, Henry gas solubility optimization algorithm, chaotic gravitational search algorithm, and J-Segmentation in terms of qualitative and quantitative measures. The result analysis shows that the proposed RGB-D segmentation method outperforms the other considered methods.

RGB[formula omitted]D: Learning depth-weighted RGB patches for RGB-D indoor semantic segmentation

Significant advances have been made in designing CNNs for RGB semantic segmentation. However, these CNNs are not widely adopted for RGB-D segmentation, due to the asymmetry between the RGB and depth modalities. Instead, dedicated architectures are designed to fuse them for effective RGB-D segmentation, wherein complex structures are often employed, resulting in much increased computational cost. In this paper, we propose a novel way to learn the fusion of RGB and depth information in an early stage. This enables our method to easily adopt existing RGB segmentation networks with minimal modification. Our method is simple yet effective to build a bridge between RGB and RGBD semantic segmentation, so as to avoid designing a far more complex network structure for RGBD segmentation. The proposed method treats RGB and depth information in an inherently asymmetric manner, and to the best of our knowledge, this is the first approach that learns to fuse them in a multiplicative manner for RGB-D segmentation; thus, we call it RGB×D. Extensive experiments and ablation studies on the challenging NYUDv2, SUN RGB-D and Cityscapes semantic segmentation benchmarks show that the proposed RGB×D offers a consistent improvement over several baselines.

A salient region detection model combining background distribution measure for indoor robots.

Vision system plays an important role in the field of indoor robot. Saliency detection methods, capturing regions that are perceived as important, are used to improve the performance of visual perception system. Most of state-of-the-art methods for saliency detection, performing outstandingly in natural images, cannot work in complicated indoor environment. Therefore, we propose a new method comprised of graph-based RGB-D segmentation, primary saliency measure, background distribution measure, and combination. Besides, region roundness is proposed to describe the compactness of a region to measure background distribution more robustly. To validate the proposed approach, eleven influential methods are compared on the DSD and ECSSD dataset. Moreover, we build a mobile robot platform for application in an actual environment, and design three different kinds of experimental constructions that are different viewpoints, illumination variations and partial occlusions. Experimental results demonstrate that our model outperforms existing methods and is useful for indoor mobile robots.

3D Environment Mapping Using the Kinect V2 and Path Planning Based on RRT Algorithms

This paper describes a 3D path planning system that is able to provide a solution trajectory for the automatic control of a robot. The proposed system uses a point cloud obtained from the robot workspace, with a Kinect V2 sensor to identify the interest regions and the obstacles of the environment. Our proposal includes a collision-free path planner based on the Rapidly-exploring Random Trees variant (RRT*), for a safe and optimal navigation of robots in 3D spaces. Results on RGB-D segmentation and recognition, point cloud processing, and comparisons between different RRT* algorithms, are presented.

Automatic Sleep System Recommendation by Multi-modal RBG-Depth-Pressure Anthropometric Analysis

This paper presents a novel system for automatic sleep system recommendation using RGB, depth and pressure information. It consists of a validated clinical knowledge-based model that, along with a set of prescription variables extracted automatically, obtains a personalized bed design recommendation. The automatic process starts by performing multi-part human body RGB-D segmentation combining GrabCut, 3D Shape Context descriptor and Thin Plate Splines, to then extract a set of anthropometric landmark points by applying orthogonal plates to the segmented human body. The extracted variables are introduced to the computerized clinical model to calculate body circumferences, weight, morphotype and Body Mass Index categorization. Furthermore, pressure image analysis is performed to extract pressure values and at-risk points, which are also introduced to the model to eventually obtain the final prescription of mattress, topper, and pillow. We validate the complete system in a set of 200 subjects, showing accurate category classification and high correlation results with respect to manual measures.

Joint Color-Spatial-Directional Clustering and Region Merging (JCSD-RM) for Unsupervised RGB-D Image Segmentation.

Recent advances in depth imaging sensors provide easy access to the synchronized depth with color, called RGB-D image. In this paper, we propose an unsupervised method for indoor RGB-D image segmentation and analysis. We consider a statistical image generation model based on the color and geometry of the scene. Our method consists of a joint color-spatial-directional clustering method followed by a statistical planar region merging method. We evaluate our method on the NYU depth database and compare it with existing unsupervised RGB-D segmentation methods. Results show that, it is comparable with the state of the art methods and it needs less computation time. Moreover, it opens interesting perspectives to fuse color and geometry in an unsupervised manner.

3D Reasoning from Blocks to Stability.

Objects occupy physical space and obey physical laws. To truly understand a scene, we must reason about the space that objects in it occupy, and how each objects is supported stably by each other. In other words, we seek to understand which objects would, if moved, cause other objects to fall. This 3D volumetric reasoning is important for many scene understanding tasks, ranging from segmentation of objects to perception of a rich 3D, physically well-founded, interpretations of the scene. In this paper, we propose a new algorithm to parse a single RGB-D image with 3D block units while jointly reasoning about the segments, volumes, supporting relationships, and object stability. Our algorithm is based on the intuition that a good 3D representation of the scene is one that fits the depth data well, and is a stable, self-supporting arrangement of objects (i.e., one that does not topple). We design an energy function for representing the quality of the block representation based on these properties. Our algorithm fits 3D blocks to the depth values corresponding to image segments, and iteratively optimizes the energy function. Our proposed algorithm is the first to consider stability of objects in complex arrangements for reasoning about the underlying structure of the scene. Experimental results show that our stability-reasoning framework improves RGB-D segmentation and scene volumetric representation.