Depth-based Segmentation Research Articles

DepthCut: improved depth edge estimation using multiple unreliable channels

In the context of scene understanding, a variety of methods exists to estimate different information channels from mono or stereo images, including disparity, depth, and normals. Although several advances have been reported in the recent years for these tasks, the estimated information is often imprecise particularly near depth discontinuities or creases. Studies have however shown that precisely such depth edges carry critical cues for the perception of shape, and play important roles in tasks like depth-based segmentation or foreground selection. Unfortunately, the currently extracted channels often carry conflicting signals, making it difficult for subsequent applications to effectively use them. In this paper, we focus on the problem of obtaining high-precision depth edges (i.e., depth contours and creases) by jointly analyzing such unreliable information channels. We propose DepthCut, a data-driven fusion of the channels using a convolutional neural network trained on a large dataset with known depth. The resulting depth edges can be used for segmentation, decomposing a scene into depth layers with relatively flat depth, or improving the accuracy of the depth estimate near depth edges by constraining its gradients to agree with these edges. Quantitatively, we compare against 18 variants of baselines and demonstrate that our depth edges result in an improved segmentation performance and an improved depth estimate near depth edges compared to data-agnostic channel fusion. Qualitatively, we demonstrate that the depth edges result in superior segmentation and depth orderings. (Code and datasets will be made available.)

Saliency and depth‐based unsupervised object segmentation

This study demonstrates an unsupervised segmentation algorithm for video sequences acquired from a moving camera with results comparable to semi-supervised (interactive) methods. The authors employ depth cues from multiple views stereo to enhance the hypothesis of a potential object based on saliency scores. The resulting object and background hypotheses are then used to model foreground and background distributions for a graph-cut-based segmentation. The authors’ graph-cut framework simultaneously optimises over depth and colour information to produce automatically segmented objects in challenging unstructured scenes. They refer to this saliency and depth-based segmentation method as ‘SDCut’. The proposed method is fully automatic without requiring any intervention. Experiments demonstrate that their method can achieve accurate segmentation results which are comparable with several well-known human interactive semi-supervised segmentation methods.

Robust Depth-based Planar Segmentation Algorithm based on Gradient of Depth feature

In this work, a new algorithm for segmenting planar regions from depth images is proposed. Despite the numerous algorithms that extract planar regions from depth images, developing a new algorithm was motivated by the results of other State-of-Art algorithms which are tailored to segment depth images acquired by laser scanners or structure light cameras, but failed to segment depth images generated from stereo vision of normal textured objects which are found a lot in many indoor scenarios. Unlike other State-of-Art methods which are based mainly on the local surface normal feature for the segmentation process; the proposed planar segmentation algorithm is based on a novel feature to be called Gradient of Depth feature (DoG). The proposed DoG feature is parameter-free, easy to compute in terms of complexity and faster to compute compared to the local surface normal (i.e. the GoD feature is computed in the 2D image space vs. the surface normal which is computed in the 3D point cloud). The proposed feature is implemented into a robust algorithm that utilizes the 1D dimension feature space of the GoD feature which in turn increases the robustness of the algorithm to parameters change. The proposed algorithm can be used on different scenes acquired by different cameras. In terms of segmentation accuracy, it does not only meet (or even pass in some cases) the performance of other State-of-Art algorithms, but it is able to segment robustly planar regions from disparity images on which other algorithms have failed. Additionally, the GoD - based algorithm segments planar regions from non-planar objects where classical planar segmentation algorithms fail to segment.

Depth-Assisted Frame Rate Up-Conversion for Stereoscopic Video

In this letter, we propose a depth-assisted frame rate up-conversion (DA-FRUC) scheme which finds applications in 3D video processing. By considering the depth cue in video plus depth representation, we categorize the blocks of the interpolated frame as depth-continuous and depth-discontinuous groups. The motion vector (MV) outliers of the depth continuous blocks are then detected and corrected by layer-constrained MV refinement method. Moreover, a depth-based adaptive interpolation and block segmentation method is proposed to deal with the disocclusion and occlusion at the boundary of the foreground area. Experimental results show that the proposed method achieves better subjective and objective performance for the interpolated color frames. Additionally, the interpolated depth frames obtained by the proposed method is more accurate, which benefit the video quality in view synthesis.

Object-based selection operating on a spatial representation made salient by dimensional segmentation mechanisms: a re-investigation of Egly and Homa (1984)

Three experiments re-investigated selective attention in the 'ring-cueing' paradigm of Egly and Homa (J Exp Psychol: Human Percept Perform 10:778-793, 1984). Observers were cued to attend to one of three concentric rings of radius 1 degrees, 2 degrees, or 3 degrees, and their signal detection accuracy for cued and uncued rings was measured. Experiment 1, which used a central color cue to indicate a like-colored ring, replicated ring-cueing effects along the lines of Egly and Homa. Experiments 2 and 3 examined whether these effects were produced by observers exploiting secondary-depth cues possibly inherent in the display layout. With color cues, the availability of secondary-depth information had no influence on the ring-cueing effects. However, making the rings monochrome and using central size cues significantly reduced the ring-cueing effects when the depth information was disrupted. The results suggest that selection was object-based, operating on a spatial 'grouped-array' representation of the cued ring made salient by color- or depth-based segmentation mechanisms.

Extracting View-Dependent Depth Maps from a Collection of Images

Stereo correspondence algorithms typically produce a single depth map. In addition to the usual problems of occlusions and textureless regions, such algorithms cannot model the variation in scene or object appearance with respect to the viewing position. In this paper, we propose a new representation that overcomes the appearance variation problem associated with an image sequence. Rather than estimating a single depth map, we associate a depth map with each input image (or a subset of them). Our representation is motivated by applications such as view interpolation and depth-based segmentation for model-building or layer extraction. We describe two approaches to extract such a representation from a sequence of images. The first approach, which is more classical, computes the local depth map associated with each chosen reference frame independently. The novelty of this approach lies in its combination of shiftable windows, temporal selection, and graph cut optimization. The second approach simultaneously optimizes a set of self-consistent depth maps at multiple key-frames. Since multiple depth maps are estimated simultaneously, visibility can be modeled explicitly and disparity consistency imposed across the different depth maps. Results, which include a difficult specular scene example, show the effectiveness of our approach.

Depth-based segmentation

The tool presented in this paper performs an automatic segmentation of stereoscopic image sequences, based on the modeling of distance maps obtained by image processing. Two three-dimensional (3-D) image analysis algorithms are combined: (i) estimation of dense depth maps from stereoscopic image sequences and (ii) depth-based object segmentation. A Markovian statistical approach for such a segmentation of a dense depth map into arbitrarily shaped and oriented planar surfaces is described in detail. The simulation results on sequences Fun fair and Tunnel, provided on video tape to the MPEG-4 tests of November 1995, are discussed.