Single Image Segmentation with Estimated Depth

Abstract

Object segmentation is a fundamental problem in computer vision. Although many segmentation methods have been proposed, most of them still rely on the appearances of images (i.e., colors or textures) [1, 2, 3, 4, 6, 8]. Consequently, they have a difficulty in distinguishing an object from the background with a similar appearance to the object. To overcome this difficulty, we employ a depth map of an input image as an additional cue to the object segmentation. The main contribution of this work is to introduce a novel segmentation framework that utilizes the depth map combined with a color image to describe the features of objects and backgrounds, where the depth map is estimated from the color image. While a depth map has great potential for use in segmentation, finding a way of integrating two completely different physical quantities, namely the color and depth, has remained unclear. We introduce an integration of the color and depth likelihood on objectness and backgroundness, which simply and effectively extends a traditional segmentation framework based on the Markov random fields (MRF) [2]. By refining the likelihood with the depth information, our proposed method can suppress the incorrect detection of misleading backgrounds. A single image is expressed by K, where K includes color information C = {Cx ∈R}x∈Ω, and in our case, depth informationZ = {Zx ∈R}x∈Ω (x is a position in the image domain Ω ⊂ N2). Object segmentation is the problem of assigning the label A = {Ax}x∈Ω, which gives a label Ax = {0,1} to each pixel, where the labels 1 and 0 at x respectively correspond to the object and background. The statistical relationship between K and A can be described by an MRF, and the appropriate configuration of the labels can be derived by minimizing the following energy function E: