Consistent Foreground Co-segmentation

Abstract

When the foreground objects have variegated appearance and/or manifest articulated motion, not to mention the momentary occlusions by other unintended objects, a segmentation method based on single video and a bottom-up approach is often insufficient for their extraction. In this paper, we present a video co-segmentation method to address the aforementioned challenges. Departing from the objectness attributes and motion coherence used by traditional figure-ground separation methods, we place central importance in the role of “common fate”, that is, the different parts of the foreground should persist together in all the videos. To accomplish this idea, we first extract seed superpixels by a motion-based figure/ground segmentation method. We then formulate a set of linkage constraints between these superpixels based on whether they exhibit the characteristics of common fate or not. An iterative constrained clustering algorithm is then proposed to trim away the incorrect and accidental linkage relationships. The clustering algorithm also performs automatic model selection to estimate the number of individual objects in the foreground (e.g. male and female birds in courtship), while at the same time binding the parts of a variegated object together in a unified whole. Finally, a multiclass labeling Markov randome field is used to obtain a refined segmentation result. Our experimental results on two datasets show that our method successfully addresses the challenges in the extraction of complex foreground and outperforms the state-of-the-art video segmentation and co-segmentation methods.