Video Object Segmentation Method Research Articles

Online Meta Adaptation for Fast Video Object Segmentation

Conventional deep neural networks based video object segmentation (VOS) methods are dominated by heavily fine-tuning a segmentation model on the first frame of a given video, which is time-consuming and inefficient. In this paper, we propose a novel method which rapidly adapts a base segmentation model to new video sequences with only a couple of model-update iterations, without sacrificing performance. Such attractive efficiency benefits from the meta-learning paradigm which leads to a meta-segmentation model and a novel continuous learning approach which enables online adaptation of the segmentation model. Concretely, we train a meta-learner on multiple VOS tasks such that the meta model can capture their common knowledge and gains the ability to fast adapt the segmentation model to new video sequences. Furthermore, to deal with unique challenges of VOS tasks from temporal variations in the video, e.g., object motion and appearance changes, we propose a principled online adaptation approach that continuously adapts the segmentation model across video frames by exploiting temporal context effectively, providing robustness to annoying temporal variations. Integrating the meta-learner with the online adaptation approach, the proposed VOS model achieves competitive performance against the state-of-the-arts and moreover provides faster per-frame processing speed.

Online Semantic Object Segmentation for Vision Robot Collected Video

Video segmentation is the foundation of many senior computer vision applications. For robotic vision applications, video object segmentation is facing more difficulties. Since the videos are usually collected from those complicated scenes and the camera perspective may be static for some time, object saliency and motion clues which are widely used in traditional video segmentation approaches could be unreliable. Besides, its online processing ability is of crucial importance. In this paper, we propose an online video object segmentation method, which is organized as a segmentation-by-detection framework. It first effectively segments out meaningful objects in short video frame batches with the help of object detectors, and then the segmentation of different batches can be associated with each other via bi-directional notebook based connecting strategy. Experiments conducted on public available datasets verify the above-mentioned property and show good performance of the proposed method.

Video Salient Object Detection Using Spatiotemporal Deep Features.

This paper presents a method for detecting salient objects in videos, where temporal information in addition to spatial information is fully taken into account. Following recent reports on the advantage of deep features over conventional handcrafted features, we propose a new set of spatiotemporal deep (STD) features that utilize local and global contexts over frames. We also propose new spatiotemporal conditional random field (STCRF) to compute saliency from STD features. STCRF is our extension of CRF to the temporal domain and describes the relationships among neighboring regions both in a frame and over frames. STCRF leads to temporally consistent saliency maps over frames, contributing to accurate detection of salient objects' boundaries and noise reduction during detection. Our proposed method first segments an input video into multiple scales and then computes a saliency map at each scale level using STD features with STCRF. The final saliency map is computed by fusing saliency maps at different scale levels. Our experiments, using publicly available benchmark datasets, confirm that the proposed method significantly outperforms the state-of-the-art methods. We also applied our saliency computation to the video object segmentation task, showing that our method outperforms existing video object segmentation methods.

Video Object Segmentation via Global Consistency Aware Query Strategy

In this paper, we propose a video object segmentation method via global consistency aware query strategy. The aim is to obtain higher segmentation accuracy with less user annotation. Intuitively, we hope to annotate some frames to obtain better segmentation performance than to annotate other frames, which can be modeled by active learning framework. Specifically, we first generate a sample space of potential annotation regions via an object proposals method for each frame. Then, the annotation likelihood for the region is calculated in terms of annotation history and global consistency for the object in the video. Third, the segmentation result of the annotation region can be obtained by minimizing an MRF energy function. Fourth, the algorithm will provide the user with the most valuable frame to annotate, which has high annotation likelihood and large segmentation result change. Finally, the annotation is added to the framework to begin the next iteration. Experiments on a number of video sequences demonstrate that the proposed method can reduce the user effort and obtain the higher segmentation accuracy compared with the state-of-the-art methods.

Gamifying Video Object Segmentation.

Video object segmentation can be considered as one of the most challenging computer vision problems. Indeed, so far, no existing solution is able to effectively deal with the peculiarities of real-world videos, especially in cases of articulated motion and object occlusions; limitations that appear more evident when we compare the performance of automated methods with the human one. However, manually segmenting objects in videos is largely impractical as it requires a lot of time and concentration. To address this problem, in this paper we propose an interactive video object segmentation method, which exploits, on one hand, the capability of humans to identify correctly objects in visual scenes, and on the other hand, the collective human brainpower to solve challenging and large-scale tasks. In particular, our method relies on a game with a purpose to collect human inputs on object locations, followed by an accurate segmentation phase achieved by optimizing an energy function encoding spatial and temporal constraints between object regions as well as human-provided location priors. Performance analysis carried out on complex video benchmarks, and exploiting data provided by over 60 users, demonstrated that our method shows a better trade-off between annotation times and segmentation accuracy than interactive video annotation and automated video object segmentation approaches.

Video object segmentation based on supervoxel for multimedia corpus construction

Multimedia corpus is efficient for second language acquisition when we tag texts, scripts or objects in language materials and retrieve the corresponding video or audio parts. This paper presents an automatic video object segmentation method based on supervoxel in a multimedia corpus. In this method, a novel algorithm is proposed for generating the supervoxel. In 3D video volume, motion consistence of moving object is a frequent challenge for segmentation algorithm and a robust, long-term supervoxel could help in this situation. The segmentation process is considered as a minimization of an energy function where variables are labels of supervoxels. The supervoxel-based energy function consists of two terms, which are data term and smoothness term. The data term estimates the likelihood of a supervoxel to be labeled as moving object or background according to an appearance model and a motion model. The appearance model is based on an automatic processing for color collection. The motion model is derived from acting center-surround filter on an optical flow field. The supervoxel-based appearance and geometry cues are used to build smoothness term. The final segmentation can be obtained by minimizing the energy function which is defined on a spatio-temporal 3D video volume. We present experimental results of our method which outperforms the state-of-the-art methods on standard test sequences.

递推式视频前景分割方法

This paper presents a kind of recursive video object segmentation method. Previous video cutout methods present three major limitations especially for complex video scene: firstly they lack the ability to deal with the inseparable color problem between foreground and background scenes, and the occlusion/disocclusion problem caused by large movement, new exposed regions or topology change are also difficult to be solved. Lastly, the consideration on how to build the color model for the following probability estimation, which plays a critical role in the final result, has always been ignored in most of existing methods. In our method, all above limitations are taken full consideration. A kind of motion prediction method based on local coherence is adopted to separate the inseparable color. Then a self-adaptive extended sampling method is used to repair the video discontinuity caused by occlusion/disocclusion problem, and also we built the color model by sampling all the pixels from selected regions in order to make it clean and representative. Lastly, the final segmentation result is generated by using 3D Graph Cut algorithm according to the spatio-temporal coherence of the video. The experimental results are presented to demonstrate the effectiveness of the proposed method at achieving high quality results, as well as the robustness of the proposed method against several challenging test inputs.

Object segmentation and key-pose based summarization for motion video

This paper proposes a key-pose based video summarization system for a video shot facilitated by using a video object segmentation method. Firstly, we detect the camera motion and extract video objects by a 3D graph-based algorithm. Once the objects are obtained, each of them is represented by a shape descriptor. Secondly, in order to find representative frames which preserve scene content as much accurately as possible, the proposed method calculates difference between pairs of frames based on shape descriptors of objects in the video shot. Finally, key-poses (representative frames) are extracted in a global manner by clustering these shapes. Experimental results on motion video shots show that the proposed method outputs satisfactory summarizations.

Video-object segmentation and 3D-trajectory estimation for monocular video sequences

In this paper, we describe a video-object segmentation and 3D-trajectory estimation method for the analysis of dynamic scenes from a monocular uncalibrated view. Based on the color and motion information among video frames, our proposed method segments the scene, calibrates the camera, and calculates the 3D trajectories of moving objects. It can be employed for video-object segmentation, 2D-to-3D video conversion, video-object retrieval, etc. In our method, reliable 2D feature motions are established by comparing SIFT descriptors among successive frames, and image over-segmentation is achieved using a graph-based method. Then, the 2D motions and the segmentation result iteratively refine each other in a hierarchically structured framework to achieve video-object segmentation. Finally, the 3D trajectories of the segmented moving objects are estimated based on a local constant-velocity constraint, and are refined by a Hidden Markov Model (HMM)-based algorithm. Experiments show that the proposed framework can achieve a good performance in terms of both object segmentation and 3D-trajectory estimation.

Compressed Domain Video Object Segmentation

We present a compressed domain video object segmentation method for the MPEG encoded video sequences. For a fraction of the raw domain analysis, compressed domain segmentation provides the essential <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> information to many vision tasks from surveillance to transcoding that require fast processing of large volumes of data where pixel-resolution boundary extraction is not required. Our method generates accurate segmentation maps in block resolution at hierarchically varying object levels, which empowers application to determine the most pertinent partition of images. It exploits the block structure of the compressed video to minimize the amount of data to be processed. All the available motion flow within a group of pictures is projected onto a single layer, which also consists of the frequency decomposition of color pattern. Then, by starting from the blocks where the spatial energy is small, it expands homogeneous regions while automatically adapting local similarity criteria. We also formulate an alternative solution that applies a kernel-based clustering where separate spatial, transform, and motion kernels are used to establish the affinity. We show that both region expansion and mean shift produce similar results as the computationally expensive raw domain segmentation. Finally, a binary clustering iteratively merges the most similar regions to generate a hierarchical partition tree.

Spatiotemporal Region Enhancement and Merging for Unsupervized Object Segmentation

This paper proposes an unsupervized offline video object segmentation method that introduces a number of improvements to existing work in the area. It consists of the following steps. The initial segmentation utilizes object color and motion variance to more accurately classify image pixels in the first frame. Histogram-based merging is then employed to reduce oversegmentation of the first frame. During object tracking, segmentation quality measures based on object color and motion contrast are taken. These measures are then used to enhance video objects through selective pixel reclassification. After object enhancement, cumulative histogram-based merging, occlusion handling, and island detection are used to help group regions into meaningful objects. Compared to two reference methods, greater success and improved accuracy in segmenting video objects are first demonstrated by subjectively examining selected frames from a set of standard video sequences. Objective results are obtained through the use of a set of measures that aim at evaluating the accuracy of object boundaries and temporal stability through the use of color, motion, and histograms.

Robust Global Motion Estimation Oriented to Video Object Segmentation

Most global motion estimation (GME) methods are oriented to video coding while video object segmentation methods either assume no global motion (GM) or directly adopt a coding-oriented method to compensate for GM. This paper proposes a hierarchical differential GME method oriented to video object segmentation. A scheme which combines three-step search and motion parameters prediction is proposed for initial estimation to increase efficiency. A robust estimator that uses object information to reject outliers introduced by local motion is also proposed. For the first frame, when the object information is unavailable, a robust estimator is proposed which rejects outliers by examining their distribution in local neighborhoods of the error between the current and the motion-compensated previous frame. Subjective and objective results show that the proposed method is more robust, more oriented to video object segmentation, and faster than the referenced methods.

Multiresolution segmentation of video objects in the compression domain

This paper proposes a multiresolution method for video object segmentation in the compression domain. We first calculate global motion parameters using only background macroblocks with tiny residual dc coefficients of the P frame, and then get true motion vectors projected to the immediate adjoined I frame. The basic layer image is obtained with only dc coefficients of the I frame. The enhancement texture characteristics are provided by the ac coefficients for partial decoding. The true object motion vectors and the basic layer image are fed into a morphological motion filter to get the lowest-resolution regions of moving objects, called the layer4 region of interest (L4-ROI). Only some of the ac coefficients in L4-ROI are decoded to obtain a higher-resolution image, called layer3, that mainly consists of blocks of the moving object. The moving object of interest in the highest resolution is obtained from a morphological motion filter with L2-ROI and the true object motion vectors. The number of ac coefficients determines the resulting resolution. Experiments show that the new algorithm can extract multiresolution moving objects efficiently.

A Lvq-Based Temporal Tracking for Semi-Automatic Video Object Segmentation

This paper presents a Learning Vector Quantization (LVQ)-based temporal tracking method for semi-automatic video object segmentation. A semantic video object is initialized using user assistance in a reference frame to give initial classification of video object and its background regions. The LVQ training approximates video object and background classification and use them for automatic segmentation of the video object on the following frames thus performing temporal tracking. For LVQ training input, we sampling each pixel of a video frame as a 5-dimensional vector combining 2-dimensional pixel position (X,Y) and 3-dimensional HSV color space. This paper also demonstrates experiments using some MPEG-4 standard test video sequences to evaluate the accuracy of the proposed method.

Performance Measures for Video Object Segmentation and Tracking

We propose measures to evaluate quantitatively the performance of video object segmentation and tracking methods without ground-truth (GT) segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its predecessors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or they can be combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without GT have been demonstrated by canonical correlation analysis with another set of measures with GT on a set of sequences (where GT information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation approaches.

2-D mesh-based video object segmentation and tracking with occlusion resolution

This paper integrates fully automatic video object segmentation and tracking including detection and assignment of uncovered regions in a 2-D mesh-based framework. Particular contributions of this work are (i) a novel video object segmentation method that is posed as a constrained maximum contrast path search problem along the edges of a 2-D triangular mesh, and (ii) a 2-D mesh-based uncovered region detection method along the object boundary as well as within the object. At the first frame, an optimal number of feature points are selected as nodes of a 2-D content-based mesh. These points are classified as moving (foreground) and stationary nodes based on multi-frame node motion analysis, yielding a coarse estimate of the foreground object boundary. Color differences across triangles near the coarse boundary are employed for a maximum contrast path search along the edges of the 2-D mesh to refine the boundary of the video object. Next, we propagate the refined boundary to the subsequent frame by using motion vectors of the node points to form the coarse boundary at the next frame. We detect occluded regions by using motion-compensated frame differences and range filtered edge maps. The boundaries of detected uncovered regions are then refined by using the search procedure. These regions are either appended to the foreground object or tracked as new objects. The segmentation procedure is re-initialized when unreliable motion vectors exceed a certain number. The proposed scheme is demonstrated on several video sequences.