Video Object Plane Research Articles

Spatiotemporal segmentation for compact video representation

In this paper, a novel hierarchical object-oriented video segmentation and representation algorithm is proposed. The local variance contrast and the frame difference contrast are jointly exploited for structural spatiotemporal video segmentation because these two visual features can indicate the spatial homogeneity of the grey levels and the temporal coherence of the motion fields efficiently, where the two-dimensional (2D) spatiotemporal entropic technique is further selected for generating the 2D thresholding vectors adaptively according to the variations of the video components. After the region growing and edge simplification procedures, the accurate boundaries among the different video components are further exploited by an intra-block edge extraction procedure. Moreover, the relationships of the video components among frames are exploited by a temporal tracking procedure. This proposed object-oriented spatiotemporal video segmentation algorithm may be useful for MPEG-4 system generating the video object plane (VOP) automatically.

Power and Speed-Efficient Code Transformation of Video Compression Algorithms for RISC Processors

Upcoming multi-media compression applications will require high memory bandwidth. In this paper, we estimate that a software reference implementation of an MPEG-4 video decoder typically requires 200 Mtransfers/s to memory to decode 1 CIF (352×288) Video Object Plane (VOP) at 30 frames/s. This imposes a high penalty in terms of power but also performance. However, we also show that we can heavily improve on the memory transfers, without sacrificing speed (even gaining about 10% on cache misses and cycles for a DEC Alpha), by aggressive code transformations. For this purpose, we have manually applied an extended version of our data transfer and storage exploration (DTSE) methodology, which was originally developed for custom hardware implementations.

Automatic key video object plane selection using the shape information in the MPEG-4 compressed domain

Object-based video representation, such as the one suggested by the MPEG-4 standard, offers a framework that is better suited for object-based video indexing and retrieval. In such a framework, the concept of a frame is replaced by that of a video object plane. In this paper, we propose a method for key video object plane selection using the shape information in the MPEG-4 compressed domain. The shape of the video object (VO) is approximated using the shape coding modes of I, P, and B video object planes (VOPs) without decoding the shape information in the MPEG-4 bit stream. Two popular shape distance measures, the Hamming and Hausdorff distance measures, are modified to measure the similarities between the approximated shapes of the video objects. Although they feature different computational and implementation complexity tradeoffs, the corresponding algorithms achieve essentially the same performance levels in selecting key video object planes that represent efficiently the salient content of the video objects.

Automatic video sequence segmentation using object tracking

Emerging video standards address content-based functionalities and interactive multimedia. MPEG-4 segments the frames into video object planes (VOP) with each VOP corresponding to one semantically meaningful object. In this paper a new automatic video sequence segmentation algorithm is presented that can extract such VOPs. A binary model for moving objects is automatically derived and tracked in subsequent frames using the generalised Hausdorff distance. The model is updated every frame to accommodate for rotation and changes in shape. For that an update mechanism consisting of two components is proposed. To improve the segmentation and reduce the computational complexity the stationary background is filtered by a novel technique. Experimental results demonstrate the performance of our proposed algorithm.

Incomplete 3-D multiview representation of video objects

This paper introduces a new form of representation for three-dimensional (3-D) video objects. We have developed a technique to extract disparity and texture data from video objects that are captured simultaneously with multiple-camera configurations. For this purpose, we derive an "area of interest" (AOI) for each of the camera views, which represents an area on the video object's surface that is best visible from this specific camera viewpoint. By combining all AOIs, we obtain the video object plane as an unwrapped surface of a 3-D object, containing all texture data visible from any of the cameras. This texture surface can be encoded like any 2-D video object plane, while the 3-D information is contained in the associated disparity map. It is then possible to reconstruct different viewpoints from the texture surface by simple disparity-based projection. The merits of the technique are efficient multiview encoding of single video objects and support for viewpoint adaptation functionality, which is desirable in mixing natural and synthetic images. We have performed experiments with the MPEG-4 video verification model, where the disparity map is encoded by use of the tools provided for grayscale alpha data encoding. Due to its simplicity, the technique is suitable for applications that require real-time viewpoint adaptation toward video objects.

A VOP generation tool: automatic segmentation of moving objects in image sequences based on spatio-temporal information

The new MPEG-4 video coding standard enables content-based functionalities. In order to support the philosophy of the MPEG-4 visual standard, each frame of video sequences should be represented in terms of video object planes (VOPs). In other words, video objects to be encoded in still pictures or video sequences should be prepared before the encoding process starts. Therefore, it requires a prior decomposition of sequences into VOPs so that each VOP represents a moving object. This paper addresses an image segmentation method for separating moving objects from the background in image sequences. The proposed method utilizes the following spatio-temporal information. (1) For localization of moving objects in the image sequence, two consecutive image frames in the temporal direction are examined and a hypothesis testing is performed by comparing two variance estimates from two consecutive difference images, which results in an F-test. (2) Spatial segmentation is performed to divide each image into semantic regions and to find precise object boundaries of the moving objects. The temporal segmentation yields a change detection mask that indicates moving areas (foreground) and nonmoving areas (background), and spatial segmentation produces spatial segmentation masks. A combination of the spatial and temporal segmentation masks produces VOPs faithfully. This paper presents various experimental results.

Boundary block-merging (BBM) technique for efficient texture coding of arbitrarily shaped object

We present an efficient texture coding method which enhances the coding efficiency of conventional discrete cosine transform (DCT) with padding techniques for arbitrarily shaped objects in object-based video coding where shape information is provided. The BBM (boundary block-merging) technique is applied to the boundary macroblocks of 16/spl times/16 pixels of a VOP (video object plane) which consist of both background and object pixels. A macroblock consists of four subblocks of 8/spl times/8 pixels. For boundary subblocks consisting of object and background pixels, padding is performed in the background region. For a pair of padded boundary subblocks in a macroblock of which alignment belongs to a predefined set, one subblock is rotated 180/spl deg/ and merged into another one if object pixels do not overlap. After merging, the boundary macroblock is coded using the conventional DCT coding. The merging process reduces the number of subblocks to be DCT coded, and high correlation between adjacent subblocks makes the number of DCT coding bits small. Experimentation has been done on various test sequences under different test conditions, and verifies significant coding efficiency improvement: reduction of coding bits for luminance boundary blocks by 5.7-11.9% at the same PSNR values compared with the padding-based DCT without BBM.

Segmentation and tracking of moving objects for content-based video coding

To enable content-based functionalities in video coding, a decomposition of the scene into physical objects is required. Such objects are normally not characterised by homogeneous colour, intensity, or optical flow. Therefore, conventional techniques based on these low-level features cannot perform the desired segmentation. The authors address segmentation and tracking of moving objects and presents a new video object plane (VOP) segmentation algorithm that extracts semantically meaningful objects. A morphological motion filter detects physical objects by identifying areas that are moving differently from the background. A new filter criterion is introduced that measures the deviation of the estimated local motion from the synthesised global motion. A two-dimensional binary model is derived for the object of interest and tracked throughout the sequence by a Hausdorff object tracker. To accommodate for rotations and changes in shape, the model is updated every frame by a two-stage method that accounts for rigid and non-rigid moving parts of the object. The binary model then guides the actual VOP extraction, whereby a novel boundary post-processor ensures high boundary accuracy. Experimental results demonstrate the performance of the proposed algorithm.

Video segmentation for content-based coding

To provide multimedia applications with new functionalities, the new video coding standard MPEG-4 relies on a content-based representation. This requires a prior decomposition of sequences into semantically meaningful, physical objects. We formulate this problem as one of separating foreground objects from the background based on motion information. For the object of interest, a 2D binary model is derived and tracked throughout the sequence. The model points consist of edge pixels detected by the Canny operator. To accommodate rotation and changes in shape of the tracked object, the model is updated every frame. These binary models then guide the actual video object plane (VOP) extraction. Thanks to our new boundary postprocessor and the excellent edge localization properties of the Canny operator, the resulting VOP contours are very accurate. Both the model initialization and update stages exploit motion information. The main assumption underlying our approach is the existence of a dominant global motion that can be assigned to the background. Areas that do not follow this background motion indicate the presence of independently moving physical objects. Two alternative methods to identify such objects are presented. The first one employs a morphological motion filter with a new filter criterion, which measures the deviation of the locally estimated optical flow from the corresponding global motion. The second method computes a change detection mask by taking the difference between consecutive frames. The first version is more suitable for sequences with little motion, whereas the second version is better at dealing with faster moving or changing objects. Experimental results demonstrate the performance of our algorithm.

Motion-based analysis and segmentation of image sequences using 3-D scene models

In this paper we present an algorithm for automatic extraction and tracking of multiple objects from a video sequence. Our approach is model-based in the sense that we first use a robust structure-from-motion algorithm to identify multiple objects and to recover initial 3-D-shape models. Then, these models are used to identify and track the objects over multiple frames of the video sequence. The procedure starts with recovering a dense depth map of the scene using two frames at the beginning of the sequence, and representing the scene as a 3-D wire frame computed from the depth map. Texture extracted from the video frames is mapped onto the model. Once the initial models are available we use a linear and low-complexity algorithm to recover the motion parameters and scene structure of the objects for the subsequent frames. Combining the new estimates of depth and the initially computed 3-D models into an unstructured set of 3-D points with associated color information, we obtain updates of the 3-D scene description for each additional frame. We show that the usage of a 3-D scene model is suitable to analyze complex scenes with several objects. In our experimental results, we apply the approach presented in this paper to the problem of video sequence segmentation, object tracking, and video object plane (VOP) generation. We separate the video sequences into different layers of depth and combine the information from multiple frames to a compact and complete description of these layers.

Prediction of image partitions using Fourier descriptors: application to segmentation-based coding schemes

This paper presents a prediction technique for partition sequences. It uses a region-by-region approach that consists of four steps: region parameterization, region prediction, region ordering, and partition creation. The time evolution of each region is divided into two types: regular motion and shape deformation. Both types of evolution are parameterized by means of the Fourier descriptors and they are separately predicted in the Fourier domain. The final predicted partition is built from the ordered combination of the predicted regions, using morphological tools. With this prediction technique, two different applications are addressed in the context of segmentation-based coding approaches. Noncausal partition prediction is applied to partition interpolation, and examples using complete partitions are presented. In turn, causal partition prediction is applied to partition extrapolation for coding purposes, and examples using complete partitions as well as sequences of binary images--shape information in video object planes (VOPs)--are presented.

Automatic segmentation of moving objects for video object plane generation

The new video coding standard MPEG-4 is enabling content-based functionalities. It takes advantage of a prior decomposition of sequences into video object planes (VOPs) so that each VOP represents one moving object. A comprehensive review summarizes some of the most important motion segmentation and VOP generation techniques that have been proposed. Then, a new automatic video sequence segmentation algorithm that extracts moving objects is presented. The core of this algorithm is an object tracker that matches a two-dimensional (2-D) binary model of the object against subsequent frames using the Hausdorff distance. The best match found indicates the translation the object has undergone, and the model is updated every frame to accommodate for rotation and changes in shape. The initial model is derived automatically, and a new model update method based on the concept of moving connected components allows for comparatively large changes in shape. The proposed algorithm is improved by a filtering technique that removes stationary background. Finally, the binary model sequence guides the extraction objects of the VOPs from the sequence. Experimental results demonstrate the performance of our algorithm.

Triangular active mesh for motion estimation

Abstract This paper introduces a motion estimation tool based on triangular active mesh. This tool can be used to model the deformation of various kinds of objects, especially frames and arbitrarily shaped regions known as video object planes (VOPs) in the MPEG-4 context. In the latter case, a polygon approximation of the region is performed in order to define border nodes and to triangulate the whole considered domain. Object motion is represented by a piecewise affine transformation whose coefficients are estimated by means of motion estimation of triangle vertices. Within the context of very low bit-rate coding, this tool appears to be useful for image prediction, temporal interpolation and may achieve the optimal sharing between motion and error information encoding. Simulations show that the proposed motion estimation tool provides higher PSNR than the classical block matching algorithm.