Object-based Video Research Articles

Bi-directional 2-D mesh representation for video object rendering, editing and superresolution in the presence of occlusion

In this paper, we propose a new bi-directional 2-D mesh representation of video objects, which utilizes forward and backward reference frames (keyframes). This framework extends the previous uni-directional mesh representation to enable efficient rendering, editing, and superresolution of video objects in the presence of occlusion by allowing bi-directional texture mapping as in MPEG B-frames. The video object of interest is tracked between two successive keyframes (which can be automatically or interactively selected) both in forward and backward directions. Keyframes provide the texture of the video object, whereas its motion is modeled by forward and backward 2-D meshes. In addition, we employ “validity maps”, associated with each 2-D mesh, which allow selective texture mapping from the keyframes. Experimental results for efficient video object editing and object-based video resolution enhancement in the presence of self-occlusion are presented to demonstrate the effectiveness of the proposed representation.

Object-based video abstraction for video surveillance systems

Key frames are the subset of still images which best represent the content of a video sequence in an abstracted manner. In other words, video abstraction transforms an entire video clip to a small number of representative images. We present a scheme for object-based video abstraction facilitated by an efficient video-object segmentation (VOS) system. In such a framework, the concept of a "key frame" is replaced by that of a "key video-object plane (VOP)." In order to achieve an online object-based framework such as an object-based video surveillance system, it becomes essential that semantically meaningful video objects are directly accessed from video sequences. Moreover, the extraction of key VOPs needs to be automated and context dependent so that they maintain the important contents of the video while removing all redundancies. Once a VOP is extracted, the shape of the VOP needs to be well described. To this end, both region-based and contour-based shape descriptors are investigated, and the region-based descriptor is selected for the proposed system. The key VOPs are extracted in a sequential manner by successive comparison with the previously declared key VOP. Experimental results on the proposed online processing scheme combined with efficient VOS show the proposed integrated scheme generates desirable summarizations of surveillance videos.

Object-based video coding by global-to-local motion segmentation

We describe an object-based video compression scheme based on the derivation and efficient coding of motion boundaries. First, we recursively identify a small number of global movement classes, each represented by two or more motion parameters. Second, we assign regions of a spatial segmentation to movement classes. Third, we merge segments using various similarity heuristics, while adding movement classes for small objects, if necessary. Finally, the boundaries of motion are coded using an efficient asymmetric binary tree coding scheme. Experimental results on standard test sequences qualitatively show that the proposed algorithm gives good segmentation, and that it is suitable for very-low-data-rate object-based coding.

Object tracking using the Gabor wavelet transform and the golden section algorithm

The paper presents an object tracking method for object-based video processing which uses a two-dimensional (2D) Gabor wavelet transform (GWT) and a 2D golden section algorithm. An object in the current frame is modeled by local features from a number of the selected feature points, and the global placement of these feature points. The feature points are stochastically selected based on the energy of their GWT coefficients. Points with higher energy have a higher probability of being selected since they are visually more important. The amplitudes of the GWT coefficients of a feature point are then used as the local feature. The global placement of the feature points is determined by a 2D mesh whose feature is the area of the triangles formed by the feature points. In this way, a local feature is represented by a GWT coefficient amplitude vector, and a global feature is represented by a triangle area vector. One advantage of the 2D mesh is that the direction of its triangle area vector is invariant to affine transform. Consequently, the similarity between two local features or two global features can be defined as a function of the angle and the length ratio between two vectors, and the overall similarity between two objects is a weighted sum of the local and global similarities. In order to find the corresponding object in the next frame, the 2D golden section algorithm is employed. Our results show that the method is robust to object deformation and supports object tracking in noisy video sequences.

Semi-automatic object-based video segmentation with labeling of color segments

In this paper we propose a semi-automatic method for general object-based segmentation of image sequences. A label field is initialized by the user in the first frame of the sequence and then it is automatically tracked for the rest of the frames based on the color and motion properties of the various objects in the scene. We propose a novel statistical modeling which is based on the local objects’ properties. The locality introduced in our modeling allows the tracking of complex objects that, globally, might be inhomogeneous in their color and motion characteristics. The labeling criterion is the maximization of the joint probability of the labels and the observed color and motion properties. The proposed method utilizes an initial color-based segmentation which is obtained for each frame of the sequence. Both the modeling and the optimization are expressed in terms of the color-segments’ statistical properties. Experimental results are presented on real-image sequences that include complex human motion in cluttered backgrounds. r 2002 Elsevier Science B.V. All rights reserved.

A low power VLSI architecture for mesh-based video motion tracking

This paper proposes a low-power very large-scale integration (VLSI) architecture for motion tracking. It uses a hierarchical adaptive structured mesh that generates a content-based video representation. The proposed mesh is a coarse-to-fine hierarchical two-dimensional mesh that is formed by recursive triangulation of the initial coarse mesh geometry. The structured mesh offers a significant reduction in the number of bits that describe the mesh topology. The motion of the mesh nodes represents the deformation of the video object. The architecture consists of motion estimation and motion compensation units. The motion estimation architecture generates a progressive mesh code and the motion vectors of the mesh nodes. It reduces the power consumption, uses a simpler approach for mesh construction, approximates the mesh nodes motion vector by using the three step search algorithm and uses a parallel motion estimation core to evaluate the mesh nodes motion vectors. Moreover, it maximizes the lifetime of the internal buffers. The motion compensation architecture uses a multiplication-free algorithm for affine transformation, which significantly reduces the complexity of the motion compensation architecture. Moreover, using pipelined affine units contributes to the power savings. The video motion compensation architecture processes a reference frame, mesh nodes and motion vectors to predict a video frame. It implements parallel threads in which each thread implements a pipelined chain of scalable affine units. This motion compensation algorithm allows the use of one simple warping unit to map a hierarchical structure. The affine unit warps the texture of a patch at any level of hierarchical mesh independently. The processor uses a memory serialization unit, which interfaces the memory to the parallel units. The architecture has been prototyped using top-down low-power design methodology. The performance analysis shows that this processor can be used in online object-based video applications such as in MPEG and VRML.

Algorithm-based low-power VLSI architecture for 2D mesh video-object motion tracking

The new VLSI architecture for video object (VO) motion tracking uses a novel hierarchical adaptive structured mesh topology. The structured mesh offers a significant reduction in the number of bits that describe the mesh topology. The motion of the mesh nodes represents the deformation of the VO. Motion compensation is performed using a multiplication-free algorithm for affine transformation, significantly reducing the decoder architecture complexity. Pipelining the affine unit contributes a considerable power saving. The VO motion-tracking architecture is based on a new algorithm. It consists of two main parts: a video object motion-estimation unit (VOME) and a video object motion-compensation unit (VOMC). The VOME processes two consequent frames to generate a hierarchical adaptive structured mesh and the motion vectors of the mesh nodes. It implements parallel block matching motion-estimation units to optimize the latency. The VOMC processes a reference frame, mesh nodes and motion vectors to predict a video frame. It implements parallel threads in which each thread implements a pipelined chain of scalable affine units. This motion-compensation algorithm allows the use of one simple warping unit to map a hierarchical structure. The affine unit warps the texture of a patch at any level of hierarchical mesh independently. The processor uses a memory serialization unit, which interfaces the memory to the parallel units. The architecture has been prototyped using top-down low-power design methodology. Performance analysis shows that this processor can be used in online object-based video applications such as MPEG-4 and VRML.

Localizing and segmenting text in images and videos

Many images, especially those used for page design on Web pages, as well as videos contain visible text. If these text occurrences could be detected, segmented, and recognized automatically, they would be a valuable source of high-level semantics for indexing and retrieval. We propose a novel method for localizing and segmenting text in complex images and videos. Text lines are identified by using a complex-valued multilayer feed-forward network trained to detect text at a fixed scale and position. The network's output at all scales and positions is integrated into a single text-saliency map, serving as a starting point for candidate text lines. In the case of video, these candidate text lines are refined by exploiting the temporal redundancy of text in video. Localized text lines are then scaled to a fixed height of 100 pixels and segmented into a binary image with black characters on white background. For videos, temporal redundancy is exploited to improve segmentation performance. Input images and videos can be of any size due to a true multiresolution approach. Moreover, the system is not only able to locate and segment text occurrences into large binary images, but is also able to track each text line with sub-pixel accuracy over the entire occurrence in a video, so that one text bitmap is created for all instances of that text line. Therefore, our text segmentation results can also be used for object-based video encoding such as that enabled by MPEG-4.

サッカー番組における動作カテゴリー化処理を伴う動画検索手法

Focusing on soccer programs, we have developed a method of retrieving scenes by categorizing the motion of important objects like a goalkeeper. It is designed for an object-based video database system that manages information about video objects in video sequences. The scene retrieval consists of two steps for the comparison process : (I) categorizing the motion and (II) calculating the similarity in distance to a reference video example of the same category. Motion categorization is carried out by probabilistically learning the sequences of motion parameters for a goalkeeper. We have implemented this method in an object relational database system containing various soccer scenes and verified it.

Mathematical formulation of shape-adaptive 2-D transforms

This paper presents a mathematical formulation of shape-adaptive (SA) two-dimensional (2-D) transforms. A generic SA transform is represented by a concatenation of three linear operators that characterize the one-dimensional (1-D) vertical transforms, the alignment of vertical coefficients, and the 1-D horizontal transforms. We state and prove several powerful properties of these operators. The mathematical formulation described may serve as an important theoretical basis for future developments in the area of object-based video coding.

User-aware object-based video transmission over the next generation Internet

In this paper, we present a new user-aware adaptive object-based video transmission approach to heterogeneous users over the next generation Internet. Firstly, we describe a new transport framework for complex multimedia applications over the next generation Internet, which provides differentiation functionality within one IP session as well as among different IP sessions. It includes application-aware intelligent resource control at the edge of the network, fast transcoding and signaling in the network. Secondly, we propose a new bitstream classification , prioritization and packetization scheme in which different types of data such as shape, motion and texture are reassembled, assigned to different priority classes, and packetized separately based on their priorities. Thirdly, we present a simple but effective mechanism of object-based dynamic rate control and adaptation by selectively dropping packets in conjunction with differentiated services (Diffserv) to minimize the end-to-end quality distortion. Finally, we perform the queuing analysis for our mechanism and explore how to extend our approach to the multicast case. Experimental results demonstrate effectiveness of our proposed approach.

Object-based transcoding for adaptable video content delivery

This paper introduces a new framework for video content delivery that is based on the transcoding of multiple video objects. Generally speaking, transcoding can be defined as the manipulation or conversion of data into another more desirable format. We consider manipulations of object-based video content, and more specifically, from one set of bit streams to another. Given the object-based framework, we present a set of new algorithms that are responsible for manipulating the original set of video bit streams. Depending on the particular strategy that is adopted, the transcoder attempts to satisfy network conditions or user requirements in various ways. One of the main contributions of this paper is to discuss the degrees of freedom within an object-based transcoder and demonstrate the flexibility that it has in adapting the content. Two approaches are considered: a dynamic programming approach and an approach that is based on available meta-data. Simulations with these two approaches provide insight regarding the bit allocation among objects and illustrates the tradeoffs that can be made in adapting the content. When certain meta-data about the content is available, we show that bit allocation can be significantly improved, key objects can be identified, and varying the temporal resolution of objects can be considered.

Image restoration for frame- and object-based video coding using an adaptive constrained least-squares approach

Especially at low bit-rates current DCT-based video coding standards suffer from the disadvantage that coarse quantization and a rigid block structure result in noticeable blocking and ringing noise. In this paper we propose a spatially adaptive method for reduction of these coding artifacts based on the principle of constrained least-squares image restoration. A strictly local filter is developed which adapts to the spatial image characteristics as well as to the coding conditions. Due to the small filter kernel, the method can be applied to frame as well as object-based video coding and is also suited for quality improvement in arbitrarily shaped MPEG-4 coded video material. The proposal is numerically scalable and yields visually pleasing results for intra- as well as interframe coded images. This is also reflected by consistent PSNR improvements between 0.2 and 1.3 dB. As post-processing technique, it is compatible to all existing image and video coding standards.

Automatic key video object plane selection using the shape information in the MPEG-4 compressed domain

Object-based video representation, such as the one suggested by the MPEG-4 standard, offers a framework that is better suited for object-based video indexing and retrieval. In such a framework, the concept of a frame is replaced by that of a video object plane. In this paper, we propose a method for key video object plane selection using the shape information in the MPEG-4 compressed domain. The shape of the video object (VO) is approximated using the shape coding modes of I, P, and B video object planes (VOPs) without decoding the shape information in the MPEG-4 bit stream. Two popular shape distance measures, the Hamming and Hausdorff distance measures, are modified to measure the similarities between the approximated shapes of the video objects. Although they feature different computational and implementation complexity tradeoffs, the corresponding algorithms achieve essentially the same performance levels in selecting key video object planes that represent efficiently the salient content of the video objects.

A contour-based approach to binary shape coding using a multiple grid chain code

This paper presents a contour-based approach to efficiently code binary shape information in the context of object-based video coding. This approach meets some of the most important requirements identified for the MPEG-4 standard, notably efficient coding and low delay. The proposed methods support both object-based lossless and quasi-lossless coding modes. For the cases where low delay is a primary requirement, a macroblock-based coding mode is proposed which can take advantage of inter-frame coding to improve the coding efficiency. The approach presented here relies on a grid different from that used for the pixels to represent the shape – the hexagonal grid – which simplifies the task of contour coding. Using this grid, an appraoch based on a differential chain code (DCC) is proposed for the lossless mode while, for the quasi-lossless case, an approach based on the multiple grid chain code (MGCC) principle is proposed. The MGCC combines both contour simplification and contour prediction to reduce the number of bits needed to code the shapes. Results for alpha plane coding of MPEG-4 video test sequences are presented in order to illustrate the performance of the several modes of operation, and a comparison is made with the shape-coding tool chosen by MPEG-4.

Shape compression of moving objects using context-based arithmetic encoding

This paper deals with an approach to the compression of 2-D shape information. Specifically, the approach utilises a block-based context-based arithmetic encoding method to exploit both spatial and temporal redundancy in sequences of shapes. The method is shown to be highly efficient in comparative tests. In addition, it is demonstrated that context-based arithmetic encoding is easily extended for the coding of interlaced shape sources and for encoding any source in a scalable and/or error resilient manner. The application of the technique is in the compression of video object shapes within the framework of object-based video coding. The proposed algorithm is part of the MPEG-4 video specification.

Optimal 2-D hierarchical content-based mesh design and update for object-based video

Representation of video objects (VOs) using hierarchical 2-D content-based meshes for accurate tracking and level of detail (LOD) rendering have been previously proposed, where a simple suboptimal hierarchical mesh design algorithm was employed. However, it was concluded that the performance of the tracking and rendering very much depends on how well each level of the hierarchical mesh structure fits the VO under consideration. To this effect, this paper proposes an optimized design of hierarchical 2-D content-based meshes with a shape-adaptive simplification and a temporal update mechanism for object-based video. Particular contributions of this work are: (1) analysis of optimal number of nodes for the initial fine level-of-detail mesh design; (2) adaptive shape simplification across hierarchy levels; (3) optimization of the interior-node decimation method to remove only a maximal independent set to preserve Delaunay topology across hierarchy levels for better bitrate versus quality performance; and (4) a mesh-update mechanism which serves to update a temporally 2-D dynamic mesh in case of occlusion due to 3-D motion and self-occlusion. The proposed optimized and temporally updated hierarchical mesh representation can be applied in object-based video coding, retrieval, and manipulation.

Coding tools in MPEG-4 for interlaced video

Previous developments in digital video compression, transmission, and displays have made object-based video viable for many applications, e.g., coding chroma-keyed video for digital TV and manipulating video objects on interactive multimedia terminals, etc. To facilitate these applications, there is a demand on international standards for coding methods and transmission formats for object-based natural and synthetic video. For the past few years, the Moving Picture Experts Group (MPEG) of the International Standards Organization (ISO), which successfully created the MPEG-1/2 standards, has been working to establish a new standard, called MPEG-4. MPEG-4 will provide standardized technological elements enabling the integration of the production, distribution, and content-access paradigms in four fields: wireless communication, digital TV, interactive graphics, and the World Wide Web. To meet the needs of interlaced video applications, MPEG-4 video adopted interlaced coding tools similar to those in MPEG-2 and features schemes to code multiple video objects. This paper provides an overview of MPEG-4 interlaced coding tools, and focuses in detail on the new shape and texture-coding algorithms for interlaced video.

Motion-based region growing segmentation of image sequences

This paper proposes a motion-based region growing segmentation scheme for the object-based video coding, which segments an image into homogeneous regions characterized by a coherent motion. It adopts a block matching algorithm to estimate motion vectors and uses morphological tools such as open-close by reconstruction and the region-growing version of the watershed algorithm for spatial segmentation to improve the temporal segmentation. In order to determine the reliable motion vectors, this paper also proposes a change detection algorithm and a multi-candidate pre- screening motion estimation method. Preliminary simulation results demonstrate that the proposed scheme is feasible. The main advantage of the scheme is its low computational load.

Introduction to the special issue on object-based video coding and description

Introduction to the special issue on object-based video coding and description