Video Object Plane Research Articles

Edge Segment-Based Automatic Video Surveillance

This paper presents a moving-object segmentation algorithm using edge information as segment. The proposed method is developed to address challenges due to variations in ambient lighting and background contents. We investigated the suitability of the proposed algorithm in comparison with the traditional-intensity-based as well as edge-pixel-based detection methods. In our method, edges are extracted from video frames and are represented as segments using an efficiently designed edge class. This representation helps to obtain the geometric information of edge in the case of edge matching and moving-object segmentation; and facilitates incorporating knowledge into edge segment during background modeling and motion tracking. An efficient approach for background initialization and robust method of edge matching is presented, to effectively reduce the risk of false alarm due to illumination change and camera motion while maintaining the high sensitivity to the presence of moving object. Detected moving edges are utilized along with watershed algorithm for extracting video object plane (VOP) with more accurate boundary. Experiment results with real image sequence reflect that the proposed method is suitable for automated video surveillance applications in various monitoring systems.

Region-of-interest based rate control for low-bit-rate video conferencing

We present a region of interest (ROI) based rate control for H.263 compatible video conferencing. A face detection and tracking scheme with very low complexity is proposed for segmentation. By analyzing quadratic rate models in frame layer, video object plane (VOP) layer, and macroblock (MB) layer extracted from the test data, a quadratic rate model at the MB layer with a modified physical meaning is proposed to improve the model accuracy. The basic idea is to use a group of uncoded MBs in the current VOP instead of individual MBs to update model parameters. A joint VOP layer and MB layer rate control algorithm is proposed. The VOP layer rate control assigns target bit rate for each VOP based on the coding complexity and visual importance, and determines an average quantization parameter (QP) for each VOP. Some new features of MB layer rate control are designed to utilize both average statistics of a VOP and individual statistics of MB together. The performance is compared with conventional TMN8 and object-based VM8 rate control, better peak SNR (PSNR) for ROI, and more accurate rate control can be achieved for various video sequences. The proposed rate control algorithm can be extended for H.264 ROI-based scalable video coding.

Extracting Objects Using Contour Evolutions in Edge-Based Object Tracking

We propose edge-based object extraction targeting automatic video object plane (VOP) generation in MPEG-4 content-based video coding. In an edge-based VOP generation framework proposed by Meier, the object is represented as a binary edge image that does not generally form a closed contour and that also contains many extra edges, making extracting the object contour accurately less straightforward in such situations. To solve this problem, we adopt a PDE-based contour evolution approach to evolve initial multiple contours contained inside the object toward its boundary based on evolution equations, and to finally merge them into a single contour that accurately represents the object’s shape. Our experimental results using an MPEG standard image sequence show that object contours obtained as we propose appear subjectively more natural in shape compared with those obtained by two conventional methods, especially when the binary object model is not in good condition.

An automatic segmentation method for moving objects based on the spatial-temporal information of video

The new MPEG-4 video coding standard enables content-based functions. In order to support the new standard, frames should be decomposed into Video Object Planes (VOP), each VOP representing a moving object. This paper proposes an image segmentation method to separate moving objects from image sequences. The proposed method utilizes the spatial-temporal information. Spatial segmentation is applied to divide each image into connected areas and to find precise object boundaries of moving objects. To locate moving objects in image sequences, two consecutive image frames in the temporal direction are examined and a hypothesis testing is performed with Neyman-Pearson criterion. Spatial segmentation produces a spatial segmentation mask, and temporal segmentation yields a change detection mask that indicates moving objects and the background. Then spatial-temporal merging can be used to get the final results. This method has been tested on several images. Experimental results show that this segmentation method is efficient.

Modified intelligent scissors and adaptive frame skipping for video object segmentation

MPEG-4 introduces the concept of video object to support content-based functionalities. Video object segmentation is a crucial step for object-based coding and manipulation. In this paper, a robust semi- automatic video object segmentation scheme is proposed. To efficiently and accurately define the initial object contour, modified intelligent scissors is proposed on the basis of original intelligent scissors. It can improve about 6–8 times the processing speed with only slight sacrifice of accuracy, which just meets the requirements of initial object extraction for semi-automatic approach. To avoid errors accumulating and propagating during object tracking, an adaptive frame skipping scheme is proposed to decompose video sequence into video clips. For rigid and non-rigid video objects, two different image segmentation algorithms are utilized, and then region-based backward projection technique is adopted to interpolate the video object plane (VOPs) of other frames within every video clip. The proposed approach can cope with occlusion/disocclusion problem to most extent. Experimental results demonstrate the effectiveness and robustness of the method.

Asynchronous rate control for multi-object videos

Object-based coding can potentially achieve a higher degree of compression and better visual quality. The objects in a scene are not always synchronous, that is, they have different temporal resolutions. In some applications, it is more efficient to transmit asynchronous objects with different temporal rates so as to achieve a better tradeoff between temporal and spatial resolutions. This requires to balance the qualities of individual objects while ensuring an overall visual quality with a given bit rate. This paper proposes a rate control algorithm for multiple video object encoding, which is suitable for both synchronous and asynchronous transmissions. The algorithm aims to maximize scene quality with an accurate bit rate, while efficiently handling buffer fullness. Using an efficient bit allocation strategy, the algorithm achieves accurate target bit rates, provides good coding quality, and decreases buffer overflow/underflow. The proposed algorithm also allows flexible priority adjustment among multiple objects to ensure overall better visual perception. Designed primarily for asynchronous objects, the algorithm treats the synchronous objects as a special case. Experimental results demonstrate that, when giving suitable asynchronous video object planes rates, the proposed algorithm achieves good temporal-spatial tradeoff while yielding accurate rate regulation and effective buffer control.

Semi-automatic modifications of video object trajectories for video compositing applications

This paper proposes a new approach allowing efficient and semi-automatic object-based manipulation for video editing or post-production applications, such as the modification of the trajectories of video objects in a natural scene. For that purpose, an image is seen as a superposition of video object planes (VOP) on an original image or a mosaic view of a fixed background. The method is based on the definition of an editing scenario which contains all the information required to create the edited sequence. A warping model is used to represent the location and the modification of the video object in the 2-D scene. Such an application is by nature semi-automatic since the operator has to specify the modifications that he wants to perform on the original sequence. The proposed method has been designed in order to reduce as much as possible the human workload by using the notion of key VOP and by calibrating in depth the fixed background area. Experimental results show that a high level of realism can be obtained on edited images in which the original trajectories of non-rigid video objects have been strongly modified.

Error concealment for shape in MPEG-4 object-based video coding

In asynchronus transfer mode networks, cell loss or channel errors can cause data to be dropped in the channel. When digital images/videos are transmitted over these networks, one must be able to reconstruct the missing data so that the impact of the errors is minimized. In this paper, we present an error-concealment technique for shape in MPEG-4 object-based video coding. This method, which is based on using global motion estimation and compensation techniques for boundary recovery, consists of three steps: (1) boundary extraction from shape; (2) boundary patching using global motion compensation; and (3) boundary filling to reconstruct the shape of the damaged video object planes. Global motion parameters are inserted as part of the USER_DATA field in the compressed stream and are utilized in reconstructing the damaged boundaries of compressed video object planes.

Intellectual property management on MPEG-4 video for hand-held device and mobile video streaming service

MPEG-4 video streaming service is being regarded as a de facto standard service for current mobile multimedia video streaming services such as video conferencing, voice over IP, and even state-of-the-art digital multimedia broadcasting technology. There have been many illegal users who have used copyrighted material without getting permission and/or paying for its use. Therefore, much attention has been paid for digital rights management (DRM) for digital media. We constructed a protection scheme for MPEG-4 video file format, which can be applied, to the video services that are exploiting the MPEG-4 standard. In this protection scheme, minimum segments of every video object plane in a MPEG-4 video file could be encrypted with a symmetric cryptography system like a DES (data encryption standard) so that people who have not received permission and/or paid to use the contents would not be able to view them. We applied this scheme to all kinds of MPEG-4 VOP (video object plane) types, that is I-, P-, and B-VOPs respectively. The prototype system was developed for proofing the feasibility of this scheme for hand-held devices and for mobile multimedia service environment. The efficiency of this scheme is reported at the end of this paper.

An efficient methodology for multimedia digital rights management on mobile handset

Wireless multimedia streaming applications have been widely propagated with the rapid evolution of mobile handsets and wireless Internet technologies, especially MPEG-4 video streams over wireless communication networks. We propose an efficient methodology for multimedia DRM which is applicable to the mobile terminal software. The methodology is comprised of the encryption methods on the media data that are encoded with MPEG-4 video format. We constructed 3 methods, one is the way we can encrypt macroblocks (MBs) of I-VOP (video object plane), another is P-VOP'S MBs encryption and the other is the combination of these two methods. They are designed to DRM services for a wireless video streaming system without performance degradation that could be a burden to a mobile handset, where there is inherently preferred system that costs for low-power, low-network bandwidth. We report competitiveness of these methods on the perspective of performance evaluation with respect to the plain MPEG-4 files.

Performance Measures for Video Object Segmentation and Tracking

We propose measures to evaluate quantitatively the performance of video object segmentation and tracking methods without ground-truth (GT) segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its predecessors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or they can be combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without GT have been demonstrated by canonical correlation analysis with another set of measures with GT on a set of sequences (where GT information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation approaches.

Automatic segmentation of video object planes in MPEG-4 based on spatio-temporal information

Segmentation of semantic Video Object Planes (VOP’s) from video sequence is a key to the standard MPEG-4 with content-based video coding. In this paper, the approach of automatic Segmentation of VOP’s Based on Spatio-Temporal Information (SBSTI) is proposed. The proceeding results demonstrate the good performance of the algorithm.

A fast automatic VOP generation using boundary block segmentation

With the increase of multimedia applications and content-based functionalities, efficient video coding methods are necessary. The moving picture experts group-4 (MPEG-4) provides content-based functionalities by introducing the concept of the video object plane (VOP). Each frame of the input sequence is segmented into a number of arbitrarily shaped image regions or VOP's so that each VOP describes a semantically meaningful object or video content of interest. For real-time applications of MPEG-4, a fast automatic object segmentation method of video sequences is needed. We propose a fast automatic VOP generation algorithm composed of two parts: object block segmentation and boundary block segmentation. The former defines block-based object regions in a frame and the latter generates the pixel-based object mask. Block-based object region search and restriction of segmentation regions to boundary blocks reduce a computational load. Experimental results with two test sequences show the effectiveness of the proposed algorithm in terms of the visual quality of segmentation results and the computation time.

A Wavelet-Based Sprite Codec

The International Standards Organization (ISO) has proposed a family of standards for compression of image and video sequences, including the JPEG, MPEG-1, and MPEG-2. The latest MPEG-4 standard has many new dimensions to coding and manipulation of visual content. A video sequence usually contains a background object and many foreground objects. Portions of this background may not be visible in certain frames due to the occlusion of the foreground objects or camera motion. MPEG-4 introduces the novel concepts of video object planes (VOPs) and Sprites. A VOP is a visual representation of real world objects with shapes that need not be rectangular. Sprite is a large image composed of pixels belonging to a video object visible throughout a video segment. Since a sprite contains all parts of the background that were at least visible once, it can be used for direct reconstruction of the background VOP. Sprite reconstruction is dependent on the mode in which it is transmitted. In the static sprite mode, the entire sprite is decoded as an Intra VOP before decoding the individual VOPs. Since sprites consist of the information needed to display multiple frames of a video sequence, they are typically much larger than a single frame of video. Therefore, a static sprite can be considered as a large static image. In this paper, a novel solution to address the problem of spatial scalability has been proposed, where the sprite is encoded in discrete wavelet transform (DWT). A lifting kernel method of DWT implementation has been used for encoding and decoding sprites. Modifying the existing lifting scheme while maintaining it to be shape-adaptive results in a reduced complexity. The proposed scheme has the advantages of: 1) avoiding the need for any extensions to image or tile border pixels and is hence superior to the discrete cosine transform-based low latency scheme (used in the current MPEG-4 verification model) and 2) mapping the in place computed wavelet coefficients into a zero-tree structure without actually rearranging them, thereby saving allocation of additional memory. The proposed solutions provide efficient implementation of the sprite decoder, making possible a VLSI realization with a reduced real estate.

Initial object segmentation for video object plane generation using cellular neural networks

MPEG-4 is a basic tool for interactivity and manipulation of video sequences. Video object segmentation is a key issue in defining the content of any video sequence, which is often divided into two steps: initial object segmentation and object tracking. In this paper, an initial object segmentation method for video object plane(VOP) generation using color information is proposed. Based on 3 by 3 linear templates, a cellular neural network (CNN) is used to implemented object segmentation. The Experimental results are presented to verify the efficiency and robustness of this approach.

Object-based video abstraction for video surveillance systems

Key frames are the subset of still images which best represent the content of a video sequence in an abstracted manner. In other words, video abstraction transforms an entire video clip to a small number of representative images. We present a scheme for object-based video abstraction facilitated by an efficient video-object segmentation (VOS) system. In such a framework, the concept of a "key frame" is replaced by that of a "key video-object plane (VOP)." In order to achieve an online object-based framework such as an object-based video surveillance system, it becomes essential that semantically meaningful video objects are directly accessed from video sequences. Moreover, the extraction of key VOPs needs to be automated and context dependent so that they maintain the important contents of the video while removing all redundancies. Once a VOP is extracted, the shape of the VOP needs to be well described. To this end, both region-based and contour-based shape descriptors are investigated, and the region-based descriptor is selected for the proposed system. The key VOPs are extracted in a sequential manner by successive comparison with the previously declared key VOP. Experimental results on the proposed online processing scheme combined with efficient VOS show the proposed integrated scheme generates desirable summarizations of surveillance videos.

Fast motion estimation of arbitrarily shaped video objects in MPEG-4

Block-based motion estimation is widely used for exploiting temporal redundancies in arbitrarily shaped video objects, which is computationally the most demanding part within the MPEG-4 standard. One of the main differences between MPEG-4 video and the previously standardized video coding schemes is the support of arbitrarily shaped video objects, for which most of the existing fast motion estimation algorithms are not suitable. In MPEG-4, motion estimation is performed on two kinds of macroblocks (MBs): the boundary MB and the opaque MB. Since the motion activities in opaque MBs are highly correlated with the neighbouring boundary MBs, a new priority search algorithm (PSA) for motion estimation is proposed in this paper, which performs motion estimation on all boundary MBs first in contrast to the conventional raster-scanning approach. This search strategy works well if the motion vectors in the boundary MBs truly represent the moving video object. Consequently, the full search algorithm is applied to the boundary MB in order to ensure its accuracy. However, the computational burden of motion estimation of the boundary MBs must be reduced. Fast search algorithms offered in the past tend to reduce the amount of computation by limiting the number of locations to be searched. Nearly all of these algorithms assume that the distortion function increases monotonically as the search location moves away from the global minimum. Unfortunately, this is usually not true in boundary MBs. We can reasonably assume that it is monotonic in a small neighbourhood around the global minimum. Consequently, one simple strategy, but perhaps the most efficient and reliable one, is to place the checking point as close as possible to the global minimum. In this paper, we also propose a fast search algorithm, which incorporates the binary alpha-plane to predict accurately the motion vectors of boundary MBs such that these motion vectors can be used in the PSA. Experimental results show that, when compared to conventional methods, our approach requires low computational complexity and provides significant improvement in terms of accuracy in motion-compensated video object planes.

Automatic segmentation of moving objects in video sequences: a region labeling approach

The emerging video coding standard MPEG-4 enables various content-based functionalities for multimedia applications. To support such functionalities, as well as to improve coding efficiency, MPEG-4 relies on a decomposition of each frame of an image sequence into video object planes (VOP). Each VOP corresponds to a single moving object in the scene. This paper presents a new method for automatic segmentation of moving objects in image sequences for VOP extraction. We formulate the problem as graph labeling over a region adjacency graph (RAG), based on motion information. The label field is modeled as a Markov random field (MRF). An initial spatial partition of each frame is obtained by a fast, floating-point based implementation of the watershed algorithm. The motion of each region is estimated by hierarchical region matching. To avoid inaccuracies in occlusion areas, a novel motion validation scheme is presented. A dynamic memory, based on object tracking, is incorporated into the segmentation process to maintain temporal coherence of the segmentation. Finally, a labeling is obtained by maximization of the a posteriori probability of the MRF using motion information, spatial information and the memory. The optimization is carried out by highest confidence first (HCF). Experimental results for several video sequences demonstrate the effectiveness of the proposed approach.

Fast and automatic video object segmentation and tracking for content-based applications

The new video-coding standard MPEG-4 enables content-based functionality, as well as high coding efficiency, by taking into account shape information of moving objects. A novel algorithm for segmentation of moving objects in video sequences and extraction of video object planes (VOPs) is proposed . For the case of multiple video objects in a scene, the extraction of a specific single video object (VO) based on connected components analysis and smoothness of VO displacement in successive frames is also discussed. Our algorithm begins with a robust double-edge map derived from the difference between two successive frames. After removing edge points which belong to the previous frame, the remaining edge map, moving edge (ME), is used to extract the VOP. The proposed algorithm is evaluated on an indoor sequence captured by a low-end camera as well as MPEG-4 test sequences and produces promising results.

Moving object segmentation in video sequences by user interaction and automatic object tracking

Abstract The new MPEG-4 video coding standard consists of object based coding schemes for multimedia and enables content based functionalities. In order to apply the MPEG-4 visual standard, each frame of video sequences should be represented in terms of video object planes (VOPs). Video objects in still pictures or video sequences should be first identified before the encoding process starts. It requires a prior decomposition of sequences into VOPs so that each VOP represents an object. This paper addresses a user-assisted video segmentation method for separating moving objects from the background in video sequences. The proposed method comprises of intra-frame and inter-frame segmentation modules. The intra-frame segmentation incorporates user interaction in defining a high level semantic object of interest to be segmented and detects precise object boundary. The inter-frame segmentation involves boundary and region tracking to capture temporal coherence of moving objects with accurate object boundary information. The proposed method exhibits a consistent and faithful segmentation performance for various types of video sequences.