Background Sprite Research Articles

Semi-automatic 2D-to-3D video conversion based on background sprite generation

This paper presents a technique for semi-automatic 2D-to-3D stereo video conversion, which is known to provide user intervention in assigning foreground/background depths for key frames and then get depth maps for non-key frames via automatic depth propagation. Our algorithm treats foreground and background separately. For foregrounds, kernel pixels are identified and then used as the seeds for graph-cut segmentation for each non-key frame independently, resulting in results not limited by objects’ motion activity. For backgrounds, all video frames, after foregrounds being removed, are integrated into a common background sprite model (BSM) based on a relay-frame-based image registration algorithm. Users can then draw background depths for BSM in an integrated manner, thus reducing human efforts significantly. Experimental results show that our method is capable of retaining more faithful foreground depth boundaries (by 1.6–2.7 dB) and smoother background depths than prior works. This advantage is helpful for 3D display and 3D perception.

Key-Frame-Based Background Sprite Generation for Hole Filling in Depth Image-Based Rendering

In this paper, we propose a new depth image-based rending scheme for 3DTV applications, where a background sprite model is utilized for dis-occlusion/hole filling purpose. Dissimilar to traditional spatial (e.g., interpolation or inpainting) and temporal methods, our algorithm is capable of recovering the holes with true background information by incrementally integrating the spatial and temporal information of the video in a unified background sprite model. The technique of background sprite model construction in this paper is featured of resolving camera motions existing in most of the consumer videos, accurate registration of multiframe information, and efficient memory use and computation in realistic 3DTV application. To register/stitch each input frame to the background sprite model accurately, foreground removal considering color and depth information, and an adaptive key-frame-based scheme are developed for transform computation. Experimental results show that our proposed scheme has a large temporal reference distance and can retrieve true background information accurately, thus leading to better quality after novel view synthesis compared to existing spatial or spatio-temporal algorithms, especially for videos with significant camera motions or complex backgrounds.

SpriteCam: virtual camera control using sprite

Editing or browsing video based on spatial content without distortion or cropping has been challenging because of providing unavailable information (view) in a single frame. Virtual camera control enables the user to view the video from her perspective by using camera functions such as pan, tilt, and zoom on a recorded video. In this paper, we propose our SpriteCam system that provides virtual camera controls by first generating the background sprite or mosaic of the video. We provide the theoretical framework and then explain how pan, tilt, and zoom operations are applied using the sprite. SpriteCam allows centralizing objects, aspect ratio conversion, and fixing the camera view without distortion or information loss for videos which sprite can be generated.

Rate-Distortion Optimized Video Coding Using Automatic Sprites

Object-based video coding has been of interest for many years. Recent work by the authors has shown that a video coding system using background Sprites with automatic segmentation and background subtraction can indeed perform better than the H.264/AVC coder. In this paper, we extend this work by developing a rate-distortion optimization approach for an object-based coder. A key issue addressed by this approach is the joint choice of quantization parameters for the foreground and background. The performance of the resulting rate-distortion-optimized coder is far superior to that of H.264/AVC for source material with dominant global motion, across a range of bit rates.

Automatic MPEG-4 sprite coding—Comparison of integrated object segmentation algorithms

Sprite coding, as standardized in MPEG-4 Visual, can result in superior performance compared to common hybrid video codecs. We consider sprite coding, which significantly increases the objective as well as the subjective quality of coded video content. The main challenge of this approach is the segmentation of the foreground objects in a preprocessing step. We evaluate automatic object segmentation methods based on global motion estimation and background sprite generation. The objects are coded using the MPEG-4 Visual Main Profile and compared with the Advanced Simple Profile.

Enabling arbitrary rotational camera motion using multisprites with minimum coding cost

Object-oriented coding in the MPEG-4 standard enables the separate processing of foreground objects and the scene background (sprite). Since the background sprite only has to be sent once, transmission bandwidth can be saved. We have found that the counter-intuitive approach of splitting the background into several independent parts can reduce the overall amount of data. Furthermore, we show that in the general case, the synthesis of a single background sprite is even impossible and that the scene background must be sent as multiple sprites instead. For this reason, we propose an algorithm that provides an optimal partitioning of a video sequence into independent background sprites (a multisprite), resulting in a significant reduction of the involved coding cost. Additionally, our sprite-generation algorithm ensures that the sprite resolution is kept high enough to preserve all details of the input sequence, which is a problem especially during camera zoom-in operations. Even though our sprite generation algorithm creates multiple sprites instead of only a single background sprite, it is fully compatible with the existing MPEG-4 standard. The algorithm has been evaluated with several test sequences, including the well-known Table-tennis and Stefan sequences. The total coding cost for the sprite VOP is reduced by a factor of about 2.6 or even higher, depending on the sequence.

Content-based image retrieval in presence of foreground disturbances

The image retrieval problem in the presence of possible foreground disturbances is considered. The foreground may be irrelevant for the retrieval purposes but it occludes the background and hence reduces the retrieval accuracy. The accrued inaccuracy is quantified in terms of cardinality of the occluding region. The use of the sprite generated from a video clip is proposed as a query, so that the effect of moving foreground can be eliminated. The segmented foreground region is filled in by constructing the background sprite to increase the retrieval accuracy. The performance of a retrieval scheme under foreground disturbance is presented here.

SPRITE GENERATION FROM MPEG VIDEO USING MOTION INFORMATION

Sprite coding, accepted by the emerging MPEG-4 standard is a very efficient method for representing the background video object. Still this sprite generation is an open issue due to the foreground objects which obstructs the accuracy of camera motion estimation and blurs the generated sprite. In this paper we propose a method for constructing the background sprite with partial decoding of the MPEG stream. Initially the Independently Moving Objects (IMO) are segmented out from the background by clustering the pre-processed motion vectors of MPEG video. The camera motion parameters are obtained from the motion information corresponding to the background region which is used for frame alignment.

Very low bit rate video compression using MPEG‐4 sprite coding

AbstractThe new MPEG‐4 image coding standard that has recently been established offers higher image quality in low‐bit‐rate coding than conventional coding methods such as H.26X, MPEG‐1, and MPEG‐2, as well as coding for each object unit (a new function).This paper proposes an algorithm for the automatic extraction of video objects consisting of a foreground object and a background sprite. It is based on “sprite coding,” which can greatly reduce the number of code bits compared to conventional methods. The foreground and background video objects processed by the proposed method are subjected to MPEG‐4 object coding and sprite coding, respectively, and are compared with MPEG‐4 simple profile (normal) coding. In particular, the influence of the frame rate and the foreground object rate on the number of code bits was investigated. It was found that the proposed method can reproduce images with the same quality while using about one‐fourth to one‐half as many code bits as the normal coding method when the foreground rate is 10 to 15% of the whole image, regardless of the frame rate. In addition, the proposed method can attain more than double the frame rate of the normal coding method at a very low bit rate (e.g., 128 and 64 kbit/s) with a similar SNR. © 2003 Wiley Periodicals, Inc. Electron Comm Jpn Pt 2, 86(6): 83–93, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjb.10158

Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques

An efficient static sprite-generation and compression scheme is proposed for background video coding. First, considering both short- and long-term motion influences, a hybrid technique is proposed in the sprite generation to search the background motion relative to the generated sprite image. In order to prevent the sprite image from being blurred due to the inaccurate segmentation, a reliability-based blending technique is developed. Both the background video at every frame and the sprite image are categorized into reliable, unreliable, and undefined regions. Each region uses a unique strategy in blending. This makes the proposed technique work well even in the case of coarse segmentation. Second, an arbitrary-shape spatial prediction technique is proposed to efficiently compress the sprite image. Due to the property of the background video object, the sprite image often contains many flat and texture-correlation regions. Therefore, utilizing directional spatial prediction can significantly improve the efficiency of sprite coding. In general, the generated sprite is not a rectangular image. Since traditional spatial-prediction methods have difficulty in handling those contour blocks, a padding technique derived from rate-distortion optimization is proposed. Experimental results show that the proposed scheme can generate the background sprite image with better visual quality. Compared with the MPEG-4 sprite-coding scheme, it can also significantly improve the coding efficiency up to 3.0 dB at low bit rates.

Sprite generation and coding in multiview image sequences

A novel algorithm for the generation of background sprite images from multiview image sequences is presented. A dynamic programming algorithm, first proposed by Grammalidis and Strintzis (see IEEE Trans. Circuits Syst. Video Technol., vol.8, p.328-44, 1998) using a multiview matching cost, as well as pure geometrical constraints, is used to provide an estimate of the disparity field and to identify occluded areas. By combining motion, disparity, and occlusion information, a sprite image corresponding to the first (main) view at the first time instant is generated. Image pixels from other views that are occluded in the main view are also added to the sprite. Finally, the sprite coding method defined by MPEG-4 is extended for multiview image sequences based on the generated sprite. Experimental results are presented, demonstrating the performance of the proposed technique and comparing it with standard MPEG-4 coding methods applied independently to each view.