MPEG-2 Bitstream Research Articles

MPEG-Encoded Light Fields

Light fields (or Lumigraphs) are an image-based rendering method. As light fields require a large number of images, memory and disk space requirements are a significant barrier to the practical use of light fields. In this paper, we explain how to greatly reduce the space demands of light fields by encoding them in an MPEG format. The encoding is comprised of an MPEG bitstream and two tables of precomputed information to aid real-time decoding. By restricting the parameters used for MPEG encoding, and by employing a caching system, new views of the compressed light field can be rendered at fast interactive frame rates on consumer hardware. Compression ratios of around 50:1 are demonstrated. Combined, the speed and compression make the use of light fields practical on consumer hardware.

Soft source decoding with applications

Motivated by previous results in joint source-channel coding and decoding, we consider the problem of decoding of variable-length codes using soft channel values. We present results of decoding of selected codes using the maximum a posteriori (MAP) decoder and the sequential decoder, and show the performance gains over decoding using hard decisions alone. The objective behind this work is to provide motivation for decoding of data compressed by standard source coding schemes, that is, to view the compressed bitstreams as being the output of variable-length coders and to make use of the redundancy in the bitstreams to assist in decoding. In order to illustrate the performance achievable by soft decoding, we provide results for decoding of MPEG-4 reversible variable-length codes as well as for decoding of MPEG-4 overhead information, under the assumption that this information is transmitted without channel coding over an additive white Gaussian noise channel. Finally, we present a method of unequal error protection for an MPEG-4 bitstream using the MAP and sequential source decoders, and show results comparable to those achievable by serial application of source and channel coding.

Encoding and reconstruction of incomplete 3-D video objects

A new approach for compact representation, MPEG-4 encoding, and reconstruction of video objects captured by an uncalibrated system of multiple cameras is presented. The method is based on the incomplete 3-D (I3D) technique, which was initially investigated for stereo video objects captured by parallel cameras. Non-overlapping portions of the object are extracted from the reference views, each view having the corresponding portion with the highest resolution. This way, the redundancy of the initial multiview data is reduced. The areas which are extracted from the basis views are denoted as areas of interest. The output of the analysis stage, i.e., the areas of interest and the corresponding parts of the disparity fields are encoded in the MPEG-4 bitstream. Disparity fields define the correspondence relations between the reference views. The view synthesis is performed by disparity-oriented reprojection of the areas of interest into the virtual view plane and can be seen as an intermediate postprocessing stage between the decoder and the scene compositor. This work performs an extension from parallel stereo views to arbitrary configured multi-views with new analysis and synthesis algorithms. Moreover, a two-way interaction is built between the analysis and reconstruction stages, which provides the tradeoff between the final image quality and amount of data transmitted. The focus is on a low-complexity solution enabling online processing capability while preserving the MPEG-4 compatibility of the I3D representation. It is finally shown that our method yields quite convincing results despite the minimal data used and the approximations involved.

Backward compatible user-defined data insertion into MPEG-4 bitstreams

It is often useful to provide information related to a video bitstream that is unsupported by the official standard. A method is described by which data can be inserted in an MPEG-4 bitstream , while still retaining compatibility with standard MPEG-4 decoders. Its effectiveness is demonstrated by results from insertion of cyclic redundancy check (CRC) codes.

Analysis of a DVB-T compliant receiver simulation under various multipath conditions

Since 1966, coded orthogonal frequency division multiplexing (COFDM) has been investigated to determine the possibility of reducing the overall throughput of a digitally modulated terrestrial television channel. In the investigations, many assumptions have emerged. One common misconception is that in a terrestrial environment, COFDM has an inherent immunity to multipath interference. A theoretical analysis of a multipath channel, along with simulation results has shown that this assumption does not hold the information is considered when including the radio frequency modulation and demodulation. This paper presents a background into the inception of COFDM, a mathematical analysis of the digitally modulated television signal under multipath conditions and the results of a European Digital Video Broadcasting-Terrestrial (DVB-T) compliant simulation model with MPEG-2 bitstreams transmitted under various multipath conditions.

Fast Transcoding of Compressed Bitstreams by Reusing Incoming Motion Vectors

Transcoding will be an important activity in the digital studio. People will transcode MPEG-2 bitstreams to resize images, support progressive or interlaced receivers, optimize for editability or storage/transmission efficiency, etc. Transcoding involves decoding a bitstream, possibly modifying the decoded pictures, and re-encoding. Transcoding is expensive because encoding is expensive, and the major cost of encoding is motion estimation (ME). A program, Motion Vector Reuse by Extrapolation (MVREX), has been developed that reuses incoming motion vectors during transcoding. MVREX assumes that motion is linear, and extrapolates known motion from input macroblocks to output macroblocks (MBs), avoiding ME. MVREX has transcoded both progressive and interlaced video between different group of picture (GOP) structures, and supports all MPEG-2 prediction modes for frame pictures, except dual prime. In the cases examined, MVREX produces video with peak signal-to-noise ratio (PSNR) within 1 dB of that produced by a full search, but with orders of magnitude less computation. MVREX continues to work well after multiple generations of transcoding and appears to be a promising approach to transcoding. After a brief review of MPEG-2, this paper describes the concept of motion extrapolation, derives basic formulas, and presents several results and observations.

Shot change detection from partially decoded MPEG data

In this paper, we propose a scene decomposition algorithm from MPEG compressed video data. As preprocessing for scene decomposition, partial reconstruction methods of DC images for P- and B-pictures as well as I-pictures directly from an MPEG bitstream are used. As for detection algorithms, we have exploited several methods for detection of abrupt shot changes, dissolves, and wipe transitions using comparison of DC images between frames and coding information such as motion vectors. A method is also proposed for exclusion of undesired detection such as flashlight scenes in order to enhance shot change detection accuracy. More than 95% decomposition accuracy has been obtained in an experiment using a TV program longer than 1 hour. It is also found that in the proposed algorithm, shot change detection can be performed more than 5 times faster than normal playback speed using a 130MIPS workstation. © 1999 Scripta Technica, Syst Comp Jpn, 30(8): 11–22, 1999

Performance evaluation of multimedia systems with MPEG-2 bitstreams

MPEG-2 has become a compression standard for multimedia applications. Performance analysts, designers, and researchers have been using mpeg-2 bitstreams for evaluating the multimedia performance of their systems. There exist a significant difference in the characteristics and behavior of such bitstreams. The bitstream set, used for performance evaluation, that does not demonstrate such variations results in erroneous evaluations and projections. In this paper, we have proposed a methodology for improving the performance estimations/projections with mpeg-2 data. Our simulation results demonstrate variations in the mpeg-2 bitstreams and the consequence it can have when data from only one of the several classes is chosen for system measurement and evaluation

Low-complexity and low-delay video transcoding for compressed MPEG-2 bit-stream

Low-complexity and low-delay video transcoding for compressed MPEG-2 bit-stream

Extensions to nonlinear editing encompassing MPEG-2 4:2:2 profile

This paper covers extensions to nonlinear editing for MPEG with emphasis on the representation of compositions, how they are created, edited, and played back. In order to emphasize the differences between JPEG intraframe compression and MPEG interframe capability, we limit the discussion to video. We discuss aspects of editing enabled by the JPEG compression standard. A brief description of MPEG follows with an explanation of the issues of editing and playing back MPEG compressed media. We extend the composition model (specifically the frame index in the media representation) to allow use of MPEG compressed media for editing and present a new hardware system which enables this capability. Next we explain the formatting of MPEG bitstreams and the transformation of arbitrary MPEG streams needed to accommodate edit and playback.

Efficient MPEG compressed video analysis using macroblock type information

Efficient indexing methods are required to handle the rapidly increasing amount of visual information within video databases. Video analysis that partitions the video into clips or extracts interesting frames is an important preprocessing step for video indexing. We develop a novel method for video analysis using the macroblock (MB) type information of MPEG compressed video bitstreams. This method exploits the comparison operations performed in the motion estimation procedure, which results in specific characteristics of the MB type information when scene changes occur or some special effects are applied. Only a simple analysis on MB types of frames is needed to achieve very fast scene change, gradual transition, flashlight, and caption detection. The advantages of this novel approach are its direct extraction from the MPEG bitstreams after VLC decoding, very low complexity analysis, frame-based detection accuracy and high sensitivity.

Design and implementation of a DSP based MPEG-1 audio encoder

The speed of current PCs enable them to decode and play an MPEG bitstream in real-time. The encode process, however, cannot be done in real-time. This paper presents a low-cost real-time fixed point DSP implementation of an MPEG-1 layer 1 audio encoder. The DSP transmits the encoded MPEG bitstream to a PC through a standard bi-directional parallel port, which the PC can immediately save to disk. The encoder is capable of transmitting bitstreams to the PC at bit-rates from 32 kbps-320 kbps.

Weighted boundary matching algorithm for error concealment in the MPEG-2 video bit stream

A novel error concealment procedure is presented for recovering the lost macroblock while decoding an erroneous MPEG-2 bitstream. The proposed weighted boundary matching algorithm (WBMA) refines the conventional boundary matching algorithm. Different from the conventional BMAs, the boundary pixels of a lost macroblock are compared with those of the previous picture located in the same position. The weighting coefficients are also adaptively adjusted to work robustly in either smooth or abruptly-changing regions. Experimental results show that the proposed algorithm outperforms the conventional algorithms, especially for the interlaced sequences having large motion or complicated textures.

MPEG Splicing: A New Standard for Television — SMPTE 312M

Commercial insertion, promo insertion, studio routing, camera switching, tape editing—each of these basic operations involves switching video and audio. We want to continue to perform these operations even as we move from using uncompressed video to using MPEG-2 compressed video, but they are far from trivial. This paper describes the MPEG bitstream constraints required to facilitate switching—or “splicing”—MPEG bit-streams. It also describes the refinements to these constraints that have led to a new standard, SMPTE 312M, Television, Splice Points for MPEG-2 Transport Streams, which specifies bitstream constraints for both seamless and nonseamless splicing. It is ironic that, although splicing is likely the simplest method of switching between MPEG-2 bitstreams, it is probably the most difficult to understand. This paper intends to alleviate the mystery of splicing by explaining the underlying principles clearly so that this simple yet effective technique may be put into wide use throughout the emerging digital television industry.

A comparison of motion-compensated interlace-to-progressive conversion methods

The conversion of interlaced video to progressive format may be carried out in a number of ways, including vertical interpolation, fixed or motion-compensated vertical-temporal interpolation, and approaches based on motion compensation using non-uniform sampling theory. Motion-compensated methods potentially offer the best performance, although the accuracy and reliability of the vectors has a significant impact on their performance. For applications such as display conversion using vectors recovered from an MPEG-2 bitstream, the vectors available may not be of the highest quality. This paper considers several motion-compensated interlace-to-progressive conversion methods, and assesses their performance with both accurate and inaccurate vectors using objective and subjective techniques. The method found to work best uses a motion-compensated three-field vertical-temporal filter, in which the vertical component of the vector is rounded to the nearest even number of picture lines per field period. This method reduced the RMS error of converted picture sequences by almost 20% compared to simple vertical filtering when using vectors from an MPEG-2 bitstream. Conversely, some other motion-compensated methods were found to increase the error by a similar amount.

Video transcoding architecture with minimum buffer requirement for compressed MPEG-2 bitstream

In this paper, an extremely efficient MPEG-2 video transcoding method is proposed, which has a low buffer requirement and results in low delay. Most importantly, the proposed method does not need the computational intensive motion estimation. Simulation results demonstrate that the proposed approach results in very consistent video quality and maintains the buffer level effectively. Compared with direct decode/encode approach, the proposed approach only suffers slight quality degradation.

Watermarking of uncompressed and compressed video

In this paper, methods for embedding additive digital watermarks into uncompressed and compressed video sequences are presented. The basic principle borrows from spread spectrum communications. It consists of addition of an encrypted, pseudo-noise signal to the video that is invisible, statistically unobtrusive, and robust against manipulations. For practical applications, watermarking schemes operating on compressed video are desirable. A method for watermarking of MPEG-2 encoded video is presented. The scheme is a compatible extension of the scheme operating on uncompressed video. The watermark is generated exactly in the same manner as for uncompressed video, transformed using the discrete cosine transform (DCT) and embedded into the MPEG-2 bit-stream without increasing the bit-rate. The watermark can be retrieved from the decoded video and without knowledge of the original, unwatermarked video. Although an existing MPEG-2 bit-stream is partly altered, the scheme avoids visible artifacts by addition of a drift compensation signal. The proposed method is robust and of much lower complexity than a complete decoding process followed by watermarking in the pixel domain and re-encoding. Fast implementations exist which have a complexity comparable to a video decoder. Experimental results are given. The scheme is also applicable to other hybrid transform coding schemes like MPEG-1, MPEG-4, H.261, and H.263.

DCT convolution and its application in compressed domain

Conventional processing of JPEG or MPEG compressed image or video data involves first decompressing the data and applying the desired processing function, and then the processed data are recompressed for the purposes of transmission or storage. We propose an alternate processing pipeline which involves direct manipulation of the JPEG or MPEG compressed domain representation to achieve the desired spatial domain processing. For direct manipulation in the compressed domain, we develop a discrete cosine transform (DCT)-domain convolution theorem which besides exploiting the sparseness of the DCT-domain representation also exploits the orthogonality and symmetry in the DCT-domain representation. These properties lead to efficient compressed domain-based processing methods unlike their spatial domain counterparts, where such properties are not available. This theorem can be used in a variety of image and video editing functions when the image and video data are available only as a JPEG or MPEG bitstream. We illustrate the use of the DCT-domain convolution theorem in a typical video editing application such as video bluescreen editing.

A cost-effective architecture for HDTV video decoder in ATSC receivers

We describe the architecture of an HDTV video decoder, Vincent5, for MPEG2 MP@HL video decoding and format conversion of all 18 ATSC DTV formats in real-time. Vincent5 adopts a dataflow architecture for its main decoding functions in contrast to the conventional decoders that use a strict pipelined structure. Consequently, this makes it possible for us to explore wide design choices in the architecture decision for each decoding function. In Vincent5 we introduce a new memory control scheme of reducing the memory bandwidth, which is necessary in MPEG2 MP@HL decoding for a cost-effective solution. Without increasing the hardware complexity of Vincent5, we embed three programmable cores into the dedicated hardware to maximize its programmability. Vincent5 was described using the VHDL and its functionality was verified with standard MPEG2 bitstreams. Vincent5 includes 115 K logic gates, 118 Kb RAM, and 32 Kb ROM after logic synthesis and had been fabricated utilizing 3 ML 0.5 /spl mu/m CMOS technology.

DCT-based watermarking for video

An MPEG-based technique for embedding digital into digital video is proposed. The watermarking technique has been proposed as a method to hide secret information in the signals so as to discourage unauthorized copying or attest the origin of the media. In the proposed method, we take advantage of prediction types of MPEG bitstreams to embed watermarks into both intraframe and non-intraframe with different residual masks. The experimental results show that the proposed watermarking technique results in an almost invisible difference between the watermarked frames and the original frames, and is also robust to cropping operations and MPEG compression.