MPEG-2 Encoder Research Articles

MPEG-Encoded Light Fields

Light fields (or Lumigraphs) are an image-based rendering method. As light fields require a large number of images, memory and disk space requirements are a significant barrier to the practical use of light fields. In this paper, we explain how to greatly reduce the space demands of light fields by encoding them in an MPEG format. The encoding is comprised of an MPEG bitstream and two tables of precomputed information to aid real-time decoding. By restricting the parameters used for MPEG encoding, and by employing a caching system, new views of the compressed light field can be rendered at fast interactive frame rates on consumer hardware. Compression ratios of around 50:1 are demonstrated. Combined, the speed and compression make the use of light fields practical on consumer hardware.

Probabilistic partial-distance fast matching algorithms for motion estimation

Motion search is by far the most complex operation to be performed in a video encoder. This paper proposes a novel fast matching algorithm to help speed up the computation of the matching (distance) metric used in the search, e.g., the sum of absolute difference (SAD). Based on a partial distance technique, our algorithm reduces complexity by terminating the SAD calculation early once it becomes clear that, given the partial SAD, it is likely that the total SAD will exceed that of the best candidate encountered so far in the search. The key idea is to introduce models to describe the probability distribution of the total distance given a measured partial distance. These models enable us to evaluate the risk involved in trusting a distance estimate obtained from a partial distance. By varying the amount of risk we are willing to take, we ran increase the speed, but we may also eliminate some good candidates too early, and thus increase the distortion of the decoded sequence. Because our approach requires knowledge of the statistical characteristics of the input, we also propose two approaches that allow these models to be obtained online. Our experimental results (based on an actual software implementation of an MPEG encoder) demonstrate that significant gains can be achieved with this approach. For example, reductions in the motion estimation computation time as compared with the original partial-distance search (where computation stops if the partial SAD is already larger than the SAD of the best candidate so far) can be as high as 45% for 2-D log search and 65% for exhaustive full search with a small penalty of 0.1-dB degradation in PSNR of the reconstructed sequences.

Applications of MPEG-4: digital multimedia broadcasting

The new MPEG-4 based digital multimedia broadcasting (DMB) system offers multimedia information with an excellent audio- and video quality to the mobile society. The combination of MPEG-4 encoding and decoding with the DAB-standard (digital audio broadcasting) enables for the first time distortion free TV-reception in fast moving vehicles. New concepts and algorithms which are required for real-time MPEG-4 audio and video encoding were developed, implemented and verified by their integration into the DMB-system. To cope with the demanding real-time requirements of MPEG-4, the corresponding hardware architecture also has been completely re-designed.

A Novel, Automatic Quality Control Scheme for Real Time Image Transmission

A novel scheme to compute energy on‐the‐fly and thereby control the quality of the image frames dynamically is presented along with its FPGA implementation. This scheme is suitable for incorporation in image compression systems such as video encoders. In this new scheme, processing is automatically stopped when the desired quality is achieved for the image being processed by using a concept called pruning. Pruning also increases the processing speed by a factor of more than two when compared to the conventional method of processing without pruning. An MPEG‐2 encoder implemented using this scheme is capable of processing good quality monochrome and color images of sizes up to 1024 × 768 pixels at the rate of 42 and 28 frames per second, respectively, with a compression ratio of over 17:1. The encoder is also capable of working in the fixed pruning level mode with user programmable features.

Improving the Performance of MPEG Compatible Encoding at Low Bit Rates Using Adaptive Neural Networks

A new approach is presented in this paper for improving the performance of MPEG encoders, especially in videophone or videoconferencing applications, through allocation of a greater number of bits in objects that belong to the foreground of image frames, than in objects that belong to the background. A human face and body detector followed by a neural network classifier are used for foreground/background object extraction. The derived image segmentation is used to modify the rate control of MPEG schemes so as to allocate more bits to foreground objects than to background, while retaining compatibility with MPEG encoders. Experimental results are presented, including image sequences with complex backgrounds, which illustrate the performance of the proposed scheme. Both a subjective image quality improvement and a PSNR increase of about 1.35 db on average have been obtained.

Rate control algorithm using SOFM-based neuralclassifier

A neural-net based rate control algorithm for MPEG encoders is introduced. It is based on the global rate-distortion model constructed by a self-organising feature map. The performances of the proposed algorithm are better than those for MPEG-4 VM5+ frame-based rate control, based on the criteria of the average number of bits per frame and PSNR.

DSP-based transcoding of digital video signals with MPEG-2 format

MPEG-2 video coding is widely used in broadcasting and increasingly in studio applications. For reasons of limited storage and/or transmission capacity, it may be necessary to reduce the bit rate of MPEG-2 video bitstreams. We present an implementation of a frequency domain transcoder (FDTC) on a media processor which reduces the bit rate of an incoming video bitstream in real time. The FDTC has a low complexity and low memory requirements. A rate control with low delay provides a bitstream with constant bit rate (CBR) at the output, independent whether the input bit rate is variable (VBR) or constant (CBR). The FDTC achieves a better or same peak signal-to-noise ratio (PSNR) than a bulky cascade of a complete MPEG-2 video decoder and encoder.

Encoding and reconstruction of incomplete 3-D video objects

A new approach for compact representation, MPEG-4 encoding, and reconstruction of video objects captured by an uncalibrated system of multiple cameras is presented. The method is based on the incomplete 3-D (I3D) technique, which was initially investigated for stereo video objects captured by parallel cameras. Non-overlapping portions of the object are extracted from the reference views, each view having the corresponding portion with the highest resolution. This way, the redundancy of the initial multiview data is reduced. The areas which are extracted from the basis views are denoted as areas of interest. The output of the analysis stage, i.e., the areas of interest and the corresponding parts of the disparity fields are encoded in the MPEG-4 bitstream. Disparity fields define the correspondence relations between the reference views. The view synthesis is performed by disparity-oriented reprojection of the areas of interest into the virtual view plane and can be seen as an intermediate postprocessing stage between the decoder and the scene compositor. This work performs an extension from parallel stereo views to arbitrary configured multi-views with new analysis and synthesis algorithms. Moreover, a two-way interaction is built between the analysis and reconstruction stages, which provides the tradeoff between the final image quality and amount of data transmitted. The focus is on a low-complexity solution enabling online processing capability while preserving the MPEG-4 compatibility of the I3D representation. It is finally shown that our method yields quite convincing results despite the minimal data used and the approximations involved.

An innovative, high quality and search window independent motion estimation algorithm and architecture for MPEG-2 encoding

This paper describes an innovative, pipelined, cache-based architecture for a motion estimation coprocessor. The coprocessor is based on a predictive/recursive algorithm whose computational complexity is low and not dependent on the search window. The synergies between the architecture and algorithm's features allow a high quality, low-area, low-bandwidth, search window implementation for standard resolution (PAL/NTSC), low resolution (CIF/QCIF) or high resolution (ATSC) sequences.

Real-Time MPEG-2 Video Codec System Using Multiple Digital Signal Processors

A real-time MPEG-2 video codec system, called the KAIST image computing system (KICS), was developed using multiple multimedia video processors (MVP, TMS320C80). The number of computations and optimum implementation of the MPEG-2 algorithm were analyzed for real-time processing. An MVP simulator was used to perform the timing analysis for the real-time MPEG-2 encoding and decoding, and it verified the design of the KICS. Task partitioning and mapping of the MPEG-2 algorithm to multiple MVPs were performed based on load-balancing, minimum data I/O, and the properties of the MVP. The KICS uses five MVPs for MPEG-2 real-time encoding of a main profile at the main level whereas it uses three MVPs for real-time decoding.

MPEG-2 Coded Video Traces Transmitted Over aSatellite Link: Scalable and Non-Scalable Solutionsin Rain Fading Conditions

The literature is poor in the analyses of the effects produced by corrupted bits in compressed video bitstreams. This paper presents the results of a transmission experiment of MPEG-2 coded video data over a satellite link affected by noise, in order to investigate under which conditions this type of transmission is economically feasible. The signal-to-noise ratio scalability feature of the MPEG-2 encoder was used to produce different bitstreams of the same movie sequence. The scope of the study was to verify which are the best combinations of video and channel codings in the presence of attenuation on the satellite link, in order to optimize the bandwidth utilisation for a requested image quality. The results obtained give indications about the data channel codings to be used to counter the rain fade on the transmission link, which is a non negligible problem especially when satellite transmissions are in the Ka band. Moreover, the results highlight the flexibility of the scalable video coding in the examined scenario.

New fast binary pyramid motion estimation for MPEG2 and HDTV encoding

A novel fast binary pyramid motion estimation (FBPME) algorithm is presented in this paper. The proposed FBPME scheme is based on binary multiresolution layers, exclusive-or (XOR) Boolean block matching, and a N-scale tiling search scheme. Each video frame is converted into a pyramid structure of K-1 binary layers with resolution decimation, plus one integer layer at the lowest resolution. At the lowest resolution layer, the N-scale tiling search is performed to select initial motion vector candidates. Motion vector fields are gradually refined with the XOR Boolean block-matching criterion and the N-scale tiling search schemes in higher binary layers. FBPME performs several thousands times faster than the conventional full-search block-matching scheme at the same PSNR performance and visual quality. It also dramatically reduces the bus bandwidth and on-chip memory requirement. Moreover, hardware complexity is low due to its binary nature. Fully functional software MPEG-2 MP@ML encoders and Advanced Television Standard Committee high definition television encoders based on the FBPME algorithm have been implemented. FBPME hardware architecture has been developed and is being incorporated into single-chip MPEG encoders. A wide range of video sequences at various resolutions has been tested. The proposed algorithm is also applicable to other digital video compression standards such as H.261, H.263, and MPEG4.

Efficient modeling of VBR MPEG-1 coded video sources

The performance evaluation of broadband networks requires statistical analysis and modeling of the actual network traffic. Since multimedia services, and especially variable bit rate (VBR) MPEG-coded video streams are expected to be a major traffic component carried by these networks, modeling of such services and accurate estimation of network resources are crucial for proper network design and congestion-control mechanisms that can guarantee the negotiated quality of service at a minimum cost. The layer modeling of MPEG-1 coded video streams and statistical analysis of their traffic characteristics at each layer is proposed, along with traffic models capable of estimating the network resources over asynchronous transfer mode (ATM) links. First, based on the properties of the entire MPEG-1 sequence (frame layer signal), a model (Model A) is presented by correlating three stochastic processes in discrete time (autoregressive models), each of which corresponds to the three types of frames of the MPEG encoder (I, P, and B frames). To simplify the traffic Model A and to reduce the required number of parameters, we study the MPEG stream at a higher layer by considering a signal, which expresses the average properties of I, P, and B frames over a group of picture (GOP) period. However, models on this layer cannot accurately estimate the network resources, especially in multiplexing schemes. For this reason, an intermediate layer is introduced, which exploits and efficiently combines information of both the aforementioned layers, producing a model (Model B), which requires much smaller number of parameters than Model A and simultaneously provides satisfactory results as far as the network resources are concerned. Evaluation of the validity of the proposed models is performed through experimental studies and computer simulations, using several long duration VBR MPEG-1 coded sequences, different from that used in modeling. The results indicate that both Models A and B are good estimators of video traffic behavior over ATM links at a wide range of utilization.

Usage parameter control schemes for improving MPEG video quality over ATM networks

An MPEG encoder compresses moving pictures into I-, P-, and B-frames, where the most important I-frames generate traffic with the highest bit rate. For MPEG videos transmitted over ATM networks, invalid cells are discarded by a usage parameter control scheme at the user network interface and thus cause error propagation in both spatial and temporal domains. For spatial error propagation, the effect of a lost cell depends on the position located in a slice: the farther the position is, the lesser affected are the data. Besides, lost data near the edge of a screen is more tolerable than those at the central part. In order to improve the quality of the received video, we propose two methods to impute the violation of non-conforming cells to others located at a less important position of the same slice. Simulation results reveal that the quality is improved based on both PSNR and subjective visual feeling.

Factors to Consider When Choosing an MPEG-2 Encoder

This paper describes some important criteria to consider when selecting a video encoder. In addition to discussing features, it will outline the practical tests that users can perform to assess the relative merits of MPEG-2 encoder products. There are a variety of encoder designs but also a correspondingly wide deviation in the picture quality for a given bit rate. We are now seeing some admirable devices that go some way to measuring picture quality, but they only tell part of the story. This paper will explain how to augment objective measurements with additional tests and feature analysis. It should place the buyer in a better position to unzip the hype and fully evaluate the strengths and weaknesses of an encoder.

Packet loss resilient MPEG-4 compliant video coding for the Internet

Targeting multimedia communications over the Internet, this paper describes a set of complementary techniques in the direction of both improved packet loss resiliency of video-compressed streams and e$cient usage of available network resources. Aiming rst at a best trade-o! between compression e$ciency and packet loss resiliency, a procedure for adapting the video coding modes to varying network characteristics is introduced. The coding mode selection is based on a rate-distortion procedure with global distortion metrics incorporating channel characteristics under the form of a two-state Markov model. This procedure has been incorporated in an MPEG-4 video encoder. It has been observed that, in error-free environments, the channel adaptive mode selection technique does not bring any penalty in terms of compression, with respect to the initial MPEG-4 encoder, while allowing a signicant gain with respect to simple conditional replenishment. On the other hand, under the same loss conditions, it is shown that this procedure signicantly improves the encoder’s performance with respect to the original MPEG-4 encoder, to approach the robustness of conditional replenishment mechanisms. This intrinsic robustication of the encoder allows to minimize the e!ects of packet losses on the visual quality of the received video; however, it does not avoid losses. A rate-based #ow control mechanism is then developed and introduced into the encoder, in order to match the bandwidth requirements of the source to the bandwidth available over the path of the connection, for both &social’ and &individual’ benets. The control mechanism developed combines an RTT-based control loop allowing early reaction to congestion and a TCPfriendly rate prediction model getting into play under lossy conditions. This hybrid control mechanism allows full rate control (even in loss-free conditions) and smooth rate variations together with high responsiveness. The introduction of the rate control in the MPEG-4 compliant encoder allows to maintain a stable PSNR and visual quality while decreasing signicantly the source throughput, hence reducing congestion and loss provoked by the same video source at a constant bit-rate. ( 1999 Elsevier Science B.V. All rights reserved.

Two-pass MPEG-2 variable-bit-rate encoding

Many MPEG-2 encoding applications are real-time; this implies that the video signal must be encoded with no significant lookahead. However, there exist non-real-time applications that do enable us to first analyze a video sequence entirely, and, using the analysis results, to optimize a second encoding pass of the same data. One example of such an application is the digital video disk (DVD), which is designed to facilitate a variable-bit-rate (VBR) output stream. In that case, it is possible to let the MPEG-2 encoder produce a video sequence with a constant visual quality over time. This is in contrast to constant-bit-rate (CBR) systems, where the rate is constant but the visual quality varies with the coding difficulty. This paper describes a two-pass encoding system that has as its objective to produce an optimized VBR data stream in a second pass. In a first pass, the video sequence is encoded with CBR, while statistics concerning coding complexity are gathered. Next, the first-pass data is processed to prepare the control parameters for the second pass, which performs the actual VBR compression. In this off-line processing stage, we determine the target number of bits for each picture in the sequence, such that we realize the VBR objective. This means that the available bits are appropriately distributed over the different video segments such that constant visual quality is obtained. To be able to quantify the constant visual quality, perceptual experiments are described and a practical model is fitted to them. Exceptional cases such as scene changes and fades are detected and dealt with appropriately. We also ensure that the second-pass compression process does not violate the decoder buffer boundaries. Finally, the encoding is performed again, but now under control of the processed first-pass data. During the running of this second pass, a run-time bit-production control mechanism monitors the accuracy and validity of the first-pass data, correcting errors in prediction and observing the buffer boundaries. Results are compared to CBR operation.

Single-pass constant- and variable-bit-rate MPEG-2 video compression

Most real-time MPEG-2 encoders are designed to perform in a constant-bit-rate (CBR) mode, in which buffer constraints are imposed to circumvent large deviations from a desired rate at any instant in time. Although such streams are generally good-quality sequences, certain types of operations or environments call for a more efficient real-time CBR encoder. The first part of the paper describes how a better-quality CBR video stream can be produced by estimating the relative complexity of a picture in comparison with the average complexity of the partially encoded stream and using it to adjust the compression parameters in a single-pass mode of operation. Our CBR encoder is particularly attractive for digital broadcast and editing environments, in which representations of higher-fidelity video objects in both display and freeze modes are constantly pursued. The second part of the paper describes the real-time generation of video streams with a variable-bit-rate (VBR) encoder. This mode of operation is high ly desirable for home entertainment and recreational events. We propose a robust single-pass VBR video encoder algorithm which is capable of learning and adapting itself to the complexity of image segments and thereafter creating streams which have constant visual picture quality. The new VBR scheme displays a better performance than the CBR encoder, particularly when special effects such as scene transitions, fades, or luminance changes are to be compressed. Both CBR and VBR encoders are fully compliant with the MPEG-2 standard and are easily implementable with IBM encoder architecture. Compression results for the new single-pass encoding algorithms and comparisons with previous CBR schemes are provided. The result suggests the suitability of our VBR approach for record/playback in storage media such as digital video disc (DVD) players, disk-based camcorders, and digital videocassette recorders (DVCRs). It further reflects the importance of our single-pass CBR scheme for providers of broadcast services , for which it allows more video programs to be allocated to a selected communication link, and for in-studio applications, for which it greatly facilitates visual analysis of captured streams.

Real-time interactive MPEG-4 system encoder using a cluster of workstations

MPEG-4 currently being finalized by the Moving Pictures Experts Group of the ISO is a multimedia standard, MPEG-4 aims to support content-based coding of audio, text, image, and video (synthetic and natural) data, multiplexing of coded data, as well as composition and representation of audiovisual scenes. One of the most critical components of an MPEG-4 environment is the system encoder. An MPEG-4 scene may contain several audio and video objects, images, and text, each of which must be encoded individually and then multiplexed to form the system bitstream. Due to its flexible features, object-based nature, and provision for user interaction, MPEG-4 encoder is highly suitable for a software-based implementation. A full-scale software-based MPEG-J system encoder with real-time encoding speed is a nontrivial task and requires massive computation. We have built such an encoder using a cluster of workstations collectively working as a virtual parallel machine. Parallel processing of MPEG-4 encoder needs to be carried out carefully as objects may appear or disappear dynamically in a scene. In addition, objects may be synchronized with each other. User interactions may also prohibit a straightforward parallelization. We propose a modeling methodology that captures the spatio-temporal relationship between various objects and user interaction. We then propose a number of scheduling algorithms that periodically allocate MPEG-4 objects to multiple workstations ensuring load balancing and synchronization requirements among multiple objects. Each scheduling algorithm has its own performance and complexity characteristics. The experimental results, while showing real-time encoding rates, exhibit tradeoffs between load balancing, scheduling overhead cost, and global performance.

Hardware/software co-synthesis with memory hierarchies

This paper introduces the first hardware/software co-synthesis algorithm of distributed real-time systems that optimizes the memory hierarchy along with the rest of the architecture. Memory hierarchies (caches) are essential for modern embedded cores to obtain high performance. They also represents a significant portion of the cost, size and power consumption of many embedded systems. Our algorithm synthesizes a set of real-time tasks with data dependencies onto a heterogeneous multiprocessor architecture that meets the performance constraints with minimized cost. Unlike previous work in co-synthesis, our algorithm not only synthesizes the hardware and software portions of the applications, but also the memory hierarchies. It chooses cache sizes and allocates tasks to caches as part of cosynthesis. The algorithm is built upon a task-level performance model for memory hierarchies. Experimental results, including examples from the literature and results on real-life examples such as an MPEG-2 encoder, show that our algorithm is efficient, and compared with existing algorithms, it can reduce the overall cost of the synthesized system.