Hybrid Video Coding Research Articles

A new distortion measure for motion estimation in motion-compensated hybrid video coding

In the video coding standards MPEG-x and H.26x, a motion-compensated prediction technique is used for enhancing the coding performance of bitrate reduction or peak signal to noise ratio (PSNR) improvement. This technique takes advantage of the correlation between consecutive frames in the time domain, which is relatively higher than that between adjacent blocks in the spatial domain. In order to utilize the correlation between consecutive frames, the conventional video coding standards have used the motion estimation (ME) and compensation technique, where the Sum of the Absolute Differences (SAD) is usually used as the distortion measure. The ME estimates the reference block that could minimize the residual signal between the current and reference blocks. However, the SAD is not appropriate to the specific sequences that have global or local illumination changes. In addition, the high-resolution video sequences have higher spatial correlation than the low-resolution video sequences in general. Therefore, a new distortion measure that can consider spatial and temporal correlation simultaneously may be helpful to enhance the coding performance. The proposed distortion measure searches for a reference block that minimizes the motion-compensated residual signal when the DC-component is predicted. In our proposed algorithm, the maximum BD-rate improvement is up to 13.6% for illumination-changed video sequences, and the average BD-rate improvement is 6.6% for various high-resolution video sequences in the baseline profile.

A Hybrid Video Coder Based on Extended Macroblock Sizes, Improved Interpolation, and Flexible Motion Representation

This paper describes a video coding technology proposal submitted by Qualcomm in response to a joint call for proposals (CfP) issued by ITU-T SG16 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in January 2010. The proposed video codec follows a hybrid coding approach based on temporal prediction, followed by transform, quantization, and entropy coding of the residual. Some of its key features are extended block sizes (up to 64 <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\,\times\,$</tex></formula> 64), single pass switched interpolation filters with offsets, mode-dependent directional transforms for intra-coding, luma and chroma high precision filtering, geometric motion partitions, adaptive motion vector resolution and efficient 16-point transforms. It also incorporates internal bit-depth increase and modified quadtree-based adaptive loop filtering. Simulation results are presented to demonstrate the high compression efficiency achieved by the proposed video codec at the expense of moderate increase in encoding and decoding complexity compared to the advanced video coding standard (AVC/H.264). For the random access and low delay configurations, it achieved average bit rate reductions of 30.9% and 33.0% for equivalent peak signal-to-noise ratio, respectively, compared to the corresponding AVC anchors. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

DCT-domain coder for digital video applications

Recent home networking services such as next generation game consoles, video-on-demand, HDTV, etc. require faster video encoders and transcoders that avoid over-complicated transformations. In most popular digital video coding standards such as MPEG-X or H.26x, discrete cosine transform (DCT), inverse DCT (IDCT) transform, and block matching algorithms are essential elements in the coding process. The transforms between the pixel and DCT domains, are taking place during the operations and being only intermediate calculations, which complicate the encoding process. Thus, it is advantageous to reduce the number of transform operations between the spatial and frequency domains in order to simplify the overall encoder structure. In this paper we propose a DCT domain video encoder for reducing processing time. This is achieved by excluding the intermediate DCT and IDCT transforms from the conventional hybrid encoder. This is achieved by calculating the motion estimation and motion compensation via a spatial relationship between the DCT coefficients of a specific block and its sub- blocks. Experimental results of this method were compared to those of the conventional hybrid video encoder in the sense of visual quality and computational complexity. The comparison shows that the computational complexity of the proposed encoder is lower by 12.5% in comparison to conventional hybrid video coder for the same visual quality.

Direction-Adaptive Partitioned Block Transform for Color Image Coding

The direction-adaptive partitioned block transform (DA-PBT) is proposed to exploit the directional features in color images to improve coding performance. Depending on the directionality in an image block, the transform either selects one of the eight directional modes or falls back to the nondirectional mode equivalent to the conventional 2-D DCT. The selection of a directional mode determines the transform direction that provides directional basis functions, the block partitioning that spatially confines the high-frequency energy, the scanning order that arranges the transform coefficients into a 1-D sequence for efficient entropy coding, and the quantization matrix optimized for visual quality. The DA-PBT can be incorporated into image coding using a rate-distortion optimized framework for direction selection, and can therefore be viewed as a generalization of variable blocksize transforms with the inclusion of directional transforms and nonrectangular partitions. As a block transform, it can naturally be combined with block-based intra or inter prediction to exploit the directionality remaining in the residual. Experimental results show that the proposed DA-PBT outperforms the 2-D DCT by more than 2 dB for test images with directional features. It also greatly reduces the ringing and checkerboard artifacts typically observed around directional features in images. The DA-PBT also consistently outperforms a previously proposed directional DCT. When combined with directional prediction, gains are less than additive, as similar signal properties are exploited by the prediction and the transform. For hybrid video coding, significant gains are shown for intra coding, but not for encoding the residual after accurate motion-compensated prediction.

Lagrange multiplier selection in wavelet-based scalable video coding for quality scalability

In this paper, a method for Lagrange multiplier selection is proposed in the context of rate-distortion optimisation for wavelet-based scalable video coding targeting quality scalability. Despite the prevalence of the conventional method for Lagrange multiplier selection in hybrid video coding, the underlying formulation is not applicable to wavelet-based scalable video coding. To address the inherent challenges, a thorough analysis of the rate-distortion models for transform video coding is provided with regard to low and middle-to-high bit-rates, respectively. Based on the analysis, the models are consolidated according to experimental observations and the consolidated rate-distortion models serve as the basis for the derivation of the Lagrange multiplier. Considering the influence of the open-loop prediction structure on the rate-distortion performance, the Lagrange multiplier is initially derived for a single-targeted bit-rate. Moreover, the method for Lagrange multiplier selection in scalable video coding aiming at multiple-targeted bit-rates is proposed in a general sense of bit-rate range, varying from low to high bit-rates, building on the initially derived Lagrange multiplier for a single-targeted bit-rate. The proposed Lagrange multiplier is content adaptive and well suited for wavelet-based scalable video coding where quantisation steps are unavailable. Detailed performance evaluation of the proposed method for wavelet-based scalable video coding is provided with regard to a given targeted bit-rate and multiple-targeted bit-rates, respectively. The experimental results have demonstrated the effectiveness of the proposed Lagrange multiplier for rate-distortion optimisation considering quality scalability in wavelet-based scalable video coding.

A Novel Analytic Quantization-Distortion Model for Hybrid Video Coding

A proper theoretical quantization-distortion model for hybrid video coding is always desirable, since this allows us to explain the behavior of existing codecs and to design better ones. However, due to the existence of motion-compensated prediction, hybrid video coding introduces interframe dependency into the encoded video, which makes its quantization-distortion characteristics difficult to analyze. In this paper, a joint analysis of quantization and motion-compensated prediction is presented. For a complete analysis, we investigate not only the distortion that quantization introduces into video signal, but also its effect on motion-compensated prediction. Based on the joint analysis, a quantization-distortion model of hybrid video coding is proposed. Our extensive experimental results show that the proposed model can estimate the quantization-distortion curve of hybrid video coding with high accuracy. Furthermore, the estimation accuracy remains high for various video sequences and encoder configurations.

Laplace Distribution Based Lagrangian Rate Distortion Optimization for Hybrid Video Coding

In today's hybrid video coding, Rate-Distortion Optimization (RDO) plays a critical role. It aims at minimizing the distortion under a constraint on the rate. Currently, the most popular RDO algorithm for one-pass coding is the one recommended in the H.264/AVC reference software. It, or HR-lambda for convenience, is actually a kind of universal method which performs the optimization only according to the quantization process while ignoring the properties of input sequences. Intuitively, it is not efficient all the time and an adaptive scheme should be better. Therefore, a new algorithm Lap- lambda is presented in this paper. Based on the Laplace distribution of transformed residuals, the proposed Lap-lambda is able to adaptively optimize the input sequences so that the overall coding efficiency is improved. Cases which cannot be well captured by the proposed models are considered via escape methods. Comprehensive simulations verify that compared with HR-lambda , Lap-lambda shows a much better or similar performance in all scenarios. Particularly, significant gains of 1.79 dB and 1.60 dB in PSNR are obtained for slow sequences and B-frames, respectively.

Spatiotemporal Selective Extrapolation for 3-D Signals and Its Applications in Video Communications

In this paper, we derive a spatiotemporal extrapolation method for 3-D discrete signals. Extending a discrete signal beyond a limited number of known samples is commonly referred to as discrete signal extrapolation. Extrapolation problems arise in many applications in video communications. Transmission errors in video communications may cause data losses which are concealed by extrapolating the surrounding video signal into the missing area. The same principle is applied for TV logo removal. Prediction in hybrid video coding is also interpreted as an extrapolation problem. Conventionally, the unknown areas in the video sequence are estimated from either the spatial or temporal surrounding. Our approach considers the spatiotemporal signal including the missing area in a volume and replaces the unknown samples by extrapolating the surrounding signal from spatial, as well as temporal direction. By exploiting spatial and temporal correlations at the same time, it is possible to inherently compensate motion. Deviations in luminance occurring from frame to frame can be compensated, too.

JPEG2000-Compatible Scalable Scheme for Wavelet-Based Video Coding

We present a simple yet efficient scalable scheme for wavelet-based video coders, able to provide on-demand spatial, temporal, and SNR scalability, and fully compatible with the still-image coding standard JPEG2000. Whereas hybrid video coders must undergo significant changes in order to support scalability, our coder only requires a specific wavelet filter for temporal analysis, as well as an adapted bit allocation procedure based on models of rate-distortion curves. Our study shows that scalably encoded sequences have the same or almost the same quality than nonscalably encoded ones, without a significant increase in complexity. A full compatibility with Motion JPEG2000, which tends to be a serious candidate for the compression of high-definition video sequences, is ensured.

1-10 DCT/DST相互予測符号化における予測精度改善方法(第1部門 画像・動画処理1)

1-10 DCT/DST相互予測符号化における予測精度改善方法(第1部門 画像・動画処理1)

Lossy coding technique for digital holographic signal

We propose an efficient coding method of digital holograms (or fringe patterns) using standard compression tools for video and image. The fringe pattern is generated by a computer-generated hologram (CGH) algorithm with both an object image and its depth information. The proposed coding consists of preprocessing to separate RGB color space components, localization by segmenting the fringe pattern, frequency transform using M×N (segment size), 2-D discrete cosine transform (DCT) for extracting redundancy, segment scanning the segmented fringe pattern to form a video sequence, classification of coefficients, and hybrid video coding, which uses MPEG-2/4, H.264/AVC, differential pulse code modulation (DPCM), and lossless coding. The proposed algorithm illustrates that it has better properties for reconstruction, from four to eight times higher compression rate than previous research.

SNR scalability by coefficient refinement for hybrid video coding

A novel quality-scalable hybrid video coding scheme based on coefficient refinement and encoder quality constraints is developed. The technique, which also enables region-of-interest coding, is tested as a possible extension to the H.264 standard with excellent results. That is, the bit-rate increase typically encountered with scalable hybrid coding schemes is, for certain coding parameters, only small to moderate. The double- and triple-layer constellations of the system are shown to perform superior to other systems.

Hybrid Video Coding Based on Bidimensional Matching Pursuit

Hybrid video coding combines together two stages: first, motion estimation and compensation predict each frame from the neighboring frames, then the prediction error is coded, reducing the correlation in the spatial domain. In this work, we focus on the latter stage, presenting a scheme that profits from some of the features introduced by the standard H.264/AVC for motion estimation and replaces the transform in the spatial domain. The prediction error is so coded using the matching pursuit algorithm which decomposes the signal over an appositely designed bidimensional, anisotropic, redundant dictionary. Comparisons are made among the proposed technique, H.264, and a DCT-based coding scheme. Moreover, we introduce fast techniques for atom selection, which exploit the spatial localization of the atoms. An adaptive coding scheme aimed at optimizing the resource allocation is also presented, together with a rate-distortion study for the matching pursuit algorithm. Results show that the proposed scheme outperforms the standard DCT, especially at very low bit rates.

Motion- and aliasing-compensated prediction for hybrid video coding

In order to reduce the bit rate of video signals, the standardized coding techniques apply motion-compensated prediction in combination with transform coding of the prediction error. By mathematical analysis, it is shown that aliasing components are deteriorating the prediction efficiency. In order to compensate the aliasing, two-dimensional (2-D) and three-dimensional interpolation filters are developed. As a result, motion- and aliasing-compensated prediction with 1/4-pel displacement vector resolution and a separable 2-D Wiener interpolation filter provide a coding gain of up to 2 dB when compared to 1/2-pel displacement vector resolution as it is used in H.263 or MPEG-2. An additional coding gain of 1 dB can be obtained with 1/8-pel displacement vector resolution when compared to 1/4-pel displacement vector resolution. In consequence of the significantly improved coding efficiency, a Wiener interpolation filter and 1/4-pel displacement vector resolution is applied in H.264/AVC and in MPEG-4 (advanced simple profile).

Matching pursuit video coding .I. Dictionary approximation

We have shown in previous works that overcomplete signal decomposition using matching pursuits is an efficient technique for coding motion-residual images in a hybrid video coder. Others have shown that alternate basis sets may improve the coding efficiency or reduce the encoder complexity. In this work, we introduce for the first time a design methodology which incorporates both coding efficiency and complexity in a systematic way. The key to the method is an algorithm which takes an arbitrary 2-D dictionary and generates approximations of the dictionary which have fast two-stage implementations according to the method of Redmill et al. (see Proc. IEEE Int. Conf. Image Processing, p.769-773, 1998). By varying the quality of the approximation, we can explore a systematic tradeoff between the coding efficiency and complexity of the resulting matching pursuit video encoder. As a practical result, we show that complexity reduction factors of up to 1000 are achievable with negligible coding efficiency losses of about 0.1-dB PSNR.

Removal of motion uncertainty and quantization noise in motion compensation

In a standard hybrid video coder, there are two important factors affecting motion-compensated prediction: motion uncertainty and quantization noise, motion uncertainty results from the use of block-decimated motion compensation, which cannot specify the motion for each pixel. The propagation of quantization noise results from interframe prediction. We analyze both these effects, and their block-structured nonstationary properties. We propose a block-adaptive linear filtering framework to reduce motion uncertainty and quantization noise propagation simultaneously. This new motion-compensated predictor can be viewed as joint application of overlapped block motion compensation and loop filtering (LF). Several system configurations are evaluated. We show that this linear filtering framework achieves better rate-distortion performance than the single use of either overlapped block motion compensation or LF.

Modulus quantization for matching-pursuit video coding

Overcomplete signal decomposition using matching pursuits has been shown to be an efficient technique for coding motion-residual images in a hybrid video coder. Unlike orthogonal decomposition, mat...

Error-resilient video transmission using long-term memory motion-compensated prediction

Long-term memory prediction extends the spatial displacement vector utilized in hybrid video coding by a variable time delay, permitting the use of more than one reference frame for motion compensation. This extension leads to improved rate-distortion performance. However, motion compensation in combination with transmission errors leads to temporal error propagation that occurs when the reference frames at the coder and decoder differ. In this paper, we present a framework that incorporates an estimated error into rate-constrained motion estimation and mode decision. Experimental results with a Rayleigh fading channel show that long-term memory prediction significantly outperforms the single-frame prediction H.263-based anchor. When a feedback channel is available, the decoder can inform the encoder about successful or unsuccessful transmission events by sending positive (ACK) or negative (NACK) acknowledgments. This information is utilized for updating the error estimates at the encoder. Similar concepts, such as the ACK and NACK mode known from the H.263 standard, are unified into a general framework providing superior transmission performance.

Analysis of video transmission over lossy channels

A theoretical analysis of the overall mean squared error (MSE) in hybrid video coding is presented for the case of error prone transmission. Our model covers the complete transmission system including the rate-distortion performance of the video encoder, forward error correction, interleaving, and the effect of error concealment and interframe error propagation at the video decoder. The channel model used is a 2-state Markov model describing burst errors on the symbol level. Reed-Solomon codes are used for forward error correction. Extensive simulation results using an H.263 video codec are provided for verification. Using the model, the optimal tradeoff between INTRA and INTER coding as well as the optimal channel code rate can be determined for given channel parameters by minimizing the expected MSE at the decoder. The main focus of this paper is to show the accuracy of the derived analytical model and its applicability to the analysis and optimization of an entire video transmission system.

Modulus quantization for matching-pursuit video coding

Overcomplete signal decomposition using matching pursuits has been shown to be an efficient technique for coding motion-residual images in a hybrid video coder. Unlike orthogonal decomposition, matching pursuit uses an in-the-loop modulus quantizer which must be specified before coding begins. This complicates the quantizer design, since the optimal quantizer depends on the statistics of the matching-pursuit coefficients which in turn depend on the in loop quantizer actually used. In this paper, we address the modulus quantizer design issue, specifically developing frame-adaptive quantization schemes for the matching-pursuit video coder. Adaptive dead-zone subtraction is shown to reduce the information content of the modulus source, and a uniform threshhold quantizer is shown to be optimal for the resulting source. Practical two-pass and one-pass algorithms are developed to jointly determine the quantizer parameters and the number of coded basis functions in order to minimize coding distortion for a given rate. The compromise one-pass scheme performs nearly as well as the full two-pass algorithm, but with the same complexity as a fixed-quantizer design. The adaptive schemes are shown to outperform the fixed quantizer used in earlier works, especially at high bit rates, where the gain is as high as 1.7 dB.