Rate-distortion Complexity Research Articles

Design Space Exploration of Practical VVC Encoding for Emerging Media Applications

Versatile Video Coding (VVC/H.266) is the latest video coding standard designed for a broad range of next-generation media applications. This paper explores the design space of practical VVC encoding by profiling the Fraunhofer Versatile Video Encoder (VVenC). All experiments were conducted over five 2160p video sequences and their downsampled versions under the random access (RA) condition. The exploration was performed by analyzing the rate-distortion-complexity (RDC) of the VVC block structure and coding tools. First, VVenC was profiled to provide a breakdown of coding block distribution and coding tool utilization in it. Then, the usefulness of each VVC coding tool was analyzed for its individual impact on overall RDC performance. Finally, our findings were elevated to practical implementation guidelines: the highest coding gains come with the multi type tree (MTT) structure, adaptive loop filter (ALF), cross component linear model (CCLM), and bi-directional optical flow (BDOF) coding tools, whereas multi transform selection (MTS) and affine motion estimation are the primary candidates for complexity reduction. To the best of our knowledge, this is the first work to provide a comprehensive RDC analysis for practical VVC encoding. It can serve as a basis for practical VVC encoder implementation or optimization on various computing platforms.

Fuzzy SVM-Based Coding Unit Decision in HEVC

The latest video compression standard, High Efficiency Video Coding (HEVC), has greatly improved the coding efficiency compared to the predecessor H.264/AVC. However, equipped with the quadtree structure of coding tree unit partition and other sophisticated coding tools, HEVC brings a significant increase in the computational complexity. To address this issue, a coding unit (CU) decision method based on fuzzy support vector machine (SVM) is proposed for rate-distortion-complexity (RDC) optimization, where the process of CU decision is formulated as a cascaded multi-level classification task. The optimal feature set is selected according to a defined misclassification cost and a risk area is introduced for an uncertain classification output. To further improve the RDC performance, different regulation parameters in SVM are adopted and outliers in training samples are eliminated. Additionally, the proposed CU decision method is incorporated into a joint RDC optimization framework, where the width of risk area is adaptively adjusted to allocate flexible computational complexity to different CUs, aiming at minimizing computational complexity under a configurable constraint in terms of RD performance degradation. Experimental results show that the proposed approach can reduce 58.9% and 55.3% computational complexity on average with the values of Bjønteggard delta peak-signal-to-noise ratio as -0.075 dB and -0.085 dB and the values of Bjøntegaard delta bit rate as 2.859% and 2.671% under low delay P and random access configurations, respectively, which has outperformed the state-of-the-art fast algorithms based on statistical information and machine learning.

A fine-grain distortion and complexity aware parameter tuning model for the H.264/AVC encoder

Most existing video encoders currently used in mobile applications are unable to gracefully degrade their output quality as the battery life nears its end. In other words, they cannot manage power consumption to efficiently utilize the available power resources. To be able to effectively adapt to changes in the encoder′s software and hardware platforms, especially due to the power limitations of mobile devices, the effect of encoder parameters on the encoding quality and power consumption has to be represented using a Rate–Distortion–Complexity (R–D–C) model. Most existing R–D–C models only consider macroblock level parameters, and overlook other higher level parameters that may have a more significant impact on complexity. In this paper, the distortion and complexity of the H.264/AVC encoder is controlled considering a subset of higher level encoding parameters consisting of search range, number of reference frames, and motion vector resolution. First, the complexity of full and fast motion estimation methods is modeled in an implementation and platform independent manner. Then, using this complexity model, a common encoding parameter setting table is derived, which leads to the least amount of distortion for each complexity condition. Finally, a complexity control mechanism is proposed which tunes the encoding parameters in a real-time manner. The proposed model can be combined with other existing macroblock level models in order to design a two-phase fine grain complexity controller. Simulation results indicate that when our method is integrated with the direct resource allocation (DRA) approach, performance increases by an average of 1.02dB and 1.06dB for full and fast motion estimation approaches, respectively.

Flexible distribution of complexity by hybrid predictive-distributed video coding

There is currently limited flexibility for distributing complexity in a video coding system. While rate-distortion-complexity (RDC) optimization techniques have been proposed for conventional predictive video coding with encoder-side motion estimation, they fail to offer true flexible distribution of complexity between encoder and decoder since the encoder is assumed to have always more computational resources available than the decoder. On the other hand, distributed video coding solutions with decoder-side motion estimation have been proposed, but hardly any RDC optimized systems have been developed. To offer more flexibility for video applications involving multi-tasking or battery-constrained devices, in this paper, we propose a codec combining predictive video coding concepts and techniques from distributed video coding and show the flexibility of this method in distributing complexity. We propose several modes to code frames, and provide complexity analysis illustrating encoder and decoder computational complexity for each mode. Rate distortion results for each mode indicate that the coding efficiency is similar. We describe a method to choose which mode to use for coding each inter frame, taking into account encoder and decoder complexity constraints, and illustrate how complexity is distributed more flexibly.

Analytical Rate-Distortion-Complexity Modeling of Wavelet-Based Video Coders

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Analytical modeling of the performance of video coders is essential in a variety of applications, such as power-constrained processing, complexity-driven video streaming, etc., where information concerning rate, distortion, or complexity (and their interrelation) is required. In this paper, we present a novel rate-distortion-complexity (R-D-C) analysis for state-of-the-art wavelet video coding methods by explicitly modeling several aspects found in operational coders, i.e., embedded quantization, quadtree decompositions of block significance maps and context-adaptive entropy coding of subband blocks. This paper achieves two main goals. First, unlike existing R-D models for wavelet video coders, the proposed derivations reveal for the first time the expected coding behavior of specific coding algorithms (e.g., quadtree decompositions, coefficient significance, and refinement coding) and, therefore, can be used for a variety of coding mechanisms incorporating some or all the coding algorithms discussed. Second, the proposed modeling derives for the first time analytical estimates of the expected number of operations (complexity) of a broad class of wavelet video coding algorithms based on stochastic source models, the coding algorithm characteristics and the system parameters. This enables the formulation of an analytical model characterizing the complexity of various video decoding operations. As a result, this paper complements prior complexity-prediction research that is based on operational measurements. The accuracy of the proposed analytical R-D-C expressions is justified against experimental data obtained with a state-of-the-art motion-compensated temporal filtering based wavelet video coder, and several new insights are revealed on the different tradeoffs between rate-distortion performance and the required decoding complexity. </para>

Region-of-Interest Based Resource Allocation for Conversational Video Communication of H.264/AVC

Due to the complexity of H.264/AVC, it is very challenging to apply this standard to design a conversational video communication system. This problem is addressed in this paper by using region-of-interest (ROI) based bit allocation and computational power allocation schemes. In our system, the ROI is first detected by using the direct frame difference and skin-tone information. Several coding parameters including quantization parameter, candidates for mode decision, the number of referencing frames, accuracy of motion vectors and the search range of motion estimation are adaptively adjusted at the macroblock (MB) level according to the relative importance of each MB. Subsequently, the encoder could allocate more resources such as bits and computational power to the ROI, and the decoding complexity is also optimized at the encoder side by utilizing an ROI based rate-distortion-complexity (R-D-C) cost function. The encoder is thus simplified and decoding-friendly, and the overall subjective visual quality can also be improved.

Complexity scalable motion compensated wavelet video encoding

We present a framework for the systematic analysis of video encoding complexity, measured in terms of the number of motion estimation (ME) computations, that we illustrate on motion compensated wavelet video coding schemes. We demonstrate the graceful complexity scalability of these schemes through the modification of the spatiotemporal decomposition structure and the ME parameters, and the use of spatiotemporal prediction. We generate a wide range of rate-distortion-complexity (R-D-C) operating points for different sequences, by modifying these options. Using our analytical framework we derive closed form expressions for the number of ME computations for these different coding modes and show that they accurately capture the computational complexity independent of the underlying content characteristics. Our framework for complexity analysis can be combined with rate-distortion modeling to determine the encoding structure and parameters for optimal R-D-C tradeoffs.