Coding Tree Unit Research Articles

High Speed Coding Unit Depth Identification Based on Texture Image Information Using SVM

High-Efficiency Video Coding (HEVC/H.265) is a new video coding standard with half the bit rate of its predecessor, Advanced Video Coding (AVC/H.264). AVC/H.264 uses macroblocks, processing units between 4×4 and 16×16 pixels in size. H.265 uses Coding Tree Units (CTUs), a more complicated block structure that lets images be as large as 64×64 pixels. However, changing from macroblocks to coding tree units is essential for H.265 to become more efficient. Using the quadtree structure to divide the Coding Unit (CU) makes it harder for HEVC to find the optimal rate distortion. This paper presents a Support Vector Machine (SVM)-based method for finding the fastest coding unit division in intra-prediction HEVC without compromising compression efficiency. All partitions of CTU are assessed using five characteristics: Standard Deviation (SD), Root Mean Square Error (RMSE), Sub CU Complexity Difference (SCCD), Directional Complexity (DC), and Quantization Parameter (QP) to optimize the intra-prediction of HEVC in all intra-configurations. Simulations have been carried out to estimate the performance of the proposed machine learning-based algorithm using test sequences with different resolutions. Simulation results have shown that combining directional complexity and standard deviation gives a more accurate classification. SVM has been used to separate split-unsplit samples, and the standard rate-distortion optimization technique has been used to separate samples that are hard to separate. The results have shown a reduction of 67.44% in encoding time with a slight increase in bit rate.

Adaptive quadtree splitting parallelization (AQSP) algorithm for the VVC standard

The Versatile Video Coding (VVC) standard, also known as H.266, was released in 2020 as the natural successor to the High Efficiency Video Coding (HEVC) standard. Among its innovative coding tools, VVC extended the concept of quadtree (QT) splitting to the multi-type tree (MTT) structure, introducing binary and ternary partitions to enhance HEVC’s coding efficiency. While this brought significant compression improvements, it also resulted in a substantial increase in encoding time, primarily due to VVC’s larger Coding Tree Unit (CTU) size of 128×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document}128 pixels. To mitigate this, this work introduces a flexible parallel approach for the QT traversal and splitting scheme of the VVC encoder, called adaptive quadtree splitting parallelization (AQSP) algorithm. This approach is based on the distribution of coding units (CUs) among different threads using the current depth level of the QT as a basis to minimize the number of broken dependencies. In this way, the algorithm achieves a good trade-off between time savings and coding efficiency. Experimental results show that, when compared with the original VVC encoder, AQSP achieves an acceleration factor of 2.04×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} with 4 threads at the expense of a low impact in terms of BD rate. These outcomes position AQSP competitively in comparison with other state-of-the-art approaches.

Coding Tree Unit Partition Prediction Algorithm Utilizing Residual AttentionBased Long Short-Term Memory for Video Intra Coding

Coding Tree Unit Partition Prediction Algorithm Utilizing Residual AttentionBased Long Short-Term Memory for Video Intra Coding

Efficient VVC Encoding Using Hierarchical Parallelization: A Comprehensive Analysis

This paper presents and analyzes different parallelization strategies in VVenC, an open and optimized software encoder implementation of the Versatile Video Coding (VVC) standard. VVC has been developed to address the increasing demand for higher compression of digital video data, and it reduces the bitrate by around 50% for the same perceived quality compared to its predecessor, the High-Efficiency Video Coding (HEVC) standard. However, this increase in compression efficiency comes with an increase in computational complexity, particularly on the encoder side. VVenC integrates algorithmic optimizations for each coding tool in VVC and defines a set of five presets from faster to slower that provide Pareto-optimal tradeoffs between runtime and efficiency. Multithreading is employed to further reduce runtime while preserving most of the compression efficiency of each preset. With a hierarchical combination of pre-processing, picture-level and in-picture parallelization, VVenC achieves a 4× speedup for four threads. Further speedup using a higher number of threads depends on the video resolution and used encoder preset. For 16 threads, it ranges from 6 to 9 for high definition to 10-12 for ultra-high-definition video. Compared to previous work, the usage of temporal prediction for the adaptive loop filter reduces the associated coding efficiency loss from 0.4% to almost zero. A better scaling for higher numbers of threads can be achieved at the cost of a higher coding efficiency loss. In the presented framework, increased speedup by smaller block Coding Tree Unit (CTU) sizes, a combination of wavefront parallel processing and various tiles picture partitioning configurations is examined. Furthermore, results on a 20-core ARM-based Apple M1 computer indicate a better scaling for multithreading compared to × 86-based architectures. The analysis is complemented by profiling, which exhibits the overhead by idle threads and identifies the mode estimation as the main bottleneck of the presented framework.

An efficient versatile video coding motion estimation hardware

Versatile Video Coding (VVC) is the latest video coding standard. It provides higher compression efficiency than the previous video coding standards at the cost of significant increase in computational complexity. Motion estimation (ME) is the most time consuming and memory intensive module in VVC encoder. Therefore, in this paper, we propose an efficient VVC ME hardware. It is the first VVC ME hardware in the literature. It has real time performance with small hardware area. This efficiency is achieved by using a 64 × 64 systolic processing element array to support maximum coding tree unit (CTU) size of 128 × 128 and by using a novel memory-based sum of absolute differences (SAD) adder tree to calculate SADs of 128 × 128 CTUs. The proposed VVC ME hardware reduces memory accesses significantly by using an efficient data reuse method. It can process up to 30 4 K (3840 × 2160) video frames per second.

Two-Stage Perceptual Quality Oriented Rate Control Algorithm for HEVC

As a practical technique in mainstream video coding applications, rate control dominates important to ensure compression quality with limited bitrates constraints. However, most rate control methods mainly focus on objective quality while ignoring the perceptual quality improvement for human eyes. In this paper, we propose a two-stage rate control algorithm to optimize the perceptual quality at the frame encoding stage and the coding tree unit (CTU) encoding stage for high efficiency video coding (HEVC), respectively. Firstly, for the frame encoding stage, with inter-frame distortion dependency consideration, a frame-level rate control method is presented by adjusting the frame-level Lagrange multiplier adaptively with a preprocessing method. Secondly, for the CTU encoding stage, we propose a saliency-based CTU-level perceptual quality rate control algorithm, which employs CTU-level saliency weight to adjust the perceptual rate-distortion (R-D) model. We conduct the CTU-level rate control by an optimized Lagrange multiplier and quantization parameter (QP) to achieve perceptual quality optimization. Extensive experimental results reveal that, compared with state-of-the-art rate control methods on HEVC, our algorithm achieves significant perceptual coding performance with improved subjective visual quality.

A convolutional neural network-based rate control algorithm for VVC intra coding

The Versatile Video Coding (VVC) has shown significant improvements in Rate-Distortion (R-D) performance compared to its predecessor, High Efficiency Video Coding (HEVC). However, it still encounters several challenges. One of these challenges is the efficient allocation of bits among all Coding Tree Units (CTUs). Additionally, there is a lack of prior information for intra-frame coding, particularly for the first frame. After CTU-level bit allocation, only fixed parameters can be used to determine the λ for CTUs, which does not result in optimal rate–distortion performance. To tackle above challenges, we propose a rate control solution based on Convolutional Neural Network (CNN). This approach utilizes CNN to predict the key parameters α and β in the R-D model, addressing the problem of lacking prior information in intra-frame coding. Subsequently, the predicted α and β values are used to adaptively allocate bits for each CTU. Our proposed algorithm is implemented in VTM-16.0 under Common Test Conditions (CTC). Experimental results show that, compared to the default rate control algorithm in VTM-16.0, our proposed algorithm enhances R-D performance by 0.96% while maintaining rate control accuracy.

Fast HEVC inter-frame coding based on LSTM neural network technology

High Efficiency Video Coding (HEVC) is the most commonly used video coding standard. However, its high coding complexity is a heavy burden for real-time video applications. But, coding tools designed based on traditional coding frameworks have reached limits. Furthermore, existing low-complexity video coding methods have not thoroughly analyzed the characteristics of compressed video, making it impossible to develop targeted models to reduce coding complexity. Therefore, in this research, a fast HEVC inter-frame coding technique is proposed. Firstly, we perform a characteristics analysis of HEVC inter-frame coding to explore the correlation between video frames. Secondly, we develop an Inter-Frame Feature Transfer-Long Short-Term Memory (IFFT-LSTM) model to obtain the optimal coding tree unit (CTU) partition structure. Thirdly, we embed the IFFT-LSTM model into the HEVC test platform. The experimental results show that suggested method can effectively reduce the HEVC inter-frame coding complexity with a small amount of rate-distortion (RD) performance loss while maintaining acceptable subjective video quality.

Research on VVC Intra-Frame Bit Allocation Scheme Based on Significance Detection

This research is based on an intra-frame rate control algorithm based on the Versatile Video Coding (VVC) standard, considering that there is the phenomenon of over-allocating the bitrate of the end coding tree units (CTUs) in the bit allocation process, while the front CTUs are not effectively compressed. Fusing a Canny-based edge detection algorithm, a color contrast-based saliency detection algorithm, a Sum of Absolute Transformed Differences (SATD) based CTU coding complexity measure, and a Partial Least Squares (PLS) regression model, this paper proposes a CTU-level bit allocation improvement scheme for intra-mode code rate control of the VVC standard. First, natural images are selected to produce a lightweight dataset. Second, different metrics are utilized to obtain the significance and complexity values of each coding unit, the relatively important coding units in the whole frame are selected, which are adjusted with different weights, and the optimal adjustment multiplicity is supplemented into the dataset. Finally, the PLS regression model was used to obtain regression equations to refine the weights for adjusting the bit allocation. The proposed bit allocation scheme improves the average rate control accuracy by 0.453%, Y-PSNR by 0.05 dB, BD-rate savings by 0.33%, and BD-PSNR by 0.03 dB compared to the VVC standard rate control algorithm.

λ-Domain Rate Control via Wavelet-Based Residual Neural Network for VVC HDR Intra Coding.

High dynamic range (HDR) video offers a more realistic visual experience than standard dynamic range (SDR) video, while introducing new challenges to both compression and transmission. Rate control is an effective technology to overcome these challenges, and ensure optimal HDR video delivery. However, the rate control algorithm in the latest video coding standard, versatile video coding (VVC), is tailored to SDR videos, and does not produce well coding results when encoding HDR videos. To address this problem, a data-driven λ-domain rate control algorithm is proposed for VVC HDR intra frames in this paper. First, the coding characteristics of HDR intra coding are analyzed, and a piecewise R-λ model is proposed to accurately determine the correlation between the rate (R) and the Lagrange parameter λ for HDR intra frames. Then, to optimize bit allocation at the coding tree unit (CTU)-level, a wavelet-based residual neural network (WRNN) is developed to accurately predict the parameters of the piecewise R-λ model for each CTU. Third, a large-scale HDR dataset is established for training WRNN, which facilitates the applications of deep learning in HDR intra coding. Extensive experimental results show that our proposed HDR intra frame rate control algorithm achieves superior coding results than the state-of-the-art algorithms. The source code of this work will be released at https://github.com/TJU-Videocoding/WRNN.git.

Research on Quantization Parameter Decision Scheme for High Efficiency Video Coding

High-Efficiency Video Coding (HEVC) is one of the most widely studied coding standards. It still uses the block-based hybrid coding framework of Advanced Video Coding (AVC), and compared to AVC, it can double the compression ratio while maintaining the same quality of reconstructed video. The quantization module is an important module in video coding. In the process of quantization, quantization parameter is an important factor in determining the bitrate in video coding, especially in the case of limited channel bandwidth. It is particularly important to select a reasonable quantization parameter to make the bitrate as close as possible to the target bitrate. Aiming at the problem of unreasonable selection of quantization parameters in codecs, this paper proposes using a differential evolution algorithm to assign quantization parameter values to the coding tree unit (CTU) in each frame of 360-degree panoramic video based on HEVC so as to strike a balance between bitrate and distortion. Firstly, the number of CTU rows in a 360-degree panoramic video frame is considered as the dimension of the optimization problem. Then, a trial vector is obtained by randomly selecting the vectors in the population for mutation and crossover. In the mutation step, the algorithm generates a new parameter vector by adding the weighted difference between two population vectors to a third vector. And the elements in the new parameter vector are selected according to the crossover rate. Finally, the trial vector is regarded as the quantization parameter of each CTU in the CTU row to encode, and the vector with the least rate distortion is selected. The algorithm will produce the optimal quantization parameter combination for the current video. The experimental results show that compared to the benchmark algorithm of HEVC reference software HM-16.20, the proposed algorithm can provide a bit saving of 1.86%, while the peak signal-to-noise ratio (PSNR) can be improved by 0.07 dB.

Neural Network Based Rate Control for Versatile Video Coding

In this work, we propose a neural network based rate control algorithm for Versatile Video Coding (VVC). The proposed method relies on the modeling of the Rate-Quantization (R-Q) and Distortion-Quantization (D-Q) relationships in a data driven manner based upon the characteristics of prediction residuals. In particular, a pre-analysis framework is adopted, in an effort to obtain the prediction residuals which govern the Rate-Distortion (R-D) behaviors. By inferring from the prediction residuals with deep neural networks, the Coding Tree Unit (CTU) level R-Q and D-Q model parameters are derived, which could efficiently guide the optimal bit allocation. Subsequently, the coding parameters, including Quantization Parameter (QP) and λ, at both frame and CTU levels, are obtained according to allocated bit-rates. We implement the proposed rate control algorithm on VVC Test Model (VTM-13.0). Experimental results exhibit that the proposed rate control algorithm achieves 0.77% BD-Rate savings under Low Delay B (LDB) configurations when compared to the default rate control algorithm used in VTM-13.0. For Random Access (RA) configurations, 1.77% BD-Rate savings can be observed. Furthermore, with better bit-rate estimation, more stable buffer status can be observed, further demonstrating the advantages of the proposed rate control method.

Reducing Video Coding Complexity Based on CNN-CBAM in HEVC

High-efficiency video coding (HEVC) outperforms H.264 in coding efficiency. However, the rate–distortion optimization (RDO) process in coding tree unit (CTU) partitioning requires an exhaustive exploration of all possible quad-tree partitions, resulting in high encoding complexity. To simplify this process, this paper proposed a convolution neural network (CNN) based optimization algorithm combined with a hybrid attention mechanism module. Firstly, we designed a CNN compatible with the current coding unit (CU) size to accurately predict the CU partitions. In addition, we also designed a convolution block to enhance the information interaction between CU blocks. Then, we introduced the convolution block attention module (CBAM) into CNN, called CNN-CBAM. This module concentrates on important regions in the image and attends to the target object correctly. Finally, we integrated the CNN-CBAM into the HEVC coding framework for CU partition prediction in advance. The proposed network was trained, validated, and tested using a large scale dataset covering various scenes and objects, which provides extensive samples for intra-frame CU partition prediction in HEVC. The experimental findings demonstrate that our scheme can reduce the coding time by up to 64.05% on average compared to a traditional HM16.5 encoder, with only 0.09 dB degradation in BD-PSNR and a 1.94% increase in BD-BR.

A CTU-Level Screen Content Rate Control for Low-Delay Versatile Video Coding

In this paper, a rate control scheme for screen content video coding is proposed for the Versatile Video Coding (VVC) standard. In view of the critical challenges arising from the spatial and temporal unnaturalness of screen content sequences, the proposed method relies on the specifically designed pre-analysis such that the content information regarding the scene complexity can be obtained. As such, the estimated residual complexity is then incorporated into the proposed complexity-aware rate models and distortion models, leading to the optimal bit allocations for each frame and coding tree unit (CTU). In particular, the optimization problem can be analytically solved with the proposed models, and the coding parameters such as Lagrangian multiplier λ and quantization parameter of each frame and CTU could be delicately derived according to the allocated bits through the proposed analytical models. Extensive experiments have been conducted to evaluate the effectiveness of the proposed method. Compared to the default hierarchical λ-domain rate control and other screen content rate control algorithms, the proposed method could achieve obvious RD performance gain, and the bit-rate accuracy could be improved.

A High-Performance Rate Control Algorithm in Versatile Video Coding Based on Spatial and Temporal Feature Complexity

This paper presents a high-performance rate control (HPRC) algorithm for the Versatile Video Coding (VVC) standard that aims to achieve higher coding efficiency by considering spatial and temporal feature complexity. The HPRC algorithm differs from conventional rate control (RC) algorithms in that it uses an adaptive Canny operator (ACO) with a novel double-threshold algorithm to obtain the feature complexity of video content accurately. Moreover, an advanced bit allocation strategy at the frame-level and coding tree unit (CTU)-level is constructed to realize rational RC by thoroughly exploring the relationship between coding bits and complexity. To improve the precision of RC, an appropriate parameter updating based on the quasi-Newton algorithm is also proposed. Experimental results demonstrate that the proposed HPRC algorithm outperforms the default RC algorithm in VVC Test Model (VTM) 18.0, with Bjontegaard Delta Rate (BD-Rate) savings of 8.77 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> and 10.34 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> under Low Delay_P and Random Access configurations, respectively. Furthermore, the proposed algorithm also shows performance enhancements compared to other advanced algorithms.

Efficient Rate Control in Versatile Video Coding With Adaptive Spatial–Temporal Bit Allocation and Parameter Updating

Despite the fact that Versatile Video Coding (VVC) has achieved superior coding performance, two major problems remain for the rate control (RC) model in VVC. First, the regions concerned by human eyes are not clear enough in the coded video due to the deviation between the target bit allocation strategy of the coding tree unit (CTU) in RC and the human visual attention mechanism (HVAM). Second, there are significant quality fluctuations in the coded video frames due to the inappropriate updating speed. To address the above problems, we propose an efficient rate control (ERC) model. Specifically, in order to make the coded video more consistent with the attention of human eyes, we extract texture and motion-based spatial-temporal information to guide the bit allocation at the CTU level. Furthermore, based on the quasi-Newton algorithm and bit error, we propose an adaptive parameter updating (APU) method with the proper updating speed to precisely control the bits per frame. The proposed ERC outperforms the default RC model of VVC Test Model (VTM) 9.1 by saving the average Bjøntegaard Delta Rate (BD-Rate) on full-frame video sequences by 3.60% and 4.94% under low delay P (LDP) and random access (RA) configurations respectively, with higher bitrate accuracy. Moreover, the Peak Signal-to-Noise Ratio (PSNR) and actual coded bits per frame in the video coded by the proposed ERC are more stable.

Low calculation cost of HEVC coding unit size based on spatial homogeneity detection

For the same video quality, HEVC gives 25% to 50% bitrate savings, compared to its predecessor the Advanced Video Coding H.264 and thus supports resolutions up to 8 K UHD. However, the reduction in bitrates provided by the HEVC leads to an increase in the computational cost of the encoding operation. This complexity can become a true handicap especially for real-time video streaming and also for VANET (Vehicular Ad-Hoc Network) applications such as traffic safety and Video surveillance. The improvement in the bitrates and also the increase in the calculation cost are due to the use of large and multi-sized coding, prediction and transform blocks. Indeed, the H264 coder is based on structure macroblocks with sizes 4 × 4, 8 × 8 and 16 × 16, while H.265 depends on Coding Tree Units (CTUs), CTUs select sizes 4 × 4, 8 × 8, 16 × 16, 32 × 32 and 64 × 64 blocks. This paper proposes a fast CU (Coding Unit) size decision method to reduce the HEVC calculation cost based on spatial homogeneity. Compared with the HM16.13 benchmark test model, the average coding time is reduced by around 40% for CIF / QCIF video sequences, 35% to 43% for class A, B and C test sequences. These important reductions in coding time are obtained with negligible loss of quality and an average increase in bitrates which does not exceed 0.89% for the three configuration modes (All intra, Random Access and Low Delay).

A classification‐prediction joint framework to accelerate QTMT‐based CU partition of inter‐mode VVC

AbstractAiming at accelerating the inter coding of versatile video coding (VVC), the existing deep‐learning‐based methods utilize a single convolutional neural network (CNN) to directly predict the quadtree plus multi‐type tree (QTMT)‐based partition of the whole coding tree unit (CTU). However, these methods adopt one prediction network for unevenly distributed CTUs and ignore that the different CTUs have different partition prediction difficulties, leading to performance degradation and computation waste. To overcome these limitations, a classification‐prediction joint framework is proposed to accelerate inter coding of VVC in this letter, which combines classification and prediction to process different CTUs through different networks with appropriate capacities. To achieve effective partition prediction of the whole CTU, the QTMT‐based partition is first modeled as the partition homogeneity map (PHM), which is a value map reflecting the partition of each 8× 8 unit. Second, the classification module classifies the CTUs into different classes according to their partition prediction difficulty, and then different prediction sub‐networks with appropriate capacities are utilized to predict the PHM for the corresponding CTU class. Finally, the decision tree (DT) is adopted to determine the optimal split modes based on the predicted PHM. Experimental results show that the approach achieves 44.5% time saving with 1.94% BD‐BR increase, outperforming state‐of‐the‐art approaches.

Learning-Based Rate Control for High Efficiency Video Coding

High efficiency video coding (HEVC) has dramatically enhanced coding efficiency compared to the previous video coding standard, H.264/AVC. However, the existing rate control updates its parameters according to a fixed initialization, which can cause errors in the prediction of bit allocation to each coding tree unit (CTU) in frames. This paper proposes a learning-based mapping method between rate control parameters and video contents to achieve an accurate target bit rate and good video quality. The proposed framework contains two main structural codings, including spatial and temporal coding. We initiate an effective learning-based particle swarm optimization for spatial and temporal coding to determine the optimal parameters at the CTU level. For temporal coding at the picture level, we introduce semantic residual information into the parameter updating process to regulate the bit correctly on the actual picture. Experimental results indicate that the proposed algorithm is effective for HEVC and outperforms the state-of-the-art rate control in the HEVC reference software (HM-16.10) by 0.19 dB on average and up to 0.41 dB for low-delay P coding structure.

Lookahead Search-Based Low-Complexity Multi-Type Tree Pruning Method for Versatile Video Coding (VVC) Intra Coding

The latest versatile video coding (VVC) introduces some novel techniques such as quadtree with nested multi-type tree (QTMT), multiple transform selection (MTS) and multiple reference line (MRL). These tools improve compression efficiency compared with the previous standard H.265/HEVC, but they suffer from very high computational complexity. One of the most time-consuming parts of VVC intra coding is the coding tree unit (CTU) structure decision. In this paper, we propose a low-complexity multi-type tree (MT) pruning method for VVC intra coding. This method consists of lookahead search and MT pruning. The lookahead search process is performed to derive the approximate rate-distortion (RD) cost of each MT node at depth 2 or 3. Subsequently, the improbable MT nodes are pruned by different strategies under different cost errors. These strategies are designed according to the priority of the node. Experimental results show that the overall proposed algorithm can achieve 47.15% time saving with only 0.93% Bjøntegaard delta bit rate (BDBR) increase over natural scene sequences, and 45.39% time saving with 1.55% BDBR increase over screen content sequences, compared with the VVC reference software VTM 10.0. Such results demonstrate that our method achieves a good trade-off between computational complexity and compression quality compared to recent methods.