Coding Unit Partition Research Articles

Fast CU decision method based on texture characteristics and decision tree for depth map intra-coding

As the demand for higher quality 3D videos continues to grow, the 3D extensions of the Efficient Video Coding (3D-HEVC) standard are gradually unable to meet the needs of users. Versatile Video Coding Standard (H. 266/VVC), as an advanced video coding standard, adopts the nested Multi-Type Tree quadtree (QTMT) partitioning structure that current fast depth map coding unit (CU) partitioning methods cannot apply. Therefore, we have designed a fast intra-frame CU partitioning algorithm for VVC 3D video depth maps. Our proposed algorithm in this article consists of two steps, including two sub-algorithms. First, by analyzing the relationship between image entropy and variance and depth map CU division, we establish a bi-criterion decision algorithm to determine whether the texture complexity of the current CU is low enough to terminate its partitioning process. Then, for CUs that have been determined by the first algorithm to need further partitioning, we use a decision tree model based on Light Gradient Boosting Machine (LGBM) to predict which direction of Rate-Distortion Optimization (RDO) calculation can be skipped, which can avoid some unnecessary RDO calculations in a certain direction. The final experiment demonstrated the effect of the proposed algorithm, which can reduce 47.65% complexity of VVC 3D video intra-coding with negligible 0.23% Bjøntegaard Delta Bitrate (BDBR) increase, superior to other advanced methods.

Learning-based CU partition prediction for fast panoramic video intra coding

In recent years, Virtual Reality (VR) technology has been applied in various aspects of life. As an indispensable component of VR, panoramic video offers viewers immersive experiences, thereby attracting widespread attention. The high resolution of panoramic video poses significant challenges for its storage and transmission, requiring efficient video compression. The Versatile Video Coding (VVC) exhibits superior coding efficiency, albeit at the expense of a several dozen-fold increase in encoding complexity. In this paper, we propose a fast Coding Unit (CU) partition algorithm for intra-coding based on Support Vector Machine (SVM), which selects appropriate features for online training of multiple classifiers based on frame texture characteristics. Additionally, in response to distortion issues in panoramic video coding, we divide video frames into different regions and adjust the Quantization Parameter (QP) values of CUs within each region to improve the encoding efficiency. Experimental results show that the proposed method achieves an average encoding time reduction of 46.21%, with an average BDBR increase of only 1.01%, achieving a good trade-off between time-saving and encoding efficiency.

Fast Depth Map Coding Algorithm for 3D-HEVC Based on Gradient Boosting Machine

Three-Dimensional High-Efficiency Video Coding (3D-HEVC) has been extensively researched due to its efficient compression and deep image representation, but encoding complexity continues to pose a difficulty. This is mainly attributed to redundancy in the coding unit (CU) recursive partitioning process and rate–distortion (RD) cost calculation, resulting in a complex encoding process. Therefore, enhancing encoding efficiency and reducing redundant computations are key objectives for optimizing 3D-HEVC. This paper introduces a fast-encoding method for 3D-HEVC, comprising an adaptive CU partitioning algorithm and a rapid rate–distortion-optimization (RDO) algorithm. Based on the ALV features extracted from each coding unit, a Gradient Boosting Machine (GBM) model is constructed to obtain the corresponding CU thresholds. These thresholds are compared with the ALV to further decide whether to continue dividing the coding unit. The RDO algorithm is used to optimize the RD cost calculation process, selecting the optimal prediction mode as much as possible. The simulation results show that this method saves 52.49% of complexity while ensuring good video quality.

Fast Coding Unit Partitioning Algorithm for Video Coding Standard Based on Block Segmentation and Block Connection Structure and CNN

The recently introduced Video Coding Standard, VVC, presents a novel Quadtree plus Nested Multi-Type Tree (QTMTT) block structure. This structure enables a more flexible block partition and demonstrates enhanced compression performance compared to its predecessor, HEVC. However, The introduction of the new structure has led to a more complex partition search process, resulting in a considerable increase in time complexity. The QTMTT structure yields diverse Coding Unit (CU) block sizes, posing challenges for CNN model inference. In this study, we propose a representation structure termed Block Segmentation and Block Connection (BSC), rooted in texture features. This ensures that partial CU blocks are uniformly represented in size. To address different-sized CUs, various levels of CNN models are designed for prediction. Moreover, we introduce a post-processing method and a multi-thresholding scheme to further mitigate errors introduced by CNNs. This allows for flexible and adjustable acceleration, achieving a trade-off between coding time complexity and performance. Experimental results indicate that, in comparison to VTM-10.0, our “Fast” scheme reduces the average complexity by 57.14% with a 1.86% increase in BDBR. Meanwhile, the “Moderate” scheme reduces average complexity by 50.14% with only a 1.39% increase in BDBR.

Divide-and-conquer-based RDO-free CU Partitioning for 8K Video Compression

8K (7689×4320) ultra-high definition (UHD) videos are growing popular with the improvement of human visual experience demand. Therefore, the compression of 8K UHD videos has become a top priority in the third-generation audio video coding standard (AVS3). However, as an important part of the coding standard promotion, the real-time hardware implementation for AVS3-based 8K UHD video intra coding is severely hindered, especially in the coding unit (CU) partition stage. To break through the limitation, this article proposes a divide-and-conquer-based rate-distortion-optimization-free (RDO-free) CU partitioning algorithm for efficient hardware implementation. Aimed at the complex CU partition in AVS3, we separately design a lightweight optimization for original partitioning rules to improve division efficiency and a decision tree-based RDO-free CU decision framework to eliminate the latency caused by the waiting for rate-distortion cost calculation in RDO strategy. Afterward, a divide-and-conquer-based hardware-friendly gradient difference calculating approach is devised to accelerate the learning feature extracting speed. To ensure that the proposed algorithm is sufficient to support the real-time CU partition for 8K videos, we also develop a hardware architecture based on FPGA. Experimental results illustrate that the software coding performance of our algorithm is significantly ahead of the efficient implementation uAVS3e for AVS3 and the reference software HM-16.20 for High Efficiency Video Coding (HEVC), even though there is 9.96% loss on BD-Rate Y. Considering its importance for the hardware implementation of AVS3-based 8K real-time encoder, the coding loss is acceptable. Moreover, the hardware simulation results on VU440 FPGA with Vivado 2019 show that our algorithm can support 61.12 frames per second (fps) CU partition for 8K UHD videos with only 0.00%, 0.00%, 1.01%, and 7.78% consumption of BRAM_18K, DSP48E, FF, and LUT, respectively. Additionally, with dual-path parallelism, 122.24 fps also can be implemented with controllable resource utilization, which achieves the state-of-the-art performance.

Low-complexity enhancement VVC encoder for vehicular networks

In intelligent transportation systems, real-time video streaming via vehicle networks has been seen as a vital difficulty. The goal of this paper is to decrease the computational complexity of the versatile video coding (VVC) encoder for VANETs. In this paper, a low-complexity enhancement VVC encoder is designed for vehicular communication. First, a fast coding unit (CU) partitioning scheme based on CU texture features is proposed in VVC, which aims to decide the final type of CU partition by calculating CU texture complexity and gray-level co-occurrence matrix (GLCM). Second, to reduce the number of candidate prediction mode types in advance, a fast chroma intra-prediction mode optimization technique based on CU texture complexity aims to combine intra-prediction mode features. Moreover, the simulation outcomes demonstrate that the overall approach may substantially reduce encoding time, while the loss of coding efficiency is reasonably low. The encoding time can be reduced by up to 53.29% when compared to the VVC reference model, although the average BD rate is only raised by 1.26%. The suggested VVC encoder is also hardware-friendly and has a minimal level of complexity for video encoders used in connected vehicle applications.

Lightweight fully connected network-based fast CU size decision for video-based point cloud compression

Video-based point cloud compression (V-PCC) utilizes high efficiency video coding (HEVC) to compress geometry and attribute videos generated from dynamic point cloud projection. However, the HEVC exhaustive coding unit (CU) size decision process is complex and hinders the real-time application of V-PCC. To reduce the coding complexity of V-PCC, this paper proposes a method that combined hand-crafted features and lightweight neural network to accurately predict the best CU partition in advance. First, we extract hand-crafted features, including direct features (DFs) and indirect feature (IF), as mixed features. DFs are simple and require no additional calculation, while IF is obtained indirectly by transforming the global and local distortions of the CU extracted before size decision determination. Second, we propose a lightweight fully connected network (LFCN) as the backbone network, two feature types are used as inputs to the LFCN to predict whether the CU should be split into sub-CUs, and the LFCN can be fully integrated into the encoder with only about 1.58KB of additional parameters. Experimental results show that the proposed method reduces coding complexity by an average of 51.2% while Luma’s BD-TotalRate only increases by 0.1% on average under the All Intra (AI) configuration.

Paired decision trees for fast intra decision in H.266/VVC

The Versatile Video Coding (H.266/VVC) has demonstrated significant improvements to its predecessor High-Efficiency Video Coding (H.265/HEVC) in terms of compression efficiency. One of its advancements is attributed to the introduction of the QuadTree with nested Multi-type Tree (QTMT). However, the accompanying computing complexity challenges practical applications. To this end, this paper proposes a fast encoding algorithm to accelerate the encoding process of H.266/VVC while maintaining its compression efficiency as much as possible. Essentially, the Coding Unit (CU) partition can be modeled as a classification problem, and thus conventional machine learning methods such as the decision tree can be used to solve it. Previous decision tree-based works generally use one decision tree to dyadically select one partition from a set of candidate partition modes, easily leading to a local optimum. In contrast, this work models the classification from a new viewpoint and devises Paired Decision Trees (PDT) to tackle it. Specifically, the first decision tree determines whether to divide the current CU or not, and the second decision tree further determines whether to divide the CU horizontally or vertically. However, inaccurate results in the first decision tree can be propagated to the second one, resulting in error accumulation. To address this issue, we introduce the Indeterminate Space to both decision trees and perform Rate-Distortion Optimization (RDO) on samples falling within this space. Experimental results show that compared with H.266/VVC reference software VTM-15.0, the proposed PDT method saves 52.86% encoding time on average while maintaining a small BDBR increase of only 1.36%, which significantly outperforms previous works.

Fast CU Partition Algorithm for Intra Frame Coding Based on Joint Texture Classification and CNN.

High-efficiency video coding (HEVC/H.265) is one of the most widely used video coding standards. HEVC introduces a quad-tree coding unit (CU) partition structure to improve video compression efficiency. The determination of the optimal CU partition is achieved through the brute-force search rate-distortion optimization method, which may result in high encoding complexity and hardware implementation challenges. To address this problem, this paper proposes a method that combines convolutional neural networks (CNN) with joint texture recognition to reduce encoding complexity. First, a classification decision method based on the global and local texture features of the CU is proposed, efficiently dividing the CU into smooth and complex texture regions. Second, for the CUs in smooth texture regions, the partition is determined by terminating early. For the CUs in complex texture regions, a proposed CNN is used for predictive partitioning, thus avoiding the traditional recursive approach. Finally, combined with texture classification, the proposed CNN achieves a good balance between the coding complexity and the coding performance. The experimental results demonstrate that the proposed algorithm reduces computational complexity by 61.23%, while only increasing BD-BR by 1.86% and decreasing BD-PSNR by just 0.09 dB.

Extreme Learning Machine-Enabled Coding Unit Partitioning Algorithm for Versatile Video Coding

The versatile video coding (VVC) standard offers improved coding efficiency compared to the high efficiency video coding (HEVC) standard in multimedia signal coding. However, this increased efficiency comes at the cost of increased coding complexity. This work proposes an efficient coding unit partitioning algorithm based on an extreme learning machine (ELM), which can reduce the coding complexity while ensuring coding efficiency. Firstly, the coding unit size decision is modeled as a classification problem. Secondly, an ELM classifier is trained to predict the coding unit size. In the experiment, the proposed approach is verified based on the VVC reference model. The results show that the proposed method can reduce coding complexity significantly, and good image quality can be obtained.

A Low-Complexity Fast CU Partitioning Decision Method Based on Texture Features and Decision Trees

The rapid advancement of information technology, particularly in artificial intelligence and communication, is driving significant transformations in video coding. There is a steadily increasing demand for high-definition video in society. The latest video coding standard, versatile video coding (VVC), offers significant improvements in coding efficiency compared with its predecessor, high-efficiency video coding (HEVC). The improvement in coding efficiency is achieved through the introduction of a quadtree with nested multi-type tree (QTMT). However, this increase in coding efficiency also leads to a rise in coding complexity. In an effort to decrease the computational complexity of VVC coding, our proposed algorithm utilizes a decision tree (DT)-based approach for coding unit (CU) partitioning. The algorithm uses texture features and decision trees to efficiently determine CU partitioning. The algorithm can be summarized as follows: firstly, a statistical analysis of the new features of the VVC is carried out. More representative features are considered to extract to train classifiers that match the framework. Secondly, we have developed a novel framework for rapid CU decision making that is specifically designed to accommodate the distinctive characteristics of QTMT partitioning. The framework predicts in advance whether the CU needs to be partitioned and whether QT partitioning is required. The framework improves the efficiency of the decision-making process by transforming the partition decision of QTMT into multiple binary classification problems. Based on the experimental results, it can be concluded that our method significantly reduces the coding time by 55.19%, whereas BDBR increases it by only 1.64%. These findings demonstrate that our method is able to maintain efficient coding performance while significantly saving coding time.

Fast CU Decision Algorithm Based on CNN and Decision Trees for VVC

Compared with the previous generation of High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC) introduces a quadtree and multi-type tree (QTMT) partition structure with nested multi-class trees so that the coding unit (CU) partition can better match the video texture features. This partition structure makes the compression efficiency of VVC significantly improved, but the computational complexity is also significantly increased, resulting in an increase in encoding time. Therefore, we propose a fast CU partition decision algorithm based on DenseNet network and decision tree (DT) classifier to reduce the coding complexity of VVC and save more coding time. We extract spatial feature vectors based on the DenseNet network model. Spatial feature vectors are constructed by predicting the boundary probabilities of 4 × 4 blocks in 64 × 64 coding units. Then, using the spatial features as the input of the DT classifier, through the classification function of the DT classifier model, the top N division modes with higher prediction probability are selected, and other division modes are skipped to reduce the computational complexity. Finally, the optimal partition mode is selected by comparing the RD cost. Our proposed algorithm achieves 47.6% encoding time savings on VTM10.0, while BDBR only increases by 0.91%.

Machine Learning Based Fast QTMTT Partitioning Strategy for VVenC Encoder in Intra Coding

The newest video compression standard, Versatile Video Coding (VVC), was finalized in July 2020 by the Joint Video Experts Team (JVET). Its main goal is to reduce the bitrate by 50% over its predecessor video coding standard, the High Efficiency Video Coding (HEVC). Due to the new advanced tools and features included in VVC, it actually provides high coding performances—for instance, the Quad Tree with nested Multi-Type Tree (QTMTT) involved in the partitioning block. Furthermore, VVC introduces various techniques that allow for superior performance compared to HEVC, but with an increase in the computational complexity. To tackle this complexity, a fast Coding Unit partition algorithm based on machine learning for the intra configuration in VVC is proposed in this work. The proposed algorithm is formed by five binary Light Gradient Boosting Machine (LightGBM) classifiers, which can directly predict the most probable split mode for each coding unit without passing through the exhaustive process known as Rate Distortion Optimization (RDO). These LightGBM classifiers were offline trained on a large dataset; then, they were embedded on the optimized implementation of VVC known as VVenC. The results of our experiment show that our proposed approach has good trade-offs in terms of time-saving and coding efficiency. Depending on the preset chosen, our approach achieves an average time savings of 30.21% to 82.46% compared to the VVenC encoder anchor, and a Bjøntegaard Delta Bitrate (BDBR) increase of 0.67% to 3.01%, respectively.

Reducing Complexity on Coding Unit Partitioning in Video Coding: A Review

Reducing Complexity on Coding Unit Partitioning in Video Coding: A Review

Analysis of coding unit partitioning and complexity reduction at intra-prediction mode of HEVC

ABSTRACT The emerging trend of ultra-high definition (4 K, UHD) video services necessitates coding efficiency, which is much higher than H.264. High-Efficiency Video Coding (HEVC) has been designed with the intention of drastically reducing the bit rate without compromising the video quality. The key feature is the introduction of a larger block structure with flexible block sizes from 64 × 64 to 8 × 8 samples. The enhanced feature has imposed higher complexity, specifically at the intra-prediction mode level of HEVC. The proposed research work has presented a novel hybrid approach to reducing the number of candidate Coding Units (CU) passing through irrelevant partition patterns and time-consuming iterative Rate-Distortion (RD) cost estimation process. The hybrid approach is implemented in two steps. In the first step gradient evaluation is performed and the CU is categorized as homogeneous or non-homogeneous. In the second step Early Termination is adopted by using machine learning for partitioning non-homogeneous CU. As the number of candidate Coding Units undergoing the time-consuming RD process is reduced by using the two-step hybrid approach, the proposed approach proves to be the best of its kind. The reduction in the number of candidate units undergoing the RD process reduces the complexity of the intra-prediction process. Encoding time is the measure of complexity and the proposed hybrid approach reduces the encoding time by an average of 36% compared with the HEVC reference model and also by an average of 26% as compared with the previous work. The Peak Signal to Noise Ratio (PSNR) reduced by an average of 0.1% indicates a slight reduction in the encoded image quality. The proposed approach significantly reduced the encoding time without affecting the RD performance of the encoder. The image quality is not compromised substantially when considering the improvement in the encoding time and the reduction in the complexity at an intra-prediction level.

A quality enhancement network with coding priors for constant bit rate video coding

Video coding can effectively compress data, while the introduction of compression artifacts degrades the visual quality and the performance of artificial intelligence video applications. Video quality enhancement (VQE) methods can improve compressed video quality without modifying the coding standard’s main modules. The existing VQE works mainly aim to improve video quality in the constant quantization parameter (CQP) coding mode. However, constant bit rate (CBR) coding mode is widely used in some streaming playback video applications, and VQE at CBR is more challenging than that at CQP. This article presents a novel Constant bit rate Video quality enhancement Network combined with Coding priors (CVCN). The proposed CVCN can be inserted into High Efficiency Video Coding (HEVC) codec as a CNN-based in-loop filter (LF) module or a post-processing module out of the codec. Moreover, we design a two-pass training strategy for the LF module to overcome multiple filtering. To adapt the QP diversity of CBR videos, we utilize the coding unit (CU)-wise QP prior by constructing a CU-wise QP adaptive module (CQAM) and a QP adaptive multi-scale residual block (QAMSRB) based on CQAM. We construct the CU-partition prior (CPP) by exploiting the relationship between the compression noise and CU partition information. A novel CPP spatial-attention block is proposed to combine the CPP with the self-attention module. Furthermore, the proposed CVCN can effectively enhance CBR videos at different bits per pixel (BPP) via a single model. Extensive experimental results show the superiority of the CVCN over state-of-the-art VQE approaches for CBR videos.

QTMT‐LNN: A fast intra CU partition using lightweight neural network for 360‐degree video coding on VVC

In the newest generation of video coding standard, Versatile Video Coding (VVC), a new technique called Quad Tree with nested Multi-type Tree (QTMT) structure is introduced. QTMT significantly improves the coding efficiency, but the improvement in compression performance comes at the cost of drastically increased complexity. This paper proposes a fast intra partition algorithm using Lightweight Neural Network (LNN) to skip QTMT partition steps which are unlikely to be chosen as the best split modes. Specifically, five LNNs (for five QTMT split modes) are trained, using features that consider the characteristic of 360-degree videos. The experimental results demonstrate that the proposed QTMT-LNN can reduce the encoding time from 52.28% to 72.17% on average, with coding efficiency losses ranging from 0.71% to 1.78%. Compared to other fast algorithms for intra coding unit (CU) partition, the method outperforms the related works in terms of Bjontegaard delta bit-rate (BDBR) and encoding time reduction.

Fast intra partition and mode prediction for equirectangular projection 360‐degree video coding

360-degree videos have drawn great attention from both the academia and the industry. For transmission and storage of 360-degree video, the joint video exploration team proposed the Versatile Video Coding (VVC) standard. VVC can significantly reduce the bitrate while maintaining the same subjective visual quality compared to the preceding high efficiency video coding. However, the computational complexity of VVC is extremely high which hinders the interactive applications. This paper proposes a fast intra partition method and mode prediction algorithm for equirectangular projection 360-degree video coding. First, a latitude-based preprocessing is introduced to early terminate the Coding Unit (CU) partition in the polar region. Second, the support vector machine is used to predict the CU partition type. Third, the fast intra mode search method accelerates the intra mode prediction. Experimental results show that the proposed algorithm can significantly obtain an average time reduction rate of 60.40% and a Bjontegaard delta rate increase of 1.96%.

Temporal Prediction Model-Based Fast Inter CU Partition for Versatile Video Coding.

Versatile video coding (VVC) adopts an advanced quad-tree plus multi-type tree (QTMT) coding structure to obtain higher compression efficiency, but it comes at the cost of a considerable increase in coding complexity. To effectively reduce the coding complexity of the QTMT-based coding unit (CU) partition, we propose a fast inter CU partition method based on a temporal prediction model, which includes early termination QTMT partition and early skipping multi-type tree (MT) partition. Firstly, according to the position of the current CU, we extract the optimal CU partition information of the position corresponding to the previously coded frames. We then establish a temporal prediction model based on temporal CU partition information to predict the current CU partition. Finally, to reduce the cumulative of errors of the temporal prediction model, we further extract the motion vector difference (MVD) of the CU to determine whether the QTMT partition can be terminated early. The experimental results show that the proposed method can reduce the inter coding complexity of VVC by 23.19% on average, while the Bjontegaard delta bit rate (BDBR) is only increased by 0.97% on average under the Random Access (RA) configuration.

Detecting double H.266/VVC compression with the same coding parameters

The detection of video double compression with the same coding parameters is a difficult problem in video forensics, since the recompression traces are extremely slight in this case. Although a series of methods have been proposed, detection of Versatile Video Coding (H.266/VVC) double compression with the same coding parameters is rarely reported. To address this issue, we propose a novel algorithm for the detection of H.266/VVC double compression with the same coding parameters in this paper. We first analyze the generation of H.266/VVC double compression traces, and explore the variation of coding modes caused by multiple compressions. Then we define Minimum Unit Mapping (MUM) and Subunit Prediction Mapping (SPM) to facilitate feature construction. The strength of the variation in the number of coding unit (CU) in each compression is calculated to form CU partition modes based feature set. We also use the consistency ratios of adjacent prediction mode pairs in the horizontal, vertical, major diagonal, and minor diagonal directions after multiple compressions to obtain prediction modes based feature set. These feature sets are further concatenated into a fusion feature for detection. With the thorough comparison to the state-of-the-art algorithms, it is verified that the proposed algorithm outperforms other current works in the field. Besides, the proposed method is robust against various encoding configurations.