CU Partition Research Articles

Improved Just Noticeable Difference Model Based Algorithm for Fast CU Partition in V-PCC

Improved Just Noticeable Difference Model Based Algorithm for Fast CU Partition in V-PCC

Fast CU partition algorithm based on swin-transformer for depth intra coding in 3D-HEVC

Fast CU partition algorithm based on swin-transformer for depth intra coding in 3D-HEVC

Fast CU size decision and intra-prediction mode decision method for H.266/VVC

AbstractH.266/Versatile Video Coding (VVC) is the most recent video coding standard developed by the Joint Video Experts Team (JVET). The quad-tree with nested multi-type tree (QTMT) architecture that improves the compression performance of H.266/VVC is introduced. Moreover, H.266/VVC contains a greater number of intra-prediction modes than H.265/High Efficiency Video Coding (HEVC), totalling 67. However, these lead to extremely the coding computational complexity. To cope with the above issues, a fast intra-coding unit (CU) size decision method and a fast intra-prediction mode decision method are proposed in this paper. Specifically, the trained Support Vector Machine (SVM) classifier models are utilized for determining CU partition mode in a fast CU size decision scheme. Furthermore, the quantity of intra-prediction modes added to the RDO mode set decreases in a fast intra-prediction mode decision scheme based on the improved search step. Simulation results illustrate that the proposed overall algorithm can decrease 55.24% encoding runtime with negligible BDBR.

Low Complexity Decision Algorithm for CU Partition Based on Image Texture Information and LGBM

Low Complexity Decision Algorithm for CU Partition Based on Image Texture Information and LGBM

Low Complexity Coding Unit Decision for Video-Based Point Cloud Compression.

With growing demand for point cloud coding, Video-based Point Cloud Compression (V-PCC) is released for dynamic point clouds, relying on mature 2D video coding techniques. However, the huge computational complexity of 2D video codec is inherited by V-PCC, thereby resulting in a notably time-consuming encoding process for the projection videos. To accelerate the compression, this paper proposes a low complexity coding unit decision algorithm for V-PCC intra coding. First, the 2D sequences (occupancy, geometry, and attribute sequences) are projected from same 3D point could frames in V-PCC. By exploring the strong correlations among them, the cross-projection information is creatively proposed for improving the predication performance of CU partition. Second, considering the disparate coding losses generated by incorrect partitioning decisions of different CUs, we develop a rate-distortion-oriented learning approach aimed at increasing the decision accuracy of the CUs, severely affecting coding performance. Third, to accommodate the particular coding architecture of V-PCC intra configuration, we further devise an overall framework, including targeted feature extraction and partitioning decision for intra and inter coding of geometry and attribute sequences. The final experimental results strongly demonstrate the effectiveness of our proposed algorithm. The time consumption of total projection sequence compression can be reduced by 57.80%, while the coding losses on Geom.BD-TotalRate (D1 and D2) and Attr.BD-TotalRate Luma component are only 0.08%, 0.33%, and 0.16%, respectively, which can be negligible. To the best of our knowledge, the proposed algorithm achieves state-of-the-art performance for accelerating the projection sequence compression in V-PCC All-Intra configuration.

A Fast CU Partition Algorithm for AVS3 Based on Adaptive Tree Search and Pruning Optimization

A Fast CU Partition Algorithm for AVS3 Based on Adaptive Tree Search and Pruning Optimization

Fast CU partition strategy based on texture and neighboring partition information for Versatile Video Coding Intra Coding

Fast CU partition strategy based on texture and neighboring partition information for Versatile Video Coding Intra Coding

An early CU partition mode decision algorithm in VVC based on variogram for virtual reality 360 degree videos

360-degree videos have become increasingly popular with the application of virtual reality (VR) technology. To encode such kind of videos with ultra-high resolution, an efficient and real-time video encoder becomes a key requirement. The Versatile Video Coding (VVC) standard has good coding performance. However, it has pretty high computational complexity which increasing the application cost of 360-degree videos. Among them, the decision of the quadtree with nested multi-type tree (QTMT) partitioning structure is one of the time-consuming procedures. In this paper, based on the characteristics of 360-degree video with Equirectangular projection (ERP) format, the empirical variogram combined with Mahalanobis distance is introduced to measure the difference between the horizontal and vertical directions of the CU, and a fast partition algorithm is proposed. The experimental results show that the algorithm saves 32.13% of the coding time with only an increase of 0.66% in BDBR.

Accelerating QTMT-based CU partition and intra mode decision for versatile video coding

The H.266/VVC achieves about 50% bitrate saving compared to its predecessor H.265/HEVC at the expense of exponentially increased computational complexity. The most efficient but complex technique for H.266/VVC intra frame coding is the QuadTree with a nested Multi-type Tree encoding structure (QTMT), which usually requires traversing the Rate-Distortion (R-D) cost of each partition and each mode for the best option. To alleviate such computational burden while preserving the coding efficiency as much as possible, this paper develops a multi-feature guided Fast CU Partition (FCP) and Laplacian guided Fast Mode Selection (FMS) to accelerate the intra QTMT decision together. For FCP, we regard the CU partition as a classification problem and adopt the Support Vector Machine (SVM) for its low-complexity implementation; after evaluating the contribution of a set of features, three representative features of video textures are selected and used to train the SVM model. Additionally, an advanced technique is applied by adopting a soft decision in SVM for a more flexible trade-off between the complexity and R-D performance. For FMS, we utilize the Laplace operator to determine the most probable directions of the current CU and skip half of the candidate modes for runtime saving. Experimental results demonstrate that the proposed FCP reduces the encoding time of H.266/VVC by 51.03% with 1.65% Bjøntegaard Delta Bit-Rate (BDBR) increase; the proposed FMS reduces the encoding time by 12.68% with 0.09% BDBR loss. Their direct combination and advanced combination finally lead to 54.84% encoding time reduction with 1.74% BDBR increase and 40.39% encoding time reduction with 1.33% BDBR increase, respectively, outperforming state-of-the-art approaches significantly.

Machine learning approaches with HEVC intra prediction on CU partition for complexity reduction

Machine learning approaches with HEVC intra prediction on CU partition for complexity reduction

A classification‐prediction joint framework to accelerate QTMT‐based CU partition of inter‐mode VVC

AbstractAiming at accelerating the inter coding of versatile video coding (VVC), the existing deep‐learning‐based methods utilize a single convolutional neural network (CNN) to directly predict the quadtree plus multi‐type tree (QTMT)‐based partition of the whole coding tree unit (CTU). However, these methods adopt one prediction network for unevenly distributed CTUs and ignore that the different CTUs have different partition prediction difficulties, leading to performance degradation and computation waste. To overcome these limitations, a classification‐prediction joint framework is proposed to accelerate inter coding of VVC in this letter, which combines classification and prediction to process different CTUs through different networks with appropriate capacities. To achieve effective partition prediction of the whole CTU, the QTMT‐based partition is first modeled as the partition homogeneity map (PHM), which is a value map reflecting the partition of each 8× 8 unit. Second, the classification module classifies the CTUs into different classes according to their partition prediction difficulty, and then different prediction sub‐networks with appropriate capacities are utilized to predict the PHM for the corresponding CTU class. Finally, the decision tree (DT) is adopted to determine the optimal split modes based on the predicted PHM. Experimental results show that the approach achieves 44.5% time saving with 1.94% BD‐BR increase, outperforming state‐of‐the‐art approaches.

A Hardware-Friendlyand High-Efficiency H.265/HEVC Encoder for Visual Sensor Networks.

Visual sensor networks (VSNs) have numerous applications in fields such as wildlife observation, object recognition, and smart homes. However, visual sensors generate vastly more data than scalar sensors. Storing and transmitting these data is challenging. High-efficiency video coding (HEVC/H.265) is a widely used video compression standard. Compare to H.264/AVC, HEVC reduces approximately 50% of the bit rate at the same video quality, which can compress the visual data with a high compression ratio but results in high computational complexity. In this study, we propose a hardware-friendly and high-efficiency H.265/HEVC accelerating algorithm to overcome this complexity for visual sensor networks. The proposed method leverages texture direction and complexity to skip redundant processing in CU partition and accelerate intra prediction for intra-frame encoding. Experimental results revealed that the proposed method could reduce encoding time by 45.33% and increase the Bjontegaard delta bit rate (BDBR) by only 1.07% as compared to HM16.22 under all-intra configuration. Moreover, the proposed method reduced the encoding time for six visual sensor video sequences by 53.72%. These results confirm that the proposed method achieves high efficiency and a favorable balance between the BDBR and encoding time reduction.

A Fast CU Partition Algorithm Based on Gradient Structural Similarity and Texture Features

The H.266/Versatile Video Coding (VVC) standard poses a great challenge for encoder design due to its high computational complexity and long encoding time. In this paper, the fast partitioning decision of coding blocks is investigated to reduce the computational complexity and save the coding time of VVC intra-frame predictive coding. A fast partitioning algorithm of VVC intra-frame coding blocks based on gradient structure similarity and directional features is proposed. First, the average gradient structure similarity of four sub-coding blocks under the current coding block is calculated, and two thresholds are set to determine whether the current coding block terminates the partitioning early or performs quadtree partitioning. Then, for the coding blocks that do not satisfy the above thresholds, the standard deviation of the vertical and horizontal directions of the current coding block is calculated to determine the texture direction and skip unnecessary partitioning to reduce computational complexity. Based on the VTM10.0 platform, this paper evaluates the performance of the designed fast algorithm for partitioning within the VVC coding unit. Compared with VTM10.0, the encoding rate is improved by 1.38% on average, and the encoder execution time is reduced by 49.32%. The overall algorithm achieves a better optimization of the existing VVC intra-frame coding technique.

A Fast Decision Algorithm for VVC Intra-Coding Based on Texture Feature and Machine Learning.

Due to the development and application of information technology, a series of modern information technologies represented by 5G, big data, and artificial intelligence are changing rapidly, and people's requirements for video coding standards have become higher. In the High-Efficiency Video Coding (HEVC) standard, the coding block division is not flexible enough, and the prediction mode is not detailed enough. A new generation of Versatile Video Coding (VVC) standards was born. VVC inherits the hybrid coding framework adopted by HEVC, improves the original technology of each module, introduces a series of new coding technologies, and builds on this greatly improving the coding efficiency. Compared with HEVC, the block division structure of VVC has undergone great changes, retaining the quad-tree (QT) division method and increasing the multi-type tree (MTT) division method, which brings high coding complexity. To reduce the computational complexity of VVC coding block division, a fast decision algorithm for VVC intra-frame coding based on texture characteristics and machine learning is proposed. First, we analyze the characteristics of the CU partition structure decision and then use the texture complexity of the CU partition structure decision to terminate the CU partition process early; for CUs that do not meet the early termination of the partition, use the global sample information, local sample information, and context information. The three-category feature-trained tandem classifier framework predicts the division type of CU. The experimental results show that in the full intra mode, compared with the existing VTM10.0, the encoding output bit rate is increased by 1.36%, and the encoding time is saved by 52.63%.

HG-FCN: Hierarchical Grid Fully Convolutional Network for Fast VVC Intra Coding

As one of the key technologies of Versatile Video Coding (VVC), a flexible quad-tree with a nested multi-type tree (QTMT) partition structure significantly improves the rate-distortion (RD) performance. However, this structure brings additional complexity due to the recursive search for the best partition type. Traditional fast partition methods in previous encoders, cannot adapt to this new complex structure, because it’s too complicated to predict each block size from one layer to another layer. Some indirect bottom-up designed methods are simple enough, but cannot predict specific split structures, making the acceleration capacity limited. Therefore, in this paper, we propose a learning-based approach to effectively predict the QTMT structure without having to heuristically explore the partitions of each layer. Firstly, we propose a hierarchy grid fully convolutional network (HG-FCN) framework, which concisely requires inference only once to obtain the entire partition information of the current CU and sub-CUs, and the inference is highly parallel. Secondly, we design a representation of complicated QTMT of CU partition in the form of hierarchy grid map (HGM), which can directly and effectively predict the specific hierarchical split structure. Lastly, a dual-threshold decision scheme is adopted to automatically control the trade-off between coding performance and complexity. Extensive experiments demonstrate the effectiveness of HG-FCN, which can reduce 51.15% <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim ~65.53$ </tex-math></inline-formula> % complexity of VVC intra coding with negligible 1.17% <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim ~2.19$ </tex-math></inline-formula> % BD-BR increase, superior to other state-of-the-art methods.

RDNet: Rate–Distortion-Based Coding Unit Partition Network for Intra-Prediction

High efficiency video coding (HEVC) has been finalized as the most widely utilized video coding standard, jointly developed by ITU-T, VCEG, and MPEG. In HEVC, the quad-tree structure of the coding unit partition is one of the most substantial modules and provides significant coding gains following huge coding time. In this paper, a rate–distortion-based coding unit partition network (RDNet) is proposed to make partition decisions based on the statistical features. RDNet is composed of a prediction sub-network and a target sub-network, where the prediction sub-network is used to predict the CU partition modes of the intra-prediction and the target sub-network is designed to optimize the network parameters by evaluating the rate–distortion cost, respectively. To balance the prediction accuracy and the rate–distortion loss, a parameter-exchanging strategy is applied to control the parameters’ sharing between two networks. Experimental results prove that our model can reduce the encoding time of HEVC by 55.83~71.72% with an efficient BD-BR of 2.876~3.347%, and the ablation study evaluates the ability of our strategy on balancing the trade-off between coding accuracy and inference speed.

Deep CNN Co-design for HEVC CU Partition Prediction on FPGA–SoC

Deep CNN Co-design for HEVC CU Partition Prediction on FPGA–SoC

ResNet-Based Fast CU Partition Decision Algorithm for VVC

H.266/VVC introduces new technologies based on previous video coding standards, including the QTMT partition structure and support for 65 angular intra-prediction modes. The introduced new technologies significantly improve the coding efficiency, but also lead to a great increase in coding complexity. This paper proposes a ResNet-based fast CU partition decision algorithm to reduce the coding complexity of VVC. First, we count and analyze the proportion of CU split modes to explore the appropriate CNN models. Then, the ResNet-based CNN models are designed to predict CU split mode, and the first convolutional layer of the CNN models combines symmetric and asymmetric convolutional kernels to extract features efficiently. We also introduce the RD cost into the loss function to improve the prediction accuracy of CNN models. Finally, two threshold schemes are used to achieve a compromise between coding complexity and coding performance. The experimental results show that the “Fast” scheme saves 55.12% of the encoding time with 1.83% BDBR increase and the “Moderate” scheme saves 47.03% of the encoding time with only 1.27% BDBR increase compared with VTM7.0.

CNN-LSTM Learning Approach-Based Complexity Reduction for High-Efficiency Video Coding Standard

High-Efficiency Video Coding provides a better compression ratio compared to earlier standard, H.264/Advanced Video Coding. In fact, HEVC saves 50% bit rate compared to H.264/AVC for the same subjective quality. This improvement is notably obtained through the hierarchical quadtree structured Coding Unit. However, the computational complexity significantly increases due to the full search Rate-Distortion Optimization, which allows reaching the optimal Coding Tree Unit partition. Despite the many speedup algorithms developed in the literature, the HEVC encoding complexity still remains a crucial problem in video coding field. Towards this goal, we propose in this paper a deep learning model-based fast mode decision algorithm for HEVC intermode. Firstly, we provide a deep insight overview of the proposed CNN-LSTM, which plays a kernel and pivotal role in this contribution, thus predicting the CU splitting and reducing the HEVC encoding complexity. Secondly, a large training and inference dataset for HEVC intercoding was investigated to train and test the proposed deep framework. Based on this framework, the temporal correlation of the CU partition for each video frame is solved by the LSTM network. Numerical results prove that the proposed CNN-LSTM scheme reduces the encoding complexity by 58.60% with an increase in the BD rate of 1.78% and a decrease in the BD-PSNR of -0.053 dB. Compared to the related works, the proposed scheme has achieved a best compromise between RD performance and complexity reduction, as proven by experimental results.

An Efficient CU Partition Algorithm for VVC Intra Coding

VVC currently under development is the latest video coding standard. VVC adopts many new coding technology or improve the original technology to enhance the coding efficiency on the basis of HEVC, at the cost of significantly increased computational complexity, which limits the industrialization and application of the VVC. To address this issue, an efficient CU partition method is presented. This method can effectively reduce computational complexity while preserving encoding efficiency. The proposed method utilizes the texture information and residual coefficient distribution of the CU to skip the low-probability CU partition types in advance. Experimental results illustrate that under the common test conditions, our proposed fast algorithm achieves 52.3% of the coding time saving while the BD-BR only increases by 1.31%, and the objective quality of the video remains basically unchanged.