Delta Bit Research Articles

Unified Cross-Component Linear Model in VVC Based on a Subset of Neighboring Samples

To compress industrial video content efficiently, H.266/Versatile Video Coding (VVC) introduces cross-component linear model (CCLM) prediction as a new coding tool, in which chroma components are predicted from the luma component based on a linear model. In this article, we propose a subset-based CCLM (S-CCLM), in which the model parameters are derived based on a subset of neighboring samples. To choose the most proper subset, we build the relationship between the prediction error and the geometric distance and resolve the optimal subset construction problem by minimizing the geometric distance. With the well-designed subset, a weight-guided parameter derivation algorithm is further proposed to improve the accuracy of the model parameters. The experimental results show that the proposed S-CCLM can achieve Bjontegaard delta bitrate (BD-rate) reductions of 0.14%, 0.64%, and 0.75% for the Y, Cb, and Cr components, respectively, when the number of samples in the subset, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> , is 4 and BD-rate reductions of 0.22%, 0.80%, and 0.95% when <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> is 8. Given a small fixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> , fewer memory access operations are needed during the CCLM calculation, and a unified CCLM process can be achieved for coding blocks with different sizes and different modes. Due to its hardware-friendly architecture, the S-CCLM has been partially adopted by H.266/VVC.

An Adjacency Encoding Information-Based Fast Affine Motion Estimation Method for Versatile Video Coding

Versatile video coding (VVC), a new generation video coding standard, achieves significant improvements over high efficiency video coding (HEVC) due to its added advanced coding tools. Despite the fact that affine motion estimation adopted in VVC takes into account the translational, rotational, and scaling motions of the object to improve the accuracy of interprediction, this technique adds a high computational complexity, making VVC unsuitable for use in real-time applications. To address this issue, an adjacency encoding information-based fast affine motion estimation method for VVC is proposed in this paper. First, this paper counts the probability of using the affine mode in interprediction. Then we analyze the trade-off between computational complexity and performance improvement based on statistical information. Finally, by exploring the mutual exclusivity between skip and affine modes, an enhanced method is proposed to reduce interprediction complexity. Experimental results show that compared with the VVC, the proposed low-complexity method achieves 10.11% total encoding time reduction and 40.85% time saving of affine motion estimation with a 0.16% Bjøontegaard delta bitrate (BDBR) increase.

QTMT‐LNN: A fast intra CU partition using lightweight neural network for 360‐degree video coding on VVC

In the newest generation of video coding standard, Versatile Video Coding (VVC), a new technique called Quad Tree with nested Multi-type Tree (QTMT) structure is introduced. QTMT significantly improves the coding efficiency, but the improvement in compression performance comes at the cost of drastically increased complexity. This paper proposes a fast intra partition algorithm using Lightweight Neural Network (LNN) to skip QTMT partition steps which are unlikely to be chosen as the best split modes. Specifically, five LNNs (for five QTMT split modes) are trained, using features that consider the characteristic of 360-degree videos. The experimental results demonstrate that the proposed QTMT-LNN can reduce the encoding time from 52.28% to 72.17% on average, with coding efficiency losses ranging from 0.71% to 1.78%. Compared to other fast algorithms for intra coding unit (CU) partition, the method outperforms the related works in terms of Bjontegaard delta bit-rate (BDBR) and encoding time reduction.

Temporal Prediction Model-Based Fast Inter CU Partition for Versatile Video Coding.

Versatile video coding (VVC) adopts an advanced quad-tree plus multi-type tree (QTMT) coding structure to obtain higher compression efficiency, but it comes at the cost of a considerable increase in coding complexity. To effectively reduce the coding complexity of the QTMT-based coding unit (CU) partition, we propose a fast inter CU partition method based on a temporal prediction model, which includes early termination QTMT partition and early skipping multi-type tree (MT) partition. Firstly, according to the position of the current CU, we extract the optimal CU partition information of the position corresponding to the previously coded frames. We then establish a temporal prediction model based on temporal CU partition information to predict the current CU partition. Finally, to reduce the cumulative of errors of the temporal prediction model, we further extract the motion vector difference (MVD) of the CU to determine whether the QTMT partition can be terminated early. The experimental results show that the proposed method can reduce the inter coding complexity of VVC by 23.19% on average, while the Bjontegaard delta bit rate (BDBR) is only increased by 0.97% on average under the Random Access (RA) configuration.

Fast CU Partition Decision Algorithm for VVC Intra Coding Using an MET-CNN

The newest video coding standard, the versatile video coding standard (VVC/H.266), came into effect in November 2020. Different from the previous generation standard—high-efficiency video coding (HEVC/H.265)—VVC adopts a more flexible block division structure, the quad-tree with nested multi-type tree (QTMT) structure, which improves its coding performance by 24%. However, it also causes a substantial increase in computational complexity. Therefore, this paper first proposes the concept of a stage grid map, which divides the overall division of a 32 × 32 coding unit (CU) into four stages and represents it as a structured output. Second, a multi-stage early termination convolutional neural network (MET-CNN) model is devised to predict the full partition information of a CU with a size of 32 × 32. Finally, a fast CU partition decision algorithm for VVC intra coding based on an MET-CNN is proposed. The algorithm can predict all partition information of a CU with a size of 32 × 32 and its sub-CUs in one run, completely replacing the complex rate-distortion optimization (RDO) process. It also has an early exit mechanism, thereby greatly reducing the encoding time. The experimental results illustrate that the scheme proposed in this paper reduces the encoding time by 49.24% on average, while the Bjøntegaard Delta Bit Rate (BDBR) only increases by 0.97%.

Fast Decision Algorithm of CU Size for HEVC Intra-Prediction Based on a Kernel Fuzzy SVM Classifier

High Efficiency Video Coding (HEVC) achieves a significant improvement in compression efficiency at the cost of extremely high computational complexity. Therefore, large-scale and wide deployment applications, especially mobile real-time video applications under low-latency and power-constrained conditions, are more challenging. In order to solve the above problems, a fast decision method for intra-coding unit size based on a new fuzzy support vector machine classifier is proposed in this paper. The relationship between the depth levels of coding units is accurately expressed by defining the cost evaluation criteria of texture and non-texture rate-distortion cost. The fuzzy support vector machine is improved by using the information entropy measure to solve the negative impact of data noise and the outliers problem. The proposed method includes three stages: the optimal coded depth level “0” early decision, coding unit depth early skip, and optimal coding unit early terminate. In order to further improve the rate-distortion complexity optimization performance, more feature vectors are introduced, including features such as space complexity, the relationship between coding unit depths, and rate-distortion cost. The experimental results showed that, compared with the HEVC reference test model HM16.5, the proposed algorithm can reduce the encoding time of various test video sequences by more than 53.24% on average, while the Bjontegaard Delta Bit Rate (BDBR) only increases by 0.82%. In addition, the proposed algorithm is better than the existing algorithms in terms of comprehensively reducing the computational complexity and maintaining the rate-distortion performance.

A GCN-based fast CU partition method of intra-mode VVC

In this paper, a global convolutional network (GCN)-based fast coding unit (CU) partition method of intra-mode VVC is proposed. By using the GCN module with large kernel size convolutions, the proposed method can capture global information in CUs, leading to an accurate partition mode prediction in the quad-tree plus multi-type tree (QTMT) structure. Ranked according to predicted probabilities, the partition modes with lower probabilities are discarded, which reduces the computational complexity of VVC. Additionally, tradeoffs between performance and complexity can be achieved with different strategies. Experimental results demonstrated that the proposed method can reduce encoding time by 51.06%∼61.15% while increasing Bjøntegaard delta bit-rate (BD-BR) by 0.84%∼1.52% when implemented in VTM 10.0, outperforming the state-of-the-art methods, and that the proposed method can be used to accelerate VVenC 1.0 at the preset slower, achieving higher performance and lower complexity compared with the original VVenC 1.0 at the presets slow and medium.

HEVC’s intra mode process expedited using Histogram of Oriented Gradients

The brute-force behavior of High Efficiency Video Coding (HEVC) is the biggest hurdle in the communication of multimedia content. Therefore, two novel methods will be presented here to expedite the intra mode decision process of HEVC. In the first algorithm, the feasibility of Histogram of Oriented Gradients (HOG) for early intra mode decision is presented by using statistical evidence. Then, HOG of the current block and 35 intra predictions are obtained. The intra-prediction that gives the least sum of absolute difference (SAD) with the HOG of the current block is selected as the termination point. In the second algorithm, the difference between the Hardmard-cost of intra modes is modeled to achieve fast intra mode decision. The proposed algorithms accelerated the encoding process of the HEVC by 5% and 35.57%, while their Bjontegaard Delta Bit Rate (BD-BR) is 1.09% and 1.61%, respectively.

Low-Complexity Multiple Transform Selection Combining Multi-Type Tree Partition Algorithm for Versatile Video Coding.

Despite the fact that Versatile Video Coding (VVC) achieves a superior coding performance to High-Efficiency Video Coding (HEVC), it takes a lot of time to encode video sequences due to the high computational complexity of the tools. Among these tools, Multiple Transform Selection (MTS) require the best of several transforms to be obtained using the Rate-Distortion Optimization (RDO) process, which increases the time spent video encoding, meaning that VVC is not suited to real-time sensor application networks. In this paper, a low-complexity multiple transform selection, combined with the multi-type tree partition algorithm, is proposed to address the above issue. First, to skip the MTS process, we introduce a method to estimate the Rate-Distortion (RD) cost of the last Coding Unit (CU) based on the relationship between the RD costs of transform candidates and the correlation between Sub-Coding Units’ (sub-CUs’) information entropy under binary splitting. When the sum of the RD costs of sub-CUs is greater than or equal to their parent CU, the RD checking of MTS will be skipped. Second, we make full use of the coding information of neighboring CUs to terminate MTS early. The experimental results show that, compared with the VVC, the proposed method achieves a 26.40% reduction in time, with a 0.13% increase in Bjøontegaard Delta Bitrate (BDBR).

A Deep Learning Approach in Scalable High Efficiency Video Coding for Fast Coding Unit Size Decision

The High Efficiency Video Coding (HEVC) with scalable extension known as SHVC allows encoding the same video with different resolutions in a single bitstream. However, this process increases the encoding complexity. The complexity is increased mainly due to the motion estimation process and the Rate-Distortion Optimization (RDO) search process. The Test Zonal (TZ) algorithm with fast search patterns helps to accelerate the motion estimation process. However, the search patterns may get trapped to local minima, leading to inaccurate motion vectors. Moreover, the RDO search process used to determine the Coding Unit (CU) size increases the complexity. We proposed the Horizontal Subsampling Motion Estimation (HSME) method to find the accurate motion vectors with reduced complexity. The experimental results prove that the HSME method saves the encoding time by 53.03% with a 6.45% increase in Bjøntegaard delta bit rate and 0.28 dB loss in Bjøntegaard Delta Peak Signal-to-Noise Ratio (BD-PSNR) compared to standard SHM-12.1. In addition, we designed the Early Terminated Long- and Short-Term Memory (ET-LSTM) network that predicts the CU partition by taking the output features of the Early Terminated Convolutional Neural Network (ET-CNN). The ET-CNN learns the CU partitions from the residual Coding Tree Unit (CTU) using the deep learning approach. Our proposed method (HSME + ET-CNN + ET-LSTM) achieves 53% savings in encoding time, which is significantly higher than state-of-the-art methods.

Fast coding unit partitioning algorithms for versatile video coding intra coding

Versatile video coding (VVC) is the newest video compression standard. It adopts quadtree with nested multi-type tree (QT-MTT) to encode square or rectangular coding units (CUs). The QT-MTT coding structure is more flexible for encoding video texture, but it is also accompanied by many time-consuming algorithms. So, this work proposes fast algorithms to determine horizontal or vertical split for binary or ternary partition of a 32 × 32 CU in the VVC intra coding to replace the rate-distortion optimization (RDO) process, which is time-consuming. The proposed fast algorithms are actually a two-step algorithm, including feature analysis method and deep learning method. The feature analysis method is based on variances of pixels, and the deep learning method applies the convolution neural networks (CNNs) for classification. Experimental results show that the proposed method can reduce encoding time by 28.94% on average but increase Bjontegaard delta bit rate (BDBR) by about 0.83%.

High Efficiency Intra Video Coding Based on Data-Driven Transform

In this work, we propose a high efficiency intra video coding based on data-driven transform, which is able to learn the source distributions of intra prediction residuals and improve the subsequent transform coding efficiency. Firstly, we model learning based transform design as an optimization problem of maximizing energy compaction or decorrelation. A data-driven Subspace Approximation with Adjusted Bias (Saab) transform is analyzed and compared with the mainstream Discrete Cosine Transform (DCT) on their energy compaction and decorrelation capabilities. Secondly, we propose a Saab transform based intra video coding framework with offline Saab transform learning. Meanwhile, intra mode dependent Saab transform is developed. Then, Rate-Distortion (RD) gain of Saab transform based intra video coding is theoretically and experimentally analyzed in detail. Finally, three strategies that apply the Saab transform to intra video coding are developed to improve the coding efficiency. Experimental results demonstrate that the proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8\times 8$ </tex-math></inline-formula> Saab transform based intra coding can achieve Bjønteggard Delta Bit Rate (BDBR) from −1.19% to −10.00% and −3.07% on average as compared with the mainstream <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8\times 8$ </tex-math></inline-formula> DCT based intra coding. In case of variable size transform unit setting, the proposed algorithm achieves BDBR from −0.17% to −6.09% and −1.80% on average, which outperform the DCT based and the neural network based schemes.

Interlayer Restoration Deep Neural Network for Scalable High Efficiency Video Coding

This paper applies an interlayer restoration deep neural network (IRDNN) for scalable high efficiency video coding (SHVC) to improve visual quality and coding efficiency. It is the first time to combine deep neural network (DNN) and SHVC. Considering the coding architecture of SHVC, we elaborate a multi-frame and multi-layer neural network to restore the interlayer of SHVC by utilizing both the adjacent reconstructed frames of the base layer (BL) and enhancement layer (EL). Moreover, we analyze the temporal motion relationship of frames in one layer and the compression degradation relationship of frames between different layers, and propose the synergistic mechanism of motion restoration and compression restoration in our IRDNN. The network can generate an interlayer with higher quality serving for the EL coding and thus enhance the coding efficiency. A large-scale and various-quality-degradation dataset is self-made for the task of interlayer restoration of SHVC. The experimental results show that with our implementation on SHVC, the EL Bj<inline-formula> <tex-math notation="LaTeX">$\phi $ </tex-math></inline-formula>ntegaard delta bit-rate (BD-BR) reduction is 9.291&#x0025; and 6.007&#x0025; in signal-to-noise ratio scalability and spatial scalability, respectively. The code is available at <i><uri>https://github.com/icecherylXuli/IRDNN</uri></i>.

Probability-Based Fast Intra Prediction Algorithm for Spatial SHVC

Due to multi-layer encoding and Inter-layer prediction, Scalable High Efficiency Video Coding (SHVC) has extremely high coding complexity. It is very crucial to improve its coding speed so as to promote widespread and cost-effective SHVC applications. In this paper, we propose a new probability-based fast Intra prediction algorithm for spatial SHVC. More specifically, first, we integrate depth probabilities with textural based all-zero blocks and all-nonzero blocks through Lagrange Interpolation Polynomial (LIP) to derive thresholds to early skip unlikely depths and early terminate depth selection. Second, to early skip Intra mode prediction, we combine mode probabilities with Jarque-Bera test through LIP to test the residual coefficients of Inter-layer reference (ILR) mode. Third, we adopt the difference of Hadamard Costs (HCs) of horizontal, vertical and diagonal directional modes to predict candidate directional modes (DMs), and then exploit both the percentages of gradient amplitudes and HCs for DM early termination. Experimental results demonstrate that the proposed algorithm can achieve a speed up gain of more than 61&#x0025; with 0.03&#x0025; decrease in Bj&#x00F8;ntegaard Delta Bit Rate on average.

Tunable early CU size decision for depth map intra coding in 3D-HEVC using unsupervised learning

To further improve coding gain for depth map in 3D extension of High Efficiency Video Coding (3D-HEVC), many new coding techniques are introduced, which drastically increase the coding complexity. In this work, an early intra coding unit (CU) size decision scheme is proposed for 3D-HEVC intra depth map coding based on unsupervised learning approach. First, we treat the early CU size decision as a clustering problem. Then, three clustering models are developed for those CUs with the sizes of 64×64, 32×32 and 16×16 to early determine whether they be further split or not. Among them, the center of the clustering model is obtained by using intra learning method. Finally, in order to meet the user's specific application preference, the similarity distance is introduced into the early CU size decision. By adjusting the similarity distance, a tunable early CU size decision is achieved to obtain different levels of coding complexity reduction. Experimental results show that the proposed scheme achieves encoding time reduction ranging from 54.0% to 68.4% on average for depth map intra coding, while the Bjontegaard delta bit rate (BDBR) of synthesized views only increased by 0.01% to 0.73%, it outperforms the state-of-the-art works in term of coding complexity reduction and BDBR increase.

SSIM-Variation-Based Complexity Optimization for Versatile Video Coding

To date, Versatile Video Coding (VVC) has a more magnificent overall performance than High Efficiency Video Coding (HEVC). The Quadtree with Nested Multi-Type Tree (QTMT) coding block structure can substantially enhance video coding quality in VVC. However, the coding gain also leads to a greater coding complexity. Therefore, this letter proposes a Fast Decision Scheme Based on Structural Similarity Index Metric Variation (FDS-SSIMV) to solve this problem. Firstly, the Structural Similarity Index Metric Variation (SSIMV) characteristic among the sub coding units of the spit mode is illustrated. Next, to evaluate the SSIMV value, SSIMV measure strategies are designed for different split modes in this letter. Then, the desired split modes are selected by the SSIMV values. Experimental results show that the proposed method achieves 64.74\% average encoding Time Saving (TS) with a 2.79\% Bj$\varnothing$ntegaard Delta Bit Rate (BDBR), outperforming the benchmarks.

High Efficiency Intra CU Partition and Mode Decision Method for VVC

Versatile video coding (VVC/H.266) is the newest video compression standard, which is developed by the Joint Video Experts Team. Compared with previous encoding schemes, VVC achieves higher compression efficiency by introducing a new partition structure and additional intra prediction modes but results in high computational complexity. To efficiently solve this problem of redundant processing in quad-tree with nested multi-type tree structures and intra mode prediction, we propose a texture analysis&#x2013;based ternary tree (TT) and binary tree (BT) partition strategy, and a gradient-based intra mode decision method to accelerate TT and BT partition and intra mode prediction, separately. The texture complexity and prediction direction of coding unit (CU) is calculated by texture detection method. A texture analysis&#x2013;based TT and BT partition strategy is established by using the regression method based on analyzing the texture complexity of the CU. Then, a texture analysis&#x2013;based TT and BT partition strategy is applied to reduce the redundant partition for each CU. By using the prediction direction of CU, a gradient-based intra mode decision method is established for skipping the impossible modes for each CU. Experimental results revealed that the proposed method could save 49.49% in encoding time and increase the Bjontegaard delta bit rate (BDBR) by only 0.56%. It confirms that the proposed method achieved high efficiency and a good balance between the BDBR and time saving.

Notice of Violation of IEEE Publication Principles: VVC/H.266 Intra Mode QTMT Based CU Partition Using CNN

Notice of Violation of IEEE Publication Principles <br><br>“VVC/H. 266 Intra Mode QTMT Based CU Partition Using CNN” <br>by Sameena Javaid, Safdar Rizvi, Muhammad Talha Ubaid, Abdullah Tariq <br>in IEEE Access, April 2022 <br><br>After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE’s Publication Principles. <br><br>This paper copied content from the article below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission. <br><br> “DeepQTMT: A Deep Learning Approach for Fast QTMT-Based CU Partition of Intra-Mode VVC” <br> by Tianyi Li, Mai Xu, Runzhi Tang, Ying Chen, Qunliang Xing <br>in IEEE Transactions on Image Processing, Vol 30, 2021 <br><br> <br/> The latest standard for video coding is versatile video coding (VVC) / H.266 which is developed by the joint video exploration team (JVET). Its coding structure is a multi-type tree (MTT) structure, which consists of two types of trees that are Ternary Tree (TT) and Binary Tree (BT). Due to the use of brute force quest for residual rate-distortion the quad-tree and multi-type tree (QTMT) structure of the coding unit (CU) split and contributes over 98% of the encoding time. This structure is efficient in coding, however, increases computational complexity. The current paper proposes a deep learning technique to predict the QTMT based CU split rather than just the brute-force QTMT method to substantially speed up the time of the encoding process for VVC/H.266 intra mode. In the first phase, we developed an extensive database containing ample CU splitting patterns and various streaming videos. In the second phase, we suggest a multi-level exit CNN (MLE-CNN) model with a redundancy removal mechanism at different levels to determine the CU partition. In the third phase, for the training of MLECNN model we have established the adaptive loss function and analyzing the both unknown number of partition modes and the focus on RD cost minimization. Finally, a variable threshold decision system is established to achieve the targeted low complexity and RD performance. Ultimately experimental findings show that VVC/H.266 encoding time has reduced to 69.11% from 47.91% with insignificant bjontegaard delta bit rate (BDBR) to 2.919% from 1.023% which performs better than the existing futuristic and modern approaches.

Fast Intra Mode Selection in HEVC Using Statistical Model

Comprehension algorithms like High Efficiency Video Coding (HEVC) facilitates fast and efficient handling of multimedia contents. Such algorithms involve various computation modules that help to reduce the size of content but preserve the same subjective viewing quality. However, the brute-force behavior of HEVC is the biggest hurdle in the communication of multimedia content. Therefore, a novel method will be presented here to accelerate the encoding process of HEVC by making early intra mode decisions for the block. Normally, the HEVC applies 35 intra modes to every block of the frame and selects the best among them based on the RD-cost (rate-distortion). Firstly, the proposed work utilizes neighboring blocks to extract available information for the current block. Then this information is converted to the probability that tells which intra mode might be best in the current situation. The proposed model has a strong foundation as it is based on the probability rule-2 which says that the sum of probabilities of all outcomes should be 1. Moreover, it is also based on optimal stopping theory (OST). Therefore, the proposed model performs better than many existing OST and classical secretary-based models. The proposed algorithms expedited the encoding process by 30.22% of the HEVC with 1.35% Bjontegaard Delta Bit Rate (BD-BR).

A Complexity Reduction Method for VVC Intra Prediction Based on Statistical Analysis and SAE-CNN

Compared with High Efficiency Video Coding (HEVC), the latest video coding standard Versatile Video Coding Standard (VVC), due to the introduction of many novel technologies and the introduction of the Quad-tree with nested Multi-type Tree (QTMT) division scheme in the block division method, the coding quality has been greatly improved. Due to the introduction of the QTMT scheme, the encoder needs to perform rate–distortion optimization for each division mode during Coding Unit (CU) division, so as to select the best division mode, which also leads to an increase in coding time and coding complexity. Therefore, we propose a VVC intra prediction complexity reduction algorithm based on statistical theory and the Size-adaptive Convolutional Neural Network (SAE-CNN). The algorithm combines the establishment of a pre-decision dictionary based on statistical theory and a Convolutional Neural Network (CNN) model based on adaptively adjusting the size of the pooling layer to form an adaptive CU size division decision process. The algorithm can make a decision on whether to divide CUs of different sizes, thereby avoiding unnecessary Rate–distortion Optimization (RDO) and reducing coding time. Experimental results show that compared with the original algorithm, our suggested algorithm can save 35.60% of the coding time and only increases the Bjøntegaard Delta Bit Rate (BD-BR) by 0.91%.