High Efficiency Video Coding Research Articles

A Study on Fast and Low-Complexity Algorithms for Versatile Video Coding.

Versatile Video Coding (VVC)/H.266, completed in 2020, provides half the bitrate of the previous video coding standard (i.e., High-Efficiency Video Coding (HEVC)/H.265) while maintaining the same visual quality. The primary goal of VVC/H.266 is to achieve a compression capability that is noticeably better than that of HEVC/H.265, as well as the functionality to support a variety of applications with a single profile. Although VVC/H.266 has improved its coding performance by incorporating new advanced technologies with flexible partitioning, the increased encoding complexity has become a challenging issue in practical market usage. To address the complexity issue of VVC/H.266, significant efforts have been expended to develop practical methods for reducing the encoding and decoding processes of VVC/H.266. In this study, we provide an overview of the VVC/H.266 standard, and compared with previous video coding standards, examine a key challenge to VVC/H.266 coding. Furthermore, we survey and present recent technical advances in fast and low-complexity VVC/H.266, focusing on key technical areas.

Analysis of coding unit partitioning and complexity reduction at intra-prediction mode of HEVC

ABSTRACT The emerging trend of ultra-high definition (4 K, UHD) video services necessitates coding efficiency, which is much higher than H.264. High-Efficiency Video Coding (HEVC) has been designed with the intention of drastically reducing the bit rate without compromising the video quality. The key feature is the introduction of a larger block structure with flexible block sizes from 64 × 64 to 8 × 8 samples. The enhanced feature has imposed higher complexity, specifically at the intra-prediction mode level of HEVC. The proposed research work has presented a novel hybrid approach to reducing the number of candidate Coding Units (CU) passing through irrelevant partition patterns and time-consuming iterative Rate-Distortion (RD) cost estimation process. The hybrid approach is implemented in two steps. In the first step gradient evaluation is performed and the CU is categorized as homogeneous or non-homogeneous. In the second step Early Termination is adopted by using machine learning for partitioning non-homogeneous CU. As the number of candidate Coding Units undergoing the time-consuming RD process is reduced by using the two-step hybrid approach, the proposed approach proves to be the best of its kind. The reduction in the number of candidate units undergoing the RD process reduces the complexity of the intra-prediction process. Encoding time is the measure of complexity and the proposed hybrid approach reduces the encoding time by an average of 36% compared with the HEVC reference model and also by an average of 26% as compared with the previous work. The Peak Signal to Noise Ratio (PSNR) reduced by an average of 0.1% indicates a slight reduction in the encoded image quality. The proposed approach significantly reduced the encoding time without affecting the RD performance of the encoder. The image quality is not compromised substantially when considering the improvement in the encoding time and the reduction in the complexity at an intra-prediction level.

An Efficient Video Encryption Algorithm Based on the Pseudorandom Number Generator of Zipf Distribution

In this study, a pioneer selective video encryption (PSVE) algorithm is proposed based on the pseudorandom number generator (PRNG) of the Zipf distribution (Z-PRNG). It is a general algorithm with high efficiency and security. The encryption process is completely separable from the video coding process. In the PSVE algorithm, Z-PRNG is designed based on the 3D SCL-HMC hyperchaotic map. Firstly, encapsulated byte sequence payloads (EBSPs) are extracted from the video bitstream. Secondly, random numbers of the Zipf distribution are generated by Z-PRNG, and they are used to randomly select encrypted data from each EBSP. Lastly, the extracted data are encrypted by AES-CTR to obtain the encrypted video. Compared with existing algorithms, the encryption position is more flexible, and the key space is further enhanced. The high efficiency video coding (HEVC) video and the advanced video coding (AVC) video are taken as examples to test the PSVE algorithm. The analysis results show that the proposed scheme can effectively resist common attacks, and its time complexity is much less than most existing algorithms.

GiTMO: An HDR10 Compatible Generic Tone-Mapping for Real-Time Video Play

Though the market of HDR displays for consumers is gaining maturity, low dynamic range (LDR) displays still hold a majority share in the consumer market. Many tone-mapping operators (TMO) have been developed to convert the HDR contents into a form suitable for LDR displays. This work proposes a generic tone-mapping operator (GiTMO) that can represent all global TMOs in a generic form of a one-dimensional array of a few coefficients. We use these coefficients and the existing HDR10 delivery pipeline to produce a tone-mapped version of the HDR content at the receiver end in real time. In our framework, coefficients are sent to the receiver as supplemental enhancement information (SEI) message of High Efficiency Video Coding (HEVC). Thus, HDR10, which is designed for HDR display, becomes compatible with LDR displays too. This can relieve the content producers from the need to release separate copies of HDR and LDR content. GiTMO, being generic and efficient, can generate LDR faithfully for all global TMOs, making the consumer end oblivious of the TMO used at the transmitter side. We perform extensive evaluations to validate the accuracy and efficiency of the GiTMO and the proposed content delivery format, and their benefits over the existing approaches.

An Adjacency Encoding Information-Based Fast Affine Motion Estimation Method for Versatile Video Coding

Versatile video coding (VVC), a new generation video coding standard, achieves significant improvements over high efficiency video coding (HEVC) due to its added advanced coding tools. Despite the fact that affine motion estimation adopted in VVC takes into account the translational, rotational, and scaling motions of the object to improve the accuracy of interprediction, this technique adds a high computational complexity, making VVC unsuitable for use in real-time applications. To address this issue, an adjacency encoding information-based fast affine motion estimation method for VVC is proposed in this paper. First, this paper counts the probability of using the affine mode in interprediction. Then we analyze the trade-off between computational complexity and performance improvement based on statistical information. Finally, by exploring the mutual exclusivity between skip and affine modes, an enhanced method is proposed to reduce interprediction complexity. Experimental results show that compared with the VVC, the proposed low-complexity method achieves 10.11% total encoding time reduction and 40.85% time saving of affine motion estimation with a 0.16% Bjøontegaard delta bitrate (BDBR) increase.

Hardware implementation of HEVC CABAC binarization/de-binarization

High efficiency video coding (HEVC) video codec applies different techniques in order to achieve high compression ratios and video quality that supports real-time applications. One of the critical techniques in HEVC is the Context adaptive Binary Arithmetic Coding (CABAC) which is type of entropy coding. CABAC comes at the cost of increased computational complexity, especially for parallelization and pipeline of these blocks: binarization, context modeling and binary arithmetic encoding. The Binarization (BZ) and de-Binarization (DBZ) methods are considered as important techniques in HEVC CABAC encoder and decoder respectively. Indeed, an important goal is to get high throughput in hardware architectures of CABAC BZ and DBZ in order to achieve high resolution applications. This work is the only one found on recent literature which focuses on design and implementation of full BZ and full DBZ compatible with H.265 and H.264. Consequently, a hardware architectures of BZ and DBZ are designed and implemented by using VHDL language, targeted an FPGA virtex4 xc4vsx25-12ff668 board and emulated with ModelSim. As a result, the implementation of BZ and DBZ can process 2 bins/cycle for each syntax element when operated at 697.83 MHz and 789.26 MHz, respectively. The proposed designs exhibits an improved high-throughput of 1395.66 Mbins/s for BZ and 1578.52 Mbins/s for the DBZ. The obtained Area Efficiencies in our proposed BZ and DBZ are about 0.544 Mbins/s/slices and 0.606 Mbins/s/slices, respectively, and it is better than many recent works.

A quality enhancement network with coding priors for constant bit rate video coding

Video coding can effectively compress data, while the introduction of compression artifacts degrades the visual quality and the performance of artificial intelligence video applications. Video quality enhancement (VQE) methods can improve compressed video quality without modifying the coding standard’s main modules. The existing VQE works mainly aim to improve video quality in the constant quantization parameter (CQP) coding mode. However, constant bit rate (CBR) coding mode is widely used in some streaming playback video applications, and VQE at CBR is more challenging than that at CQP. This article presents a novel Constant bit rate Video quality enhancement Network combined with Coding priors (CVCN). The proposed CVCN can be inserted into High Efficiency Video Coding (HEVC) codec as a CNN-based in-loop filter (LF) module or a post-processing module out of the codec. Moreover, we design a two-pass training strategy for the LF module to overcome multiple filtering. To adapt the QP diversity of CBR videos, we utilize the coding unit (CU)-wise QP prior by constructing a CU-wise QP adaptive module (CQAM) and a QP adaptive multi-scale residual block (QAMSRB) based on CQAM. We construct the CU-partition prior (CPP) by exploiting the relationship between the compression noise and CU partition information. A novel CPP spatial-attention block is proposed to combine the CPP with the self-attention module. Furthermore, the proposed CVCN can effectively enhance CBR videos at different bits per pixel (BPP) via a single model. Extensive experimental results show the superiority of the CVCN over state-of-the-art VQE approaches for CBR videos.

Fast intra partition and mode prediction for equirectangular projection 360‐degree video coding

360-degree videos have drawn great attention from both the academia and the industry. For transmission and storage of 360-degree video, the joint video exploration team proposed the Versatile Video Coding (VVC) standard. VVC can significantly reduce the bitrate while maintaining the same subjective visual quality compared to the preceding high efficiency video coding. However, the computational complexity of VVC is extremely high which hinders the interactive applications. This paper proposes a fast intra partition method and mode prediction algorithm for equirectangular projection 360-degree video coding. First, a latitude-based preprocessing is introduced to early terminate the Coding Unit (CU) partition in the polar region. Second, the support vector machine is used to predict the CU partition type. Third, the fast intra mode search method accelerates the intra mode prediction. Experimental results show that the proposed algorithm can significantly obtain an average time reduction rate of 60.40% and a Bjontegaard delta rate increase of 1.96%.

Dynamic convolutional capsule network for In‐loop filtering in HEVC video codec

Recently, several in-loop filtering algorithms based on convolutional neural network (CNN) have been proposed to improve the efficiency of HEVC (High Efficiency Video Coding). Conventional CNN-based filters only apply a single model to the whole image, which cannot adapt well to all local features from the image. To solve this problem, an in-loop filtering algorithm based on a dynamic convolutional capsule network (DCC-net) is proposed, which embeds localized dynamic routing and dynamic segmentation algorithms into capsule network, and integrate them into the HEVC hybrid video coding framework as a new in-loop filter. The proposed method brings average 7.9% and 5.9% BD-BR reductions under all intra (AI) and random access (RA) configurations, respectively, as well as, 0.4 dB and 0.2 dB BD-PSNR gains, respectively. In addition, the proposed algorithm has an outstanding performance in terms of time efficiency.

Comparative Objective Analysis of video quality Between H.265/HEVC and H.264/AVC

This paper shows the difference in video quality between two compressed videos using H.264 AVC (Advanced Video Coding) and H.265 HEVC (High Efficiency Video Coding) encoders. To evaluate video completely it should be prepared video files that have a variety of bit rates and content. There are many video quality assessment methods. We can divide the min to subjective and objective methods. Subjective are conducted by a human perception and objective are conducted by a computer software which is calculating the video quality. All of these methods have theirs advantages and disadvantages. To generate compressed videos from the original video FFmpeg (Fast Forward-moving picture experts group) converter has been used. MSU-VQMT (Moscow State University’s Video Quality Measure- mentTool) was used to perform comparative objective analysis of video quality. Delta, MSE (mean square Error), MSAD (Mean Sum of Absolute Difference), PSNR (Peak Signal-to-Noise Ratio), and SSIM (Structural Similarity Index Measure) metrics were measured. The result from FFmpeg shows that the size of the compressed video using the H.265 codec has been decreased by 50% compared to the compressed video using the H.264 codec. The comparison of metrics shows that delta, MSAD, PSNR, and SSIM values of H.265 encoded video was decreased, while Delta and MSE value was increased compared to H.264 encoded Video. That’s mean the overall video quality was decreased but the video size was enhanced.

High-Quality Video Watermarking Based on Deep Neural Networks for Video with HEVC Compression.

This article presents a method for transparent watermarking of high-capacity watermarked video under H.265/HEVC (High-Efficiency Video Coding) compression conditions while maintaining high-quality encoded image. The aim of this paper is to present a method for watermark embedding using neural networks under conditions of subjecting video to lossy compression of the HEVC codec using the YUV420p color model chrominance channel for watermarking. This paper presents a method for training a deep neural network to embed a watermark when a compression channel is present. The discussed method is characterized by high accuracy of the video with an embedded watermark compared to the original. The PSNR (peak signal-to-noise ratio) values obtained are over 44 dB. The watermark capacity is 96 bits for an image with a resolution of 128 × 128. The method enables the complete recovery of a watermark from a single video frame compressed by the HEVC codec within the range of compression values defined by the CRF (constant rate factor) up to 22.

Intraprediction Complexity Control Algorithm Based on Visual Saliency

Intraprediction is one of the most complex parts of High-Efficiency Video Coding (HEVC), because it selects the best prediction mode by calculating the cost of every Coding Unit (CU), which provides higher complexity of intracoding. Visual saliency map can show the attention regions of the human eyes, which is generated by certain static and space-time saliency detection method. By analyzing the percentage of coding time for different size CU, and the relation of visual saliency and CU depth, an intraprediction complexity control algorithm based on visual saliency is proposed in this paper. Based on the feature of the video and the target level, the saliency threshold is adapted to determine whether the current CU in the intraprediction processing should be split into smaller CUs or the division processing should be stopped early. Three samples were compared by the proposed algorithm and other algorithm, and the proposed algorithm has better performance in PSNR, BitRate, and coding time. Experimental results show that this algorithm can effectively control the coding complexity of intraprediction with minimal visual loss and can be applied to a number of scenarios, such as real-time video coding.

Transformer-Based Data-Driven Video Coding Acceleration for Industrial Applications

With the exploding development of edge intelligence and smart industry, deep learning-based intelligent industrial solutions are promptly applied in the manufacturing process. Many intelligent industrial solutions such as automatic manufacturing inspection are computer vision based and require fast and efficient video encoding techniques so that video streams can be processed as quickly as possible either at the edge cluster or over the cloud. As one of the most popular video coding standards, the high efficiency video coding (HEVC) standard has been applied to various industrial scenes. However, HEVC brings not only a higher compression rate but also a significant increase in encoding complexity, which hinders its practical application in industrial scenarios. Fortunately, a large amount of video coding data makes it possible to accelerate the encoding process in the industry. To speed up the video coding process in some industrial scenes, this paper proposes a data-driven fast approach for coding tree unit (CTU) partitioning in HEVC intracoding. First, we propose a method to represent the partition result of a CTU as a column vector of length 21. Then, we employ lots of encoding data produced in normal industry scenes to train transformer models used to predict the partitioning vector of the CTU. Finally, the final partitioning structure of the CTU is generated from the partitioning vector after a postprocessing operation and used by an industrial encoder. Compared with the original HEVC encoder used by some industrial applications, experiment results show that our approach achieves 58.77% encoding time reduction with 3.9% bit rate loss, which indicates that our data-driven approach for video coding has great capacity working in industrial applications.

Fast CU Partition Decision Algorithm for VVC Intra Coding Using an MET-CNN

The newest video coding standard, the versatile video coding standard (VVC/H.266), came into effect in November 2020. Different from the previous generation standard—high-efficiency video coding (HEVC/H.265)—VVC adopts a more flexible block division structure, the quad-tree with nested multi-type tree (QTMT) structure, which improves its coding performance by 24%. However, it also causes a substantial increase in computational complexity. Therefore, this paper first proposes the concept of a stage grid map, which divides the overall division of a 32 × 32 coding unit (CU) into four stages and represents it as a structured output. Second, a multi-stage early termination convolutional neural network (MET-CNN) model is devised to predict the full partition information of a CU with a size of 32 × 32. Finally, a fast CU partition decision algorithm for VVC intra coding based on an MET-CNN is proposed. The algorithm can predict all partition information of a CU with a size of 32 × 32 and its sub-CUs in one run, completely replacing the complex rate-distortion optimization (RDO) process. It also has an early exit mechanism, thereby greatly reducing the encoding time. The experimental results illustrate that the scheme proposed in this paper reduces the encoding time by 49.24% on average, while the Bjøntegaard Delta Bit Rate (BDBR) only increases by 0.97%.

A Fast Decision Algorithm for VVC Intra-Coding Based on Texture Feature and Machine Learning.

Due to the development and application of information technology, a series of modern information technologies represented by 5G, big data, and artificial intelligence are changing rapidly, and people's requirements for video coding standards have become higher. In the High-Efficiency Video Coding (HEVC) standard, the coding block division is not flexible enough, and the prediction mode is not detailed enough. A new generation of Versatile Video Coding (VVC) standards was born. VVC inherits the hybrid coding framework adopted by HEVC, improves the original technology of each module, introduces a series of new coding technologies, and builds on this greatly improving the coding efficiency. Compared with HEVC, the block division structure of VVC has undergone great changes, retaining the quad-tree (QT) division method and increasing the multi-type tree (MTT) division method, which brings high coding complexity. To reduce the computational complexity of VVC coding block division, a fast decision algorithm for VVC intra-frame coding based on texture characteristics and machine learning is proposed. First, we analyze the characteristics of the CU partition structure decision and then use the texture complexity of the CU partition structure decision to terminate the CU partition process early; for CUs that do not meet the early termination of the partition, use the global sample information, local sample information, and context information. The three-category feature-trained tandem classifier framework predicts the division type of CU. The experimental results show that in the full intra mode, compared with the existing VTM10.0, the encoding output bit rate is increased by 1.36%, and the encoding time is saved by 52.63%.

Fast Decision Algorithm of CU Size for HEVC Intra-Prediction Based on a Kernel Fuzzy SVM Classifier

High Efficiency Video Coding (HEVC) achieves a significant improvement in compression efficiency at the cost of extremely high computational complexity. Therefore, large-scale and wide deployment applications, especially mobile real-time video applications under low-latency and power-constrained conditions, are more challenging. In order to solve the above problems, a fast decision method for intra-coding unit size based on a new fuzzy support vector machine classifier is proposed in this paper. The relationship between the depth levels of coding units is accurately expressed by defining the cost evaluation criteria of texture and non-texture rate-distortion cost. The fuzzy support vector machine is improved by using the information entropy measure to solve the negative impact of data noise and the outliers problem. The proposed method includes three stages: the optimal coded depth level “0” early decision, coding unit depth early skip, and optimal coding unit early terminate. In order to further improve the rate-distortion complexity optimization performance, more feature vectors are introduced, including features such as space complexity, the relationship between coding unit depths, and rate-distortion cost. The experimental results showed that, compared with the HEVC reference test model HM16.5, the proposed algorithm can reduce the encoding time of various test video sequences by more than 53.24% on average, while the Bjontegaard Delta Bit Rate (BDBR) only increases by 0.82%. In addition, the proposed algorithm is better than the existing algorithms in terms of comprehensively reducing the computational complexity and maintaining the rate-distortion performance.

Fast mode decision for CU coding based on CNN for the VVC standard

Versatile Video Coding (VVC) is the latest generation of the video coding standard. In VVC, the advanced quadtree with a nested multitype tree (QTMT) partition structure provides more flexible coding unit (CU) partition sizes compared with the quadtree (QT) decision tree structure applied in the previous High Efficiency Video Coding (HEVC) standard. This flexibility, achieved by the new QTMT partitioning improvement, considerably improves the coding performance while increasing the coding computational complexity caused mainly by the rate distortion optimization processing. To overcome the complexity issue, a fast deep intra QTMT decision tree approach based on a convolution neural network (CNN) is adopted to determine the QTMT depth decision of each 128 × 128 Coding Tree Unit (CTU). The proposed algorithm predicts both the BT depths at 32 × 32 CUs and the QT depths at 64 × 64 using trained CNNs designed for each structure instead of processing the RDcost. Experimental results prove that the suggested deep QTMT approach achieves an important complexity reduction of up to 55.51% compared with the original reference software VTM3.0, with an average of about 35% encoding time reduction accompanied by an insignificant loss in encoding performance.

Scalable extension of HEVC with adaptive de-blocking filter for bandwidth-limited applications

A primary latest video coding standard H.265/SHVC (Scalable High Efficiency Video Coding), an extension of High Efficiency Video Coding (HEVC) is dealt in this paper. To transfer high quality video over any network that usually occupies large bandwidth, suffers bandwidth limitation. It can be controlled by trading-off between the quality and the coding efficiency. The trade-off between these two parameters is considered in this paper to keep the difference at the minimum level. A novel Neural Network based Adaptive De-blocking Filter (NNADF) technique is proposed to maintain good PSNR with minimum bit rate in SHVC encoder. The adaptive de-blocking filter (ADF) performs block matching for nonlinear predictions to all frames in the video. Moreover, the ADF performs frame de-noising hence good quality of the video is achieved and trains a neural network (NN) model. The NN model restrains their outputs to achieve bit rate reduction, thereby coding efficiency is achieved. The simulation results show that the proposed NNADF technique delivers an average increment of 0.62 dB in BD-PSNR and an average decrement of 24 % in BD-BR for combined quality and spatial ratios respectively compared with the previously proposed algorithm.

A high-performance two-dimensional transform architecture of variable block sizes for the VVC standard.

The versatile video coding standard H.266/VVC release has been accompanied with various new contributions to improve the coding efficiency beyond the high-efficiency video coding (HEVC), particularly in the transformation process. The adaptive multiple transform (AMT) is one of the new tools that was introduced in the transform module. It involves five transform types from the discrete cosine transform/discrete sine transform families with larger block sizes. The DCT-II has a fast computing algorithm, while the DST-VII relies on a complex matrix multiplication. This has led to an additional computational complexity. The approximation of the DST-VII can be used for the transform optimization. At the hardware level, this method can provide a gain in power consumption, logic resources use and speed. In this paper, a unifed two-dimensional transform architecture that enables exact and approximate DST-VII computation of sizes 8times 8, 8times 16, 8times 32, 16times 8, 16times 16, 16times 32, 32times 8, 32times 16 and 32times 32 is proposed. The exact transform computation can be processed using either multipliers or the MCM algorithm, while the approximate transform computation is based on additions and bit-shifting operations. All the designs are implemented under the Arria 10 FPGA device. The synthesis results show that the proposed design implementing the approximate transform matrices is the most efficient method with only 4% of area consumption. It reduces the logic utilization by more than 65% compared to the multipliers-based exact transform design, while about 53% of hardware cost saving is obtained when compared to the MCM-based computation. Furthermore, the approximate-based 2D transform architecture can operate at 78 MHz allowing a real-time coding for 2K and 4K videos at 100 and 25 frames/s, respectively.

Anti-recompression video watermarking algorithm without bit-rate increase

Watermarking technology based on high-efficiency video coding (HEVC) is evolving such that it can be used in various scenarios, such as copyright protection, ownership certification, tamper detection, and authentication. However, video watermarking algorithms based on HEVC still have poor resistance to recompression attack, which seriously affects the practicability of watermarking algorithms. To solve this limitation, we propose a robust HEVC watermarking algorithm based on the analysis of the characteristics of HEVC recompression. The influencing factors of choosing embedding carrier are discussed, and the calculation formula of texture complexity is then deduced. According to the texture complexity, the watermark is embedded into the stable 4 × 4 luma blocks. Specifically, the watermark information is completely embedded after modulating the signs of the quantized discrete sinusoidal transform coefficients of the carrier. Experimental results show that the algorithm is robust to the recompression attack with constant quantization parameter. There is no obvious artifact after watermark embedding, which reflects the proposed algorithm has good imperceptibility. In addition, the embedding and extraction process of the algorithm is simple and fast, and the embedding of the watermark does not increase the video bit rate.