High Efficiency Video Coding Research Articles

Enhanced Standard Compatible Image Compression Framework Based on Auxiliary Codec Networks.

Recent deep neural network-based research to enhance image compression performance can be divided into three categories: learnable codecs, postprocessing networks, and compact representation networks. The learnable codec has been designed for end-to-end learning beyond the conventional compression modules. The postprocessing network increases the quality of decoded images using example-based learning. The compact representation network is learned to reduce the capacity of an input image, reducing the bit rate while maintaining the quality of the decoded image. However, these approaches are not compatible with existing codecs or are not optimal for increasing coding efficiency. Specifically, it is difficult to achieve optimal learning in previous studies using a compact representation network due to the inaccurate consideration of the codecs. In this paper, we propose a novel standard compatible image compression framework based on auxiliary codec networks (ACNs). In addition, ACNs are designed to imitate image degradation operations of the existing codec, which delivers more accurate gradients to the compact representation network. Therefore, compact representation and postprocessing networks can be learned effectively and optimally. We demonstrate that the proposed framework based on the JPEG and High Efficiency Video Coding standard substantially outperforms existing image compression algorithms in a standard compatible manner.

Fast CU Partition for VVC Using Texture Complexity Classification Convolutional Neural Network

Versatile video coding (H.266/VVC), which was newly released by the Joint Video Exploration Team (JVET), introduces quad-tree plus multi-type tree (QTMT) partition structure on the basis of quad-tree (QT) partition structure in High Efficiency Video Coding (H.265/HEVC). More complicated coding unit (CU) partitioning processes in H.266/VVC significantly improve video compression efficiency, but greatly increase the computational complexity compared. The ultra-high encoding complexity has obstructed its real-time applications. In order to solve this problem, a CU partition algorithm using convolutional neural network (CNN) is proposed in this paper to speed up the H.266/VVC CU partition process. Firstly, 64 × 64 CU is divided into smooth texture CU, mildly complex texture CU and complex texture CU according to the CU texture characteristics. Second, CU texture complexity classification convolutional neural network (CUTCC-CNN) is proposed to classify CUs. Finally, according to the classification results, the encoder is guided to skip different RDO search process. And optimal CU partition results will be determined. Experimental results show that the proposed method reduces the average coding time by 32.2% with only 0.55% BD-BR loss compared with VTM 10.2.

Medical Image Compression Method Using Lightweight Multi-Layer Perceptron for Mobile Healthcare Applications

As video compression is one of the core technologies required to enable seamless medical data streaming in mobile healthcare applications, there is a need to develop powerful media codecs that can achieve minimum bitrates while maintaining high perceptual quality. Versatile Video Coding (VVC) is the latest video coding standard that can provide powerful coding performance with a similar visual quality compared to the previously developed method that is High Efficiency Video Coding (HEVC). In order to achieve this improved coding performance, VVC adopted various advanced coding tools, such as flexible Multi-type Tree (MTT) block structure which uses Binary Tree (BT) split and Ternary Tree (TT) split. However, VVC encoder requires heavy computational complexity due to the excessive Rate-distortion Optimization (RDO) processes used to determine the optimal MTT block mode. In this paper, we propose a fast MTT decision method with two Lightweight Neural Networks (LNNs) using Multi-layer Perceptron (MLP), which are applied to determine the early termination of the TT split within the encoding process. Experimental results show that the proposed method significantly reduced the encoding complexity up to 26% with unnoticeable coding loss compared to the VVC Test Model (VTM).

Neural Reference Synthesis for Inter Frame Coding

This work proposes the neural reference synthesis (NRS) to generate high-fidelity reference block for motion estimation and motion compensation (MEMC) in inter frame coding. The NRS is comprised of two submodules: one for reconstruction enhancement and the other for reference generation. Although numerous methods have been developed in the past for these two submodules using either handcrafted rules or deep convolutional neural network (CNN) models, they basically deal with them separately, resulting in limited coding gains. By contrast, the NRS proposes to optimize them collaboratively. It first develops two CNN-based models, namely EnhNet and GenNet. The EnhNet only uses spatial correlations within the current frame for reconstruction enhancement and the GenNet is then augmented by further aggregating temporal correlations across multiple frames for reference synthesis. However, a direct concatenation of EnhNet and GenNet without considering the complex temporal reference dependency across inter frames would implicitly induce iterative CNN processing and cause the data overfitting problem, leading to visually-disturbing artifacts and oversmoothed pixels. To tackle this problem, the NRS applies a new training strategy to coordinate the EnhNet and GenNet for more robust and generalizable models, and also devises a lightweight multi-level R-D (rate-distortion) selection policy for the encoder to adaptively choose reference blocks generated from the proposed NRS model or conventional coding process. Our NRS not only offers state-of-the-art coding gains, e.g., >10% BD-Rate (Bjøntegaard Delta Rate) reduction against the High Efficiency Video Coding (HEVC) anchor for a variety of common test video sequences encoded at a wide bit range in both low-delay and random access settings, but also greatly reduces the complexity relative to existing learning-based methods by utilizing more lightweight DNNs. All models are made publicly accessible at https://github.com/IVC-Projects/NRS for reproducible research.

L-Shape-Based Iterative Prediction and Residual Median Edge Detection (LIP-RMED) Algorithm for VVC Intra-Frame Lossless Compression

Versatile Video Coding (VVC) is the latest video coding standard beyond High Efficiency Video Coding (HEVC), offering higher compression efficiency. The algorithm in this letter called L-shape-based iterative prediction and residual median edge detection (LIP-RMED) divides each block into a series of close neighboring 1-D inverted L-shapes. They are employed as reference samples in the subsequent recursively L-shape prediction. Because of the inherited close neighboring characteristics of L-shapes, LIP can achieve much more accurate prediction results. To further maximize the efficiency of our prediction method, this letter also proposes a residual median edge detection (RMED) method, which allows a second process over the residual that still possesses rich edge information. Experimental results on VVC Test Model (VTM) 12.3 show that our proposed merge method is able to achieve 5.43% average bit-rate savings, with a maximum of 13.67% bit-rate savings on VVC intra-frame lossless compression.

Early Termination of CU Partition Based on Boosting Neural Network for 3D-HEVC Inter-Coding

As an extension of the High Efficiency Video Coding (HEVC) standard, the 3D-HEVC needs to encode multiple texture videos and depth maps components. In the 3D-HEVC inter-coding test model, a large variety of Coding Unit (CU) sizes are adopted to select the one with the lowest Rate-Distortion (RD) cost as the best CU size. This technique provides the highest achievable coding efficiency, but it brings a huge computational complexity which limits 3D-HEVC from practical applications. In this paper, early termination of CU encoding is proposed to reduce the complexity caused by the CU size splitting process. The proposed algorithm is based on CU homogeneity and a boosting neural network clustering algorithm. The algorithm contained three main steps. The first step is for the extraction of various features from the original encoder. Then, the selection of the features, which had a high correlation with CU partition using a machine learning algorithm. In the second step, a boosting neural network model is used for training the selected features to derive the threshold values for our proposed algorithm. In the final step, an efficient early termination of CU splitting is released for texture videos and depth maps based on the extracted thresholds from the training model. The experimental results show that the proposed algorithm reduces a significant encoding time, while the loss in coding efficiency is negligible.

Deep Convolutional Neural Network Based In- Loop Filtering for High Efficiency Video Coding

Deep Convolutional Neural Network Based In- Loop Filtering for High Efficiency Video Coding

A Commonality Modeling Framework for Enhanced Video Coding Leveraging on the Cuboidal Partitioning Based Representation of Frames

Video coding algorithms attempt to minimize the significant commonality that exists within a video sequence. Each new video coding standard contains tools that can perform this task more efficiently compared to its predecessors. Modern video coding systems are block-based wherein commonality modeling is carried out only from the perspective of the block that need be coded next. In this work, we argue for a commonality modeling approach that can provide a seamless blending between global and local homogeneity information. For this purpose, at first the frame that need be coded, is recursively partitioned into rectangular regions based on the homogeneity information of the entire frame. After that each obtained rectangular region’s feature descriptor is taken to be the average value of all the pixels’ intensities encompassing the region. In this way, the proposed approach generates a coarse representation of the current frame by minimizing both global and local commonality. This coarse frame is computationally simple and has a compact representation. It attempts to preserve important structural properties of the current frame which can be viewed subjectively as well as from improved rate-distortion performance of a reference scalable HEVC coder that employs the coarse frame as a reference frame for encoding the current frame.

Application of region-based video surveillance in smart cities using deep learning

Smart video surveillance helps to build more robust smart city environment. The varied angle cameras act as smart sensors and collect visual data from smart city environment and transmit it for further visual analysis. The transmitted visual data is required to be in high quality for efficient analysis which is a challenging task while transmitting videos on low capacity bandwidth communication channels. In latest smart surveillance cameras, high quality of video transmission is maintained through various video encoding techniques such as high efficiency video coding. However, these video coding techniques still provide limited capabilities and the demand of high-quality based encoding for salient regions such as pedestrians, vehicles, cyclist/motorcyclist and road in video surveillance systems is still not met. This work is a contribution towards building an efficient salient region-based surveillance framework for smart cities. The proposed framework integrates a deep learning-based video surveillance technique that extracts salient regions from a video frame without information loss, and then encodes it in reduced size. We have applied this approach in diverse case studies environments of smart city to test the applicability of the framework. The successful result in terms of bitrate 56.92%, peak signal to noise ratio 5.35 bd and SR based segmentation accuracy of 92% and 96% for two different benchmark datasets is the outcome of proposed work. Consequently, the generation of less computational region-based video data makes it adaptable to improve surveillance solution in Smart Cities.

A Complexity Reduction Method for VVC Intra Prediction Based on Statistical Analysis and SAE-CNN

Compared with High Efficiency Video Coding (HEVC), the latest video coding standard Versatile Video Coding Standard (VVC), due to the introduction of many novel technologies and the introduction of the Quad-tree with nested Multi-type Tree (QTMT) division scheme in the block division method, the coding quality has been greatly improved. Due to the introduction of the QTMT scheme, the encoder needs to perform rate–distortion optimization for each division mode during Coding Unit (CU) division, so as to select the best division mode, which also leads to an increase in coding time and coding complexity. Therefore, we propose a VVC intra prediction complexity reduction algorithm based on statistical theory and the Size-adaptive Convolutional Neural Network (SAE-CNN). The algorithm combines the establishment of a pre-decision dictionary based on statistical theory and a Convolutional Neural Network (CNN) model based on adaptively adjusting the size of the pooling layer to form an adaptive CU size division decision process. The algorithm can make a decision on whether to divide CUs of different sizes, thereby avoiding unnecessary Rate–distortion Optimization (RDO) and reducing coding time. Experimental results show that compared with the original algorithm, our suggested algorithm can save 35.60% of the coding time and only increases the Bjøntegaard Delta Bit Rate (BD-BR) by 0.91%.

Data-independent low-complexity KLT approximations for image and video coding

The Karhunen–Loève transform (KLT) is often used for data decorrelation and dimensionality reduction. The KLT is able to optimally retain the signal energy in only few transform components, being mathematically suitable for image and video compression. However, in practice, because of its high computational cost and dependence on the input signal, its application in real-time scenarios is precluded. This work proposes low-computational cost approximations for the KLT. We focus on the blocklengths N∈{4,8,16,32} because they are widely employed in image and video coding standards such as JPEG and high efficiency video coding (HEVC). Extensive computational experiments demonstrate the suitability of the proposed low-complexity transforms for image and video compression.

Delhi High Court unbalances the seesaw of ‘fair play’ and ‘trade secret protection’: an analysis of Interdigital ‘v’. Xiaomi

The year 2020 witnessed a tussle between an Indian and a Chinese court owing to the fair, reasonable and non-discriminatory (FRAND) licensing litigation between InterDigital (a company headquartered in the USA) and Xiaomi (a company headquartered in China). On 3 June 2020, Xiaomi initiated a suit at the Wuhan Intermediate People’s Court (Wuhan Court) seeking to determine royalty rates for numerous 3G and 4G standard essential patents (SEPs) owned by InterDigital in accordance with FRAND.1 Subsequently, InterDigital filed an infringement suit at the Delhi High Court (DHC) in respect of five of its Indian SEPs2 and three HEVC (High Efficiency Video Coding) video compression standards3 against Xiaomi.4 The Wuhan Court, by its order dated 23 September 2020, issued an anti-suit injunction (ASI) against InterDigital and proscribed them to seek an injunction against Xiaomi in the DHC. The Wuhan Court denied reconsidering its order. ASIs have been common in SEP proceedings as they reduce the expenses and avoid conflicting decisions in parallel jurisdictions.5

Video Compression for Screen Recorded Sequences Following Eye Movements

With the advent of smartphones and tablets, video traffic on the Internet has increased enormously. With this in mind, in 2013 the High Efficiency Video Coding (HEVC) standard was released with the aim of reducing the bit rate (at the same quality) by 50% with respect to its predecessor. However, new contents with greater resolutions and requirements appear every day, making it necessary to further reduce the bit rate. Perceptual video coding has recently been recognized as a promising approach to achieving high-performance video compression and eye tracking data can be used to create and verify these models. In this paper, we present a new algorithm for the bit rate reduction of screen recorded sequences based on the visual perception of videos. An eye tracking system is used during the recording to locate the fixation point of the viewer. Then, the area around that point is encoded with the base quantization parameter (QP) value, which increases when moving away from it. The results show that up to 31.3% of the bit rate may be saved when compared with the original HEVC-encoded sequence, without a significant impact on the perceived quality.

Time Delay Optimization of Compressing Shipborne Radar Digital Video Based on Deep Learning

The High Efficiency Video Coding Standard (HEVC) is one of the most advanced coding schemes at present, and its excellent coding performance is highly suitable for application in the navigation field with limited bandwidth. In recent years, the development of emerging technologies such as screen sharing and remote control has promoted the process of realizing the virtual driving of unmanned ships. In order to improve the transmission and coding efficiency during screen sharing, HEVC proposes a new extension scheme for screen content coding (HEVC-SCC), which is based on the original coding framework. SCC has improved the performance of compressing computer graphics content and video by adding new coding tools, but the complexity of the algorithm has also increased. At present, there is no delay in the compression optimization method designed for radar digital video in the field of navigation. Therefore, our paper starts from the perspective of increasing the speed of encoded radar video, and takes reducing the computational complexity of the rate distortion cost (RD-cost) as the goal of optimization. By analyzing the characteristics of shipborne radar digital video, a fast encoding algorithm for shipborne radar digital video based on deep learning is proposed. Firstly, a coding tree unit (CTU) division depth interval dataset of shipborne radar images was established. Secondly, in order to avoid erroneously skipping of the intra block copy (IBC)/palette mode (PLT) in the coding unit (CU) division search process, we designed a method to divide the depth interval by predicting the CTU in advance and limiting the CU rate distortion cost to be outside the traversal calculation depth interval, which effectively reduced the compression time. The effect of radar transmission and display shows that, within the acceptable range of Bjøntegaard Delta Bit Rate (BD-BR) and Bjøntegaard Delta Peak Signal to Noise Rate (BD-PSNR) attenuation, the algorithm proposed in this paper reduces the coding time by about 39.84%, on average, compared to SCM8.7.

Residual-guided In-loop Filter Using Convolution Neural Network

The block-based coding structure in the hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing, and so on. To compensate for those artifacts, extensive filtering techniques were proposed in the loop of video codecs, which are capable of boosting the subjective and objective qualities of reconstructed videos. Recently, neural network-based filters were presented with the power of deep learning from a large magnitude of data. Though the coding efficiency has been improved from traditional methods in High-Efficiency Video Coding (HEVC), the rich features and information generated by the compression pipeline have not been fully utilized in the design of neural networks. Therefore, in this article, we propose the Residual-Reconstruction-based Convolutional Neural Network (RRNet) to further improve the coding efficiency to its full extent, where the compression features induced from bitstream in form of prediction residual are fed into the network as an additional input to the reconstructed frame. In essence, the residual signal can provide valuable information about block partitions and can aid reconstruction of edge and texture regions in a picture. Thus, more adaptive parameters can be trained to handle different texture characteristics. The experimental results show that our proposed RRNet approach presents significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating that residual signal plays a significant role in enhancing video frame reconstruction.

Decision tree accelerated CTU partition algorithm for intra prediction in versatile video coding.

Versatile video coding (VVC) achieves enormous improvement over the advanced high efficiency video coding (HEVC) standard due to the adoption of the quadtree with nested multi-type tree (QTMT) partition structure and other coding tools. However, the computational complexity increases dramatically as well. To tackle this problem, we propose a decision tree accelerated coding tree units (CTU) partition algorithm for intra prediction in VVC. Firstly, specially designated image features are extracted to characterize the coding unit (CU) complexity. Then, the trained decision tree is employed to predict the partition results. Finally, based on our newly designed intra prediction framework, the partition process is early terminated or redundant partition modes are screened out. The experimental results show that the proposed algorithm could achieve around 52% encoding time reduction for various test video sequences on average with only 1.75% Bjontegaard delta bit rate increase compared with the reference test model VTM9.0 of VVC.

Design and Implementation of an Efficient Multi-Pattern Motion Estimation Search Algorithm for HEVC/H.265

Motion Estimation (ME) is the most computationally intensive block in high efficiency video coding (HEVC) due to its complex partition schemes. It consumes a large amount of power in the encoder. So designing a ME module on hardware platform imposes significant challenges. Many hardware oriented integer ME (IME) algorithms have been proposed in the literature to meet the challenges but suffer from coding efficiency degradation. In this paper we propose a low complexity IME algorithm and its hardware implementation. The proposed algorithm has been shown for two different pattern structures (PS) with 38 search points. It achieves bjontegaard bitrate (BD-BR) of 0.5% decrease and 0.077% increase for two PSs compared to test zone search (TZS) in HM 16.8. It results 8.725% and 9.072% saving in encoding time respectively. The proposed architecture supports all the HEVC partitions. It provides real time encoding of 4096 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 2160 @60 fps operated at the maximum frequency of 353 MHz with 1591 K NAND equivalent gate count and 8.32 kB on chip memory using 90 nm technology library. Therefore, it can be used in consumer electronics applications such as security surveillance, video recording systems etc. that require high efficient HEVC encoder.

Lightweight Hardware Transform Design for the Versatile Video Coding 4K ASIC Decoders

Versatile Video Coding (VVC) is the next generation video coding standard finalized in July 2020. VVC introduces new coding tools enhancing the coding efficiency compared to its predecessor High Efficiency Video Coding (HEVC). These new tools have a significant impact on the VVC software decoder complexity estimated to 2 times HEVC decoder complexity. In particular, the transform module includes in VVC separable and non-separable transforms named Multiple Transform Selection (MTS) and Low Frequency Non-Separable Transform (LFNST) tools, respectively. In this paper, we present an area-efficient hardware architecture of the inverse transform module for a VVC decoder. The proposed design uses a total of 64 regular multipliers in a pipelined architecture targeting Application-Specific Integrated Circuit (ASIC) platforms. It consists in a multi-standard architecture that supports the transform modules of recent MPEG standards including Advanced Video Coding (AVC), HEVC and VVC. The architecture leverages all primary and secondary transforms’ optimisations including butterfly decomposition, coefficients zeroing and the inherent linear relationship between the transforms. The synthesized results show that the proposed method sustains a constant throughput of 1 sample per cycle and a constant latency for all block sizes. The proposed hardware inverse transform module operates at 600 MHz frequency enabling to decode in real-time 4K video at 30 frames per second in 4:2:2 chroma sub-sampling format. The proposed module has been integrated in an ASIC UHD decoder targeting energy-aware decoding of VVC videos on consumer devices.

HEVC Information Hiding Algorithm Based on Intra Prediction and Matrix Coding

Aiming at the problem that the data hiding algorithm of high efficiency video coding (HEVC) has great influence on the video bit rate and visual quality, an information hiding algorithm based on intra prediction mode and matrix coding is proposed. Firstly, 8 prediction modes are selected from 4×4 luminance blocks in I frame to embed the hidden information. Then, the Least Significant Bit (LSB) algorithm is used to modulate the LSB of the last prediction mode. Finally, the modulated luminance block is re-encoded to embed 4 bits secret information. Experimental results show that the algorithm improves the embedding capacity, guarantees the subjective and objective quality of the video, and the bit rate increases by 1.14% on average.

Video Steganography Using Chaos Encryption Algorithm with High Efficiency Video Coding for Data Hiding

Video Steganography Using Chaos Encryption Algorithm with High Efficiency Video Coding for Data Hiding