High Efficiency Video Coding Anchor Research Articles

Performance enhancement of HEVC lossless mode using context-based angular and planar intra predictions

Lossless mode of High-Efficiency Video Coding (HEVC), the state-of-the-art video coding standard, can be used for distortion-free reconstruction of the input data for a wide variety of applications. HEVC relies on the usage of efficient intra prediction strategies to achieve superior compression than its predecessor H.264. A large amount of spatial redundancy exists in almost all video sequences due to coherence, smoothness and the inherent correlation within the neighboring pixels. In this paper, a context-based intra prediction scheme is proposed to minimize this local redundancy by identifying the edges and textures to appropriately modify the prediction strategy at the pixel level, without further increase in the computational complexity. The variability in the sum of absolute differences and local pixel intensity values are chosen to derive the context of the nearby region around the target pixel in the planar and angular intra prediction modes respectively. The experimental results validate the superiority of the proposed method over the HEVC anchor and other state-of-the-art techniques in the literature.

Convolutional Neural Network-Based Block Up-Sampling for HEVC

Recently, convolutional neural network (CNN)-based methods have achieved remarkable progress in image and video super-resolution, which inspires research on down-/up-sampling-based image and video coding using CNN. Instead of hand-crafted filters for up-sampling, trained CNN models are believed to be more capable of improving image quality, thus leading to coding gain. However, previous studies either concentrated on intra-frame coding or performed down- and up-sampling of entire frame. In this paper, we introduce block-level down- and up-sampling into inter-frame coding with the help of CNN. Specifically, each block in the P or B frame can either be compressed at the original resolution or down-sampled and compressed at low resolution and then, up-sampled by the trained CNN models. Such block-level adaptivity is flexible to cope with the spatially variant texture and motion characteristics. We further investigate how to enhance the capability of CNN-based up-sampling by utilizing reference frames and study how to train the CNN models by using encoded video sequences. We implement the proposed scheme onto the high efficiency video coding (HEVC) reference software and perform a comprehensive set of experiments to evaluate our methods. The experimental results show that our scheme achieves superior performance to the HEVC anchor, especially at low bit rates, leading to an average 3.8%, 2.6%, and 3.5% BD-rate reduction on the HEVC common test sequences under random-access, low-delay B, and low-delay P configurations, respectively. When tested on high-definition and ultrahigh-definition sequences, the average BD-rate exceeds 5%.

Pixelwise improvised blend of predictors in HEVC lossless mode

The commendable work by the two video coding pioneers ISO/IEC and ITU-T, to handle the next-generation of multimedia services has led to the evolution of High Efficiency Video Coding (HEVC) standard. The lossless mode of HEVC is essential when no loss in fidelity is desired to aide most of the real-world applications like video analytics, web collaboration, remote desktop sharing, etc. The proposed work intends to improvise the HEVC intra prediction scheme through the application of the heuristic history-based blend of predefined sub-predictors, while in lossless mode. The prime element of the locally adaptive mechanism is the derivation of the penalizing factors that are imposed on the sub-predictors, based on the neighborhood residuals. The experimental analysis highlights that the proposed method outperforms the lossless mode of HEVC anchor and the prevalent state-of-the-art prediction techniques in terms of savings in bit-rate which is achieved without any increase in run-time.

Deep Learning-Based Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Convolutional Neural Network-Based Arithmetic Coding for HEVC Intra-Predicted Residues

Entropy coding is a fundamental technology in video coding that removes statistical redundancy among syntax elements. In high efficiency video coding (HEVC), context-adaptive binary arithmetic coding (CABAC) is adopted as the primary entropy coding method. The CABAC consists of three steps: binarization, context modeling, and binary arithmetic coding. As the binarization processes and context models are both manually designed in CABAC, the probability of the syntax elements may not be estimated accurately, which restricts the coding efficiency of CABAC. To address the problem, we propose a convolutional neural network-based arithmetic coding (CNNAC) method and apply it to compress the syntax elements of the intra-predicted residues in HEVC. Instead of manually designing the binarization processes and context models, we propose directly estimating the probability distribution of the syntax elements with a convolutional neural network (CNN), as CNNs can adaptively build complex relationships between inputs and outputs by training with a lot of data. Then, the values of the syntax elements, together with their estimated probability distributions, are fed into a multi-level arithmetic codec to perform entropy coding. In this paper, we have utilized the CNNAC to code the syntax elements of the DC coefficient; the lowest frequency AC coefficient; the second, third, fourth, and fifth lowest frequency AC coefficients; and the position of the last non-zero coefficient in the HEVC intra-predicted residues. The experimental results show that our proposed method achieves up to 6.7% BD-rate reduction and an average of 4.7% BD-rate reduction compared to the HEVC anchor under all intra (AI) configuration.

Invertibility-Driven Interpolation Filter for Video Coding.

Motion compensation with fractional motion vector has been widely utilized in the video coding standards. The fractional samples are usually generated by fractional interpolation filters. Traditional interpolation filters are usually designed based on the signal processing theory with the assumption of band-limited signal, which cannot effectively capture the non-stationary property of video content and cannot adapt to the variety of video quality. In this paper, we reveal an intuitive property of the fractional interpolation problem, named invertibility. That is, the fractional interpolation filters should not only generate fractional samples from integer samples but also recover the integer samples from the fractional samples in an invertible manner. We prove in theory that the invertibility in the spatial domain is equivalent to the constant magnitude in the Fourier transform domain. Driven by the invertibility, we then develop a learning-based method to solve the fractional interpolation problem. Inspired by the advances of convolutional neural network (CNN), we propose to establish an end-to-end scheme using CNN to train invertibility-driven interpolation filter (InvIF). Different from the previous learning-based methods, the proposed training scheme does not need hand-crafted "ground truth" of fractional samples. The proposed InvIF is integrated into high efficiency video coding (HEVC), and extensive experiments are conducted to verify its effectiveness. The experimental results show that the proposed method can achieve on average 4.7% and 3.6% BD-rate reduction compared with the HEVC anchor, under low-delay-B and random-access configurations, respectively.

Traffic surveillance video coding with libraries of vehicles and background

This paper presents a video coding scheme tailored for traffic surveillance videos, which features a pre-built library that is utilized in both encoder and decoder to pursue higher compression efficiency. We are motivated by the observation that, in traffic surveillance videos, not only the background is steady for a long while, but also the foreground (e.g. vehicles) contains high redundancy. For example, in the video taken by a traffic monitoring camera, we can observe that the passing-through vehicles are usually similar. However, the redundancy in the vehicles and the background is not fully exploited in the current video coding schemes. To address this problem, we propose a library-based video coding scheme. Specifically, for each static monitoring camera, we can collect video in a period, and build a library of vehicles and background from that video. When encoding the following video of that camera, we utilize the pre-built library by searching similar vehicles and background from the library and using the retrieved vehicles and background to help encode. Accordingly, the decoder also refers to the same library to reconstruct the video. We design efficient algorithms for building the library, searching in the library, as well as utilizing the library for encoding/decoding, to fulfill the proposed scheme. Our scheme is implemented upon the state-of-the-art video coding system–High Efficiency Video Coding (HEVC), and is tested on our own collected traffic surveillance videos. Experimental results show that, compared to the HEVC anchor, our proposed scheme achieves as high as 47.0%, 37.1%, and 34.8% BD-rate reduction, under random-access, low-delay B, and low-delay P settings, respectively. Our scheme also achieves higher compression efficiency than the existing background-based methods for surveillance video coding.

Reference Clip for Inter Prediction in Video Coding

Inter prediction is a fundamental technology in video coding to remove the temporal redundancy between video frames. Traditionally, the reconstructed frames are directly put into a reference frame buffer to serve as references for inter prediction. Using multiple reference frames increases the accuracy of inter prediction, but also incurs a waste of memory of the buffer since the content of reference frames is highly similar. To address this problem, we propose to organize the references at clip level in addition to frame level, i.e. the reference buffer stores not only reference frames, but also reference clips that are cropped regions selected from the reconstructed frames. Using clip-level references, we can manage the reference content more economically, since the content of multiple reference frames is divided into the singular content of each frame as well as the repetitive content that appears in multiple frames. For the repetitive content, only one copy is stored in reference clips so as to avoid duplicate. Moreover, using reference clips also facilitates the bit-rate allocation among reference content, i.e. the quality of each clip can be decided adaptively to achieve the rate-distortion optimization. In this paper, we propose a complete video coding framework using reference clips, and investigate the problems including how to generate reference clips as either singular content clips or repetitive content clips, how to manage the clips, how to utilize the clips for inter prediction, and how to allocate bit-rate among clips, in a systematic manner. The proposed video coding framework is implemented upon the state-of-the-art video coding scheme, High Efficiency Video Coding (HEVC). Experimental results show that our scheme achieves on average 5.1% and 5.0% BD-rate reduction than the HEVC anchor, in low-delay B and low-delay P settings, respectively. We believe that reference clip opens up a new dimension for optimizing inter prediction in video coding, and thus is worthy of further study.

Gradient-oriented directional predictor for HEVC planar and angular intra prediction modes to enhance lossless compression

Recent advancements in the capture and display technologies motivated the ITU-T Video Coding Experts Group and ISO/IEC Moving Picture Experts Group to jointly develop the High-Efficiency Video Coding (HEVC), a state-of-the-art video coding standard for efficient compression. The compression applications that essentially require lossless compression scenarios include medical imaging, video analytics, video surveillance, video streaming etc., where the content reconstruction should be flawless. In the proposed work, we present a gradient-oriented directional prediction (GDP) strategy at the pixel level to enhance the compression efficiency of the conventional block-based planar and angular intra prediction in the HEVC lossless mode. The detailed experimental analysis demonstrates that the proposed method outperforms the lossless mode of HEVC anchor in terms of bit-rate savings by 8.29%, 1.65%, 1.94% and 2.21% for Main-AI, LD, LDP and RA configurations respectively, without impairing the computational complexity. The experimental results also illustrates that the proposed predictor performs superior to the existing state-of-the-art techniques in the literature.

Convolutional Neural Network-Based Block Up-Sampling for Intra Frame Coding

Inspired by the recent advances of image super-resolution using convolutional neural network (CNN), we propose a CNN-based block up-sampling scheme for intra frame coding. A block can be down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution. Different from previous studies on down/up-sampling-based coding, the up-sampling methods in our scheme have been designed by training CNN instead of hand-crafted. We explore a new CNN structure for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning, making the network both compact and efficient. We also design different networks for the up-sampling of luma and chroma components, respectively, where the chroma up-sampling CNN utilizes the luma information to boost its performance. In addition, we design a two-stage up-sampling process, the first stage being within the block-by-block coding loop, and the second stage being performed on the entire frame, so as to refine block boundaries. We also empirically study how to set the coding parameters of down-sampled blocks for pursuing the frame-level rate-distortion optimization. Our proposed scheme is implemented into the High Efficiency Video Coding (HEVC) reference software, and a comprehensive set of experiments have been performed to evaluate our methods. Experimental results show that our scheme achieves significant bits saving compared with HEVC anchor especially at low bit rates, leading to on average 5.5% BD-rate reduction on common test sequences and on average 9.0% BD-rate reduction on ultra high definition (UHD) test sequences.

Performance enhancement of HEVC lossless mode using sample-based angular and planar predictions

High-efficiency video coding (HEVC) is the state-of-the-art video compression standard designed to handle the storage and transmission requirements of next-generation multimedia services. In the lossless mode HEVC, prevailing sample-based prediction algorithms in the literature have shown better prediction accuracy compared to the conventional block-based prediction within the HEVC anchor. This work proposes a sample-based prediction technique to modify the planar prediction mode of HEVC and a complete sample-based predictive encoder for the lossless mode of HEVC. In this work, we propose gradient adaptive sample-based intraprediction (GASP) as a replacement for the block-based planar prediction in HEVC. To obtain the benefits of sample-based prediction methods in both angular and planar prediction modes, we also propose a combination of ISAP, a sample-based angular prediction in the literature, with the newly proposed GASP (CIG). The experimental results demonstrate the superiority of GASP and CIG over other state-of-the-art sample-based prediction strategies in angular and planar prediction modes.

Selective intra prediction in HEVC planar and angular modes for efficient near-lossless video compression

In the lossless mode of HEVC (high efficiency video coding), the coding gains of sample-based prediction algorithms are always better than the conventional block-based predictions within the HEVC anchor. Both block based and sample-based prediction strategies select the best prediction mode for the current prediction unit (PU), on the basis of a cost function evaluated at the PU level. Hence, the selected prediction mode for a PU may not be generating the best prediction at the pixel level. If the selection of the best prediction mode can be performed at the pixel level, the accuracy of prediction can be increased significantly. In this work, we propose two selective intra prediction strategies (SIP) which select the best prediction mode from the block-based and sample-based predictions at the pixel level. In the proposed SIP-A algorithm the SIP strategy is applied to only angular prediction modes while the combined SIP algorithm (SIP-C) employs the SIP strategy in both the angular and planar prediction modes. The proposed SIP-C algorithm enhances the performance of the current state of the art SIP algorithm in the literature by introducing better prediction strategies for both angular and planar predictions of HEVC intra prediction. To avoid the enormous overhead required to convey the choice of prediction from the encoder to the decoder, SIP algorithms utilise the least significant bit (LSB) piggybacking strategy. The experimental results provide significant improvements in coding gain and run time for the proposed near-lossless SIP algorithms.

Adaptive Block Coding Order for Intra Prediction in HEVC

In this paper, an adaptive block coding order for intra prediction is proposed. Modern video coding standards, including the most recent High Efficiency Video Coding (HEVC), utilize fixed scan orders in processing blocks during intra coding. However, the fixed scan orders typically result in residual blocks with noticeable edge patterns. That means the fixed scan orders cannot fully exploit the content-adaptive spatial correlations between adjacent blocks, thus the bitrate after compression tends to be large. To reduce the bitrate induced by inaccurate intra prediction, the proposed approach adaptively chooses both the block and subblock coding orders by minimizing the coding cost. Specifically, determining the block coding order is formulated as a traveling salesman problem that is solved using dynamic programming. Besides the block coding order, we also design the subblock coding order in each block with an adaptive manner. The experimental results demonstrate a Bjontegaard-Delta-rate reduction of up to 4.4% compared with HEVC anchor.

Highly efficient near lossless video compression using selective intra prediction for HEVC lossless mode

Lossless coding mode of high efficiency video coding (HEVC) relies on efficient intra prediction methods for achieving higher compression. Sample based angular intra prediction (SAP) is a modification proposed for the conventional block-based intra prediction (BP) in the HEVC standard. In this paper a selective intra prediction (SIP) algorithm is proposed for lossless intra mode coding within HEVC, which uses an adaptive switching strategy between the two intra prediction methods, BP and modified SAP (MSAP), on a per pixel basis. MSAP is a modification proposed in this work to improve the prediction accuracy of SAP. The proposed SIP algorithm is a near lossless algorithm that selects the best prediction for each pixel to reduce the residual energy thereby increasing the coding efficiency. To circumvent the huge overhead required for the transmission of the selection from encoder to decoder, SIP adopts piggybacking the selection on the least-significant-bit (LSB) of the transmitted residual for each pixel. The proposed approach removes overhead at the expense of a ±1 reconstruction error for each pixel which is negligible for 8-bit or 10-bit input videos. Experimental results show significant coding gains for the proposed SIP over the HEVC anchor and similar state-of-the-art lossless coding algorithms.

Representing Visual Objects in HEVC Coding Loop

Different from the previous video coding standards that employ fixed-size coding blocks (and macroblocks), the latest high efficiency video coding (HEVC) introduces a quadtree structure to represent variable-size coding blocks in the coding loop. The main objective of this study is to investigate a novel way to reuse these variable-size blocks to represent the foreground objects in the picture. Towards this end, this paper proposes three methods, i.e., flagging the blocks lying in the object regions flagging compression blocks (FCB), adding an object tree in each Coding Tree Unit to describe the objects' shape in it additional object tree (AOT) and confining the block splitting procedure to fit the object shape confining by shape (CBS). Among them, FCB and CBS add a flag bit in the syntax description of the block to indicate whether it lies in the objects region, while AOT adds a separate quadtree to represent the objects. For all these methods, the additional bits are then fed into the HEVC entropy coding module to compress. As such, the representation of visual objects in the pictures can be implemented in the HEVC coding loop by reusing the variable-size blocks and entropy coding, without additional coding tools. The experiments on six manually-segmented HEVC testing sequences (three in 1080P and three in 720P) demonstrate the feasibility and effectiveness of our proposal. To represent the objects in the 1080P testing sequences, the BD rate increases of FCB, AOT, and CBS over the HEVC anchor are 1.57%, 3.27%, and 5.93% respectively; while for the 720P conference videos, those are 4.57%, 17.23%, and 26.93% (note that the average bitrate of the anchor is only 1009 kb/s).