Future Video Coding Research Articles

Complexity Analysis of New Future Video Coding (FVC) Standard Technology

Future Video Coding (FVC) is a modern standard in the field of video coding that offers much higher compression efficiency than the HEVC standard. FVC was developed by the Joint Video Exploration Team (JVET), formed through collaboration between the ISO/IEC MPEG and ITU-T VCEG. New tools emerging with the FVC bring in super resolution implementation schemes that are being recommended for Ultra-High-Definition (UHD) video coding in both SDR and HDR images. However, a new flexible block structure is adopted in the FVC standard, which is named quadtree plus binary tree (QTBT) in order to enhance compression efficiency. In this paper, we provide a fast FVC algorithm to achieve better performance and to reduce encoding complexity. First, we evaluate the FVC profiles under All Intra, Low-Delay P, and Random Access to determine which coding components consume the most time. Second, a fast FVC mode decision is proposed to reduce encoding computational complexity. Then, a comparison between three configurations, namely, Random Access, Low-Delay B, and Low-Delay P, is proposed, in terms of Bitrate, PSNR, and encoding time. Compared to previous works, the experimental results prove that the time saving reaches 13% with a decrease in the Bitrate of about 0.6% and in the PSNR of 0.01 to 0.2 dB.

Low-complexity QTMT partition based on deep neural network for Versatile Video Coding

Versatile Video Coding (VVC), the newest standard for future video coding, is currently under development. This proposal aimed to improve the encoder performance over the latest standard namely High Efficiency Video Coding, carried with a high increase in coding complexity. The VVC partition structure is mainly based on the quadtree with nested multi-type tree (QTMT) block scheme. Such an improvement leads to a more flexible block partition and promotes a high encoding efficiency, but generates a huge coding complexity. In order to deal with this issue, a fast QTMT intra partition algorithm, based on a deep neural network named Early Terminated Hierarchical Convolution Neural Network, is applied to predict the $$64\times $$ 64 block QT partition structure. The proposed algorithm determines the QTMT partition structure based on the decision of whether to split or skip the corresponding CU, in order to get $$128\times $$ 128 Coding Tree Unit partition architecture. In this paper, the proposed intra partition work achieves a significant speedup in encoding gain that reaches 32.96% in best cases for Ultra High Definition video sequences compared to the reference VVC software VTM-3.0. For all video sequences, 24.49% time saving is reached on average. This improvement comes with an increase of 4.18% and a decrease of 0.18 dB in terms of BDBR and BDPSNR, respectively.

Prediction With Multicross Component for Future Video Coding

There has been a consensus of cross-component prediction in video compression that the interchannel redundancies can be effectively removed through a linear prediction model. To fully explore the similarities existing in different color components, we propose prediction with multicross component (PMC) as a new chroma intracoding mode for the emerging video coding standard AVS3. The proposed PMC enjoys extremely low operational complexity and high flexibility, which advocates to reuse and reform the linear prediction model of conventional two-step cross-component prediction mode. Moreover, dual-prediction patterns of PMC are exploited to enhance the prediction efficiency. Experimental results on AVS3 test model HPM-6.0 show that PMC brings 0.32%, 1.60%, and 2.39% BD-rate savings for Y, Cb, and Cr components, respectively, under all intraconfiguration with natural-scene videos. Moreover, 0.91%, 2.24%, and 2.82% BD-rate gains can be observed on screen-content videos. Subset modes of PMC have been adopted by AVS3.

Fast intra-coding unit partition decision in H.266/FVC based on deep learning

In the recent Future Video Coding (FVC) standard developed by the Joint Video Exploration Team (JVET), the quad-tree binary-tree (QTBT) block partition module makes use of rectangular block forms and additional square block sizes compared to quad-tree (QT) block partitioning module proposed in the predecessor High-Efficiency Video Coding (HEVC) standard. This block flexibility, induced with the QTBT module, significantly improves compression performance while it dramatically increases coding complexity due to the brute force search for Rate Distortion Optimization (RDO). To cope with this issue, it is necessary to consider the unique characteristics of QTBT in FVC. In this paper, we propose a fast QT partitioning algorithm based on a deep convolutional neural network (CNN) model to predict coding unit (CU) partition instead of RDO which enhances considerably QTBT performance for intra-mode coding. Based on a suitable diversified CU partition patterns database, the optimization process is set up with three levels CNN structure developed to learn the split or non-split decision from the established database. Experimental results reveal that the proposed algorithm can accelerate the QTBT block partition structure by reducing the intra-mode encoding time by an average of 35% with a bit rate increase of 1.7%, allowing its application in practical scenarios.

Emerging Technologies toward Future Video Coding

Emerging Technologies toward Future Video Coding

Emerging Technologies toward Future Video Coding

Emerging Technologies toward Future Video Coding

Analysis of Emerging Video Codecs: Coding Tools, Compression Efficiency

The video compression community, from technologists to customers, is affected by the competition between two lines of development for future video compression technology beyond high-efficiency video coding (HEVC): AV1, as developed by the industry-driven Alliance for Open Media (AOM), and Versatile Video Coding (VVC), as developed and studied by the Joint Video Experts Team (JVET) on Future Video Coding of ITU-T Video Coding Expert Group (VCEG) and International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Motion Picture Experts Group (MPEG). Both groups have asserted that their respective codecs, AV1 and VVC, outperform the state-of-the-art HEVC. However, in many cases, related studies are evaluating video codec performances based on different test conditions, scenarios, or codec implementations, which makes comparisons sometimes unfair and at least difficult to interpret. This paper provides a neutral and well-balanced evaluation of compression and runtime performance for both AV1 and VVC codecs, using their respective reference models, in comparison to two HEVC codec implementations: HEVC Test Model (HM) as the HEVC standard reference model and x265 as a well-known open-source HEVC codec implementation. Comparison of differentiating features of AV1 and VVC over HEVC is also reported. Our main findings show that, based on an objective analysis, AV1 allows bitrate reduction of 10% in comparison to HM for the same peak signal-to-noise ratio (PSNR) and Structural SIMilarity (SSIM) quality scores while increasing encoding runtime by 55.5x. However, we show that this objective advantage in coding efficiency does not necessarily translate into gain in terms of subjective quality assessment. Overall, objective results show VVC to be the best candidate for compression technology beyond HEVC.

Deep Learning-Based Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Multiple classifier-based fast coding unit partition for intra coding in future video coding

Future Video Coding (FVC) significantly improves the compression efficiency over the preceding High Efficiency Video Coding (HEVC) standard but at the cost of extremely huge computational complexity. The flexible quad-tree plus binary tree (QTBT) block partitioning structure in FVC is largely responsible for the high computational complexity. In order to address the issue of huge computational complexity of QTBT, we propose a multiple classifier-based fast QTBT partitioning algorithm for FVC intra coding. The proposed multiple classifier-based QTBT algorithm contains three stages including horizon binary-tree decision model (HBTDM), vertical binary-tree decision model (VBTDM), and quad-tree decision model (QTDM). Consequently, the computational complexity of FVC intra coding can be drastically reduced by replacing the brute-force search with HBTDM, VBTDM, and QTDM to decide the optimal QTBT partitioning. To achieve high prediction accuracy, the descriptive features of intra QTBT partitioning decision are extracted to train corresponding classifiers. An efficient optimal parameter selection method of training classifiers is introduced to balance the computational complexity and rate distortion (RD) performance. Experimental results show that in comparison with the original FVC reference software, the proposed overall algorithm can reduce 64.54% intra coding time with negligible degradation of RD performance. Meanwhile, the proposed algorithm outperforms four state-of-the-art algorithms in terms of computational complexity reduction.

An Overview of Emerging Video Coding Standards

Today's popular video coding standards, such as H.264/AVC, are widely used to encode video into bit streams for storage and transmission. With the explosive growth of various video applications, H.264/AVC may not fully satisfy their requirements anymore. There is an increasing demand for high compression efficiency and low complexity video coding standards. In this article, we provide an overview of existing and emerging video coding standards. We review the timeline of the development of the popular H.26X family video coding standards, and introduce several emerging video coding standards such as AV1, VP9 and VVC. As for future video coding, considering the success of machine learning in various fields and hardware acceleration, we conclude this article with a discussion of several future trends in video coding.

Learning for Video Compression

One key challenge to learning-based video compression is that motion predictive coding, a very effective tool for video compression, can hardly be trained into a neural network. In this paper we propose the concept of PixelMotionCNN (PMCNN) which includes motion extension and hybrid prediction networks. PMCNN can model spatiotemporal coherence to effectively perform predictive coding inside the learning network. On the basis of PMCNN, we further explore a learning-based framework for video compression with additional components of iterative analysis/synthesis, binarization, etc. Experimental results demonstrate the effectiveness of the proposed scheme. Although entropy coding and complex configurations are not employed in this paper, we still demonstrate superior performance compared with MPEG-2 and achieve comparable results with H.264 codec. The proposed learning-based scheme provides a possible new direction to further improve compression efficiency and functionalities of future video coding.

联合多特征的未来视频快速编码

联合多特征的未来视频快速编码

Fast intra coding unit partition decision in H.266/FVC based on spatial features

With the development of technology, the requirements of hardware equipment and user expectations of visual enjoyment are increasingly gradually. The Joint Video Exploration Team (JVET) has established the latest video compression standard, Future Video Coding (FVC). FVC adopts QuadTree plus Binary Tree (QTBT) based Coding Unit (CU) structure, which not only removes the complex hierarchical structure of the CU, Prediction Unit (PU), and Transform Unit (TU) but also supports square and rectangular coding blocks based on the texture of the video content. Although the QTBT structure can provide superior coding performance, it significantly increases the encoding time, particularly in intra coding. Therefore, developing a fast intra CU partition decision algorithm is essential. In this paper, a fast CU partition decision algorithm in FVC intra coding based on spatial features is proposed. Different spatial features in the pixel domain are proposed in the binary tree and quadtree decision processes. Spatial features for the binary tree are employed for early skipping of the encoding process of CUs with binary tree depth and for early determination of binary tree split mode. Spatial features for the quadtree are employed for early splitting or termination of CUs with quadtree depth. Compared with JEM 5.0, the proposed method can save 23% encoding time on average with a slight increase of 0.62% in the Bjontegaard delta bitrate (BDBR).

Overview of Potential Technologies for Future Video Coding Standard (FVC) in JEM Software : Status and Review

In this paper, we present an investigation into several potential technologies that are being developed and prepared for Future Video Coding (FVC) standard. There are some requirements that need to be considered in delivering the FVC standard. The Joint Video Exploration Team (JVET) is now inspecting potential tools to be adopted as video coding technology under the Joint Exploration Model (JEM) reference software. This paper provides an overview of the technical features of JEM software, a brief account of the background, and the key elements according to the latest version of JEM software and its algorithm description.

A High Performance FPGA-Based Architecture for the Future Video Coding Adaptive Multiple Core Transform

Future video coding (FVC) will be the next generation video coding standard. Yet in the first stages of the standardization process, it is expected to replace high efficiency video coding beyond 2020. One of the enhancements in discussion is the adaptive multiple core transform, which uses five different types of 2-D discrete sine/cosine transforms (DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII) and up to ${64 \times 64}$ transform unit sizes. This schema, increases the computational complexity of both, encoder and decoder. In this paper, a deeply pipelined high performance architecture to implement the five transforms for ${4 \times 4}$ , ${8 \times 8}$ , ${16 \times 16}$ , and ${32 \times 32}$ sizes is proposed. The design has been described in VHDL and it has been synthesized for different Field-programmable gate array chips, being able to process up 182 fps ${@} 3840\times2160$ for ${4 \times 4}$ transform sizes. Up the best of our knowledge, this is the first implementation of an architecture for the FVC Adaptive Multiple Core Transform supporting ${4 \times 4}$ , ${8 \times 8}$ , ${16 \times 16}$ , and ${32 \times 32}$ sizes.

Probabilistic Decision Based Block Partitioning for Future Video Coding.

In the latest Joint Video Exploration Team development, the quadtree plus binary tree (QTBT) block partitioning structure has been proposed for future video coding. Compared to the traditional quadtree structure of High Efficiency Video coding (HEVC) standard, QTBT provides more flexible patterns for splitting the blocks, which results in dramatically increased combinations of block partitions and high computational complexity. In view of this, a confidence interval based early termination (CIET) scheme is proposed for QTBT to identify the unnecessary partition modes in the sense of rate-distortion (RD) optimization. In particular, a RD model is established to predict the RD cost of each partition pattern without the full encoding process. Subsequently, the mode decision problem is casted into a probabilistic framework to select the final partition based on the confidence interval decision strategy. Experimental results show that the proposed CIET algorithm can speed up QTBT block partitioning structure by reducing 54.7% encoding time with only 1.12% increase in terms of bit rate. Moreover, the proposed scheme performs consistently well for the high resolution sequences, of which the video coding efficiency is crucial in real applications.

Reconfigurable intra prediction hardware for future video coding

In this paper, two Future Video Coding (FVC) reconfigurable intra prediction hardware are proposed. They are the first FVC intra prediction hardware in the literature. The first hardware implements multiplications with constants using adders and shifters instead of using multipliers. Therefore, it can be used in ASIC implementations of FVC encoders. The second hardware implements multiplications with constants using DSP blocks in FPGA. Therefore, it can be used in FPGA implementations of FVC encoders.

High performance 2D transform hardware for future video coding

Future Video Coding (FVC) is a new international video compression standard offering much better compression efficiency than previous video compression standards at the expense of much higher computational complexity. In this paper, two different high performance FVC 2D transform hardware are designed and implemented using Verilog HDL. They are the first FVC 2D transform hardware in the literature. They perform 2D DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII operations for 4x4 and 8x8 transform units. The first (baseline) hardware uses separate datapaths for each 1D transform. Since the second (reconfigurable) hardware uses two reconfigurable datapaths for all 1D transforms, it has smaller area than the baseline hardware. Since the baseline hardware uses data gating technique, it has lower energy consumption than the reconfigurable hardware. Therefore, the proposed FVC baseline 2D transform hardware can be used in high performance and low energy FVC encoders. The proposed FVC reconfigurable 2D transform hardware can be used in high performance and low cost FVC encoders.