Joint Video Exploration Team Research Articles

A Fast VVC Intra Prediction Based on Gradient Analysis and Multi-Feature Fusion CNN

The Joint Video Exploration Team (JVET) has created the Versatile Video Coding Standard (VVC/H.266), the most up-to-date video coding standard, offering a broad selection of coding tools. The maturity of commercial VVC codecs can significantly reduce costs and improve coding efficiency. However, the latest video coding standards have introduced binomial and trinomial tree partitioning methods, which cause the coding units (CUs) to have various shapes, increasing the complexity of coding. This article proposes a technique to simplify VVC intra prediction through the use of gradient analysis and a multi-feature fusion CNN. The gradient of CUs is computed by employing the Sobel operator, the calculation results are used for predecision-making. Further decisions can be made by CNN for coding units that cannot be judged whether they should be segmented or not. We calculate the standard deviation (SD) and the initial depth as the input features of the CNN. To implement this method, the initial depth can be determined by constructing a segmented depth prediction dictionary. For the initial segmentation depth of the coding unit, regardless of its shape, it can also be determined by consulting the dictionary. The algorithm can determine whether to split CUs of varying sizes, decreasing the complexity of the CU division process and making VVC more practical. Experimental results demonstrate that the proposed algorithm can reduce encoding time by 36.56% with a minimal increase of 1.06% Bjøntegaard delta bit rate (BD-BR) compared to the original algorithm.

Fast intra partition and mode prediction for equirectangular projection 360‐degree video coding

360-degree videos have drawn great attention from both the academia and the industry. For transmission and storage of 360-degree video, the joint video exploration team proposed the Versatile Video Coding (VVC) standard. VVC can significantly reduce the bitrate while maintaining the same subjective visual quality compared to the preceding high efficiency video coding. However, the computational complexity of VVC is extremely high which hinders the interactive applications. This paper proposes a fast intra partition method and mode prediction algorithm for equirectangular projection 360-degree video coding. First, a latitude-based preprocessing is introduced to early terminate the Coding Unit (CU) partition in the polar region. Second, the support vector machine is used to predict the CU partition type. Third, the fast intra mode search method accelerates the intra mode prediction. Experimental results show that the proposed algorithm can significantly obtain an average time reduction rate of 60.40% and a Bjontegaard delta rate increase of 1.96%.

Fast CU Partition for VVC Using Texture Complexity Classification Convolutional Neural Network

Versatile video coding (H.266/VVC), which was newly released by the Joint Video Exploration Team (JVET), introduces quad-tree plus multi-type tree (QTMT) partition structure on the basis of quad-tree (QT) partition structure in High Efficiency Video Coding (H.265/HEVC). More complicated coding unit (CU) partitioning processes in H.266/VVC significantly improve video compression efficiency, but greatly increase the computational complexity compared. The ultra-high encoding complexity has obstructed its real-time applications. In order to solve this problem, a CU partition algorithm using convolutional neural network (CNN) is proposed in this paper to speed up the H.266/VVC CU partition process. Firstly, 64 × 64 CU is divided into smooth texture CU, mildly complex texture CU and complex texture CU according to the CU texture characteristics. Second, CU texture complexity classification convolutional neural network (CUTCC-CNN) is proposed to classify CUs. Finally, according to the classification results, the encoder is guided to skip different RDO search process. And optimal CU partition results will be determined. Experimental results show that the proposed method reduces the average coding time by 32.2% with only 0.55% BD-BR loss compared with VTM 10.2.

Notice of Violation of IEEE Publication Principles: VVC/H.266 Intra Mode QTMT Based CU Partition Using CNN

Notice of Violation of IEEE Publication Principles <br><br>“VVC/H. 266 Intra Mode QTMT Based CU Partition Using CNN” <br>by Sameena Javaid, Safdar Rizvi, Muhammad Talha Ubaid, Abdullah Tariq <br>in IEEE Access, April 2022 <br><br>After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE’s Publication Principles. <br><br>This paper copied content from the article below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission. <br><br> “DeepQTMT: A Deep Learning Approach for Fast QTMT-Based CU Partition of Intra-Mode VVC” <br> by Tianyi Li, Mai Xu, Runzhi Tang, Ying Chen, Qunliang Xing <br>in IEEE Transactions on Image Processing, Vol 30, 2021 <br><br> <br/> The latest standard for video coding is versatile video coding (VVC) / H.266 which is developed by the joint video exploration team (JVET). Its coding structure is a multi-type tree (MTT) structure, which consists of two types of trees that are Ternary Tree (TT) and Binary Tree (BT). Due to the use of brute force quest for residual rate-distortion the quad-tree and multi-type tree (QTMT) structure of the coding unit (CU) split and contributes over 98% of the encoding time. This structure is efficient in coding, however, increases computational complexity. The current paper proposes a deep learning technique to predict the QTMT based CU split rather than just the brute-force QTMT method to substantially speed up the time of the encoding process for VVC/H.266 intra mode. In the first phase, we developed an extensive database containing ample CU splitting patterns and various streaming videos. In the second phase, we suggest a multi-level exit CNN (MLE-CNN) model with a redundancy removal mechanism at different levels to determine the CU partition. In the third phase, for the training of MLECNN model we have established the adaptive loss function and analyzing the both unknown number of partition modes and the focus on RD cost minimization. Finally, a variable threshold decision system is established to achieve the targeted low complexity and RD performance. Ultimately experimental findings show that VVC/H.266 encoding time has reduced to 69.11% from 47.91% with insignificant bjontegaard delta bit rate (BDBR) to 2.919% from 1.023% which performs better than the existing futuristic and modern approaches.

CNN-based Fast Split Mode Decision Algorithm for Versatile Video Coding (VVC) Inter Prediction

Versatile Video Coding (VVC) is the latest video coding standard developed by Joint Video Exploration Team (JVET). In VVC, the quadtree plus multi-type tree (QT+MTT) structure of coding unit (CU) partition is adopted, and its computational complexity is considerably high due to the brute-force search for recursive rate-distortion (RD) optimization. In this paper, we aim to reduce the time complexity of inter-picture prediction mode since the inter prediction accounts for a large portion of the total encoding time. The problem can be defined as classifying the split mode of each CU. To classify the split mode effectively, a novel convolutional neural network (CNN) called multi-level tree (MLT-CNN) architecture is introduced. For boosting classification performance, we utilize additional information including inter-picture information while training the CNN. The overall algorithm including the MLT-CNN inference process is implemented on VVC Test Model (VTM) 11.0. The CUs of size 128×128 can be the inputs of the CNN. The sequences are encoded at the random access (RA) configuration with five QP values {22, 27, 32, 37, 42}. The experimental results show that the proposed algorithm can reduce the computational complexity by 11.53% on average, and 26.14% for the maximum with an average 1.01% of the increase in Bjøntegaard delta bit rate (BDBR). Especially, the proposed method shows higher performance on the sequences of the A and B classes, reducing 9.81%~26.14% of encoding time with 0.95%~3.28% of the BDBR increase.

Complexity Analysis of New Future Video Coding (FVC) Standard Technology

Future Video Coding (FVC) is a modern standard in the field of video coding that offers much higher compression efficiency than the HEVC standard. FVC was developed by the Joint Video Exploration Team (JVET), formed through collaboration between the ISO/IEC MPEG and ITU-T VCEG. New tools emerging with the FVC bring in super resolution implementation schemes that are being recommended for Ultra-High-Definition (UHD) video coding in both SDR and HDR images. However, a new flexible block structure is adopted in the FVC standard, which is named quadtree plus binary tree (QTBT) in order to enhance compression efficiency. In this paper, we provide a fast FVC algorithm to achieve better performance and to reduce encoding complexity. First, we evaluate the FVC profiles under All Intra, Low-Delay P, and Random Access to determine which coding components consume the most time. Second, a fast FVC mode decision is proposed to reduce encoding computational complexity. Then, a comparison between three configurations, namely, Random Access, Low-Delay B, and Low-Delay P, is proposed, in terms of Bitrate, PSNR, and encoding time. Compared to previous works, the experimental results prove that the time saving reaches 13% with a decrease in the Bitrate of about 0.6% and in the PSNR of 0.01 to 0.2 dB.

ViSTRA2: Video coding using spatial resolution and effective bit depth adaptation

We present a new video compression framework (ViSTRA2) which exploits adaptation of spatial resolution and effective bit depth, down-sampling these parameters at the encoder based on perceptual criteria, and up-sampling at the decoder using a deep convolution neural network. ViSTRA2 has been integrated with the reference software of both the HEVC (HM 16.20) and VVC (VTM 4.0.1), and evaluated under the Joint Video Exploration Team Common Test Conditions using the Random Access configuration. Our results show consistent and significant compression gains against HM and VVC based on Bjønegaard Delta measurements, with average BD-rate savings of 12.6% (PSNR) and 19.5% (VMAF) over HM and 5.5% (PSNR) and 8.6% (VMAF) over VTM.

Adaptive CU partition and early skip mode detection for H.266/VVC

The Joint Video Exploration Team (JVET) has started to develop the next-generation video coding standard-H.266/Versatile Video Coding (H.266/VVC) based on H.265/High Efficiency Video Coding (H.265/HEVC) to provide higher compression performance. The H.266/VVC supports the flexible quadtree with a nested multi-type tree (QTMT) partition structure including quadtree (QT), binary tree (BT), and ternary tree (TT). The coding unit (CU) sizes range from 128 to 4 for the luma component or from 64 to 2 for the chroma component in the QTMT splitting structure. The introduction of small CU size, i.e., 2×N, leads to inefficient hardware implementation because it causes pipeline delayed and needs to process 2×N pixels in the hardware architecture. In addition, the inter or bi-predicted of small CU requires a higher memory bandwidth than the bi-predicted of 8×8 CU in H.266/VVC. To solve the above issues, we introduce a fast method to accelerate CU partition and mode decision, including an adaptive CU partition method and early skip mode detection method. The proposed algorithm consists of two parts: (1) adaptive remove 2×N CUs by skipping BT and TT splitting mode; (2) early skip bi-predicted or inter prediction of small CU. The experimental results demonstrate that the proposed scheme can save 47% coding time while maintaining the coding performance.

Evaluating the Coding Performance of 360° Image Projection Formats Using Objective Quality Metrics

Recently, 360° content has emerged as a new method for offering real-life interaction. Ultra-high resolution 360° content is mapped to the two-dimensional plane to adjust to the input of existing generic coding standards for transmission. Many formats have been proposed, and tremendous work is being done to investigate 360° videos in the Joint Video Exploration Team using projection-based coding. However, the standardization activities for quality assessment of 360° images are limited. In this study, we evaluate the coding performance of various projection formats, including recently-proposed formats adapting to the input of JPEG and JPEG 2000 content. We present an overview of the nine state-of-the-art formats considered in the evaluation. We also propose an evaluation framework for reducing the bias toward the native equi-rectangular (ERP) format. We consider the downsampled ERP image as the ground truth image. Firstly, format conversions are applied to the ERP image. Secondly, each converted image is subjected to the JPEG and JPEG 2000 image coding standards, then decoded and converted back to the downsampled ERP to find the coding gain of each format. The quality metrics designed for 360° content and conventional 2D metrics have been used for both end-to-end distortion measurement and codec level, in two subsampling modes, i.e., YUV (4:2:0 and 4:4:4). Our evaluation results prove that the hybrid equi-angular format and equatorial cylindrical format achieve better coding performance among the compared formats. Our work presents evidence to find the coding gain of these formats over ERP, which is useful for identifying the best image format for a future standard.

Optimized VLSI Architecture of HEVC Fractional Pixel Interpolators with Approximate Computing

High Efficiency Video Coding (HEVC) is the latest video standard developed by the Joint Video Exploration Team. HEVC is able to offer better compression results than preceding standards but it suffers from a high computational complexity. In particular, one of the most time consuming blocks in HEVC is the fractional-sample interpolation filter, which is used in both the encoding and the decoding processes. Integrating different state-of-the-art techniques, this paper presents an architecture for interpolation filters, able to trade quality for energy and power efficiency by exploiting approximate interpolation filters and by halving the amount of required memory with respect to state-of-the-art implementations.

Fast intra-coding unit partition decision in H.266/FVC based on deep learning

In the recent Future Video Coding (FVC) standard developed by the Joint Video Exploration Team (JVET), the quad-tree binary-tree (QTBT) block partition module makes use of rectangular block forms and additional square block sizes compared to quad-tree (QT) block partitioning module proposed in the predecessor High-Efficiency Video Coding (HEVC) standard. This block flexibility, induced with the QTBT module, significantly improves compression performance while it dramatically increases coding complexity due to the brute force search for Rate Distortion Optimization (RDO). To cope with this issue, it is necessary to consider the unique characteristics of QTBT in FVC. In this paper, we propose a fast QT partitioning algorithm based on a deep convolutional neural network (CNN) model to predict coding unit (CU) partition instead of RDO which enhances considerably QTBT performance for intra-mode coding. Based on a suitable diversified CU partition patterns database, the optimization process is set up with three levels CNN structure developed to learn the split or non-split decision from the established database. Experimental results reveal that the proposed algorithm can accelerate the QTBT block partition structure by reducing the intra-mode encoding time by an average of 35% with a bit rate increase of 1.7%, allowing its application in practical scenarios.

Fast DST-VII/DCT-VIII With Dual Implementation Support for Versatile Video Coding

The Joint Video Exploration Team (JVET) recently launched the standardization of the next-generation video coding named Versatile Video Coding (VVC) with the inherited technical framework from its predecessor High-Efficiency Video Coding (HEVC). The simplified Enhanced Multiple Transform (EMT) has been adopted as the primary residual coding transform solution, termed Multiple Transform Selection (MTS). In MTS, only the transform set consisting of DST-VII and DCT-VIII remains, excluding the other transform sets and the dependency on intra prediction modes. Significant coding gains are achieved by introducing new DST/DCT transforms, but the full matrix implementation is relatively costly compared to partial butterfly in terms of both software run-time and operation counts. In this work, we exploit the inherent features existing in DST-VII and DCT-VIII. Instead of repeating the element-wise additions and multiplications in full matrix operation, these features can be leveraged to achieve more efficient implementations which only use partial elements to derive the identical results. Existing transform matrices are further tuned to utilize these (anti-)symmetric features. A partial butterfly-type fast algorithm with dual-implementation support is proposed for DST-VII/DCT-VIII transform in VVC. Complexity analysis including operation counts and software run-time are conducted to validate the effectiveness. In addition, we prove the features are perfectly supported by theory. The proposed fast methods achieve noticeable software run-time savings without compromising on coding performance by comparing with the VVC Test Model VTM-3.0. It is shown that under Common Test Condition (CTC) with inter MTS enabled, an average of 9%, 0%, and 3% decoding time savings are achieved for All Intra (AI), Random Access (RA) and Low Delay B (LDB), respectively. Under low QP test condition with inter MTS enabled, the proposed fast methods achieve 1%, 2% and 4% decoding time savings on average for AI, RA, and LDB, respectively.

The Battle of the Video Codecs in the Healthcare Domain - A Comparative Performance Evaluation Study Leveraging VVC and AV1

Video compression is the core technology in mobile (mHealth) and electronic (eHealth) health video streaming applications. With global video traffic projected to reach 82% of all Internet traffic by 2022, there is a strong need to develop efficient compression algorithms to accommodate expected future growth. For the first time in decades, and especially since ISO/IEC MPEG and ITU-T VCEG expert groups strategically joined forces to develop the highly successful H.264/AVC standard, we have two distinct initiatives competing for the best performing video codec. On the one hand, we have the Alliance for Open Media (AOM) that support a new, royalty free video codec generation, termed AV1, based on VP8 and VP9 efforts. On the other hand, the Joint Video Exploration Team (JVET) has been developing the Versatile Video Codec (VVC) as the successor of the High Efficiency Video Coding (HEVC) standard. At the same time, the breadth of applications utilizing video codecs, involving significant content variability and moving across the video resolution ladder, to satisfy different constraints, have resulted in mixed literature results, with respect to the best performing codec. In this paper, we compare the performance of emerging VVC and AV1 codecs, along with popular HEVC implementations, namely the HEVC Test Model (HM) and x265, as well as earlier, VP9 codec, and investigate their suitability for medical applications. To the best of our knowledge, this is the first performance comparison of emerging VVC and AV1 video codecs for use in the healthcare domain. Experimental evaluation based on three datasets (ultrasound, emergency scenery, and general-purpose videos) demonstrate that VVC outperforms all rival codecs while AV1 achieves better compression efficiency than HEVC in all cases but low-resolution ($560\times448$ @40Hz) ultrasound videos of the common carotid artery. Furthermore, the use of video despeckling prior to ultrasound video compression can provide significant bitrate savings.

Interweaved Prediction for Video Coding.

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%.

Fast CU Partition and Intra Mode Decision Method for H.266/VVC

The Versatile Video Coding (H.266/VVC) standard has developed by Joint Video Exploration Team (JVET). Compared with the previous generation video coding standard, the H.266/VVC is more outstanding. Since the H.266/VVC introduces multi-type tree (MTT) structure including binary tree (BT) and ternary tree (TT), which brings the significant coding efficiency but increases coding complexity. Moreover, the intra prediction modes have increased from 35 to 67, which can provide more accurate prediction than H.265/High Efficiency Video Coding (HEVC). Therefore, these can improve the encoding quality, but increase computational complexity. To reduce the computational complexity, this paper designs a fast coding unit (CU) partition and intra mode decision algorithm, which includes fast CU partition based on random forest classifier (RFC) model and fast intra prediction modes optimization based on texture region features. Simulation results indicate that the proposed scheme can save 54.91% encoding time with only 0.93% increase in BDBR.

Omnidirectional 360° Video Coding Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

Augmented reality (AR) and virtual reality (VR) applications have seen rising popularity in recent years. Omnidirectional 360° video is a video format often used in AR and VR applications. To address the industry needs, a new HEVC edition recently published includes several supplemental enhancement information (SEI) messages to enable the carriage of omnidirectional video using HEVC. However, further improvement in 360° video compression efficiency is needed. In order to address this challenge, the Joint Video Exploration Team (JVET) of ITU-T VCEG and ISO/IEC MPEG has been investigating 360° video coding technologies, including projection formats, pre- and post-processing technologies, as well as 360°-video-specific coding tools since 2016. The joint call for proposals (CfP) recently issued by ITU-T VCEG and ISO/IEC MPEG on video compression technologies beyond HEVC included a category on 360° video. Twelve CfP responses in the 360° video category were received. This paper describes technologies relevant to 360° video for VVC. A summary of projection formats, pre- and post-processing methods, and 360°-video specific coding tool modifications in these proposals is provided.

Accelerating the CU partitioning decision in an HEVC-JEM transcoder

High Efficiency Video Coding (HEVC) is currently the latest video coding standard available on the market, and it is able to offer up to twice the coding efficiency, in the range of 50% bitrate reduction for the same video quality, of the previous standard, namely H.264/Advanced Video Coding (AVC). HEVC was standardized in 2013 for videos up to a resolution of 2K. However, the popularity of 4K videos is increasing due to the growing use of video-on-demand platforms. Therefore, the ITU-T Video Coding Expert Group (VCEG) and the ISO/IEC Moving Picture Expert Group (MPEG) created the Joint Video Exploration Team (JVET) in 2015 to design the future video coding technology under the Joint Exploration Model (JEM), which its latest version achieves an improvement in coding efficiency of 30%, but at a high cost in terms of computational complexity (10×) with respect to HEVC. The new video standard is expected to be ready in 2020, so it is necessary to find efficient mechanisms to convert current content to the new format adopted in JEM. In this regard, our proposal consists in a probabilistic classifier based on Naive-Bayes that enables the prediction of the splitting decision at the first quadtree level in JEM, reducing the computational complexity of the transcoding process from HEVC to this new standard. The experimental results show a good trade-off between coding efficiency and complexity compared with the anchor transcoder, obtaining a time reduction up to 12.71% at the expense of low coding efficiency penalties in the configurations evaluated.

Understanding the Video Codec Jungle: A Comparison of TCO and Compression Efficiency

The codec landscape has rapidly accelerated, especially with the recent announcement by Apple to fully support High-Efficiency Video Coding (HEVC) and join the Alliance for Open Media (AOM) consortium. At the 2018 National Association of Broadcasters (NAB) Show, the AOM froze its specification of the AV1 codec. In addition, the Joint Video Exploration Team (JVET), a collaborative initiative between the Motion Picture Experts Group (MPEG) and International Telecommunication Union - Telecommunication (ITU-T), is making progress with the development of the next generation of MPEG compression, which will address future applications such as 8K and virtual reality (VR) six degrees of freedom (6DoF). This paper provides a thorough comparison of different codec technologies, including Advanced Video Coding (AVC), HEVC, AV1, and Versatile Video Coding (VVC). The total cost of ownership (TCO) and compression efficiency of each codec will be evaluated based on objective testing methods [i.e., peak signal-to-noise ratio (PSNR) and Video Multimethod Assessment Fusion (VMAF)] and subjective testing. Based on the factual data presented, readers will be able to make a better decision about which codec to use.

General Video Coding Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

After the development of the High-Efficiency Video Coding Standard (HEVC), ITU-T VCEG and ISO/IEC MPEG formed the Joint Video Exploration Team (JVET), which started exploring video coding technology with higher coding efficiency, including development of a Joint Exploration Model (JEM) algorithm and a corresponding software implementation. The technology explored in the last version of the JEM further increases the compression capabilities of the hybrid video coding approach by adding new tools, reaching up to 30% bit rate reduction compared to HEVC based on the Bjontegaard delta bit rate (BD-rate) metric, and further improvement beyond that in terms of subjective visual quality. This provided enough evidence to issue a joint Call for Proposals (CfP) for a new standardization activity now known as Versatile Video Coding (VVC). All technology proposed in the responses to the CfP was based on the classic block-based hybrid video coding design, extending it by new elements of partitioning, intra- and inter-picture prediction, prediction signal filtering, transforms, quantization/scaling, entropy coding, and in-loop filtering. This article provides an overview of technology that was proposed in the responses to the CfP, with a focus on techniques that were not already explored in the JEM context.

The Joint Exploration Model (JEM) for Video Compression With Capability Beyond HEVC

This paper provides an overview of the coding algorithms of the Joint Exploration Model (JEM) for video compression with capability beyond HEVC, which was developed by the Joint Video Exploration Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). The goal of the JEM development and experimentation was to provide evidence that sufficient coding efficiency improvement over the High Efficiency Video Coding (HEVC) standard can be achieved, which would justify the need for a new video coding standard with a compression capability significantly exceeding that of HEVC. The development of the JEM provided an ability to conduct studies toward that goal in a verifiable and collaborative manner and led to the launching of the project to develop the new Versatile Video Coding (VVC) standard. Objective metric gains exceeding 30% were measured for most of the tested high-resolution video content that represents current demanding new applications, and subjective testing using human observers showed even more benefit.