Versatile Video Coding Test Model Research Articles

Complexity analysis of the VVenC versus VVC encoder

The joint video experts team (JVET) has recently finalized a next-generation open-source video codec, called versatile video coding (VVC). The new standard presents a higher gain in the run time and rate compressions. Based on the reference software (VTM, VVC Test Model) of VVC, an optimized encoder was developed these last years resulting in a fast and efficient encoder, called Fraunhofer versatile video encoder (VVenC). Based on the Bjontegaard methodology and the GPROF profiling tool, this paper presents a technical complexity analysis and comparison of both VVC and VVenC. The appropriate comparisons cover the percentage taken by each block in terms of processing time and the resulting whole encoding time. The peak signal-to-noise ratio (PSNR) and bit rate between VVC and VVenC encoders based on the common test conditions manual are also analyzed and compared. The profiling results show that the VVenC encoder presents a maximum gain of runtime and bit rate of 90% and 20% respectively in classes A and D test sequences, compared to the VVC encoder.

DATRA-MIV: Decoder-Adaptive Tiling and Rate Allocation for MPEG Immersive Video

The emerging immersive video coding standard moving picture experts group (MPEG) immersive video (MIV), which is ongoing standardization by MPEG-Immersive (MPEG-I) group, enables six degrees of freedom in a virtual reality environment that represents both natural and computer-generated scenes using multi-view video compression. The MIV eliminates the redundancy between multi-view videos and merges the residuals into multiple pictures, called an atlas. Thus, bitstreams with encoded atlases are generated and corresponding number of decoders are needed, which is challenging for the lightweight device with a single decoder. This article proposes a decoder-adaptive tiling and rate allocation method for MIV to overcome the challenge. First, the proposed method divides atlases into subpictures considering two aspects: (i) subpicture bitstream extracting and merging into one bitstream to use a single decoder and (ii) separation of each source view from the atlases for rate allocation. Second, the atlases are encoded by versatile video coding (VVC), using an extractable subpicture to divide the atlases into subpictures. Third, each subpicture bitstream is extracted, and asymmetric quality allocation for each subpictures is conducted by considering the residuals in the subpicture. Fourth, mixed-quality subpictures were merged by using the proposed bitstream merger. Fifth, the merged bitstream is decoded by using a single decoder. Finally, the viewing area of the user is synthesized by using the reconstructed atlases. Experimental results with the VVC test model (VTM) show that the proposed method achieves a 21.37% Bjøntegaard delta rate saving for immersive video peak signal-to-noise ratio and a 26.76% decoding runtime saving compared to the VTM anchor configuration. Moreover, it supports bitstreams for multiple decoders and single decoder without re-encoding, transcoding, or a substantial increase of the server-side storage.

Fast Decision-Tree-Based Series Partitioning and Mode Prediction Termination Algorithm for H.266/VVC

With the advancement of network technology, multimedia videos have emerged as a crucial channel for individuals to access external information, owing to their realistic and intuitive effects. In the presence of high frame rate and high dynamic range videos, the coding efficiency of high-efficiency video coding (HEVC) falls short of meeting the storage and transmission demands of the video content. Therefore, versatile video coding (VVC) introduces a nested quadtree plus multi-type tree (QTMT) segmentation structure based on the HEVC standard, while also expanding the intra-prediction modes from 35 to 67. While the new technology introduced by VVC has enhanced compression performance, it concurrently introduces a higher level of computational complexity. To enhance coding efficiency and diminish computational complexity, this paper explores two key aspects: coding unit (CU) partition decision-making and intra-frame mode selection. Firstly, to address the flexible partitioning structure of QTMT, we propose a decision-tree-based series partitioning decision algorithm for partitioning decisions. Through concatenating the quadtree (QT) partition division decision with the multi-type tree (MT) division decision, a strategy is implemented to determine whether to skip the MT division decision based on texture characteristics. If the MT partition decision is used, four decision tree classifiers are used to judge different partition types. Secondly, for intra-frame mode selection, this paper proposes an ensemble-learning-based algorithm for mode prediction termination. Through the reordering of complete candidate modes and the assessment of prediction accuracy, the termination of redundant candidate modes is accomplished. Experimental results show that compared with the VVC test model (VTM), the algorithm proposed in this paper achieves an average time saving of 54.74%, while the BDBR only increases by 1.61%.

Suboptimal video coding for machines method based on selective activation of in‐loop filter

AbstractA conventional codec aims to increase the compression efficiency for transmission and storage while maintaining video quality. However, as the number of platforms using machine vision rapidly increases, a codec that increases the compression efficiency and maintains the accuracy of machine vision tasks must be devised. Hence, the Moving Picture Experts Group created a standardization process for video coding for machines (VCM) to reduce bitrates while maintaining the accuracy of machine vision tasks. In particular, in‐loop filters have been developed for improving the subjective quality and machine vision task accuracy. However, the high computational complexity of in‐loop filters limits the development of a high‐performance VCM architecture. We analyze the effect of an in‐loop filter on the VCM performance and propose a suboptimal VCM method based on the selective activation of in‐loop filters. The proposed method reduces the computation time for video coding by approximately 5% when using the enhanced compression model and 2% when employing a Versatile Video Coding test model while maintaining the machine vision accuracy and compression efficiency of the VCM architecture.

Sample-Based Gradient Edge and Angular Prediction for VVC Lossless Intra-Coding

Lossless coding is a compression method in the Versatile Video Coding (VVC) standard, which can compress video without distortion. Lossless coding has great application prospects in fields with high requirements for video quality. Since the current VVC standard is mainly designed for lossy coding, the compression efficiency of VVC lossless coding makes it hard to meet people’s needs. In order to improve the performance of VVC lossless coding, this paper proposes a sample-based intra-gradient edge detection and angular prediction (SGAP) method. SGAP utilizes the characteristics of lossless intra-coding to employ samples adjacent to the current sample as reference samples and performs prediction through sample iteration. SGAP aims to improve the prediction accuracy for edge regions, smooth regions and directional texture regions in images. Experimental results on the VVC Test Model (VTM) 12.3 reveal that SGAP achieves 7.31% bit-rate savings on average in VVC lossless intra-coding, while the encoding time is only increased by 5.4%. Compared with existing advanced sample-based intra-prediction methods, SGAP can provide significantly higher coding performance gain.

Coarse-to-Fine Network-Based Intra Prediction in Versatile Video Coding

After the development of the Versatile Video Coding (VVC) standard, research on neural network-based video coding technologies continues as a potential approach for future video coding standards. Particularly, neural network-based intra prediction is receiving attention as a solution to mitigate the limitations of traditional intra prediction performance in intricate images with limited spatial redundancy. This study presents an intra prediction method based on coarse-to-fine networks that employ both convolutional neural networks and fully connected layers to enhance VVC intra prediction performance. The coarse networks are designed to adjust the influence on prediction performance depending on the positions and conditions of reference samples. Moreover, the fine networks generate refined prediction samples by considering continuity with adjacent reference samples and facilitate prediction through upscaling at a block size unsupported by the coarse networks. The proposed networks are integrated into the VVC test model (VTM) as an additional intra prediction mode to evaluate the coding performance. The experimental results show that our coarse-to-fine network architecture provides an average gain of 1.31% Bjøntegaard delta-rate (BD-rate) saving for the luma component compared with VTM 11.0 and an average of 0.47% BD-rate saving compared with the previous related work.

Neural Network Based Rate Control for Versatile Video Coding

In this work, we propose a neural network based rate control algorithm for Versatile Video Coding (VVC). The proposed method relies on the modeling of the Rate-Quantization (R-Q) and Distortion-Quantization (D-Q) relationships in a data driven manner based upon the characteristics of prediction residuals. In particular, a pre-analysis framework is adopted, in an effort to obtain the prediction residuals which govern the Rate-Distortion (R-D) behaviors. By inferring from the prediction residuals with deep neural networks, the Coding Tree Unit (CTU) level R-Q and D-Q model parameters are derived, which could efficiently guide the optimal bit allocation. Subsequently, the coding parameters, including Quantization Parameter (QP) and λ, at both frame and CTU levels, are obtained according to allocated bit-rates. We implement the proposed rate control algorithm on VVC Test Model (VTM-13.0). Experimental results exhibit that the proposed rate control algorithm achieves 0.77% BD-Rate savings under Low Delay B (LDB) configurations when compared to the default rate control algorithm used in VTM-13.0. For Random Access (RA) configurations, 1.77% BD-Rate savings can be observed. Furthermore, with better bit-rate estimation, more stable buffer status can be observed, further demonstrating the advantages of the proposed rate control method.

Deep Multi-Task Learning Based Fast Intra-Mode Decision for Versatile Video Coding

The latest Versatile Video Coding (VVC) standard has significantly coding efficiency improvement compared with its ancestor High Efficiency Video Coding (HEVC) standard, but at the expense of over-high complexity. As measured by the VVC test model (VTM), the intra-mode comparison and selection in the rate-distortion optimization (RDO) search consume most of the encoding time. In this paper, we propose a deep multi-task learning based fast intra-mode decision approach via adaptively pruning off most redundant modes. First, we create a large-scale intra-mode database for VVC, including both normal angular modes and the newly introduced tools, i.e., intra sub-partition (ISP) and matrix-based intra prediction (MIP). Next, we propose a multi-task intra-mode decision network (MID-Net) model to effectively predict the most probable angular modes and whether to skip ISP and MIP modes. Then, a fast intra-coding workflow is designed accordingly, involving rough mode decision (RMD) acceleration and candidate mode list (CML) pruning. For the workflow output, the learning-oriented probability and the statistics-oriented probability are synthesized together to further improve the prediction accuracy, ensuring that only unnecessary intra-modes are skipped. Finally, experimental results show that our approach can significantly reduce 40.48% of encoding time of VVC intra-coding with negligible rate-distortion degradation, outperforming other state-of-the-art approaches.

An efficient geometric partitioning mode by adaptive blending method for screen content coding

AbstractGeometric partitioning mode (GPM) was newly adopted in Versatile Video Coding (VVC) to increase the partitioning precision of the arbitrary boundaries of actual objects. Note that the blending process of the GPM is not suitable for screen content with various sharp and strong edges. To alleviate this drawback, three new GPM matrices are introduced, and an efficient GPM is proposed to utilize four different GPM matrices and implicitly determine one of them depending on the discontinuity and gradients of the two predictors. The proposed method achieves an average BD‐rate reduction of 1.2% and 1.6% without additional computational complexity for random access and low‐delay B configurations, respectively, over the VVC test model (VTM18.0).

A High-Performance Rate Control Algorithm in Versatile Video Coding Based on Spatial and Temporal Feature Complexity

This paper presents a high-performance rate control (HPRC) algorithm for the Versatile Video Coding (VVC) standard that aims to achieve higher coding efficiency by considering spatial and temporal feature complexity. The HPRC algorithm differs from conventional rate control (RC) algorithms in that it uses an adaptive Canny operator (ACO) with a novel double-threshold algorithm to obtain the feature complexity of video content accurately. Moreover, an advanced bit allocation strategy at the frame-level and coding tree unit (CTU)-level is constructed to realize rational RC by thoroughly exploring the relationship between coding bits and complexity. To improve the precision of RC, an appropriate parameter updating based on the quasi-Newton algorithm is also proposed. Experimental results demonstrate that the proposed HPRC algorithm outperforms the default RC algorithm in VVC Test Model (VTM) 18.0, with Bjontegaard Delta Rate (BD-Rate) savings of 8.77 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> and 10.34 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> under Low Delay_P and Random Access configurations, respectively. Furthermore, the proposed algorithm also shows performance enhancements compared to other advanced algorithms.

A novel fast intra algorithm for VVC based on histogram of oriented gradient

The latest Versatile Video Coding (VVC) standard incorporates a series of effective and complex new intra coding tools, which obtains superior coding efficiency than the High Efficiency Video Coding (HEVC). However, this makes the intra coding more complicated and time-consuming. A fast algorithm for VVC from two aspects of fast mode decision and fast partition decision is proposed in this paper. For the fast mode decision, the relationship between bins with Histogram of Oriented Gradient (HOG) and intra modes is created for the mode selection, decreasing the planar modes for SATD and RDO. Moreover, we analyze the maximum bins to determine the final modes, and we use the modes of left and upper blocks as a reference for the current CU, which can early terminate RDO. Moreover, a two-step fast partition algorithm is proposed based on HOG for fast partition decision, in which two thresholds are investigated to control the uniformity of textures. The proposed fast algorithm is implemented on the VVC test model, and the experimental results show that it can achieve 69.07% time savings with only 2.96% BDBR increases averagely, which outperforms other relatively existing state-of-the-art methods. Moreover, to convince the universality of our algorithm, we further implement our method in Fraunhofer Versatile Video Encoder (VVenc) and Fraunhofer Versatile Video Decoder (VVdec), which have five settings to control the trade-off between encoding quality and efficiency for intra coding. The fast intra mode decision algorithm and fast partition algorithm decrease the complexity of intra coding for both VTM and VVenc, which shows the efficiency and universality of the proposed fast partition and fast mode decision algorithms.

Rate distortion optimization with adaptive content modeling for random-access versatile video coding

In this paper, we study the capability of improving the rate-distortion (RD) performance based on the adaptive content modeling in the Versatile Video Coding (VVC) standard. In particular, the frame-level dependent relationships and inherent RD relationship are explored. As such, the rate dependency, distortion dependency and inherent RD characteristics are fully utilized in the global rate distortion optimization (RDO) process, and the quantization parameter (QP) for each frame could be adaptively solved. To facilitate the adaptive QP calculation, a two-pass coding strategy is proposed. In the first-pass coding, the video statistics are sufficiently collected with the proposed Dual Motion Compensation and Residual Coding (DMCRC) method to generate the parameters for the content-aware models. During the second-pass coding, the optimal QP at the frame level is obtained by optimizing the global RD performance with the dependent and inherent models. The proposed algorithm is implemented on VVC test model (VTM-4.0) and achieves significant performance gain for test sequences with constant and varying scenes.

Fast Mode Decision Method of Multiple Weighted Bi-Predictions Using Lightweight Multilayer Perceptron in Versatile Video Coding

Versatile Video Coding (VVC), the state-of-the-art video coding standard, was developed by the Joint Video Experts Team (JVET) of ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) in 2020. Although VVC can provide powerful coding performance, it requires tremendous computational complexity to determine the optimal mode decision during the encoding process. In particular, VVC adopted the bi-prediction with CU-level weight (BCW) as one of the new tools, which enhanced the coding efficiency of conventional bi-prediction by assigning different weights to the two prediction blocks in the process of inter prediction. In this study, we investigate the statistical characteristics of input features that exhibit a correlation with the BCW and define four useful types of categories to facilitate the inter prediction of VVC. With the investigated input features, a lightweight neural network with multilayer perceptron (MLP) architecture is designed to provide high accuracy and low complexity. We propose a fast BCW mode decision method with a lightweight MLP to reduce the computational complexity of the weighted multiple bi-prediction in the VVC encoder. The experimental results show that the proposed method significantly reduced the BCW encoding complexity by up to 33% with unnoticeable coding loss, compared to the VVC test model (VTM) under the random-access (RA) configuration.

Efficient Rate Control in Versatile Video Coding With Adaptive Spatial–Temporal Bit Allocation and Parameter Updating

Despite the fact that Versatile Video Coding (VVC) has achieved superior coding performance, two major problems remain for the rate control (RC) model in VVC. First, the regions concerned by human eyes are not clear enough in the coded video due to the deviation between the target bit allocation strategy of the coding tree unit (CTU) in RC and the human visual attention mechanism (HVAM). Second, there are significant quality fluctuations in the coded video frames due to the inappropriate updating speed. To address the above problems, we propose an efficient rate control (ERC) model. Specifically, in order to make the coded video more consistent with the attention of human eyes, we extract texture and motion-based spatial-temporal information to guide the bit allocation at the CTU level. Furthermore, based on the quasi-Newton algorithm and bit error, we propose an adaptive parameter updating (APU) method with the proper updating speed to precisely control the bits per frame. The proposed ERC outperforms the default RC model of VVC Test Model (VTM) 9.1 by saving the average Bjøntegaard Delta Rate (BD-Rate) on full-frame video sequences by 3.60% and 4.94% under low delay P (LDP) and random access (RA) configurations respectively, with higher bitrate accuracy. Moreover, the Peak Signal-to-Noise Ratio (PSNR) and actual coded bits per frame in the video coded by the proposed ERC are more stable.

A Quality-of-Experience-Aware Framework for Versatile Video Coding-Based Video Transmission

Multimedia applications have expanded and resulted in huge demands for high resolution videos in order to enhance the end user Quality of Experience (QoE). Therefore, QoE becomes a critical factor that is often addressed in design considerations of Consumer Electronics (CE) devices. Furthermore, error prone transmission conditions can influence the QoE, potentially degrading the end-user multimedia experience. Although video coding standards such as Versatile Video Coding (VVC) take potential mitigation measures to improve the bandwidth reductions, it fails to address other significant network Quality of Service (QoS) parameters that impact transmission based QoE such as transmission delay and Packet Loss Rate (PLR) in the encoding processes. In response, this paper first derives a relationship between QoE that addresses the network QoS parameters and VVC dependent parameters. Subsequently, the derived QoE model is used with the VVC encoding process to adaptively optimize Lagrange parameters as well as the quantization parameter. Extensive experiments report significant improvements to VVC Test Model (VTM) 7.0, achieving superior QoE gains over the state-of-the-art research and up to 3.41% over the anchor implementations.

Fusion-Based Versatile Video Coding Intra Prediction Algorithm with Template Matching and Linear Prediction.

The new generation video coding standard Versatile Video Coding (VVC) has adopted many novel technologies to improve compression performance, and consequently, remarkable results have been achieved. In practical applications, less data, in terms of bitrate, would reduce the burden of the sensors and improve their performance. Hence, to further enhance the intra compression performance of VVC, we propose a fusion-based intra prediction algorithm in this paper. Specifically, to better predict areas with similar texture information, we propose a fusion-based adaptive template matching method, which directly takes the error between reference and objective templates into account. Furthermore, to better utilize the correlation between reference pixels and the pixels to be predicted, we propose a fusion-based linear prediction method, which can compensate for the deficiency of single linear prediction. We implemented our algorithm on top of the VVC Test Model (VTM) 9.1. When compared with the VVC, our proposed fusion-based algorithm saves a bitrate of 0.89%, 0.84%, and 0.90% on average for the Y, Cb, and Cr components, respectively. In addition, when compared with some other existing works, our algorithm showed superior performance in bitrate savings.

BVI-CC: A Dataset for Research on Video Compression and Quality Assessment

The video technology scenery has been very vivid over the past years, with novel video coding technologies introduced that promise improved compression performance over state-of-the-art technologies. Despite the fact that a lot of video datasets are available, representative content of the wide parameter space along with subjective evaluations of variations of encoded content from an unpartial end is required. In response to this requirement, this paper features a dataset, the BVI-CC. Three video codecs were deployed to create the variations of the encoded sequences: High Efficiency Video Coding Test Model (HM), AOMedia Video 1 (AV1), and Versatile Video Coding Test Model (VTM). Nine source video sequences were carefully selected to offer both diversity and representativeness in the spatio-temporal domain. Different spatial resolution versions of the sequences were created and encoded by all three codecs at pre-defined target bit rates. The compression efficiency of the codecs was evaluated with commonly used objective quality metrics, and the subjective quality of their reconstructed content was also evaluated through psychophysical experiments. Additionally, an adaptive bit rate (convex hull rate-distortion optimization across spatial resolutions) test case was assessed using both objective and subjective evaluations. Finally, the computational complexities of the tested codecs were examined. All data have been made publicly available as part of the dataset, which can be used for coding performance evaluation and video quality metric development.

High Efficiency Rate Control for Versatile Video Coding Based on Composite Cauchy Distribution

In this work, we propose a novel rate control algorithm for Versatile Video Coding (VVC) standard based on its distinct rate-distortion characteristics. By modelling the transform coefficients with the composite Cauchy distribution, higher accuracy compared with traditional distributions has been achieved. Based on the transform coefficient modelling, the theoretically derived R-Q and D-Q models which have been shown to deliver higher accuracy in characterizing RD characteristics for sequences with different content are incorporated into the rate control process. Furthermore, to establish an adaptive bit allocation scheme, the dependency between different levels of frames is modelled by a dependency factor to describe relationship between the reference and to-be-coded frames. Given the derived R-Q and D-Q relationships, as well as the dependency factor, an adaptive bit allocation scheme is developed for optimal bits allocation. We implement the proposed algorithm on VVC Test Model (VTM) 3.0. Experiments show that due to proper bit allocation, for low delay configuration the proposed algorithm can achieve 1.03% BD-Rate saving compared with the default rate control algorithm and 2.96% BD-Rate saving compared with fixed-QP scheme. Moreover, 1.29% BD-Rate saving and higher control accuracy have also been observed under the random access configuration.

Activity-Based Block Partitioning Decision Method for Versatile Video Coding

Versatile Video Coding (VVC), the latest international video coding standard, has more than twice the compression performance of High-Efficiency Video Coding (HEVC) through adopting various coding techniques. The multi-type tree (MTT) block structure offers more advanced flexible block partitioning by allowing the binary tree (BT) and ternary tree (TT) structures, as well as the quadtree (QT) structure. Because VVC selects the optimal block partition by performing encoding on all possible CU partitions, the encoding complexity increases enormously. In this paper, we observe the relationship between block partitions and activity that indicates block texture complexity. Based on experimental observations, we propose an activity-based fast block partitioning decision method to reduce the encoding complexity. The proposed method uses only information of the current block without using the information of neighboring or upper blocks, and also minimizes the dependency on QP. For these reasons, the proposed algorithm is simple and parallelizable. In addition, by utilizing the gradient calculation used in VVC’s ALF, a VVC-friendly fast algorithm was designed. The proposed method consists of two-step decision-making processes. The first step terminates the block partitioning early based on observed posterior probability through the relationship between the block size and activity per sample. Next, the sub-activities of the current block are used to determine the type and direction of partitioning. The experimental results show that in the all-intra configuration, the proposed method can reduce the encoding time of the VVC test model (VTM) by up to 45.15% with 2.80% BD-rate loss.

Perceptual Quality Assessment of HEVC and VVC Standards for 8K Video

With the growing data consumption of emerging video applications and users&#x2019; requirement for higher resolutions, up to 8K, a huge effort has been made in video compression technologies. Recently, versatile video coding (VVC) has been standardized by the moving picture expert group (MPEG), providing a significant improvement in compression performance over its predecessor high efficiency video coding (HEVC). In this paper, we provide a comparative subjective quality evaluation between VVC and HEVC standards for 8K resolution videos. In addition, we evaluate the perceived quality improvement offered by 8K over UHD 4K resolution. The compression performance of both VVC and HEVC standards has been conducted in random access (RA) coding configuration, using their respective reference software, VVC test model (VTM-11) and HEVC test model (HM-16.20). Objective measurements, using PSNR, MS-SSIM and VMAF metrics have shown that the bitrate gains offered by VVC over HEVC for 8K video content are around 31&#x0025;, 26&#x0025; and 35&#x0025;, respectively. Subjectively, VVC offers an average of around 41&#x0025; of bitrate reduction over HEVC for the same visual quality. A compression gain of 50&#x0025; has been reached for some tested video sequences regarding a Student&#x2019;s t-test analysis. In addition, for most tested scenes, a significant visual difference between uncompressed 4K and 8K has been noticed.