Video Coding Experts Group Research Articles

Fast Mode Decision Method of Multiple Weighted Bi-Predictions Using Lightweight Multilayer Perceptron in Versatile Video Coding

Versatile Video Coding (VVC), the state-of-the-art video coding standard, was developed by the Joint Video Experts Team (JVET) of ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) in 2020. Although VVC can provide powerful coding performance, it requires tremendous computational complexity to determine the optimal mode decision during the encoding process. In particular, VVC adopted the bi-prediction with CU-level weight (BCW) as one of the new tools, which enhanced the coding efficiency of conventional bi-prediction by assigning different weights to the two prediction blocks in the process of inter prediction. In this study, we investigate the statistical characteristics of input features that exhibit a correlation with the BCW and define four useful types of categories to facilitate the inter prediction of VVC. With the investigated input features, a lightweight neural network with multilayer perceptron (MLP) architecture is designed to provide high accuracy and low complexity. We propose a fast BCW mode decision method with a lightweight MLP to reduce the computational complexity of the weighted multiple bi-prediction in the VVC encoder. The experimental results show that the proposed method significantly reduced the BCW encoding complexity by up to 33% with unnoticeable coding loss, compared to the VVC test model (VTM) under the random-access (RA) configuration.

Overview of Versatile Video Coding (H.266/VVC) and Its Coding Performance Analysis

Versatile Video Coding (H.266/VVC) is the newest video coding standard jointly developed by the Joint Video Experts Team (JVET), which is organized by the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). H.266/VVC provides about 40% bitrate reduction compared with High Efficiency Video Coding (H.265/HEVC) for the same visual quality. This paper introduces in detail the core structure of H.266/VVC by highlighting its features within block partitioning structure, intra/inter prediction, transform, quantization, and in-loop filtering, compared to its predecessor (H.265/HEVC). H.266/VVC yields significantly improved the coding performance, but it increased the computational complexity, particularly for the encoder side, which remains a problem to be tackled for successful commercialization in the future. This paper examined the statistical performance of H.266/VVC coding tools from the bitstreams encoded by the VVC test model (VTM12.0) encoder through rate-distortion optimization under the JVET common test conditions. In addition, the complexity and performance analyses are conducted on the block partitioning structure and group of picture structure. It is expected that an optimized H.266/VVC encoder can be designed and developed by minimizing the coding loss based on the analysis data.

Low complexity inter coding scheme for Versatile Video Coding (VVC)

The latest coding standard Versatile Video Coding (VVC) developed by the Joint Video Experts Team (JVET) and Video Coding Experts Group (VCEG) was finalized in 2020. By introducing several new coding techniques, VVC improves the compression efficiency by 50% compared with H.265/HEVC. However, the coding complexity increases dramatically, which obstructs it from real-time application. To tackle this issue, a fast inter coding algorithm utilizing coding information is proposed to speed up the coding process. First, by analyzing the coding areas of the neighboring CUs, we predict the coding area of the current CU to terminate unnecessary splitting modes. Then, the temporally optimal coding mode generated during the prediction process is further utilized to shrink the candidate modes to speed up the coding process. Finally, the distribution of neighboring prediction modes are exploited to measure the motion complexity of the current CU, based on which the unnecessary prediction modes can be skipped earlier. Experimental results demonstrate that the proposed method can reduce the coding complexity by 40.08% on average with 0.07 dB BDPSNR decrease and 1.56% BDBR increase, which outperforms the state-of-the-art approach.

Overview of the Versatile Video Coding (VVC) Standard and its Applications

Versatile Video Coding (VVC) was finalized in July 2020 as the most recent international video coding standard. It was developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) to serve an ever-growing need for improved video compression as well as to support a wider variety of today’s media content and emerging applications. This paper provides an overview of the novel technical features for new applications and the core compression technologies for achieving significant bit rate reductions in the neighborhood of 50% over its predecessor for equal video quality, the High Efficiency Video Coding (HEVC) standard, and 75% over the currently most-used format, the Advanced Video Coding (AVC) standard. It is explained how these new features in VVC provide greater versatility for applications. Highlighted applications include video with resolutions beyond standard- and high-definition, video with high dynamic range and wide color gamut, adaptive streaming with resolution changes, computer-generated and screen-captured video, ultralow-delay streaming, 360° immersive video, and multilayer coding e.g., for scalability. Furthermore, early implementations are presented to show that the new VVC standard is implementable and ready for real-world deployment.

Efficient mode decision scheme based on edge detection with Gaussian pulse for Intra-prediction in H.264/AVC

Advanced Video Coding (H.264/AVC) is one of the most important video coding standards which was developed by the ITU-T Video Coding Experts Group (VCEG) together with the ISO/IEC Moving Picture Experts Group (MPEG). Due to the complex procedures needed to find the optimal mode in intra prediction mode stage of H.264/AVC, an efficient mode decision is suggested in this paper. The main objectives of the proposed work are improving the compression efficiency of video coding and minimizing the degradation of video quality. The proposed algorithm is based on the gradient method for edge detection using the Prewitt filter to determine the texture direction of a block to be predicted in the luminance component of a video frame. To decide either implemented the DC mode or the edge-detection based gradient method to predict block under consideration, a predefined threshold value is used for the homogeneity test of each macro-block. To reduce the Bit Rate of the compressed video, Gaussian pulse modulation is also suggested to the Discrete Cosine Transformation (DCT) components of each macro-block during the coding process. The proposed algorithm is carried out on a luminance component of a set of different YUV video sequences with different resolutions and different quantization parameters using MATLAB. The suggested algorithm is evaluated using different metrics which are Peak Signal to Noise Ratio (PSNR) and bit rate measured by Bjontegaard Delta method. The simulation results show the ability of the proposed algorithm in enhancing the BDPSNR averagely by 1.14 dB and reducing the BDBR averagely by 11.34 %for QCIF sequences compared to the existing state-of-the-art fast intra-prediction mode decision algorithms.

Performance Comparison of Emerging EVC and VVC Video Coding Standards with HEVC and AV1

<i>The recent dramatic increase in video content consumption requires efficient video coding standards, which is specifically true for ultrahigh-definition (UltraHD) resolutions, such as 4K and 8K (i.e., 3840&#x00D7;2160 or 7680&#x00D7;4320 resolutions in terms of luma samples, respectively). The well-known high-efficiency video coding (HEVC) [H.265/Moving Pictures Expert Group (MPEG)-H] standard was approved in 2013. Although HEVC provides approximately 50&#x0025; coding gain compared to its predecessor advanced video coding (AVC) (H.264/MPEG-4), its adoption is still relatively slow. In addition, larger bitrate savings than those provided by HEVC are currently in demand. At the same time, work on the versatile video coding (VVC) and essential video coding (EVC) standards started in 2018. After intensive development efforts that continued for two and a half years, these two video coding standards have been recently finalized. VVC (H.266/MPEG-I) was developed jointly by the MPEG and the International Telecommunication Union-Telecommunication (ITU-T) Video Coding Experts Group (VCEG). On the other hand, EVC (MPEG-5) is an MPEG-only effort. In this paper, we compare the performance of EVC and VVC, in terms of both coding gains and computational complexity, to their predecessor&#x2014;the HEVC video coding standard. In addition, given the growing popularity of the AV1 video codec, which was recently developed by the Alliance for Open Media (AOM), we also include AV1 as an alternative baseline and provide corresponding comparison results. According to the experimental results, which have been carried out in a constant bitrate (CBR) mode, EVC provides about 30&#x0025; bitrate savings compared to HEVC for encoding 4K/2160p entertainment content (such as VoD) in terms of Bj&#x00F8;ntegaard-delta bitrate (BD-BR) peak-signal-to-noise ratio (PSNR)_YUV, while introducing an encoding computational complexity increase of approximately five times. VVC provides larger bitrate savings of about 40&#x0025; at a price of a significant encoding computational complexity increase of more than nine times. When the performance of HEVC CBR encoding (i.e., with the rate control disabled) was compared to that of AV1 VBR encoding (i.e., with the rate control enabled), it was found that AV1 provides bitrate savings of about 20&#x0025; compared to HEVC for encoding 4K/2160p video sequences, as a tradeoff of an encoding computational complexity increase by a factor of approximately four. The authors find both EVC and VVC to be very promising successors to HEVC in terms of coding gains and computational complexity, but the jury is still out on the speed of their adoption</i>.

An Overview of the MPEG-5 Essential Video Coding Standard [Standards in a Nutshell

Since the 1970s, various image and video coding techniques have been explored, and some of them have been included in the video coding standards issued by the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Motion Pictures Expert Group (MPEG) and International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) Video Coding Experts Group (VCEG). MPEG is the most successful standards development organization (SDO) for multimedia compression standardization. In particular, most of the widely deployed video coding standards of the past 30 years have been developed within this working group. One of the first, standardized in 1996, was the MPEG-2 standard [1], which is still in use as a digital TV standard in many countries.

Accelerating the CU partitioning decision in an HEVC-JEM transcoder

High Efficiency Video Coding (HEVC) is currently the latest video coding standard available on the market, and it is able to offer up to twice the coding efficiency, in the range of 50% bitrate reduction for the same video quality, of the previous standard, namely H.264/Advanced Video Coding (AVC). HEVC was standardized in 2013 for videos up to a resolution of 2K. However, the popularity of 4K videos is increasing due to the growing use of video-on-demand platforms. Therefore, the ITU-T Video Coding Expert Group (VCEG) and the ISO/IEC Moving Picture Expert Group (MPEG) created the Joint Video Exploration Team (JVET) in 2015 to design the future video coding technology under the Joint Exploration Model (JEM), which its latest version achieves an improvement in coding efficiency of 30%, but at a high cost in terms of computational complexity (10×) with respect to HEVC. The new video standard is expected to be ready in 2020, so it is necessary to find efficient mechanisms to convert current content to the new format adopted in JEM. In this regard, our proposal consists in a probabilistic classifier based on Naive-Bayes that enables the prediction of the splitting decision at the first quadtree level in JEM, reducing the computational complexity of the transcoding process from HEVC to this new standard. The experimental results show a good trade-off between coding efficiency and complexity compared with the anchor transcoder, obtaining a time reduction up to 12.71% at the expense of low coding efficiency penalties in the configurations evaluated.

Analysis of Emerging Video Codecs: Coding Tools, Compression Efficiency

The video compression community, from technologists to customers, is affected by the competition between two lines of development for future video compression technology beyond high-efficiency video coding (HEVC): AV1, as developed by the industry-driven Alliance for Open Media (AOM), and Versatile Video Coding (VVC), as developed and studied by the Joint Video Experts Team (JVET) on Future Video Coding of ITU-T Video Coding Expert Group (VCEG) and International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Motion Picture Experts Group (MPEG). Both groups have asserted that their respective codecs, AV1 and VVC, outperform the state-of-the-art HEVC. However, in many cases, related studies are evaluating video codec performances based on different test conditions, scenarios, or codec implementations, which makes comparisons sometimes unfair and at least difficult to interpret. This paper provides a neutral and well-balanced evaluation of compression and runtime performance for both AV1 and VVC codecs, using their respective reference models, in comparison to two HEVC codec implementations: HEVC Test Model (HM) as the HEVC standard reference model and x265 as a well-known open-source HEVC codec implementation. Comparison of differentiating features of AV1 and VVC over HEVC is also reported. Our main findings show that, based on an objective analysis, AV1 allows bitrate reduction of 10% in comparison to HM for the same peak signal-to-noise ratio (PSNR) and Structural SIMilarity (SSIM) quality scores while increasing encoding runtime by 55.5x. However, we show that this objective advantage in coding efficiency does not necessarily translate into gain in terms of subjective quality assessment. Overall, objective results show VVC to be the best candidate for compression technology beyond HEVC.

Tools for Video Coding Beyond HEVC: Flexible Partitioning, Motion Vector Coding, Luma Adaptive Quantization, and Improved Deblocking

This paper provides a description and analysis of technology included in two contributions to the Call for Proposals for Video Compression with Capability beyond HEVC. This Call for Proposals was issued jointly by the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG). The contribution emphasized a flexible and rectangular partitioning structure, which was combined with both new and existing coding tools. New coding tools included methods for improved motion vector coding, quantization signaling, and deblocking; while existing tools were largely methods studied in the Joint Exploration Model software. Results show the efficacy of the approach. Using the sequences and test conditions defined in the Call for Proposals, the described approach provided an average bitrate reduction, relative to an HEVC anchor, of 41.2% and 35.7% for 4K and HD test sequences, respectively. Moreover, the method achieved a compression performance of 34.3% and 32.2% for high dynamic range content using the perceptual quantizer (PQ) and Hybrid-Log Gamma transfer functions, respectively.

The Joint Exploration Model (JEM) for Video Compression With Capability Beyond HEVC

This paper provides an overview of the coding algorithms of the Joint Exploration Model (JEM) for video compression with capability beyond HEVC, which was developed by the Joint Video Exploration Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). The goal of the JEM development and experimentation was to provide evidence that sufficient coding efficiency improvement over the High Efficiency Video Coding (HEVC) standard can be achieved, which would justify the need for a new video coding standard with a compression capability significantly exceeding that of HEVC. The development of the JEM provided an ability to conduct studies toward that goal in a verifiable and collaborative manner and led to the launching of the project to develop the new Versatile Video Coding (VVC) standard. Objective metric gains exceeding 30% were measured for most of the tested high-resolution video content that represents current demanding new applications, and subjective testing using human observers showed even more benefit.

A Unified Video Codec for SDR, HDR, and 360° Video Applications

The ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG) issued in October 2017 a joint Call for Proposals (CfP) on video compression with capability beyond HEVC. The joint CfP included three categories of content: standard dynamic range (SDR), high dynamic range and wide color gamut (HDR/WCG), and 360° omni-directional video (360°). This paper describes a response to the joint CfP that considers all three categories of video content. The core codec in the response is designed based on the joint exploration model (JEM) reference software. The key coding tools in the JEM are significantly simplified to reduce both average and worst-case complexity for hardware design with negligible coding performance loss. Furthermore, two additional coding tools are used to further improve coding efficiency. For the HDR and 360° categories, additional coding tools specifically designed to optimize the compression efficiency and subjective quality of that specific content category are included. Further, some SDR coding tools are modified to alleviate subjective quality problems. For the random access configuration, compared to the HEVC test model (HM) anchor, the proposed video codec achieves average luma rate savings of 35.7%, 31.3%, and 33.9% for the SDR, HDR, and 360° categories, respectively.

Fast 3D-HEVC PU size decision algorithm for depth map intra-video coding

High-Efficiency Video Coding (HEVC)-based 3D video coding (3D-HEVC) is the most recent standard and last exertion of ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) for 3D video coding using a new data video format called Multi-View Video plus Depth map (MVD). This new standard achieves a high coding improvement. In any case, one of the most critical difficulties in 3D-HEVC is time computational complexity. The depth map intra-prediction is a critical factor in 3D-HEVC intra-coding, in which, the 3D-HEVC uses a highly adaptable Coding Unit (CU) structure with a specific end goal to expand the coding efficiency of all depth map characteristics. However, it results in an enormous Rate Distortion Optimization Cost (RDO-Cost) because of the broad recursive search for the best CU size from $$64\times 64$$ down to $$4\times 4$$ . This computational complexity excludes the 3D-HEVC from true and real-time application. Hence, it is imperative to build up an algorithm to diminish the complexity of the size decision in depth map intra-coding. To determine the previously mentioned issue, this paper proposes an effective 3D-HEVC PU size decision algorithm for depth map intra-video coding based on tensor features and statistical data analyses. The experimental results demonstrate that the proposed model diminishes the complexity of depth map size decision significantly with low rate distortion increase.

Fast Depth Map Intra Coding for 3D Video Compression Based Tensor Feature Extraction and Data Analysis

3D high-efficiency video coding (3D-HEVC) is the latest standard for 3D video compression created by the ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) based on a new video format called multiview texture videos plus depth maps (MVDs). In 3D-HEVC depth map intra prediction, the test model uses the conventional HEVC intra modes and new supplementary coding tools called depth modeling modes (DMMs) to better preserve the sharp edges of depth maps. These intra prediction modes fundamentally enhance the depth map intra coding efficiency. Although this process enables high coding efficiency, it also results in a very high encoding complexity, which limits the use of the 3D-HEVC encoder in practical and real-world applications. For most cases, the conventional intra coding and DMMs can be skipped if the current region is classified as a homogenous region. Bringing this intuitive approach to practice, this paper proposes fast depth map intra coding based on tensor feature extraction and data analysis. The experimental results show that the proposed intra model decision brings a good complexity reduction with negligible loss of rate distortion performance.

HEVC optimization based on human perception for real-time environments

High-Efficiency Video Coding (HEVC) is the new emerging video coding standard of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). The HEVC standard provides a significant improvement in compression efficiency in comparison with existing standards such as H264/AVC by means of greater complexity. In this paper we will examine several HEVC optimizations based on image analysis to reduce its huge CPU, resource and memory expensive encoding process. The proposed algorithms optimize the HEVC quad-tree partitioning procedure, intra/inter prediction and mode decision by means of H264-based methods and spatial and temporal homogeneity analysis which is directly applied to the original video. The validation process of these approaches was conducted by taking into account the human visual system (HVS). The adopted solution makes it possible to perform HEVC real time encoding for HD sequences on a low cost processor with negligible quality loss. Moreover, the frames pre-processing leverages the logic units and embedded hardware available on an Intel GPU, so the execution time of these stages are negligible for the encoding processor.

Acceleration of the integer motion estimation in JEM through pre-analysis techniques

ITU-T Video Coding Expert Group and ISO/IEC Moving Picture Expert Group are studying the potential need for standardization of the future video coding technology with a compression capability that significantly exceeds that of the current High Efficiency Video Coding (HEVC) standard, including its current extensions. Both groups are working together on this exploration activity in a collaboration effort known as Joint Video Exploration Team to evaluate compression technology designs proposed by their experts in this area. Preliminary results show that the new model achieves 25% bitrate reduction, but at a cost of extremely high computational complexity (11 $$\times {}$$ ) with respect to HEVC. This paper proposes a pre-analysis algorithm designed to extract motion information of a frame, which is later used in the Motion Estimation (ME) module to speed up the encoder, showing that around 27% of the reference frames can be skipped and that more than 62% of the time is saved in the integer ME operation with a negligible impact of 0.11% in BD-rate.

Performance Study of HEVC and H.264 Video Coding Standards

This paper summarizes efforts in describing the main concepts of the two recent video coding standards H.265/MPEG-HEVC (High-Efficiency Video Coding) and H.264/MPEG-AVC. The High Efficiency Video Coding (HEVC) is the successor video coding standard to H.264/MPEG-4 AVC (Advanced Video Coding) and was developed in 2013 by the Joint Collaborative Team on Video Coding (JCT-VC) from the ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG). An overview of the technical characteristics of the High Efficiency Video Coding (HEVC) standard is presented. It has been shown that the HEVC standard provides a significant improvement on the compression performance compared with the H.264 AVC. The FFMPEG codec is used as a research tool for studying the performance analysis and comparison of the HEVC and H.264 AVC compression standers. Evaluation metrics for comparison includes the PSNR (peak-peak Signal to Noise Ratio), VQM (Video Quality Model), MSAD, Delta and SSIM of HEVC and H.264 AVC for several video sequences and bit rates.

H.264 Encoder Parameter Optimization for Encoded Wireless Multimedia Transmissions

Wireless multimedia sensor network has been renewed to the distributed model by incorporating the single board computer (i.e., camera nodes and mobile beacons). The redesigned camera nodes, such as unmanned aerial vehicle and mobile beacon nodes are limited in terms of memory, computation power, bandwidth, and node power while the real-time processing requires the multimedia contents transmission over the wireless interface in the presence of all these constraints. To tackle this problem, advanced video compression schemes have been adopted by which bit rate overhead over the network is greatly improved. H.264/moving picture experts group (MPEG)-4 is a state-of-the-art video compression standard developed jointly by International Telecommunication Union Telecommunication Standardization Sector video coding experts group and ISO/Joint Technical Committee of International Electrotechnical Commission MPEG. The H.264 provides a number of tunable parameters (e.g., quantization parameter (QP) and number of reference frames (NoRF)) for tailoring the encoding process as per requirements. In this paper, we propose a framework consisting of a joint parameter cost function to find the best trade-off between QP and NoRF with the policy of controlling the bit-rate (BR) while maintaining reasonably good video quality without compromising on processing delay (PD) and nodal power consumption (NPC). The framework has been proposed by transforming the multi-objective problem into a convex optimization problem. It is shown that an optimal trade-off between these conflicting objectives can be achieved by choosing appropriate values for QP and NoRF. Results for optimal values of QP and NoRF with corresponding trade-off between BR, quality of multimedia contents, low PD and low NPC are presented.

Efficient HEVC selective stream encryption using chaotic logistic map

At present, the digital video encryption technology has become an interest research topic as a result of very rapid evolution in the application of real-time video over the Internet. So this paper presents a new method for encrypting the selective sensitive data of the latest video coding standard, which called High-Efficiency Video Coding (HEVC). The High-Efficiency Video Coding was founded in 2013 by the Joint Collaborative Team on Video Coding (JCT-VC) from the ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG). The proposed selective encryption HEVC video technique uses the low complexity overhead chaotic logistic map (CLM) to encrypt the sign bits of the Motion Vector Difference (MVD) and the Discrete Cosine Transform (DCT) coefficients in the entropy stage of the process of video encoding. The contribution of the proposed CLM-based HEVC SE is to encrypt the sensitive video bits with the features of low complexity overhead, fast encoding time, keeping the HEVC constant bitrate and format compliant. Also, this paper introduces a comparative study between the proposed CLM-based HEVC SE and the Glenn HEVC SE that uses the Advanced Encryption Standard (AES). Experimental results demonstrate the main feature of the proposed CLM-based HEVC SE, which turned out to save the time of the video encoding with remaining of the near visual distortion of the encrypted video stream by Glenn HEVC SE. This feature is due to the low complexity of the CLM-based encryption employed in the proposed CLM-based HEVC SE scheme instead of using the AES in the Glenn HEVC SE. A course of security investigation experiments is performed for the proposed CLM-based HEVC SE including the main security performance metrics like encryption quality, key space, statistical and sensitivity tests. The achieved test results ensured the superiority of the proposed CLM-based HEVC SE for digital video streams.

Fast coding unit selection and motion estimation algorithm based on early detection of zero block quantified transform coefficients for high‐efficiency video coding standard

High-efficiency video coding (HEVC) is the newest video coding standard developed by the joint video team, consisting of ITU-T video coding experts group and ISO/IEC Moving Picture Experts Group. The HEVC standard has aggregated an exhaustive algorithm for mode decision based on a recursive quad-tree structured coding tree block. Moreover, several specific features have been incorporated into the motion estimation (ME) process to improve its coding efficiency. However, they resulted in very high computational complexity. To accelerate the encoding process, fast mode decision algorithms for the partitioning module and also for the ME module were proposed in this study. These algorithms are based on early zero block detection technique. To improve the efficiency of these algorithms, an overall algorithm which combines the two techniques has been implemented. The performance of the proposed algorithm was checked through a comparative analysis in terms of encoding time and compression rate. Compared to HEVC test model 10.0, the authors’ proposed algorithms bring a great reduction of the HEVC complexity encoder with a saving time, which can reach 25% in average for different tested videos and a slight coding loss in terms of image quality and compression rate.