Joint Exploration Model Research Articles

Performance Overview of the Latest Video Coding Proposals: HEVC, JEM and VVC.

The audiovisual entertainment industry has entered a race to find the video encoder offering the best Rate/Distortion (R/D) performance for high-quality high-definition video content. The challenge consists in providing a moderate to low computational/hardware complexity encoder able to run Ultra High-Definition (UHD) video formats of different flavours (360°, AR/VR, etc.) with state-of-the-art R/D performance results. It is necessary to evaluate not only R/D performance, a highly important feature, but also the complexity of future video encoders. New coding tools offering a small increase in R/D performance at the cost of greater complexity are being advanced with caution. We performed a detailed analysis of two evolutions of High Efficiency Video Coding (HEVC) video standards, Joint Exploration Model (JEM) and Versatile Video Coding (VVC), in terms of both R/D performance and complexity. The results show how VVC, which represents the new direction of future standards, has, for the time being, sacrificed R/D performance in order to significantly reduce overall coding/decoding complexity.

Low-Complexity Reconfigurable DCT-V Architecture

This brief presents a low-complexity, reconfigurable architecture for the Discrete Cosine Transform (DCT) of type V (DCT-V) of length 32. The proposed architecture can be reconfigured to compute five DCT-V of length 4 with negligible area overhead. As the DCT-V is one of the odd type transforms employed in the Adaptive Multiple Transform (AMT) scheme, the effect of fixed point implementation has been assessed in the Joint Exploration Model (JEM) developed by the JVET group for the Versatile-Video-Coding (VVC) forthcoming standard. Simulation results show that the proposed architecture is not only low-complexity and reconfigurable, but features also imperceptible quality loss. Moreover, when implemented in 90 nm CMOS technology it occupies only 90k eq. gates running at 187 MHz.

Hybrid Video Codec Based on Flexible Block Partitioning With Extensions to the Joint Exploration Model

This article describes the main video coding technologies included in a joint proposal submitted by Qualcomm and Technicolor, in response to a Call for Proposals (CfP) issued by ITU-T SG16 WP3 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in Oct. 2017. The proposal contains the majority of the tools that have been adopted into the Joint Exploration Model (JEM), developed in the exploratory phase that preceded the CfP. A flexible multi-tree type (MTT) block-partitioning scheme is proposed to extend the quadtree and binary tree (QTBT) based partitioning in JEM by including triple tree (TT) and asymmetric binary tree (ABT) partitions. In addition, several JEM tools in intra and inter prediction, transforms and arithmetic coding are modified, and new tools such as sign prediction and motion compensated padding are proposed. Objective standard dynamic range (SDR) gains of 43.1% and 15.5% in terms of average luma BD-rate improvement have been achieved for the CfP constraint set 1 (random-access configuration) relative to HEVC/H.265 (HM) and JEM anchors, respectively. For the CfP constraint set 2 (low-delay configuration), the average luma BD-rate improvements are 33.7% relative to the HM anchor and 12.7% relative to the JEM anchor. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

Accelerating the CU partitioning decision in an HEVC-JEM transcoder

High Efficiency Video Coding (HEVC) is currently the latest video coding standard available on the market, and it is able to offer up to twice the coding efficiency, in the range of 50% bitrate reduction for the same video quality, of the previous standard, namely H.264/Advanced Video Coding (AVC). HEVC was standardized in 2013 for videos up to a resolution of 2K. However, the popularity of 4K videos is increasing due to the growing use of video-on-demand platforms. Therefore, the ITU-T Video Coding Expert Group (VCEG) and the ISO/IEC Moving Picture Expert Group (MPEG) created the Joint Video Exploration Team (JVET) in 2015 to design the future video coding technology under the Joint Exploration Model (JEM), which its latest version achieves an improvement in coding efficiency of 30%, but at a high cost in terms of computational complexity (10×) with respect to HEVC. The new video standard is expected to be ready in 2020, so it is necessary to find efficient mechanisms to convert current content to the new format adopted in JEM. In this regard, our proposal consists in a probabilistic classifier based on Naive-Bayes that enables the prediction of the splitting decision at the first quadtree level in JEM, reducing the computational complexity of the transcoding process from HEVC to this new standard. The experimental results show a good trade-off between coding efficiency and complexity compared with the anchor transcoder, obtaining a time reduction up to 12.71% at the expense of low coding efficiency penalties in the configurations evaluated.

General Video Coding Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

After the development of the High-Efficiency Video Coding Standard (HEVC), ITU-T VCEG and ISO/IEC MPEG formed the Joint Video Exploration Team (JVET), which started exploring video coding technology with higher coding efficiency, including development of a Joint Exploration Model (JEM) algorithm and a corresponding software implementation. The technology explored in the last version of the JEM further increases the compression capabilities of the hybrid video coding approach by adding new tools, reaching up to 30% bit rate reduction compared to HEVC based on the Bjontegaard delta bit rate (BD-rate) metric, and further improvement beyond that in terms of subjective visual quality. This provided enough evidence to issue a joint Call for Proposals (CfP) for a new standardization activity now known as Versatile Video Coding (VVC). All technology proposed in the responses to the CfP was based on the classic block-based hybrid video coding design, extending it by new elements of partitioning, intra- and inter-picture prediction, prediction signal filtering, transforms, quantization/scaling, entropy coding, and in-loop filtering. This article provides an overview of technology that was proposed in the responses to the CfP, with a focus on techniques that were not already explored in the JEM context.

The Joint Exploration Model (JEM) for Video Compression With Capability Beyond HEVC

This paper provides an overview of the coding algorithms of the Joint Exploration Model (JEM) for video compression with capability beyond HEVC, which was developed by the Joint Video Exploration Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). The goal of the JEM development and experimentation was to provide evidence that sufficient coding efficiency improvement over the High Efficiency Video Coding (HEVC) standard can be achieved, which would justify the need for a new video coding standard with a compression capability significantly exceeding that of HEVC. The development of the JEM provided an ability to conduct studies toward that goal in a verifiable and collaborative manner and led to the launching of the project to develop the new Versatile Video Coding (VVC) standard. Objective metric gains exceeding 30% were measured for most of the tested high-resolution video content that represents current demanding new applications, and subjective testing using human observers showed even more benefit.

Tools for Video Coding Beyond HEVC: Flexible Partitioning, Motion Vector Coding, Luma Adaptive Quantization, and Improved Deblocking

This paper provides a description and analysis of technology included in two contributions to the Call for Proposals for Video Compression with Capability beyond HEVC. This Call for Proposals was issued jointly by the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG). The contribution emphasized a flexible and rectangular partitioning structure, which was combined with both new and existing coding tools. New coding tools included methods for improved motion vector coding, quantization signaling, and deblocking; while existing tools were largely methods studied in the Joint Exploration Model software. Results show the efficacy of the approach. Using the sequences and test conditions defined in the Call for Proposals, the described approach provided an average bitrate reduction, relative to an HEVC anchor, of 41.2% and 35.7% for 4K and HD test sequences, respectively. Moreover, the method achieved a compression performance of 34.3% and 32.2% for high dynamic range content using the perceptual quantizer (PQ) and Hybrid-Log Gamma transfer functions, respectively.

A Unified Video Codec for SDR, HDR, and 360° Video Applications

The ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG) issued in October 2017 a joint Call for Proposals (CfP) on video compression with capability beyond HEVC. The joint CfP included three categories of content: standard dynamic range (SDR), high dynamic range and wide color gamut (HDR/WCG), and 360° omni-directional video (360°). This paper describes a response to the joint CfP that considers all three categories of video content. The core codec in the response is designed based on the joint exploration model (JEM) reference software. The key coding tools in the JEM are significantly simplified to reduce both average and worst-case complexity for hardware design with negligible coding performance loss. Furthermore, two additional coding tools are used to further improve coding efficiency. For the HDR and 360° categories, additional coding tools specifically designed to optimize the compression efficiency and subjective quality of that specific content category are included. Further, some SDR coding tools are modified to alleviate subjective quality problems. For the random access configuration, compared to the HEVC test model (HM) anchor, the proposed video codec achieves average luma rate savings of 35.7%, 31.3%, and 33.9% for the SDR, HDR, and 360° categories, respectively.

A Novel Consistent Quality Driven for JEM Based Distributed Video Coding

Distributed video coding (DVC) is an attractive and promising solution for low complexity constrained video applications, such as wireless sensor networks or wireless surveillance systems. In DVC, visual quality consistency is one of the most important issues to evaluate the performance of a DVC codec. However, it is the fact that the quality of the decoded frames that is achieved in most recent DVC codecs is not consistent and it is varied with high quality fluctuation. In this paper, we propose a novel DVC solution named Joint exploration model based DVC (JEM-DVC) to solve the problem, which can provide not only higher performance as compared to the traditional DVC solutions, but also an effective scheme for the quality consistency control. We first employ several advanced techniques that are provided in the Joint exploration model (JEM) of the future video coding standard (FVC) in the proposed JEM-DVC solution to effectively improve the performance of JEM-DVC codec. Subsequently, for consistent quality control, we propose two novel methods, named key frame quantization (KF-Q) and Wyner-Zip frame quantization (WZF-Q), which determine the optimal values of the quantization parameter (QP) and quantization matrix (QM) applied for the key and WZ frame coding, respectively. The optimal values of QP and QM are adaptively controlled and updated for every key and WZ frames to guarantee the consistent video quality for the proposed codec unlike the conventional approaches. Our proposed JEM-DVC is the first DVC codec in literature that employs the JEM coding technique, and then all of the results that are presented in this paper are new. The experimental results show that the proposed JEM-DVC significantly outperforms the relevant DVC benchmarks, notably the DISCOVER DVC and the recent H.265/HEVC based DVC, in terms of both Peak signal-to-noise ratio (PSNR) performance and consistent visual quality.

Spherical Coordinates Transform-Based Motion Model for Panoramic Video Coding

Panoramic video is produced and displayed as if it is supported on sphere, but it is usually mapped to two-dimensional (2D) plane for compression purpose. The mapping from sphere to 2D plane inevitably causes deformation of video content, which shall be taken into account in panoramic video coding. In this paper, we propose a new motion model based on spherical coordinates transform to explicitly address the deformation problem. Different from the traditional 2D translational motion model, our model assumes a 3D translational motion, which is characterized by a 2D motion vector and a relative depth parameter. Based on spherical coordinates transform, we can deduce the motion compensation and motion estimation algorithms for our proposed motion model. We then integrate the algorithms into the joint exploration model (JEM) scheme for panoramic video coding, and perform experiments on some typical equirectangular projection format video sequences. Experimental results show that our proposed method leads to up to 10.7% and on average 2.6% BD-rate reduction compared to the JEM anchor on our test sequences, with no increase of decoding time.

Intra-Prediction Mode Propagation for Video Coding

Intra-prediction and inter-prediction are two major ways of prediction in video coding. In the state-of-the-art video coding schemes such as joint exploration model (JEM), these two kinds of prediction are performed independently as different modes. An inter-coded coding unit (CU) is only assigned with the motion information, without an intra-prediction mode (IPM). To capture the texture direction in an inter-coded CU with low complexity, we present an IPM propagation approach in this paper. In the proposed scheme, each $4\times 4$ sub-block in an inter-coded CU is assigned with a propagated IPM, which is fetched from a reference block, located by a motion vector. The propagated IPM is utilized in two aspects. First, the propagated IPM can help to perform inter–intra-prediction in the inter-coded CU. Inter-prediction controlled by the motion information and intra-prediction controlled by the propagated IPM are both performed. The final prediction is generated by a position-dependent weighted sum of the two kinds of prediction. Second, the propagated IPMs of neighboring inter-coded blocks can help to predict the IPM of the intra-coded CU in an inter-picture. The propagated IPM of an inter-coded neighboring block can also be treated as a most probable mode. Simulation results show that BD rate savings of 0.5%, 1.0%, and 1.0% on average for Y, Cb, and Cr components, respectively, are achieved in random access configurations under the common test condition, compared with JEM-7.1. On ultra-high definition sequences, the BD rate savings can reach 1.0%, 1.7%, and 2.3% for Y, Cb, and Cr components, respectively.

360° Video Coding Based on Projection Format Adaptation and Spherical Neighboring Relationship

This paper describes a 360° video coding scheme submitted in response to the joint call for proposal on video compression for capability beyond HEVC issued by ITU-T SG16 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in October 2017. The proposed coding scheme uses projection format adaptation and spherical neighboring relationship to improve coding efficiency. A hybrid angular cubemap projection format is used to adapt the sampling within each face. The faces are packed in an adaptive manner, and the frame packing configuration is updated at regular intervals. Geometry padding of reference photographs is used to improve inter prediction. For intra-/inter-prediction and in-loop filters, processes are modified to avoid utilizing “wrong” neighbors in the frame packed photograph. Finally, post-filtering is used to reduce artifacts at face discontinuities. Experimental results are presented to demonstrate the superior compression efficiency achieved by the proposed 360° video coding scheme. Based on the end-to-end weighted-to-spherically uniform PSNR metric, it achieves average bit rate savings of 33.9% and 13.5% over the HM and joint exploration model anchors, respectively, encoded in the padded equirectangular projection format.

Fast Computation of Integer DCT-V, DCT-VIII, and DST-VII for Video Coding.

Joint exploration model (JEM) reference codecs of ISO/IEC and ITU-T utilize multiple types of integer transforms based on DCT and DST of various transform sizes for intra- and inter-predictive coding, which has brought a significant improvement in coding efficiency. JEM adopts three types of integer DCTs (DCT-II, DCT-V, and DCT-VIII), and two types of integer DSTs (DST-I and DST-VII). The fast computations of Integer DCT-II and DST-I are well known, but few studies have been performed for the other types such as DCT-V, DCT-VIII, and DST-VII for all transform sizes. In this paper, we present fast computation methods of N-point DCT-V and DCT-VIII. For this, we first decompose the DCT-VIII into a preprocessing matrix, the DST-VII and a post-processing matrix to quickly compute it by using the linear relation between DCT-VIII and DST-VII. Then, we approximate integer kernels of N = 4, 8, 16, and 32 for DCT-V, DCT-VIII, and DST-VII with norm scaling and bit-shift to be compatible with quantization in each stage of multiplications between decomposed matrices for video coding. In various experiments, the proposed fast computation methods have shown to effectively reduce the total complexity of the matrix operations with little loss in BDBR performance. In particular, our methods reduce the number of addition and multiplication operations by 38% and 80.3%, respectively, in average, compared to the original JEM.

Short-Distance Intra Prediction of Screen Content in Versatile Video Coding (VVC)

A novel intra prediction algorithm is proposed to improve the coding performance of screen content for the emerging Versatile Video Coding (VVC) standard. The algorithm, called in-loop residual coding with scalar quantization, employs in-block pixels as reference rather than the regular out-block ones. To this end, an additional in-loop residual signal is used to partially reconstruct the block at the pixel level, during the prediction. The proposed algorithm is essentially designed to target high detail textures, where deep block partitioning structure is required. Therefore, it is implemented to operate on $4\times 4$ blocks only, where further block split is not allowed and the standard algorithm is still unable to properly predict the texture. Experiments in the Joint Exploration Model (JEM) reference software show that the proposed algorithm brings a Bjontegaard Delta (BD)-rate gain of 13% on synthetic content, with a negligible computational complexity overhead at both encoder and decoder sides.

An Improved Framework of Affine Motion Compensation in Video Coding.

Affine Motion Compensation (AMC) is a promising coding tool in Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). AMC in JEM employs a 4-parameter affine model between the current block and its reference block. With this model, Motion Vectors (MV) of each sub-block can be derived from the MVs at two control points. In this paper, we present a practical framework to further improve the AMC in JEM. First, we introduce a multi-model AMC approach, which allows the encoder to select either the 4-parameter affine model or the 6-paramter affine model adaptively. Second, we improve the affine inter-mode in two aspects. For the normative part, we present an efficient affine motion coding method, which replaces the affine MV Prediction (MVP) candidates in JEM with more accurate but simpler ones, and employs a second-order MVP. For the non-normative part, we enhance the motion estimation process for AMC, by regulating the optimization algorithm. Finally, we propose to unify the affine merge-mode and the normal merge-mode into a unified merge-mode, which combine affine merge candidates and normal merge candidates in a single merge candidate list. Partial of these methods have been adopted into the next generation video coding standard named Versatile Video Coding (VVC). Simulation results show that the proposed methods can achieve 1.67% BD rate savings in average for the random access configurations.

Machine Learning-Based Fast Angular Prediction Mode Decision Technique in Video Coding.

In this paper we propose a machine learning-based fast intra-prediction mode decision algorithm, using random forest that is an ensemble model of randomized decision trees. The random forest is used to estimate an intra-prediction mode from a prediction unit and to reduce encoding time significantly by avoiding the intensive Rate-Distortion optimization of a number of intra-prediction modes. To this aim, we develop a randomized tree model including parameterized split functions at nodes to learn directional block-based features. The feature uses only four pixels reflecting a directional property of a block, and, thus the evaluation is fast and efficient. To integrate the proposed technique into the conventional video coding standard frameworks, the intra-prediction mode derived from the proposed technique, called an inferred mode (IM), is used to shrink the pool of the candidate modes before carrying out the Rate-Distortion (R-D) optimization. The proposed technique is implemented into the High Efficiency Video Coding Test Model (HM) reference software of the state-of-the-art video coding standard and Joint Exploration Model (JEM) reference software, by integrating the random forest trained off-line into the codecs. Experimental results demonstrate that the proposed technique achieves significant encoding time reduction with only slight coding loss as compared the reference software models.

Hybrid Intraprediction Based on Local and Nonlocal Correlations

In the latest video coding standard, namely, high-efficiency video coding (HEVC), intra coding efficiency is significantly improved by a quadtree partition structure and more intra prediction modes. For intra coding, 35 intra modes are employed including 33 angular intra prediction (AIP) modes that are effective for blocks with strong directions, and a planar mode and a dc mode that are used to predict smooth regions. However, intra prediction in HEVC still cannot handle complicated blocks well. To deal with this problem, this paper proposes a hybrid intra prediction method to improve the intra prediction efficiency. The proposed hybrid intra prediction method consists of three parts: adaptive template matching prediction (ATMP) by exploring nonlocal correlation, combined local and nonlocal prediction by exploring both local correlation for AIP and non-local correlation for ATMP, and combined neighboring modes prediction that can generate smooth prediction by exploring more local correlations. Experimental results suggest that the proposed hybrid intra prediction method achieves 2.8% BD-rate reduction on average for luma compared to HEVC reference software HM-14.0 under all intra main configurations. The gain can be up to 6.9%. When integrated into joint exploration model −1.0, the proposed method still can achieve 1.2% BD-rate reduction for luma.

Enhanced Cross-Component Linear Model for Chroma Intra-Prediction in Video Coding.

Cross-Component Linear Model (CCLM) for chroma intra-prediction is a promising coding tool in Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). CCLM assumes a linear correlation between the luma and chroma components in a coding block. With this assumption, the chroma components can be predicted by the Linear Model (LM) mode, which utilizes the reconstructed neighbouring samples to derive parameters of a linear model by linear regression. This paper presents three new methods to further improve the coding efficiency of CCLM. First, we introduce a multi-model CCLM (MM-CCLM) approach, which applies more than one linear models to a coding block. With MM-CCLM, reconstructed neighbouring luma and chroma samples of the current block are classified into several groups, and a particular set of linear model parameters is derived for each group. The reconstructed luma samples of the current block are also classified to predict the associated chroma samples with the corresponding linear model. Second, we propose a multi-filter CCLM (MF-CCLM) technique, which allows the encoder to select the optimal down-sampling filter for the luma component with the 4:2:0 colour format. Third, we present a LM-angular prediction (LAP) method, which synthesizes the angular intra-prediction and the MM-CCLM intra-prediction into a new chroma intra coding mode. Simulation results show that 0.55%, 4.66% and 5.08% BD rate savings in average on Y, Cb and Cr components respectively, are achieved for All Intra (AI) configurations with the proposed three methods. MM-CCLM and MF-CCLM have been adopted into the JEM by JVET.

Overview of Potential Technologies for Future Video Coding Standard (FVC) in JEM Software : Status and Review

In this paper, we present an investigation into several potential technologies that are being developed and prepared for Future Video Coding (FVC) standard. There are some requirements that need to be considered in delivering the FVC standard. The Joint Video Exploration Team (JVET) is now inspecting potential tools to be adopted as video coding technology under the Joint Exploration Model (JEM) reference software. This paper provides an overview of the technical features of JEM software, a brief account of the background, and the key elements according to the latest version of JEM software and its algorithm description.