Inter-picture Prediction Research Articles

CNN-based Fast Split Mode Decision Algorithm for Versatile Video Coding (VVC) Inter Prediction

Versatile Video Coding (VVC) is the latest video coding standard developed by Joint Video Exploration Team (JVET). In VVC, the quadtree plus multi-type tree (QT+MTT) structure of coding unit (CU) partition is adopted, and its computational complexity is considerably high due to the brute-force search for recursive rate-distortion (RD) optimization. In this paper, we aim to reduce the time complexity of inter-picture prediction mode since the inter prediction accounts for a large portion of the total encoding time. The problem can be defined as classifying the split mode of each CU. To classify the split mode effectively, a novel convolutional neural network (CNN) called multi-level tree (MLT-CNN) architecture is introduced. For boosting classification performance, we utilize additional information including inter-picture information while training the CNN. The overall algorithm including the MLT-CNN inference process is implemented on VVC Test Model (VTM) 11.0. The CUs of size 128×128 can be the inputs of the CNN. The sequences are encoded at the random access (RA) configuration with five QP values {22, 27, 32, 37, 42}. The experimental results show that the proposed algorithm can reduce the computational complexity by 11.53% on average, and 26.14% for the maximum with an average 1.01% of the increase in Bjøntegaard delta bit rate (BDBR). Especially, the proposed method shows higher performance on the sequences of the A and B classes, reducing 9.81%~26.14% of encoding time with 0.95%~3.28% of the BDBR increase.

Deep Learning-Based Video Coding

The past decade has witnessed the great success of deep learning in many disciplines, especially in computer vision and image processing. However, deep learning-based video coding remains in its infancy. We review the representative works about using deep learning for image/video coding, an actively developing research area since 2015. We divide the related works into two categories: new coding schemes that are built primarily upon deep networks, and deep network-based coding tools that shall be used within traditional coding schemes. For deep schemes, pixel probability modeling and auto-encoder are the two approaches, that can be viewed as predictive coding and transform coding, respectively. For deep tools, there have been several techniques using deep learning to perform intra-picture prediction, inter-picture prediction, cross-channel prediction, probability distribution prediction, transform, post- or in-loop filtering, down- and up-sampling, as well as encoding optimizations. In the hope of advocating the research of deep learning-based video coding, we present a case study of our developed prototype video codec, Deep Learning Video Coding (DLVC). DLVC features two deep tools that are both based on convolutional neural network (CNN), namely CNN-based in-loop filter and CNN-based block adaptive resolution coding. The source code of DLVC has been released for future research.

General Video Coding Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

After the development of the High-Efficiency Video Coding Standard (HEVC), ITU-T VCEG and ISO/IEC MPEG formed the Joint Video Exploration Team (JVET), which started exploring video coding technology with higher coding efficiency, including development of a Joint Exploration Model (JEM) algorithm and a corresponding software implementation. The technology explored in the last version of the JEM further increases the compression capabilities of the hybrid video coding approach by adding new tools, reaching up to 30% bit rate reduction compared to HEVC based on the Bjontegaard delta bit rate (BD-rate) metric, and further improvement beyond that in terms of subjective visual quality. This provided enough evidence to issue a joint Call for Proposals (CfP) for a new standardization activity now known as Versatile Video Coding (VVC). All technology proposed in the responses to the CfP was based on the classic block-based hybrid video coding design, extending it by new elements of partitioning, intra- and inter-picture prediction, prediction signal filtering, transforms, quantization/scaling, entropy coding, and in-loop filtering. This article provides an overview of technology that was proposed in the responses to the CfP, with a focus on techniques that were not already explored in the JEM context.

Intelligent Coding Unit Partitioning using Predictive Data Mining

Increasing applications of videos in everyday life demands compressing the videos further. International bodies for Video Coding standards are working toward making it more efficient in terms of reducing bitrate so as to efficiently compress the high-resolution videos. With increasing resolution, the size of the Coding Units increases. Latest Video Coding techniques like High Efficiency Video Coding (HEVC) and Versatile Video coding (VVC) proposed Larger coding Units with flexible Quadtree decompositions. In Inter-picture prediction all the sub blocks have to find best partitioning structure during motion estimation. Due to larger coding units finding the best partitioning introduces computational complexity. In the proposed work we present a computational complexity control scheme using predictive data mining. The method helps to predict whether to split or no split the coding unit. The decision tree model trained offline in the proposed work achieves 77.73% saving in encoding time with minimal change of 0.15 in average PSNR and 0.00074 in average SSIM values.

Context-Based Fractional Sample Refinement for HEVC Compliant Encoding

The H.265/MPEG-H High Efficiency Video Coding compliant encoding process faces the challenge of high computational complexity. Particularly, in the case of inter-picture prediction, most of the computational resources are allocated for the motion estimation (ME) process. In turn, ME and motion compensation enable improving coding efficiency by addressing the blocks of video frames as corresponding displacements from one or more reference blocks. These displacements do not necessarily have to be limited to integer sample positions, but may have an accuracy of half sample or quarter sample positions, which are identified during fractional sample refinement. In this paper, a context-based scheme for fractional sample refinement is proposed. The scheme takes the advantage of already obtained information in prior ME steps and provides significant flexibility in terms of parameterization. In this way, it adaptively achieves a desired tradeoff between computational complexity and coding efficiency. According to the experimental results obtained for an example algorithm utilizing the proposed framework, a significant decrease in the number of search points can be achieved. For instance, considering only 6 instead of 16 fractional sample positions results in a tradeoff of only 0.4% Bjontegaard Delta-rate loss for high-definition video sequences compared with the conventional interpolation-and-search method.

H.263: video coding for low-bit-rate communication

This article provides an overview of H.263, the new ITU-T Recommendation for low-bit-rate video communication. H.263 specifies a coded representation for compressing the moving picture component of audio-visual signals at low bit rates. The basic structure of the video source coding algorithm is taken from ITU-T Recommendation H.261 and is a hybrid of interpicture prediction to reduce temporal redundancy and transform coding of the prediction residual to reduce spatial redundancy. The source coder can operate on five standardized picture formats: sub-QCIF, QCIF, CIF, 4CIF, and 16CIF. The decoder has motion compensation capability with half-pixel precision, in contrast to H.261 which uses full-pixel precision and employs a loop filter. H.263 includes four negotiable coding options which provide improved coding efficiency: unrestricted motion vectors, syntax-based arithmetic coding, advanced prediction, and PB-frames.