Inter Prediction Research Articles

Spatial Decomposition and Temporal Fusion Based Inter Prediction for Learned Video Compression

Spatial Decomposition and Temporal Fusion Based Inter Prediction for Learned Video Compression

Spatio-Temporal Convolutional Neural Network for Enhanced Inter Prediction in Video Coding.

This paper presents a convolutional neural network (CNN)-based enhancement to inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples. It is motivated by the theoretical consideration that neural network-based methods have a higher degree of signal adaptivity than conventional signal processing methods and that spatially neighboring reference samples have the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. We show that adding a polyphase decomposition stage to the CNN results in a significantly better trade-off between computational complexity and coding performance. Incorporating spatial reference samples in the inter prediction process is challenging: The fact that the input of the CNN for one block may depend on the output of the CNN for preceding blocks prohibits parallel processing. We solve this by introducing a novel signal plane that contains specifically constrained reference samples, enabling parallel decoding while maintaining a high compression efficiency. Overall, experimental results show average bit rate savings of 4.07% and 3.47% for the random access (RA) and low-delay B (LB) configurations of the JVET common test conditions, respectively.

DMVC: Decomposed Motion Modeling for Learned Video Compression

Inter prediction is the critical component in hybrid coding framework to deal with the temporal redundancy. Most of the neural video coding methods typically follow the motion compensation based inter coding scheme, establishing motion vector (MV) as the central role. In this paper, we innovatively propose an efficient motion modeling approach by inherently decomposing it into two components, the intrinsic motion and the compensatory motion. The intrinsic motion originates from the implicit spatiotemporal context hidden in the historical sequence, which can be intuitively captured free of bits. On the top of it, the compensatory motion acts a role of structural refinement and texture enhancement as a form of side information. In particular, the inter prediction is performed in the feature space as a manner of progressive temporal transition, conditioned on the decomposed motion. By the motion decomposition paradigm, we innovatively answer the question of motion representation, compensation and coding in the learned video compression framework. With the temporal prediction, the remaining pixel residue is signaled to obtain the reconstruction. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art coding performance on par with other end-to-end coding methods, and outperforms versatile video coding (VVC) under low-delay P (LDP) configuration in terms of MS-SSIM metric.

Neural Network-Based Reference Block Quality Enhancement for Motion Compensation Prediction

Inter prediction is a crucial part of hybrid video coding frameworks, and it is used to eliminate redundancy in adjacent frames and improve coding performance. During inter prediction, motion estimation is used to find the reference block that is most similar to the current block, and the following motion compensation is used to shift the reference block fractionally to obtain the prediction block. The closer the reference block is to the original block, the higher the coding efficiency is. To improve the quality of reference blocks, a quality enhancement network (RBENN) that is dedicated to reference blocks is proposed. The main body of the network consists of 10 residual modules, with two convolution layers for preprocessing and feature extraction. Each residual module consists of two convolutional layers, one ReLU activation, and a shortcut. The network uses the luma reference block as input before motion compensation, and the enhanced reference block is then filtered by the default fractional interpolation. Moreover, the proposed method can be used for both conventional motion compensation and affine motion compensation. Experimental results showed that RBENN could achieve a −1.35% BD rate on average under the low-delay P (LDP) configuration compared with the latest H.266/VVC.

Modelling of Depth Prediction Algorithm for Intra Prediction Complexity Reduction

Video compression gained its relevance with the boon of the internet, mobile phones, variable resolution acquisition device etc. The redundant information is explored in initial stages of compression that’s is prediction. Inter prediction that is prediction within the frame generates high computational complexity when working with traditional signal processing procedures. The paper proposes the design of a deep convolutional neural network model to perform inter prediction by crossing out the flaws in the traditional method. It briefs the modeling of network, mathematics behind each stage and evaluation of the proposed model with sample dataset. The video frame’s coding tree unit (CTU) of 64x64 is the input, the model converts and store it as a 16-element vector with the goodness of CNN network. It gives an overview of deep depth decision algorithm. The evaluation process shows that the model performs better for compression with less computational complexity.

Subblock-Based Motion Derivation and Inter Prediction Refinement in the Versatile Video Coding Standard

Efficient representation and coding of fine-granular motion information is one of the key research areas for exploiting inter-frame correlation in video coding. Representative techniques towards this direction are affine motion compensation (AMC), decoder-side motion vector refinement (DMVR), and subblock-based temporal motion vector prediction (SbTMVP). Fine-granular motion information is derived at subblock level for all the three coding tools. In addition, the obtained inter prediction can be further refined by two optical flow-based coding tools, the bi-directional optical flow (BDOF) for bi-directional inter prediction and the prediction refinement with optical flow (PROF) exclusively used in combination with AMC. The aforementioned five coding tools have been extensively studied and finally adopted in the Versatile Video Coding (VVC) standard. This paper presents technical details of each tool and highlights the design elements with the consideration of typical hardware implementations. Following the common test conditions defined by Joint Video Experts Team (JVET) for the development of VVC, 5.7% bitrate reduction on average is achieved by the five tools. For test sequences characterized by large and complex motion, up to 13.4% bitrate reduction is observed. Additionally, visual quality improvement is demonstrated and analyzed.

Neural Network-Based Enhancement to Inter Prediction for Video Coding

Inter prediction is a crucial part of hybrid video coding frameworks, utilized to exploit the temporal redundancy in video sequences and improve the coding performance. During inter prediction, a predicted block is typically derived from reference pictures using motion estimation and motion compensation. To improve the coding performance of inter prediction, a neural network based enhancement to inter prediction (NNIP) is proposed in this paper. NNIP is composed of three networks, namely residue estimation network, combination network, and deep refinement network. Specifically, first, a residue estimation network is designed to estimate the residue between current block and its predicted block using their available spatial neighbors. Second, the feature maps of the estimated residue and the predicted block are extracted and concatenated in a combination network. Finally, the concatenated feature maps are fed into a deep refinement network to generate a refined residue, which is added back to the predicted block to derive a more accurate predicted block. NNIP is integrated in HEVC to evaluate its efficiency. The experimental results demonstrate that NNIP can achieve 4.6%, 3.0%, and 2.7% BD-rate reduction on average under LDP, LDB, and RA configurations compared to HEVC.

Improved CNN-Based Learning of Interpolation Filters for Low-Complexity Inter Prediction in Video Coding

The versatility of recent machine learning approaches makes them ideal for improvement of next generation video compression solutions. Unfortunately, these approaches typically bring significant increases in computational complexity and are difficult to interpret into explainable models, affecting their potential for implementation within practical video coding applications. This paper introduces a novel explainable neural network-based inter-prediction scheme, to improve the interpolation of reference samples needed for fractional precision motion compensation. The approach requires a single neural network to be trained from which a full quarter-pixel interpolation filter set is derived, as the network is easily interpretable due to its linear structure. A novel training framework enables each network branch to resemble a specific fractional shift. This practical solution makes it very efficient to use alongside conventional video coding schemes. When implemented in the context of the state-of-the-art Versatile Video Coding (VVC) test model, 0.77%, 1.27% and 2.25% BD-rate savings can be achieved on average for lower resolution sequences under the random access, low-delay B and low-delay P configurations, respectively, while the complexity of the learned interpolation schemes is significantly reduced compared to the interpolation with full CNNs.

Compressing Neural Networks With Inter Prediction and Linear Transformation

Because of resource-constrained environments, network compression has become an essential part of deep neural networks research. In this paper, we found a mutual relationship between kernel weights termed as Inter-Layer Kernel Correlation (ILKC). The kernel weights between two different convolution layers share a substantial similarity in shapes and values. Based on this relationship, we propose a new compression method, Inter-Layer Kernel Prediction (ILKP), which represents convolutional kernels with fewer bits through similarity between kernel weights in convolutional neural networks. Furthermore, to effectively adapt the inter prediction scheme from video coding technology, we integrate a linear transformation into the prediction scheme, which significantly enhances compression efficiency. The proposed method achieved 93.77% top-1 accuracy with $4.1\times $ compression ratio compared to the ResNet110 baseline model on CIFAR10. It means that 0.04% top-1 accuracy improvement was achieved by using less memory footprint. Moreover, incorporating quantization, the proposed method achieved a $13\times $ compression ratio with little performance degradation compared to the ResNets baseline model trained on CIFAR10 and CIFAR100.

Deep Reference Generation With Multi-Domain Hierarchical Constraints for Inter Prediction

Inter prediction is an important module in video coding for temporal redundancy removal, where similar reference blocks are searched from previously coded frames and employed to predict the block to be coded. Although traditional video codecs can estimate and compensate for block-level motions, their inter prediction performance is still heavily affected by the remaining inconsistent pixel-wise displacement caused by irregular rotation and deformation. In this paper, we address the problem by proposing a deep frame interpolation network to generate additional reference frames in coding scenarios. First, we summarize the previous adaptive convolutions used for frame interpolation and propose a factorized kernel convolutional network to improve the modeling capacity and simultaneously keep its compact form. Second, to better train this network, multi-domain hierarchical constraints are introduced to regularize the training of our factorized kernel convolutional network. For spatial domain, we use a gradually down-sampled and up-sampled auto-encoder to generate the factorized kernels for frame interpolation at different scales. For quality domain, considering the inconsistent quality of the input frames, the factorized kernel convolution is modulated with quality-related features to learn to exploit more information from high quality frames. For frequency domain, a sum of absolute transformed difference loss that performs frequency transformation is utilized to facilitate network optimization from the view of coding performance. With the well-designed frame interpolation network regularized by multi-domain hierarchical constraints, our method surpasses HEVC on average 6.1% BD-rate saving and up to 11.0% BD-rate saving for the luma component under the random access configuration.

Optimal Mode Decision Method for Interframe Prediction in H.264/AVC

Studies show that encoding technologies in H.264/AVC, including prediction and conversion, are essential technologies. However, these technologies are more complicated than the MPEG-4, which is a standard method and widely adopted worldwide. Therefore, the amount of calculation in H.264/AVC is significantly up-regulated compared to that of the MPEG-4. In the present study, it is intended to simplify the computational expenses in the international standard compression coding system H.264/AVC for moving images. Inter prediction refers to the most feasible compression technology, taking up to 60% of the entire encoding. In this regard, prediction error and motion vector information are proposed to simplify the computation of inter predictive coding technology. In the initial frame, motion compensation is performed in all target modes and then basic information is collected and analyzed. After the initial frame, motion compensation is performed only in the middle 8×8 modes, and the basic information amount shifts. In order to evaluate the effectiveness of the proposed method and assess the motion image compression coding, four types of motion images, defined by the international telecommunication union (ITU), are employed. Based on the obtained results, it is concluded that the developed method is capable of simplifying the calculation, while it is slightly affected by the inferior image quality and the amount of information.

Exploiting Data Mining for Fast Inter Prediction Mode Decision in HEVC

HEVC has made significant progress in coding efficiency. Compared with the previous generation H.264/AVC video coding standard, HEVC can save 50% of code rate under the same visual quality. However, this drop in bit rate is accompanied by an increase in computational complexity, which poses a great challenge to the application and development of HEVC. After analyzing the great correlation between the partitioning mode of coding unit in the encoder and several parameters, the partitioning problem of the CU is modeled as a classification problem in data mining, and it is decided to use a decision tree to solve this classification problem. We implement the decision tree on the official software HM16.2 and test the algorithm on the test set. Experiments show that fast algorithm based on decision tree can improve the encoding speed effectively with little influence on the encoding performance.

Reference Clip for Inter Prediction in Video Coding

Inter prediction is a fundamental technology in video coding to remove the temporal redundancy between video frames. Traditionally, the reconstructed frames are directly put into a reference frame buffer to serve as references for inter prediction. Using multiple reference frames increases the accuracy of inter prediction, but also incurs a waste of memory of the buffer since the content of reference frames is highly similar. To address this problem, we propose to organize the references at clip level in addition to frame level, i.e. the reference buffer stores not only reference frames, but also reference clips that are cropped regions selected from the reconstructed frames. Using clip-level references, we can manage the reference content more economically, since the content of multiple reference frames is divided into the singular content of each frame as well as the repetitive content that appears in multiple frames. For the repetitive content, only one copy is stored in reference clips so as to avoid duplicate. Moreover, using reference clips also facilitates the bit-rate allocation among reference content, i.e. the quality of each clip can be decided adaptively to achieve the rate-distortion optimization. In this paper, we propose a complete video coding framework using reference clips, and investigate the problems including how to generate reference clips as either singular content clips or repetitive content clips, how to manage the clips, how to utilize the clips for inter prediction, and how to allocate bit-rate among clips, in a systematic manner. The proposed video coding framework is implemented upon the state-of-the-art video coding scheme, High Efficiency Video Coding (HEVC). Experimental results show that our scheme achieves on average 5.1% and 5.0% BD-rate reduction than the HEVC anchor, in low-delay B and low-delay P settings, respectively. We believe that reference clip opens up a new dimension for optimizing inter prediction in video coding, and thus is worthy of further study.

Fast CU size decision and PU mode decision algorithm for quality SHVC inter coding

As a scalable extension of the High Efficiency Video Coding (HEVC), the Scalable High Efficiency Video Coding (SHVC) encoder needs to encode multiple HEVC layers with Inter-layer predictions, which causes the significant increase in coding complexity. In this paper, we proposed a novel Inter prediction scheme to effectively reduce computational complexity in Quality SHVC. The new features of the proposed algorithm include: First, spatial and Inter-layer depth correlations are used to predict the most possible coding unit (CU) depth level candidates. Second, a statistical test method on the current CU depth level is introduced to examine whether the residual coefficients within its block present similar distribution to terminate depth selection early. Finally, during Inter prediction selection from 8 Prediction Unit (PU) sizes, spatial and Inter-layer correlations are combined with residual coefficients distribution to determine the PU partitioning mode is Symmetric Motion Partitioning (SMP) or Asymmetric Motion Partitioning (AMP). Experimental results demonstrate that the proposed algorithm can save an average of 65.33% coding time of enhancement layer (EL) while achieving a better rate distortion (RD) performance over other state-of-the-art work.

A Fast Inter Prediction Method in HEVC Using SAD Computing Algorithm Based on Bottom-up Design

A Fast Inter Prediction Method in HEVC Using SAD Computing Algorithm Based on Bottom-up Design

Accelerating HEVC Inter Prediction with Improved Merge Mode Handling

Accelerating HEVC Inter Prediction with Improved Merge Mode Handling

Inter Prediction Complexity Reduction for HEVC based on Residuals Characteristics

High Efficiency Video Coding (HEVC) or H.265 is currently the latest standard in video coding. While this new standard promises improved performance over the previous H.264/AVC standard, the complexity has drastically increased due to the various new improved tools added. The splitting of the 64x64 Largest Coding Unit (LCU) into smaller CU sizes forming a quad tree structure involves a significant number of operations and comparisons which imposes a high computational burden on the encoder. In addition, the improved Motion Estimation (ME) techniques used in HEVC inter prediction in order to ensure greater compression also contribute to the high encoding time. In this paper, a set of standard thresholds are identified based on the Mean Square (MS) of the residuals. These thresholds are used to terminate the CU splitting process and to skip some of the inter modes processing. In addition, CUs with large MS values are split at a very early stage. Experimental results show that the proposed method can effectively reduce the encoding time by 62.2% (70.8% for ME) on average, compared to HM 10, yielding a BD-Rate of only 1.14%.

Fast Coding Unit Size Decision Based on Probabilistic Graphical Model in High Efficiency Video Coding Inter Prediction

In this work, a high efficiency coding unit (CU) size decision algorithm is proposed for high efficiency video coding (HEVC) inter coding. The CU splitting or non-splitting is modeled as a binary classification problem based on probability graphical model (PGM). This method incorporates two sub-methods: CU size termination decision and CU size skip decision. This method focuses on the trade-off between encoding efficiency and encoding complexity, and it has a good performance. Particularly in the high resolution application, simulation results demonstrate that the proposed algorithm can reduce encoding time by 53.62%-57.54%, while the increased BD-rate are only 1.27%-1.65%, compared to the HEVC software model.

A Fast Inter Prediction Algorithm Based on Rate-Distortion Cost in HEVC

A Fast Inter Prediction Algorithm Based on Rate-Distortion Cost in HEVC

An Adapted Intra and Inter Prediction Algorithm for H.264

An adapted prediction algorithm for H.264 is proposed in this paper. For I frames, a new intra algorithm is proposed in which a novel Intra_4×4 mode selection method is designed. Nine Intra_4×4 modes are divided into three classes, the suitable mode class for current 4×4-block is predicted using the mode classes of neighboring left and up blocks. For inter frames, a new inter prediction algorithm is proposed, and two matrixes which are composed of sum of absolute values of the integer transform coefficients in 4×4-residual-block contained in luminance 16×16-block and 8×8-block are defined respectively, the novel MB and sub-MB mode selection criteria which use the derived gradients of the two matrixes are then proposed.