Inter Frames Research Articles

Consistent Quality Oriented Rate Control in HEVC Via Balancing Intra and Inter Frame Coding

Consistent quality oriented rate control in video coding has attracted much more attention. However, the existing efforts only focus on decreasing variations between every two adjacent frames, but neglect coding trade-off problem between intra and inter frames. In this paper, we deal with it from a new perspective, where intra frame quantization parameter (IQP) and rate control are optimized for balanced coding. First, due to the importance of intra frames, a new framework is proposed for consistent quality oriented IQP prediction, and then we remove unqualified IQP candidates using the proposed penalty term. Second, we extensively evaluate possible features, and select target bits per pixel for all remaining frames, average and standard variance of frame QPs, where equivalent acquisition methods for QP features are given. Third, predicted IQPs are clipped effectively according to bandwidth and previous information for better bit rate accuracy. Compared with High Efficiency Video Coding (HEVC) reference baseline, experiments demonstrate that our method reduces quality fluctuation greatly by 37.2% on frame-level standard variance of peak-signal-noise-ratio (PSNR) and 45.1% on that of structural similarity (SSIM). Moreover, it also can have satisfactory results on Rate-Distortion (R-D) performance, bit accuracy and buffer control.

Neural Reference Synthesis for Inter Frame Coding

This work proposes the neural reference synthesis (NRS) to generate high-fidelity reference block for motion estimation and motion compensation (MEMC) in inter frame coding. The NRS is comprised of two submodules: one for reconstruction enhancement and the other for reference generation. Although numerous methods have been developed in the past for these two submodules using either handcrafted rules or deep convolutional neural network (CNN) models, they basically deal with them separately, resulting in limited coding gains. By contrast, the NRS proposes to optimize them collaboratively. It first develops two CNN-based models, namely EnhNet and GenNet. The EnhNet only uses spatial correlations within the current frame for reconstruction enhancement and the GenNet is then augmented by further aggregating temporal correlations across multiple frames for reference synthesis. However, a direct concatenation of EnhNet and GenNet without considering the complex temporal reference dependency across inter frames would implicitly induce iterative CNN processing and cause the data overfitting problem, leading to visually-disturbing artifacts and oversmoothed pixels. To tackle this problem, the NRS applies a new training strategy to coordinate the EnhNet and GenNet for more robust and generalizable models, and also devises a lightweight multi-level R-D (rate-distortion) selection policy for the encoder to adaptively choose reference blocks generated from the proposed NRS model or conventional coding process. Our NRS not only offers state-of-the-art coding gains, e.g., >10% BD-Rate (Bjøntegaard Delta Rate) reduction against the High Efficiency Video Coding (HEVC) anchor for a variety of common test video sequences encoded at a wide bit range in both low-delay and random access settings, but also greatly reduces the complexity relative to existing learning-based methods by utilizing more lightweight DNNs. All models are made publicly accessible at https://github.com/IVC-Projects/NRS for reproducible research.

Regional Bit Allocation and Rate Distortion Optimization for Multiview Depth Video Coding With View Synthesis Distortion Model

In this paper, we propose a view synthesis distortion model (VSDM) that establishes the relationship between depth distortion and view synthesis distortion for the regions with different characteristics: color texture area corresponding depth (CTAD) region and color smooth area corresponding depth (CSAD), respectively. With this VSDM, we propose regional bit allocation (RBA) and rate distortion optimization (RDO) algorithms for multiview depth video coding (MDVC) by allocating more bits on CTAD for rendering quality and fewer bits on CSAD for compression efficiency. Experimental results show that the proposed VSDM based RBA and RDO can improve the coding efficiency significantly for the test sequences. In addition, for the proposed overall MDVC algorithm that integrates VSDM based RBA and RDO, it achieves 9.99% and 14.51% bit rate reduction on average for the high and low bit rate, respectively. It can improve virtual view image quality 0.22 and 0.24 dB on average at the high and low bit rate, respectively, when compared with the original joint multiview video coding model. The RD performance comparisons using five different metrics also validate the effectiveness of the proposed overall algorithm. In addition, the proposed algorithms can be applied to both INTRA and INTER frames.

ContentAdaptive Refined Error Concealment Schemes for H.264/AVC Video Coding

this paper, we propose content-adaptive refined error concealment schemes for video transmission over packet lossy networks. The proposed spatial and temporal error concealment algorithms improve the reconstructed video quality at the common loss rates encountered on the networks. To reduce the computational complexity, the proposed schemes make use of the existing information of Intra prediction modes from the coded bit-stream for spatial error concealment. Neighboring macroblocks mode information is used to adapt the partition size of the lost macroblock to enhance the temporal error concealment performance. In addition, overlapped motion compensation is used to avoid the spatial discontinuities. A hybrid spatio-temporal concealment scheme is also proposed to use the appropriate algorithm for the lost macroblocks in the Inter frames. Experimental results demonstrate that the proposed schemes improve the concealment performance and reduce the structural degradations.

A Fine Rate Control Algorithm With Adaptive Rounding Offsets (ARO)

Rate control plays an important role in regulating the bit rate to meet the bandwidth and storage requirement. Most existing video encoders regulate the bit rate by adjusting the quantization step size. We propose to incorporate a new dimension: the quantization rounding offset into rate control. In this paper, we present a rate control algorithm with adaptive rounding offsets (ARO) that jointly adjusts the quantization step size and the rounding offset for high bit rate accuracy. Different from the quantization step size that has a limited number of choices, the rounding offset is a continuously adjustable variable that allows the rate control algorithm to reach any precision in principle. Our extensive experimental results show that the proposed ARO algorithm significantly improves the rate control accuracy at almost no extra computational complexity. Compared with the rho-domain rate control, the ARO algorithm reduces the rate control errors from about 2% to 0.5% for INTRA frames, and 5% to 1.5% for INTER frames. Our experiments also demonstrate that ARO provides the extra benefit of smoother visual quality.

Wavelet Video Coding With Dependent Optimization

We present a new wavelet video coding algorithm and an optimization framework that allocates bits efficiently among consecutive frames at the pixel level. The video residual coder is based on set partitioning in hierarchical trees and wavelet blocks, allowing flexible bit allocation among active and inactive regions in a video frame. To optimize the encoder for efficient bit allocation, we use Lagrangian methods. First, the rate-distortion cost for each wavelet block is minimized, effectively enforcing the equal-slope rule at the pixel level. The Lagrangian method is then extended to successive frames, and an iterative algorithm is presented to solve the dependent coding problem. Finally, motion search is jointly optimized by including motion vectors in the cost function, completing an optimization framework that enforces the equal-slope rule for all bits at the macroblock level and pixels across consecutive frames. The result is a video compression algorithm that requires no training, no explicit quantization, no floating-point operation for INTER frames, no entropy coding, and allows precise rate control. Compared with the discrete cosine transform code in H.263, the new video coding algorithm has faster decoding procedures and achieves an improvement up to 1.12 dB in a PSNR or 20.6% in bit rate savings for typical sequences used in the video compression community.