Rate-distortion Optimization Scheme Research Articles

A progressive CNN in-loop filtering approach for inter frame coding

Convolutional Neural Network (CNN) structures have been designed for in-loop filtering to improve video coding performance. These CNN models are usually trained through learning the correlations between the reconstructed and the original frames, which are then applied to every single reconstructed frame to improve the overall video quality. This direct model training and deployment strategy is effective for intra coding since a locally optimal model is sufficient. However, when applied to inter coding, it causes over-filtering because the intertwined reference dependencies across inter frames are not taken into consideration. To address this issue, existing methods usually resort to the Rate–Distortion Optimization (RDO) to selectively apply the CNN model, but fail to address the limitation of using a local CNN model. In this paper, we propose a progressive approach to train and incorporate the CNN-based in-loop filters to work seamlessly with video encoders. First, we develop a progressive training method to obtain the inter model. Using transfer learning, reconstructed frames using the CNN model are progressively involved back into the training of the CNN model itself, to simulate the reference dependencies in inter coding. Next, we design a frame-level model selection strategy for the high-bitrate coding where the over-filtering effect is diluted. Experimental results show that the proposed method outperforms the RDO method that utilizes only local model. Proposed approach also achieves comparable coding performance but with less computational complexity when integrating our progressive model into the RDO scheme.

In-loop perceptual model-based rate-distortion optimization for HEVC real-time encoder

In this paper, a novel High Efficiency Video Coding (HEVC)-compliant perceptual rate-distortion optimization (RDO) scheme is proposed based on motion attention and visual distortion sensitivity models, which both fully utilize in-loop coding information of HEVC. In detail, the motion attention model is designed by using the motion vectors (MVs) estimated during the inter-prediction process. The MV field is refined based on maximum a posteriori (MAP) estimation to remove MV outliers and improve the model’s efficiency. In addition, the visual distortion sensitivity is modeled by using the spatiotemporal energy of AC coefficients, which are obtained from HEVC transform process. Then, these two models are incorporated together into the RDO process. As a result, the Lagrange multiplier and quantization parameter are adjusted adaptively in an analytical way. Since the two models are calculated within the HEVC coding loop, the complexity increase is limited. The experimental results indicate that the proposed perceptual RDO scheme can achieve significantly better rate-VQM performance than the conventional RDO scheme. Specifically, the BD-rate can reach a maximum 24.45% and an average 13.68% reduction in terms of the Bjontegaard Delta metric compared to HEVC practical encoder x265.

Weighted Rate-Distortion Optimization for Screen Content Coding

Unlike camera-captured video, screen content (SC) often contains a lot of repeating patterns, which makes some blocks used as references much more important than others. However, conventional rate-distortion optimization (RDO) schemes in video coding do not consider the dependence among image blocks, which often leads to a locally optimal parameter selection, especially for SC. In this paper, we present a weighted RDO scheme for SC coding (SCC), in which the repeating characteristics are taken into account when deciding RD tradeoff for each block. For one block, the number being referenced by the current picture and following pictures is estimated and based on the number, we set a proper weight in the RDO process to reflect its importance from a global point of view. To estimate the number being referenced, we propose a hash-based method to approximate the results to avoid the complexity of direct search. Experimental results show that compared with the High Efficiency Video Coding SCC reference software, 10.1%, 14.5%, and 2.2% on average and up to 25.7%, 39.8%, and 4.6% bit saving can be achieved by considering weights provided by our scheme for hierarchical-B, IBBB, and all intra coding structures, respectively. Thanks to our hash-based design, the complexity increase brought by the proposed scheme is marginal.

Background Error Propagation Model Based RDO in HEVC for Surveillance and Conference Video Coding

The emerging high efficiency video coding (HEVC) Standard has significantly improved the compression performance in comparison with its predecessor H.264/AVC. However, it was originally designed for generic video contents. The backgrounds are generally static in the surveillance and conference videos. The background coding errors will propagate to the subsequent frames in coding the videos. In this paper, a background error propagation (BEP) model-based rate distortion optimization (RDO) scheme in HEVC is proposed for the surveillance and conference videos. First, the R-D performance of the long-term frames is optimized globally. The global RDO scheme can efficiently exploit the background error propagation. Second, a BEP model is studied to express the linear relationship between the distortion of the first frame and that of its subsequent frames. Based on the BEP model, enhanced frames are proposed to be coded with a small quantization parameter so as to improve the global performance. Third, a decay model is proposed to investigate the variation of the error propagation ratio as the frame order increased. Based on the decay model, a periodical optimization scheme is presented by deploying the enhanced frames periodically. Experiments are tested on the surveillance and conference videos. The results show that the proposed algorithm achieves significant performance improvement.

Temporally Dependent Rate-Distortion Optimization for Low-Delay Hierarchical Video Coding.

Low-delay hierarchical coding structure (LD-HCS), as one of the most important components in the latest High Efficiency Video Coding (HEVC) standard, greatly improves coding performance. It groups consecutive P/B frames into different layers and encodes them with different quantization parameters (QPs) and reference mechanisms in such a way that temporal dependency among frames can be exploited. However, due to varying characteristics of video contents, temporal dependency among coding units differs significantly from each other in the same or different layers, while a fixed LD-HCS scheme cannot take full advantage of the dependency, leading to a substantial loss in coding performance. This paper addresses the temporally dependent rate distortion optimization (RDO) problem by attempting to exploit varying temporal dependency of different units. First, the temporal relationship of different frames under the LD-HCS is examined, and hierarchical temporal propagation chains are constructed to represent the temporal dependency among coding units in different frames. Then, a hierarchical temporally dependent RDO scheme is developed specifically for the LD-HCS based on a source distortion propagation model. Experimental results show that our proposed scheme can achieve 2.5% and 2.3% BD-rate gain in average compared with the HEVC codec under the same configuration of P and B frames, respectively, with a negligible increase in encoding time. Furthermore, coupled with QP adaption, our proposed method can achieve higher coding gains, e.g., with multi-QP optimization, about 5.4% and 5.0% BD-rate saving in average over the HEVC codec under the same setting of P and B frames, respectively.

Fast coding tree structure decision for HEVC based on classification trees

The High Efficiency Video Coding (HEVC) standard provides improved compression rates in comparison to its predecessors at the cost of large increases in computational complexity. An important share of such increases is due to the introduction of flexible Coding Tree structures, which best configuration is decided through exhaustive tests in a rate-distortion optimization (RDO) scheme. In this work, an early termination method for the decision of such structures was designed using classification trees obtained through Data Mining techniques. The classification trees were trained using intermediate encoding results from a set of video sequences and implemented in the encoder to avoid the full RDO-based decision. An average reduction of 37 % in the HEVC encoder computational complexity was achieved when using the designed classification trees, with a negligible cost of only 0.28 % in terms of Bjontegaard Delta-rate increase.

Low-Complexity Depth Coding by Depth Sensitivity Aware Rate-Distortion Optimization

In this paper, a low-complexity coding approach is explored for depth video under the framework of high efficiency video coding based 3-D video coding standard. Unlike the existing low-complexity coding approaches that optimize motion search and mode decision, the major technical innovation of this paper is to incorporate depth sensitivity fidelity (DSF) into the rate-distortion optimization (RDO) process. Specifically, a quantitative maximum tolerable depth distortion is derived to measure the DSF, and jointly estimate the view synthesis distortion to account for the DSF. Then, a DSF-aware RDO scheme is proposed by developing new quantization parameter and Lagrangian multiplier determination strategies. Extensive experimental results demonstrate that the proposed method can reduce the computational complexity of encoding without significant view synthesis performance loss.

Lagrangian Multiplier Adaptation for Rate-Distortion Optimization With Inter-Frame Dependency

Rate-distortion optimization (RDO) is widely used in video coding, which plays a critical role in enhancing the coding efficiency substantially. Currently, the RDO process is performed in a way that coding efficiency of each coding unit (CU) is maximized independently without considering the dependency among CUs. As we know, in the current hybrid video coding structure, spatial/temporal prediction techniques are extensively used, which introduce strong dependency among CUs. In this paper, we investigate RDO with inter-frame dependency, where the impact of coding performance of the current CU on that of the following frames is considered. Accordingly, an RDO scheme taking the inter-frame dependency into account is proposed by adapting the Lagrangian multiplier. The experimental results show that the proposed scheme can achieve about 3.22% and 3.19% BD-rate saving in average over the state-of-the-art High Efficiency Video Coding (HEVC) reference software HM15.0 in the low-delay $P$ (LDP) and low-delay $B$ (LDB) coding structures, respectively, with no extra encoding time. The proposed scheme can obtain a significantly higher coding gain than the multiple quantization parameter (MQP) (±3) optimization technique that would greatly increase the encoding time by a factor of about six. Coupled with MQP optimization, the proposed scheme can further achieve about 5.96% and 5.57% BD-rate savings in average over the HEVC and about 4.03% and 4.07% over the HEVC with MQP optimization, under the specified common test conditions for LDP and LDB coding structures, respectively.

A Novel Image Compression Algorithm Based on Multitree Dictionary and Perceptual-based Rate-Distortion Optimization

This paper presents a novel block-based image coding algorithm which applies a tree-structured multitree dictionary and a perceptual-based rate distortion optimization scheme. While multitree dictionary is employed to support a very large number of different tilings, perceptual-based rate distortion optimization utilizes the SSIM metric, instead of popular MSE metric, to allocate bit rate according to human visual system. Experimental results show that our proposed method outperforms many existing techniques in both subjective and objective image quality measures.

SSIM-based error-resilient rate-distortion optimization of H.264/AVC video coding for wireless streaming

The SSIM-based rate-distortion optimization (RDO) has been verified to be an effective tool for H.264/AVC to promote the perceptual video coding performance. However, the current SSIM-based RDO is not efficient for improving the perceptual quality of the video streaming application over the error-prone network, because it does not consider the transmission induced distortion in the encoding process. In this paper, a SSIM-based error-resilient RDO scheme for H.264/AVC is proposed to improve the wireless video streaming performance. Firstly, with the help of the SSE-based RDO, we present a low-complexity Lagrange multiplier decision method for the SSIM-based RDO video coding in the error-free environment. Then, the SSIM-based decoding distortion of the user end is estimated at the encoder and is correspondingly introduced into the RDO to involve the transmission induced distortion into the encoding process. Further, the Lagrange multiplier is theoretically derived to optimize the encoding mode selection in the error-resilient RDO process. Experimental results show that the proposed SSIM-based error-resilient RDO can obtain superior perceptual video quality (more structural information) to the traditional SSE-based error-resilient RDO for wireless video streaming at the same bit rate condition.

A Novel Joint Rate Distortion Optimization Scheme for Intra Prediction Coding in H.264/AVC

A Novel Joint Rate Distortion Optimization Scheme for Intra Prediction Coding in H.264/AVC

A Whole Rate-Distortion Optimize Pronounce Based on Wireless Sensor Network

Based on the Joint Video Team (JVT) of the ITU-T Video Coding Experts Group VC EG and the IS O/IEC Moving Picture Experts Group MPEG, an RD optimal Macro Block mode decision scheme for Internet error channel streaming is introduced. The scheme employs the luminance Rate Distortion (RD) optimal mode decision scheme so as to take the effects of video encoding distortion and the channel error propagation to get higher error robustness for error transmission. Based on the Wireless Sensor Network, this paper analyzes the data distortion problem when transmitting H.264 coded video stream over error-prone channel. And the authors also have discussed a widely accepted technique that introduces more intra-coded information on macro block basis. Additionally, this paper introduces a simple loss and multiplication factor estimation method, the rate-distortion optimized assessing strategy over the whole situation.

SSIM-Motivated Rate-Distortion Optimization for Video Coding

We propose a rate-distortion optimization (RDO) scheme based on the structural similarity (SSIM) index, which was found to be a better indicator of perceived image quality than mean-squared error, but has not been fully exploited in the context of image and video coding. At the frame level, an adaptive Lagrange multiplier selection method is proposed based on a novel reduced-reference statistical SSIM estimation algorithm and a rate model that combines the side information with the entropy of the transformed residuals. At the macroblock level, the Lagrange multiplier is further adjusted based on an information theoretical approach that takes into account both the motion information content and perceptual uncertainty of visual speed perception. Finally, the mode for H.264/AVC coding is selected by the SSIM index and the adjusted Lagrange multiplier. Extensive experiments show that the proposed scheme can achieve significantly better rate-SSIM performance and provide better visual quality than conventional RDO coding schemes.

Robust Video Transmission With Distributed Source Coded Auxiliary Channel

We propose a novel solution to the problem of robust, low-latency video transmission over lossy channels. Predictive video codecs, such as MPEG and H.26x, are very susceptible to prediction mismatch between encoder and decoder or "drift" when there are packet losses. These mismatches lead to a significant degradation in the decoded quality. To address this problem, we propose an auxiliary codec system that sends additional information alongside an MPEG or H.26x compressed video stream to correct for errors in decoded frames and mitigate drift. The proposed system is based on the principles of distributed source coding and uses the (possibly erroneous) MPEG/H.26x decoder reconstruction as side information at the auxiliary decoder. The distributed source coding framework depends upon knowing the statistical dependency (or correlation) between the source and the side information. We propose a recursive algorithm to analytically track the correlation between the original source frame and the erroneous MPEG/H.26x decoded frame. Finally, we propose a rate-distortion optimization scheme to allocate the rate used by the auxiliary encoder among the encoding blocks within a video frame. We implement the proposed system and present extensive simulation results that demonstrate significant gains in performance both visually and objectively (on the order of 2 dB in PSNR over forward error correction based solutions and 1.5 dB in PSNR over intrarefresh based solutions for typical scenarios) under tight latency constraints.

Wavelet-Based Joint Estimation and Encoding of Depth-Image-Based Representations for Free-Viewpoint Rendering

We propose a wavelet-based codec for the static depth-image-based representation, which allows viewers to freely choose the viewpoint. The proposed codec jointly estimates and encodes the unknown depth map from multiple views using a novel rate-distortion (RD) optimization scheme. The rate constraint reduces the ambiguity of depth estimation by favoring piecewise-smooth depth maps. The optimization is efficiently solved by a novel dynamic programming along trees of integer wavelet coefficients. The codec encodes the image and the depth map jointly to decrease their redundancy and to provide a RD-optimized bitrate allocation between the two. The codec also offers scalability both in resolution and in quality. Experiments on real data show the effectiveness of the proposed codec.

FGS enhancement layer truncation with reduced intra-frame quality variation

This paper presents a rate-distortion optimized MPEG-4 FGS enhancement-layer truncation scheme. Our objective is to minimize the quality variation within each frame when the last transmitted enhancement layer is truncated according to the available network bandwidth. By properly redistributing the bit-budget to the enhancement layer that will be truncated, we can raise the quality of the whole frame uniformly. To achieve this goal, we use a trellis-search approach to decide which “1” bits will be truncated from the enhancement layer for each block. An operational rate-distortion optimization scheme based on the Lagrange Multiplier algorithm is adopted to decide the truncation criteria. Our proposed method can improve the visual quality and reduce the intra-frame quality variation both subjectively and objectively. Compared to previously reported straightforward truncation methods, our approach reduces the intra-frame quality variation significantly, and the decoded visual quality in terms of PSNR is also improved.

Rate-distortion optimal bit allocation for object-based video coding

In object-based video encoding, the encoding of the video data is decoupled into the encoding of shape, motion, and texture information, which enables certain functionalities, like content-based interactivity and content-based scalability. The fundamental problem, however, of how to jointly encode this separate information to reach the best coding efficiency has not been studied thoroughly. In this paper, we present an operational rate-distortion optimal scheme for the allocation of bits among shape, motion, and texture in object-based video encoding. Our approach is based on Lagrangian relaxation and dynamic programming. We implement our algorithm on the MPEG-4 video verification model, although it is applicable to any object-based video encoding scheme. The performance is accessed utilizing a proposed metric that jointly captures the distortion due to the encoding of the shape and texture. Experimental results demonstrate that the gains of lossy shape encoding depend on the percentage the shape bits occupy out of the total bit budget. This gain may be small or may be realized at very low bit rates for certain typical scenes.

Efficient rate control for JPEG2000 image coding

JPEG2000 is the new image coding standard which can provide superior rate-distortion performance over the old JPEG standard. However, the conventional post-compression rate-distortion (PCRD) optimization scheme in JPEG2000 is not efficient. It requires entropy encoding all available data even though a large portion of them will not be included in the final output. In this paper, three rate control methods are proposed to efficiently reduce both the computational complexity and memory usage over the conventional PCRD method. The first method, called successive bit-plane rate allocation (SBRA), allocates the bit rate by using the currently available rate-distortion information only. The second method, called priority scanning rate allocation (PSRA), allocates bits according to certain prioritized ordering. The third method uses PSRA to achieve optimal truncation as PCRD without encoding of all the image details and is called priority scanning with optimal truncation (PSOT). Simulation results suggest that the three proposed methods provide different tradeoff among visual quality, computational complexity, coding delay and working memory size. SBRA is memoryless and causal and requires the least computational complexity, lowest coding delay and achieves good visual quality. PSRA achieves higher PSNR than SBRA at the expense of larger working memory size and longer coding delay. PSOT gives the best PSNR but requires even more computation, delay and memory.

Motion-compensated transform coding of video using adaptive displacement fields

To find good trade-offs between the rate-distortion efficiency of motion-compensated, discrete-cosine-transform video coding and its computational complexity, we explore a scheme which uses an adaptive displacement vector field for motion compensation. The sampling density of the field is nonuniform and is adapted to the image scene using a Lagrangian optimization algorithm. The algorithm is formulated to minimize an overall distortion for a given bit rate constraint on the video frame. The algorithm takes into account the contributions to the overall bit rate and distortion due to motion vector and prediction residual coding. Our simulation results demonstrate that, for various common video sequences, the proposed scheme outperforms a popular H.263 test model by about 0.5 dB of peak signal-to-noise ratio. Significant reduction in coding artifacts is observed, and visual quality improvement is highly palpable. The work demonstrates that a suitably crafted rate-distortion optimization scheme can improve performance without exacerbating complexity.

Best wavelet packet bases in a rate-distortion sense

A fast rate-distortion (R-D) optimal scheme for coding adaptive trees whose individual nodes spawn descendents forming a disjoint and complete basis cover for the space spanned by their parent nodes is presented. The scheme guarantees operation on the convex hull of the operational R-D curve and uses a fast dynamic programing pruning algorithm to markedly reduce computational complexity. Applications for this coding technique include R. Coefman et al.'s (Yale Univ., 1990) generalized multiresolution wavelet packet decomposition, iterative subband coders, and quadtree structures. Applications to image processing involving wavelet packets as well as discrete cosine transform (DCT) quadtrees are presented.