Rate-distortion Optimization Problem Research Articles

Model-Based Joint Bit Allocation Between Geometry and Color for Video-Based 3D Point Cloud Compression

Rate distortion optimization plays a very important role in image/video coding. But for 3D point cloud, this problem has not been investigated. In this paper, the rate and distortion characteristics of 3D point cloud are investigated in detail, and a typical and challenging rate distortion optimization problem is solved for 3D point cloud. Specifically, since the quality of the reconstructed 3D point cloud depends on both the geometry and color distortions, we first propose analytical rate and distortion models for the geometry and color information in video-based 3D point cloud compression platform, and then solve the joint bit allocation problem for geometry and color based on the derived models. To maximize the reconstructed quality of 3D point cloud, the bit allocation problem is formulated as a constrained optimization problem and solved by an interior point method. Experimental results show that the rate-distortion performance of the proposed solution is close to that obtained with exhaustive search but at only 0.68% of its time complexity. Moreover, the proposed rate and distortion models can also be used for the other rate-distortion optimization problems (such as prediction mode decision) and rate control technologies for 3D point cloud coding in the future.

Transform Quantization for CNN Compression.

In this paper, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous CNN quantization techniques tend to ignore the joint statistics of weights and activations, producing sub-optimal CNN performance at a given quantization bit-rate, or consider their joint statistics during training only and do not facilitate efficient compression of already trained CNN models. We optimally transform (decorrelate) and quantize the weights post-training using a rate-distortion framework to improve compression at any given quantization bit-rate. Transform quantization unifies quantization and dimensionality reduction (decorrelation) techniques in a single framework to facilitate low bit-rate compression of CNNs and efficient inference in the transform domain. We first introduce a theory of rate and distortion for CNN quantization and pose optimum quantization as a rate-distortion optimization problem. We then show that this problem can be solved using optimal bit-depth allocation following decorrelation by the optimal End-to-end Learned Transform (ELT) we derive in this paper. Experiments demonstrate that transform quantization advances the state of the art in CNN compression in both retrained and non-retrained quantization scenarios. In particular, we find that transform quantization with retraining is able to compress CNN models such as AlexNet, ResNet and DenseNet to very low bit-rates (1-2 bits).

Rate Distortion Optimization: A Joint Framework and Algorithms for Random Access Hierarchical Video Coding.

This paper revisits the problem of rate distortion optimization (RDO) with focus on inter-picture dependence. A joint RDO framework which incorporates the Lagrange multiplier as one of parameters to be optimized is proposed. Simplification strategies are demonstrated for practical applications. To make the problem tractable, we consider an approach where prediction residuals of pictures in a video sequence are assumed to be emitted from a finite set of sources. Consequently the RDO problem is formulated as finding optimal coding parameters for a finite number of sources, regardless of the length of the video sequence. Specifically, in cases where a hierarchical prediction structure is used, prediction residuals of pictures at the same prediction layer are assumed to be emitted from a common source. Following this approach, we propose an iterative algorithm to alternatively optimize the selections of quantization parameters (QPs) and the corresponding Lagrange multipliers. Based on the results of the iterative algorithm, we further propose two practical algorithms to compute QPs and the Lagrange multipliers for the RA(random access) hierarchical video coding: the first practical algorithm uses a fixed formula to compute QPs and the Lagrange multipliers, and the second practical algorithm adaptively adjusts both QPs and the Lagrange multipliers. Experimental results show that these three algorithms, integrated into the HM 16.20 reference software of HEVC, can achieve considerable RD improvements over the standard HM 16.20 encoder, in the common RA test configuration.

Rate-Distortion-Complexity Optimized Coding Mode Decision for HEVC

The newest generation of video coding standard, high efficiency video coding (HEVC), significantly improves the video compression efficiency by introducing more flexible block partitioning structures and richer coding modes than those of the previous coding standards; however, the encoders suffer from high-computational complexity, which greatly hinders their extensive application. Extensive studies on optimizing the complexity of the HEVC encoders have been conducted. However, most studies do not effectively achieve a trade-off between the rate-distortion (RD) performance loss and complexity during the rate-distortion optimization (RDO). In this paper, we mathematically define the complexity-constrained RDO problem as a constrained optimization problem of subset selection. Next, based on the classification methodology, the derivation process for this optimization problem is simplified to find the adaptive threshold function in the feature space with extremely low complexity. The proposed method is also highly general and is applicable to algorithm design for various coding mode decisions, such as coding unit splitting, prediction unit partitioning and transform unit tree decision, and the global optimum can be achieved. Compared with existing methods, the experimental results show that the proposed method can reduce the coding time by 2–16% with the same RD performance loss and can decrease the BD rate by 0.1–1.2% under the same complexity. In addition, this method is capable of flexibly adjusting the complexity under different rate-distortion complexity trade-off requirements.

Novel Integration of Frame Rate Up Conversion and HEVC Coding Based on Rate-Distortion Optimization.

Frame rate up conversion (FRUC) can improve the visual quality by interpolating new intermediate frames. However, high frame rate videos by FRUC are confronted with more bitrate consumption or annoying artifacts of interpolated frames. In this paper, a novel integration framework of FRUC and high efficiency video coding (HEVC) is proposed based on rate-distortion optimization, and the interpolated frames can be reconstructed at encoder side with low bitrate cost and high visual quality. First, joint motion estimation (JME) algorithm is proposed to obtain robust motion vectors, which are shared between FRUC and video coding. What's more, JME is embedded into the coding loop and employs the original motion search strategy in HEVC coding. Then, the frame interpolation is formulated as a rate-distortion optimization problem, where both the coding bitrate consumption and visual quality are taken into account. Due to the absence of original frames, the distortion model for interpolated frames is established according to the motion vector reliability and coding quantization error. Experimental results demonstrate that the proposed framework can achieve 21% ~ 42% reduction in BDBR, when compared with the traditional methods of FRUC cascaded with coding.

Delay–Rate–Distortion Optimization for Cloud Gaming With Hybrid Streaming

Cloud gaming as the emerging game service has attracted significant attention. However, traditional video streaming approach suffers from high bandwidth consumption, and traditional graphics streaming approach requires a long initial period to download game models. In this paper, we propose a novel hybrid streaming framework, jointly applying video streaming and graphics streaming to provide a high-quality gaming experience. In the proposed framework, cloud servers not only transmit the encoded video frames but also progressively transmit the graphics data, which are used to render a game frame to provide an additional reference to the video encoder. Based on the proposed framework, we investigate the delay–rate–distortion optimization problem, where the source rate between the video stream and the graphics stream is optimized to minimize the overall distortion under the bandwidth and response delay constraints. The experimental results demonstrate that the proposed hybrid streaming can achieve the lowest distortion under the constraints of bandwidth and response delay, compared with the traditional video streaming and graphics streaming.

Temporally Dependent Rate-Distortion Optimization for Low-Delay Hierarchical Video Coding.

Low-delay hierarchical coding structure (LD-HCS), as one of the most important components in the latest High Efficiency Video Coding (HEVC) standard, greatly improves coding performance. It groups consecutive P/B frames into different layers and encodes them with different quantization parameters (QPs) and reference mechanisms in such a way that temporal dependency among frames can be exploited. However, due to varying characteristics of video contents, temporal dependency among coding units differs significantly from each other in the same or different layers, while a fixed LD-HCS scheme cannot take full advantage of the dependency, leading to a substantial loss in coding performance. This paper addresses the temporally dependent rate distortion optimization (RDO) problem by attempting to exploit varying temporal dependency of different units. First, the temporal relationship of different frames under the LD-HCS is examined, and hierarchical temporal propagation chains are constructed to represent the temporal dependency among coding units in different frames. Then, a hierarchical temporally dependent RDO scheme is developed specifically for the LD-HCS based on a source distortion propagation model. Experimental results show that our proposed scheme can achieve 2.5% and 2.3% BD-rate gain in average compared with the HEVC codec under the same configuration of P and B frames, respectively, with a negligible increase in encoding time. Furthermore, coupled with QP adaption, our proposed method can achieve higher coding gains, e.g., with multi-QP optimization, about 5.4% and 5.0% BD-rate saving in average over the HEVC codec under the same setting of P and B frames, respectively.

Adaptive Progressive Motion Vector Resolution Selection Based on Rate-Distortion Optimization.

In the state-of-the-art H.265/HEVC video coding standard, the motion vector (MV) resolution is fixed to be 1/4-pel for the entire video sequence, while the inherent video characteristics, e.g., texture complexities and motion activities, have been largely ignored. Obviously such strategy may not suffice the demand of high-accuracy motion compensation. In this paper, we propose a specially designed rate-distortion model in terms of the MV resolution by taking the video characteristics into consideration. In particular, the MV resolution selection is formulated as a rate-distortion optimization problem by analyzing the rate-distortion cost of each MV resolution candidate. To further improve the coding performance, the progressive MV resolution strategy is employed, where the optimal progressive MV resolution is determined by decision trees constructed with the rate-distortion model. In this manner, a novel adaptive progressive motion vector resolution selection scheme can be realized and the MV resolution can be adaptively adjusted based on the properties of local content. Extensive experiments and comparisons show that the proposed algorithm significantly improves the coding performance, and 1.8% BD-rate gain on average has been achieved without introducing any noticeable computational complexity.

Efficient Quantization Based on Rate–Distortion Optimization for Video Coding

Most rate-distortion (R-D) optimized quantization methods of video coding involve an exhaustive search process to determine the optimal quantized transform coefficients of a coding block and are computationally more expensive than the conventional quantization. In this paper, we present a novel analytical method that directly solves the rate–distortion optimization problem in a closed form by employing a rate model for entropy coding. It has the appealing property of low complexity and is easy to implement. The results show that the proposed method is $4\times $ to $40\times $ faster than the previous methods and 3%–5% more efficient in bitrate than the H.264/AVC reference encoder, which uses the conventional quantization, for video coded with the IBBP structure of group of pictures (GOP) in the normal peak signal-to-noise ratio (PSNR) range (30–40 dB).

Decoder-Complexity-Aware Encoding of Motion Compensation for Multiple Heterogeneous Receivers

For mobile multimedia systems, advances in battery technology have been much slower than those in memory, graphics, and processing power, making power consumption a major concern in mobile systems. The computational complexity of video codecs, which consists of CPU operations and memory accesses, is one of the main factors affecting power consumption. In this article, we propose a method that achieves near-optimal video quality while respecting user-defined bounds on the complexity needed to decode a video. We specifically focus on the motion compensation process, including motion vector prediction and interpolation, because it is the single largest component of computation-based power consumption. We start by formulating a scenario with a single receiver as a rate-distortion optimization problem and we develop an efficient decoder-complexity-aware video encoding method to solve it. Then we extend our approach to handle multiple heterogeneous receivers, each with a different complexity requirement. We test our method experimentally using the H.264 standard for the single receiver scenario and the H.264 SVC extension for the multiple receiver scenario. Our experimental results show that our method can achieve up to 97% of the optimal solution value in the single receiver scenario, and an average of 97% of the optimal solution value in the multiple receiver scenario. Furthermore, our tests with actual power measurements show a power saving of up to 23% at the decoder when the complexity threshold is halved in the encoder.

Forward Error Correction-Based 2-D Layered Multiple Description Coding for Error-Resilient H.264 SVC Video Transmission

In this paper, we propose a novel 2-D layered multiple description coding (2DL-MDC) for error-resilient video transmission over unreliable networks. The proposed 2DL-MDC scheme allocates multiple description sub-bitstreams of a 2-D scalable bitstream to two network paths with unequal loss rates. We formulate the 2-D scalable rate-distortion problem and derive the expected distortion for the proposed scheme. To minimize the end-to-end distortion given the total rate budget and packet loss probabilities, we need to optimally allocate source and channel rates for each hierarchical sublayer of the scalable bitstream. The conventional Lagrangian multiplier method can be utilized to solve this problem but with overwhelming computational complexity. Therefore, we consider the use of the genetic algorithm to solve the rate-distortion optimization problem. The simulation results verify that the proposed method is able to achieve significant performance gain as opposed to the conventional equal rate allocation method.

Simultaneous MAP-Based Video Denoising and Rate-Distortion Optimized Video Encoding

In this paper, a simultaneous MAP-based video denoising and rate-distortion optimized video encoding algorithm is proposed. We begin with formulating the denoising problem as a maximum a posteriori (MAP) estimate problem. Then, according to the Bayes rule, we show that the MAP estimate is determined by two terms: noise conditional density model and priori conditional density model. Based on the assumptions that the noise satisfies Gaussian distribution and the priori model is measured by the bit-rate, the MAP estimate can be expressed as a rate distortion optimization problem. With this, we are able to simultaneously perform MAP-based video denoising and rate-distortion optimized video encoding under some assumptions. Moreover, we describe in details how to select suitable coding parameters, i.e., quantization parameter, mode, motion vector, reference index, and regularization parameter. Finally, we conduct several experiments to verify our proposed algorithm.

Iterative Rate-Distortion Optimization of H.264 With Constant Bit Rate Constraint

In this paper, we apply the primal-dual decomposition and subgradient projection methods to solve the rate-distortion optimization problem with the constant bit rate constraint. The primal decomposition method enables spatial or temporal prediction dependency within a group of picture (GOP) to be processed in the master primal problem. As a result, we can apply the dual decomposition to minimize independently the Lagrangian cost of all the MBs using the reference software model of H.264. Furthermore, the optimal Lagrange multiplier lambda* is iteratively derived from the solution of the dual problem. As an example, we derive the optimal bit allocation condition with the consideration of temporal prediction dependency among the pictures. Experimental results show that the proposed method achieves better performance than the reference software model of H.264 with rate control.

Rate-distortion optimal video summary generation

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth, and/or power are limited. The summarization process inevitably introduces distortion. The amount of summarization distortion is related to its "conciseness," or the number of frames available in the summary. If there are m frames in the original sequence and n frames in the summary, we define the summarization rate as m/n, to characterize this "conciseness". We also develop a new summarization distortion metric and formulate the summarization problem as a rate-distortion optimization problem. Optimal algorithms based on dynamic programming are presented and compared experimentally with heuristic algorithms. Practical constraints, like the maximum number of frames that can be skipped, are also considered in the formulation and solution of the problem.

Error concealment aware rate shaping for wireless video transport

Streaming of video, which is both source- and channel-coded, over wireless networks faces the challenge of time-varying packet loss rate and fluctuating bandwidth. Rate shaping (RS) has been proposed to reduce the bit-rate of a precoded video bitstream to adapt to the real-time bandwidth variation. In our earlier work, rate shaping was extended to consider not only the bandwidth but also the packet loss rate variations. Rate-distortion optimized rate adaptation is performed on the precoded video that is a scalable coded bitstream protected by forward error correction codes. In this paper, we propose an RS scheme that further takes into account the error concealment (EC) method used at the receiver. We refer to this scheme as EC aware RS (ECARS). When performing ECARS, first ECARS needs to know the benefit/ gain of sending each part of the precoded video, as opposed to not sending it but reconstructing it by EC. Then given a certain packet loss probability, the expected gain can be derived and be included in the rate-distortion optimization problem formulation. Finally, ECARS performs rate-distortion optimization to adapt the rate of the precoded video. A two-stage rate-distortion optimization approach is proposed to solve the ECARS rate-distortion optimization problem. In addition to ECARS, the precoding process can be EC aware to prioritize the precoded video based on the gain. We present an example EC aware precoding process by means of macroblock prioritization. Experiment results of ECARS together with EC aware precoding are shown to have excellent performance.

A new fast block matching algorithm based on complexity‐distortion optimization

AbstractMost fast block matching algorithms ignore the efficiency in motion compensation within each checking step. In order to achieve better‐compensated performance, the limited computational complexity should be allocated more carefully into each block. It means that the fast block matching algorithm can be viewed as a kind of rate‐distortion optimization problem. The complexity‐distortion optimal fast block matching algorithm should find the maximized quality of the compensated image under a target computational complexity. In order to approach the optimal complexity‐distortion solution, some strategies are developed. For example, a domination‐based motion vector prediction technique is developed to set the initial motion vector for each block. A predictive complexity‐distortion benefit list is established to predict the compensated benefit for each block. Also, a three‐level pattern searching is employed to check the candidate motion vector. Experimental results show that our proposed algorithm outperforms significantly the three‐step search. For example, in “Salesman,” the average checkpoints for one block is 33 by using the three‐step search. The average checkpoint is 1.75 by using our proposal algorithm under the same average PSNR condition. © 2002 Wiley Periodicals, Inc. Int J Imaging Syst Technol 12, 63–67, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10012

Rate control for robust video transmission over burst-error wireless channels

We study the problem of rate control for transmission of video over burst-error wireless channels, i.e., channels such that errors tend to occur in clusters during fading periods. In particular we consider a scenario consisting of packet based transmission with automatic repeat request (ARQ) error control and a back channel. We start by showing how the delay constraints in real time video transmission can be translated into rate constraints at the encoder, where the applicable rate constraints at a given time depend on future channel rates. With the acknowledgments received through the back channel we have an estimate of the current channel state. This information, combined with an a priori model of the channel, allows us to statistically model the future channel rates. Thus the rate constraints at the encoder can be expressed in terms of the expected channel behavior. We can then formalize a rate distortion optimization problem, namely, that of assigning quantizers to each of the video blocks stored in the encoder buffer such that the quality of the received video is maximized. This requires that the rate constraints be included in the optimization, since violating a rate constraint is equivalent to violating a delay constraint and thus results in losing a video block. We formalize two possible approaches. The first one seeks to minimize the distortion for the expected rate constraints given the channel model and current observation. The second approach seeks to allocate bits so as to minimize the expected distortion for the given model. We use both dynamic programming and Lagrangian optimization approaches to solve these problems. Our simulation results demonstrate that both the video distortion at the decoder and packet loss rate can be significantly reduced when incorporating the channel information provided by the feedback channel and the a priori model into the rate control algorithm.