Rate-distortion Optimization Framework Research Articles

EFFICIENT VIDEO COMPRESSION USING HEVC AND NON-LINEAR CONVOLUTIONAL MOBILEVNET BASED RATE-DISTORTION OPTIMIZATION

With the surge in demand for high-quality video content over various platforms, efficient video compression techniques have become indispensable. High-Efficiency Video Coding (HEVC) has been a cornerstone, yet further enhancements are essential for optimal compression. Despite HEVC’s advancements, achieving optimal compression while maintaining video quality remains challenging. Additionally, existing methods often overlook the computational complexity, hindering real-time applications. We propose a novel approach integrating HEVC with Non-Linear Convolutional MobileNet (NLCM) for enhanced compression efficiency. Our method employs a rate-distortion optimization framework, leveraging the capabilities of both HEVC and NLCM to achieve superior compression performance. NLCM provides adaptive filtering, enhancing spatial and temporal correlations, while HEVC ensures high compression efficiency. Through experimentation on standard video datasets, our method demonstrates significant improvements over existing techniques. Compared to HEVC alone, our approach achieves up to 30% reduction in bitrate at equivalent perceptual quality levels. Moreover, computational complexity is reduced by 15%, enabling real- time applications without compromising performance. The proposed method exhibits competitive results across various resolutions and frame rates, making it versatile for diverse video compression scenarios.

Rate Distortion Optimization: A Joint Framework and Algorithms for Random Access Hierarchical Video Coding.

This paper revisits the problem of rate distortion optimization (RDO) with focus on inter-picture dependence. A joint RDO framework which incorporates the Lagrange multiplier as one of parameters to be optimized is proposed. Simplification strategies are demonstrated for practical applications. To make the problem tractable, we consider an approach where prediction residuals of pictures in a video sequence are assumed to be emitted from a finite set of sources. Consequently the RDO problem is formulated as finding optimal coding parameters for a finite number of sources, regardless of the length of the video sequence. Specifically, in cases where a hierarchical prediction structure is used, prediction residuals of pictures at the same prediction layer are assumed to be emitted from a common source. Following this approach, we propose an iterative algorithm to alternatively optimize the selections of quantization parameters (QPs) and the corresponding Lagrange multipliers. Based on the results of the iterative algorithm, we further propose two practical algorithms to compute QPs and the Lagrange multipliers for the RA(random access) hierarchical video coding: the first practical algorithm uses a fixed formula to compute QPs and the Lagrange multipliers, and the second practical algorithm adaptively adjusts both QPs and the Lagrange multipliers. Experimental results show that these three algorithms, integrated into the HM 16.20 reference software of HEVC, can achieve considerable RD improvements over the standard HM 16.20 encoder, in the common RA test configuration.

SSIM-based distortion metric for film grain noise in HEVC

In video coding, the sum of squared differences (SSD) is traditionally used for rate-distortion optimization (RDO). However, SSD has been known that has low correlation on subjective quality. In particular, film grain noise (FGN)-synthesized video sequence is a very good example of subjective quality degradation with SSD-based RDO. Therefore, structural similarity (SSIM) has been considered for RDO owing to its simplicity and high correlation with subjective quality. The SSIM metric was not designed to be used for previous RDO framework; additional processing, such as content analysis or adaptive Lagrangian multipliers, was required in previous studies. Based on analyzing cases of degradation in SSIM-based coding, this study proposes a novel SSIM-like distortion measure. In this paper, two objectives are considered. First one is FGN-synthesized video coding using the SSIM-like distortion measure to preserve noise pattern. Seconds, the proposed metric is designed for direct application in previously developed RDO frameworks without scene-analysis-based RDO. The experimental results demonstrate that the proposed method reduces erroneous prediction blocks and the Bjontegaard delta rate by 67.46% on average compared to original SSIM-based RDO for FGN-synthesized video sequences. The results show the proposed metric is effective for film grain noise in similar bit rate, compared to a high-efficiency video coding test model (HM16.6) and the original SSIM metric.

On Rate Distortion Optimization Using SSIM

In this paper, we present a method for performing rate-distortion optimization (RDO) using a perceptual visual quality metric, the structural similarity index (SSIM), as the target of optimization. Rate-distortion optimization is widely used in modern video codecs to make various encoder decisions to optimize the rate-distortion tradeoff. Typically, the distortion measure used is either sum-of-square error or sum-of-absolute distance, both of which are convenient when used in the RDO framework but not always reflective of a perceptual visual quality. We show that SSIM can be used as the distortion metric in the RDO framework in a simple, yet effective, manner by scaling the Lagrange multiplier used in RDO based on the local variance in that region. The experimental results on the H.264/AVC reference software show that compared to traditional RDO approaches, for the same SSIM score, the proposed approach can achieve an average rate reduction of about 9% and 14% for random access and low-delay encoding configurations. At the same time, there is no significant change in the encoding runtime.

A Novel Rate Control Algorithm for H.264/AVC Standard

In this paper, we describe our rate distortion optimization framework for H.264/AVC standard. We investigate that the R-D characteristics of H.264 video signal in traditional transform-based video coding systems should be modeled for the Texture and Header components separately. Based on the proposed model, a R-D cost estimation function is also proposed to give a more accurate R-Q model. Built upon the above ideas, a rate control (RC) algorithm is developed for the H.264 encoder under the constant bit rate constraint. It is shown by experimental results that the new scheme can achieve better results in control bit rate and R-D performance compared to previously proposed approaches.

Rate-distortion optimized mode selection method for multiple description video coding

Multiple description coding (MDC) is a potential solution for video transmission over error-prone networks because it shows promising enhancement of error resilient capability. The MDC systems encode a single video stream into two or more equally important independent sub-streams, called descriptions. Therefore, if some of the descriptors get lost, remaining descriptors can be used to recover the video. Much research has proposed different information distribution methods. Since each method has different characteristic, we proposed a general rate-distortion optimization framework for MDC systems in this paper. By sophisticated rate-distortion analysis and optimization, the framework enables MDC systems to adaptively encode video considering contents and channel variation. Experimental results showed that, by comparing with the work in Tsai and You (IEEE Trans Circ Syst Video Technol 22(2):309---320, 2012), the proposed technique improves the R-D performance significantly. The improvement can be up to 2.4 dB for the channels with 0 %?~?20 % packet loss rates, and it can be even more if the loss rate increases. The proposed framework is not restricted to specific MDC tools. Ones can easily integrate their proposed coding tools into the framework and achieve better performance as long as the macroblock's bitrate and distortion information can be measured.

Scalable Coding of Depth Maps With R-D Optimized Embedding

Recent work on depth map compression has revealed the importance of incorporating a description of discontinuity boundary geometry into the compression scheme. We propose a novel compression strategy for depth maps that incorporates geometry information while achieving the goals of scalability and embedded representation. Our scheme involves two separate image pyramid structures, one for breakpoints and the other for sub-band samples produced by a breakpoint-adaptive transform. Breakpoints capture geometric attributes, and are amenable to scalable coding. We develop a rate-distortion optimization framework for determining the presence and precision of breakpoints in the pyramid representation. We employ a variation of the EBCOT scheme to produce embedded bit-streams for both the breakpoint and sub-band data. Compared to JPEG 2000, our proposed scheme enables the same the scalability features while achieving substantially improved rate-distortion performance at the higher bit-rate range and comparable performance at the lower rates.

Algorithm and Architecture Design of Bandwidth-Oriented Motion Estimation for Real-Time Mobile Video Applications

This paper proposes a data bandwidth-oriented motion estimation design for resource-limited mobile video applications using an integrated bandwidth rate distortion optimization framework. This framework predicts and allocates the appropriate data bandwidth for motion estimation under a limited bandwidth supply to fit a dynamically changing bandwidth supply. The simulation results show that our proposed algorithm can achieve 66% and 41% memory bandwidth savings while maintaining an equivalent rate-distortion performance and meeting real-time targets, when compared with conventional approaches for low-motion and high-motion D1 (704ntn 576)-size video, respectively. The final implementation costs 122 K gate counts with TSMC 0.13-μ m CMOS technology and consumes 74 mW of power for D1 resolution at 30 frames/s which is 40% of that achieved in previous designs.

Rate-Distortion Optimized Pixel-Based Motion Vector Concatenation for Reference Picture Selection

Reference picture selection (RPS) is a powerful error-control technique for video streaming. Previously, two fast block-based motion vector concatenation (MVC) algorithms were proposed for video transcoding based on forward dominant vector selection (FDVS) and activity dominant vector selection (ADVS). In this paper, we cast these algorithms in the rate-distortion (RD) optimization framework. We also present two novel RD optimized pixel-based MVC schemes for RPS. Experimental results indicate that the proposed methods provide higher video quality compared to both FDVS and ADVS. In addition, we study the complexity of various RPS algorithms as a function of the loss rate and round trip time.

Fast Intra Prediction Algorithm for H.264 High Profile Based on Image Texture

H.264/AVC standard uses rate-distortion optimization framework with heavy calculation and full search algorithm to select the best mode of each macroblock. Besides, 8×8 luma block prediction is added to high profile, which makes the search algorithm process more complicated. In order to reduce the amount of calculation, a filter method based on image texture feature and image complexity is adopted in this paper. Experiment results show that the new algorithm achieves an average saving of 43%-57% in encoding time ,with negligible effect on bitrate and PSNR.

SSIM-Based Perceptual Rate Control for Video Coding

The quality of video is ultimately judged by human eye; however, mean squared error and the like that have been used as quality metrics are poorly correlated with human perception. Although the characteristics of human visual system have been incorporated into perceptual-based rate control, most existing schemes do not take rate-distortion optimization into consideration. In this paper, we use the structural similarity index as the quality metric for rate-distortion modeling and develop an optimum bit allocation and rate control scheme for video coding. This scheme achieves up to 25% bit-rate reduction over the JM reference software of H.264. Under the rate-distortion optimization framework, the proposed scheme can be easily integrated with the perceptual-based mode decision scheme. The overall bit-rate reduction may reach as high as 32% over the JM reference software.

Perceptual Rate-Distortion Optimization Using Structural Similarity Index as Quality Metric

The rate-distortion optimization (RDO) framework for video coding achieves a tradeoff between bit-rate and quality. However, objective distortion metrics such as mean squared error traditionally used in this framework are poorly correlated with perceptual quality. We address this issue by proposing an approach that incorporates the structural similarity index as a quality metric into the framework. In particular, we develop a predictive Lagrange multiplier estimation method to resolve the chicken and egg dilemma of perceptual-based RDO and apply it to H.264 intra and inter mode decision. Given a perceptual quality level, the resulting video encoder achieves on the average 9% bit-rate reduction for intra-frame coding and 11% for inter-frame coding over the JM reference software. Subjective test further confirms that, at the same bit-rate, the proposed perceptual RDO indeed preserves image details and prevents block artifact better than traditional RDO.

RD Optimized Bandwidth Efficient Motion Estimation and Its Hardware Design With On-Demand Data Access

Data bandwidth dominates the performance and power consumption in the video encoder design. In which a low bandwidth and bandwidth aware motion estimation design enables smooth and better video quality as well as lower power consumption on data accesses. This paper proposes a bandwidth efficient motion estimation and its hardware implementation to deal with the bandwidth issues. First, an on-demand data access mechanism is proposed to acquire the reference data according to the video content for motion estimation process and thus can avoid unnecessary reference data loading. Furthermore, the available bandwidth constraint is properly modeled into our proposed rate distortion optimization framework to efficiently use the data bandwidth. Simulation results show that our proposed algorithm not only allocates proper data bandwidth for motion estimation according to video content but also saves 79.15% data bandwidth demand with 0.03 dB PSNR drop and 2.50% bitrate increase in maximum for 4 CIF resolution sequences, when compared to the fully data reuse full search motion estimation which reuses the overlapped reference data to avoid unnecessary data reloading. In addition, under the available data bandwidth constraint, our proposed algorithm can achieve 2.43%, 0.08%, and 0.20% BD-bitrate saving with 0.17 dB, 0.01 dB, and 0.01 dB BD-PSNR increase on average for high, median, and low motion sequences when compared to the full search motion estimation algorithm. The resulted design only needs 75.27 K gate counts when running at 23 MHz operating frequency for 4 CIF at 30 frames/s with 90 nm CMOS process due to its search range independent buffer design.

Key-frame reference selection for non-feedback video communication

This article addresses the problem of reference picture optimization in video communication over error prone networks. A novel estimation model for transmission distortion is proposed. This model is capable of recursively estimating the overall end-to-end distortion caused by quantization, error propagation, and error concealment. Simulation results show that this model can accurately estimate channel distortion. Then, based on the distortion estimation model, a new non-feedback key-frame reference picture selection (KRPS) algorithm is developed. The optimum reference picture minimizes the transmission distortion under the rate-distortion optimization framework. Extensive experiment results demonstrate that the proposed KRPS algorithm substantially achieves more peak signal to noise ratio (PSNR) gain over traditional prediction, especially in low bit-rate transmission.

Soft Decision Quantization for H.264 With Main Profile Compatibility

In this paper, we study the rate-distortion (RD) optimization of the H.264 main profile encoding. Specifically, a soft decision quantization (SDQ) algorithm is developed based on the context adaptive binary arithmetic coding (CABAC) method in the H.264 main profile. Given motion prediction and quantization step sizes, the proposed SDQ algorithm is proved to achieve near-optimal SDQ for residual coding in the sense of minimizing the true RD cost when the weak adjacent block dependency utilized in CABAC is ignored for optimization. The SDQ algorithm is then used in conjunction with a general RD optimization framework to jointly design motion prediction and residual coding for H.264 main profile coding given previously coded reference frames. Experiments have been conducted based on the reference encoder JM82 of H.264 main profile. Comparative studies show that the joint design method achieves on average 10% rate reduction at the same PSNR when compared with the RD method in the H.264 main-profile reference software, with half of the reduction coming from the proposed SDQ algorithm, and 20% rate reduction at the same PSNR when compared with the RD method in the H.264 baseline-profile reference software.

Joint Source and Channel Coding for 3-D Scene Databases Using Vector Quantization and Embedded Parity Objects

Three-dimensional graphic scenes contain various mesh objects in one geometric space where different objects have potentially unequal importance regarding display. This paper proposes an object-oriented system for efficiently coding and streaming 3-D scene databases in lossy and rate-constrained environments. Vector quantization (VQ) is exploited to code 3-D scene databases into multiresolution hierarchies. For the best distortion-rate performance, adaptive quantization precisions are allocated to different objects and different layers of each object based on a weighted distortion model. Upon transmission, scalably coded objects are delivered in respective packet sequences to preserve their manipulation independency. For packet loss resilience, a plurality of FEC codes are generated as "parity objects" parallel to graphic objects, which protect the graphic objects concurrently and also preferentially in regard to their unequal decoding importance. A rate-distortion optimization framework is then developed, which performs rate allocation between graphic objects and parity objects and generates the parity data properly. We show that, by treating graphic objects jointly and preferentially in source and channel coding while preserving their independencies in transport, the proposed system reduces the receiving distortion of the 3-D database significantly compared to conventional methods.

Multistreaming of 3-D Scenes With Optimized Transmission and Rendering Scalability

Three-dimensional (3D) graphic scenes require considerable network bandwidth to be transmitted and computing power to be rendered on a user's terminal. Toward high-quality display in real time, we propose a sender-driven mechanism for streaming 3D scenes in a resource-constrained environment. In doing so, objects are encoded into multiresolutions to provide transmission and rendering scalability, and a weighted distortion metric is developed to measure the quality of a scene rendered with multiresolution objects, modeling objects' unequal importance regarding display. To preserve the manipulation independency of multiple objects in data delivery while provide preferential treatment for different objects as well as different layers of each object, transmission of the objects is performed over multiple streams in a partially sequenced and partially reliable fashion. A rate-distortion optimization framework is developed, which determines an optimal level of reliability for every chunk of data in each stream, taking into account the rendering importance of the object, the distortion-rate performance of the data chunks, and the statistics of the network link. Compared with heuristical methods, simulation results show that the proposed framework maximizes the display quality of the scene while minimizing the amount of data that needs to be processed by the client's rendering engine

The Virtue of Patience in Low-Complexity Scheduling of Packetized Media With Feedback

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> We consider streaming pre-encoded and packetized media over best-effort networks in the presence of acknowledgment feedbacks. We first review a rate-distortion (RD) optimization framework that can be employed in such scenarios. As part of the framework, a scheduling algorithm selects the data to send over the network at any given time, so as to minimize the end-to-end distortion, given an estimate of channel resources and a history of previous transmissions and received acknowledgements. In practice, a greedy scheduling strategy is often considered to limit the solution search space, and reduce the computational complexity associated to the RD optimization framework. Our work observes that popular greedy schedulers are strongly penalized by early retransmissions. Therefore, we propose a scheduling algorithm that avoids premature retransmissions, while preserving the low computational complexity aspect of the greedy paradigm. Such a scheduling strategy maintains close to optimal RD performance when adapting to network bandwidth fluctuations. Our experimental results demonstrate that the proposed patient greedy scheduler provides a reduction of up to 50% in transmission rate relative to conventional greedy approaches, and that it brings up to 2 dB of quality improvement in scheduling classical MPEG-based packet video streams. </para>

RaDiO Edge: Rate-Distortion Optimized Proxy-Driven Streaming From the Network Edge

This paper addresses the problem of streaming packetized media over a lossy packet network through an intermediate proxy server to a client, in a rate-distortion optimized way. The proxy, located at the junction of the backbone network and the last hop to the client, coordinates the communication between the media server and the client using hybrid receiver/sender-driven streaming in a rate-distortion optimization framework. The framework enables the proxy to determine at every instant which packets, if any, it should either request from the media server or (re)transmit directly to the client, in order to meet constraints on the average transmission rates on the backbone and the last hop while minimizing the average end-to-end distortion. Performance gains are observed over rate-distortion optimized sender-driven systems for streaming packetized video content. The improvement in performance depends on the quality of the network path both in the backbone network and along the last hop

Network-adaptive low-latency video communication over best-effort networks

The quality of service limitation of today's best-effort networks poses major challenge for low-latency video communication. To combat network losses for real-time and on-demand video communication, which exhibits stronger dependency across packets, a network-adaptive coding scheme is employed to dynamically manage the packet dependency using optimal reference picture selection. The selection of the reference is achieved within a rate-distortion optimization framework and is adapted to the varying network conditions. For network-adaptive streaming of prestored video, based on an accurate loss-distortion model, a prescient scheme that optimizes the dependency of a group of packets is proposed to achieve global optimality as well as improved rate-distortion performance. With the improved trade-off between compression efficiency and error resilience, the proposed system does not require retransmission of lost packets, which makes less than one-second low-latency communication possible.