Rate-distortion Model Research Articles

Two-Stage Perceptual Quality Oriented Rate Control Algorithm for HEVC

As a practical technique in mainstream video coding applications, rate control dominates important to ensure compression quality with limited bitrates constraints. However, most rate control methods mainly focus on objective quality while ignoring the perceptual quality improvement for human eyes. In this paper, we propose a two-stage rate control algorithm to optimize the perceptual quality at the frame encoding stage and the coding tree unit (CTU) encoding stage for high efficiency video coding (HEVC), respectively. Firstly, for the frame encoding stage, with inter-frame distortion dependency consideration, a frame-level rate control method is presented by adjusting the frame-level Lagrange multiplier adaptively with a preprocessing method. Secondly, for the CTU encoding stage, we propose a saliency-based CTU-level perceptual quality rate control algorithm, which employs CTU-level saliency weight to adjust the perceptual rate-distortion (R-D) model. We conduct the CTU-level rate control by an optimized Lagrange multiplier and quantization parameter (QP) to achieve perceptual quality optimization. Extensive experimental results reveal that, compared with state-of-the-art rate control methods on HEVC, our algorithm achieves significant perceptual coding performance with improved subjective visual quality.

Latitude-Redundancy-Aware All-Zero Block Detection for Fast 360-Degree Video Coding.

The sphere-to-plane projection of 360-degree video introduces substantial stretched redundant data, which is discarded when reprojected to the 3D sphere for display. Consequently, encoding and transmitting such redundant data is unnecessary. Highly redundant blocks can be referred to as all-zero blocks (AZBs). Detecting these AZBs in advance can reduce computational and transmission resource consumption. However, this cannot be achieved by existing AZB detection techniques due to the unawareness of the stretching redundancy. In this paper, we first derive a latitude-adaptive redundancy detection (LARD) approach to adaptively detect coefficients carrying redundancy in transformed blocks by modeling the dependency between valid frequency range and the stretching degree based on spectrum analysis. Utilizing LARD, a latitude-redundancy-aware AZB detection scheme tailored for fast 360-degree video coding (LRAS) is proposed to accelerate the encoding process. LRAS consists of three sequential stages: latitude-adaptive AZB (L-AZB) detection, latitude-adaptive genuine-AZB (LG-AZB) detection and latitude-adaptive pseudo-AZB (LP-AZB) detection. Specifically, L-AZB refers to the AZB introduced by projection. LARD is used to detect L-AZB directly. LG-AZB refers to the AZB after hard-decision quantization and zeroing redundant coefficients. A novel latitude-adaptive sum of absolute difference estimation model is built to derive the threshold for LG-AZB detection. LP-AZB refers to the AZB in terms of rate-distortion optimization considering redundancy. A latitude-adaptive rate-distortion model is established for LP-AZB detection. Experimental results show that LRAS can achieve an average total encoding time reduction of 25.85% and 20.38% under low-delay and random access configurations compared to the original HEVC encoder, with only 0.16% and 0.13% BDBR increases and 0.01dB BDPSNR loss, respectively. The transform and quantization time savings are 60.13% and 59.94% on average.

Quality-Constrained Encoding Optimization for Omnidirectional Video Streaming

Omnidirectional video streaming is usually implemented based on the representations of tiles, where the tiles are obtained by splitting the video frame into several rectangular areas and each tile is converted into multiple representations with different resolutions and encoded at different bitrates. One key issue in omnidirectional video streaming is how to choose the optimal representations for each tile at the server to save the overall transmission bitrate to all users while offering them satisfactory quality. This is different from the adaptive bitrate-based method that optimizes the downloading procedure of individual users, where the given video representations are stored on the server. In this work, we focus on optimization for the encoding of omnidirectional video streaming by using the optimal combination of tile representations. To achieve our goal, we formulate the selection of the representations into an optimization problem in which the transmission bitrate of all the representations is minimized with a quality constraint. By using this constraint, we can improve the average quality of omnidirectional videos for users. More specifically, we first construct the tile-level rate-distortion (R-D) model and determine the available tile bandwidth based on the previous viewers’ statistics. Then, we formulate the representation selection problem based on the obtained R-D model and tile bandwidth. Finally, we solve this problem to obtain the optimal combination of tile representations so that we can transmit the omnidirectional video to users with satisfactory quality but low bitrate. The experimental results demonstrate the effectiveness of our proposed approach when it is applied to omnidirectional video streaming.

A CTU-Level Screen Content Rate Control for Low-Delay Versatile Video Coding

In this paper, a rate control scheme for screen content video coding is proposed for the Versatile Video Coding (VVC) standard. In view of the critical challenges arising from the spatial and temporal unnaturalness of screen content sequences, the proposed method relies on the specifically designed pre-analysis such that the content information regarding the scene complexity can be obtained. As such, the estimated residual complexity is then incorporated into the proposed complexity-aware rate models and distortion models, leading to the optimal bit allocations for each frame and coding tree unit (CTU). In particular, the optimization problem can be analytically solved with the proposed models, and the coding parameters such as Lagrangian multiplier λ and quantization parameter of each frame and CTU could be delicately derived according to the allocated bits through the proposed analytical models. Extensive experiments have been conducted to evaluate the effectiveness of the proposed method. Compared to the default hierarchical λ-domain rate control and other screen content rate control algorithms, the proposed method could achieve obvious RD performance gain, and the bit-rate accuracy could be improved.

Toward Low-Latency and High-Quality Adaptive 360$^\circ$ Streaming

Advanced 360-degree video streaming is essential for high-quality communication and industrial video applications, supporting novel and interactive visual experiences that promote consumption. However, due to the inherent fluctuations in communication networks, hardware resources, and power costs, a trade-off needs to be achieved between occupying network bandwidth and guaranteeing 360-degree visual quality. To address this issue, this paper proposes a quality-aware global optimization solution that combines different types of sensory characteristics to further enhance the 360-degree streaming efficiency in cellular networks. First, considering the characteristics of 360-degree video, each frame is divided into three different regions based on a proposed field-of-view (FoV) prediction method. Second, a new region-based rate-distortion model reflecting the bitrate-quality relationship is proposed. The divided regions are represented by different rate-distortion models. In addition, a rate-distortion model parameter update strategy with robustness to region changes is proposed to further guarantee transmission performance. Finally, we propose a globally optimized adaptive bitrate allocation algorithm to optimize 360-degree mobile streaming, which uses both the rate-distortion models and viewpoint prediction results. Evaluation results indicate that the proposed method outperforms state-of-the-art approaches in terms of several quality of experience (QoE) objectives under various network conditions.

Rate-Distortion Modeling for Bit Rate Constrained Point Cloud Compression

As being one of the main representation formats of 3D real world and well-suited for virtual reality and augmented reality applications, point clouds have gained a lot of popularity. In order to reduce the huge amount of data, a considerable amount of research on point cloud compression has been done. However, given a target bit rate, how to properly choose the color and geometry quantization parameters for compressing point clouds is still an open issue. In this paper, we propose a rate-distortion model based quantization parameter selection scheme for bit rate constrained point cloud compression. Firstly, to overcome the measurement uncertainty in evaluating the distortion of the point clouds, we propose a unified model to combine the geometry distortion and color distortion. In this model, we take into account the correlation between geometry and color variables of point clouds and derive a dimensionless quantity to represent the overall quality degradation. Then, we derive the relationships of overall distortion and bit rate with the quantization parameters. Finally, we formulate the bit rate constrained point cloud compression as a constrained minimization problem using the derived polynomial models and deduce the solution via an iterative numerical method. Experimental results show that the proposed algorithm can achieve optimal decoded point cloud quality at various target bit rates, and substantially outperform the video-rate-distortion model based point cloud compression scheme.

Rate Control in Versatile Video Coding with Cosh Rate–Distortion Model

In this paper, we propose a rate control (RC) scheme for versatile video coding (VVC) to enhance coding performance. First, we propose a rate–distortion (R–D) model at the coding tree unit (CTU) level to describe the R–D relationship. Second, we adjust the quantization parameter (QP) and Lagrange multiplier at the coding unit (CU) level according to visual features to improve the rationality of local coding results. Finally, we propose a model parameter updating strategy to guarantee bitrate accuracy. The experimental results demonstrate that our RC method has better R–D performance with similar bitrate accuracy and better visual quality compared to the default RC strategy in VVC reference software VTM 10.2.

Joint Decision Tree and Visual Feature Rate Control Optimization for VVC UHD Coding.

In this paper, a joint decision tree and visual feature optimization rate control scheme for ultrahigh-definition (UHD) versatile video coding (VVC) is proposed. First, we design a new rate-distortion (R-D) model for UHD videos, and we establish a decision-tree-based multiclass classification scheme to improve the prediction accuracy of the R-D model by fully considering visual features. Second, based on the proposed R-D model, the globally optimal solution is obtained through convex optimization. Finally, we embed our algorithm into the latest VVC reference software, VTM 10.2. According to our experimental results, compared with the latest algorithm in VTM 10.2 and other state-of-the-art algorithms, our method can achieve significant bit rate reductions while maintaining a given peak signal-to-noise ratio (PSNR) or structural similarity index measure (SSIM).

Bitstream-based Perceptual Quality Assessment of Compressed 3D Point Clouds.

With the increasing demand of compressing and streaming 3D point clouds under constrained bandwidth, it has become ever more important to accurately and efficiently determine the quality of compressed point clouds, so as to assess and optimize the quality-of-experience (QoE) of end users. Here we make one of the first attempts developing a bitstream-based no-reference (NR) model for perceptual quality assessment of point clouds without resorting to full decoding of the compressed data stream. Specifically, we first establish a relationship between texture complexity and the bitrate and texture quantization parameters based on an empirical rate-distortion model. We then construct a texture distortion assessment model upon texture complexity and quantization parameters. By combining this texture distortion model with a geometric distortion model derived from Trisoup geometry encoding parameters, we obtain an overall bitstream-based NR point cloud quality model named streamPCQ. Experimental results show that the proposed streamPCQ model demonstrates highly competitive performance when compared with existing classic full-reference (FR) and reduced-reference (RR) point cloud quality assessment methods with a fraction of computational cost.

Rate Control Technology for Next Generation Video Coding Overview and Future Perspective

Video data have become the main data traffic on the Internet, and their traffic is increasing explosively every year, thus increasing the pressure of video transmission. Video coding technology has become the key to compressing original videos. As an indispensable technology, rate control plays an important role in stabilizing video stream transmission. Rate control (RC) is part of rate distortion optimization (RDO) whose job is to find the optimal solution based on balancing rate and distortion. It not only needs to consider the buffer and network status but also adjust the corresponding bit rate according to the video content. This paper reviews the related technologies of rate control under high efficiency video coding (HEVC) and versatile video coding (VVC) standards so that subsequent researchers can quickly understand the field and promote the development of rate control algorithms. Firstly, the paper summarizes the various aspects of RC, including basic principles, rate-distortion models, major processes, and performance criteria. Secondly, the paper surveys, in detail, the research progress in the field of rate control and analyzes several mainstream research directions. Thirdly, we carry out relevant experiments on the standard reference software and analyze and discuss the experimental results of the existing studies. Finally, we look ahead to the future trends of rate control and provide feasible improvement suggestions.

Optimum Quality Control Algorithm for Versatile Video Coding

We propose a quality control method that replaces rate control in video coding and achieves quality control with the minimum bit rate. First, based on the rate–distortion model, we establish a Karush–Kuhn–Tucker equation considering the convergent properties of variational inequalities to solve the continuous convex problem. Second, combined with the rate–quality (R-Q) model, the proposed algorithm is implemented under versatile video coding (VVC) to obtain the optimal closed-form solution for the quantization parameters of a coding unit through the Lagrange multiplier method. Experiments reveal that our algorithm can achieve stable video coding quality while guaranteeing coding performance, achieving high quality control accuracy with constant quality.

Latitude-Based Flexible Complexity Allocation for 360-Degree Video Coding

The emerging 360-degree video has a good practical prospect in video broadcasting due to its more immersive experience than traditional videos. To maintain a high fidelity with panoramic vision, 360-degree video adopts a significantly increased resolution than planar videos, which also inevitably leads to higher encoding complexity. In this paper, we propose a flexible complexity optimization method for 360-degree video coding, in which the encoder is capable of adapting to different complexity constraints in diversified broadcasting scenarios. In particular, the proposed method is realized with a latitude-based computation complexity allocation. Firstly, it investigates the Coding Unit (CU) partition variation along the latitude (i.e., vertical axis) and divides each video frame into Largest CU (LCU)-based latitude regions. Then, it formulates the complexity allocation problem from the global complexity target to all latitude regions, by considering the intraframe dependencies and Rate-Distortion (RD) models. Finally, it calculates the solution to this complexity allocation problem and further employs it in 360-degree video coding. Comprehensive experiments on the standard dataset reveal the superiority of our method, which achieves a flexible computational time reduction from 14.53% to 68.36% whilst maintaining a high visual quality of compressed videos.

Content-based coding tree unit level rate-quantization model for intra-coding in high efficiency video coding standard using convolutional neural network

In almost all video applications, a video rate control algorithm (RCA) is used by the encoder. The RCA tunes the quantization parameter (QP) to match the encoded bit rate to the available capacity of the communication channel or storage media. Conventional RCAs usually utilize a rate-quantization (R-Q) or a rate-distortion (R-D) model for rate control. A content-based R-Q model for intra coding tree units (CTUs) of the high-efficiency video coding standard is proposed. The model is a convolutional neural network that observes pixels of a CTU and its intraprediction reference pixels and it estimates required bit counts for intracoding the CTU for all QP values simultaneously. The proposed model can be easily used by any video RCA. A given RCA just selects a proper QP for which the estimated bit counts are closer to the allocated bit budget. The evaluation results show a high accuracy for the model. According to simulation results, the mean absolute normalized bit error at CTU level is 19.66% and it decreases to 6.85% at the frame level. Compared with similar networks, the proposed structure has a very low computational complexity.

Proportional–integral–derivative-based intra rate controller for low-delay applications of high efficiency video coding screen content coding

Most screen content (SC) video applications are real time, and a low-delay transmission is needed. An effective low-delay transmission requires a precise video rate controller to fit the video bit rate to the capacity of the transmission channel. The rate controller has to support the unique standard coding tools of SC video. We propose a unique intra rate controller based on a unique content-based rate-distortion model for low delay applications of high efficiency video coding SC coding. The proposed model supports the unique SC coding tools and different modes and sizes of coding tree units. The proportional–integral–derivative control concept is used for frame-level bit allocation and rate control. The rate controller utilizes a small size virtual buffer to provide low-delay buffering constraints for the bit stream. Experimental results show that the proposed rate controller can control the bit rate of test sequences with a low buffering delay (about 20 ms) while the rate-distortion performance is maintained.

A content-based intra rate-distortion model for HEVC-SCC

A content-based intra rate-distortion model for HEVC-SCC

Structural similarity-based rate control algorithm for 3D video

Human vision easily identifies the structural degradation in a video and thus perceptual quality improvement is necessary. To adjust the video quality to match with the human vision we proposed a structural similarity (SSIM) based rate control algorithm for the 3D video. We incorporated a structural similarity index (SSIM) as the quality metric in rate control algorithm. The rate-distortion model and Lagrange multiplier are derived considering the structural dissimilarity (dSSIM) as the distortion metric to achieve the rate control with perceptual quality improvement. Furthermore, the optimal joint bit allocation scheme at texture video/depth map level, frame level, and basic unit (BU) level is modified to incorporate dSSIM for measuring distortion. The proposed algorithm is implemented in HEVC compression standard HTM-16.2. The performance of the proposed algorithm is compared using RD curves, BD-Rate comparison, and subjective evaluation. Besides, rate accuracy is also computed to measure the bit rate mismatch. Compared to the original λ-domain rate control algorithm, the proposed algorithm achieves a better SSIM along with a reduction in bit rate.

Reversible Data Hiding for JPEG Images Based on Multiple Two-Dimensional Histograms

Reversible data hiding (RDH) for JPEG images has attracted extensive attentions in recent years. However, since DCT is a de-correlation operation, it is difficult for JPEG images to employ the image redundancy for RDH design like uncompressed images. In this paper, considering better utilizing the special properties of quantized DCT coefficients, a new RDH scheme for JPEG images based on multiple two-dimensional (2D) histograms modification is proposed. Firstly, by combining every two nonzero alternating current (AC) coefficients of adjacent DCT blocks in each band as a pair, a new coefficients pairing strategy is proposed for a sharper 2D histogram. Then, with a classification process, multiple 1D and 2D histograms are generated, in which the 2D histograms are selected for data embedding and the bins in 1D histograms are shifted for reversibility. Finally, to further enhance the embedding performance, the 2D mappings are adaptively determined for different 2D histograms through a formulated rate-distortion model. Experimental results demonstrate the superiority of the proposed scheme both in visual quality and file size preservation.

Motion vector modification distortion analysis-based payload allocation for video steganography

Video steganography forms a covert communication channel by data embedding in cover elements. To consider inter-frame mutual embedding impacts, this paper proposes a payload allocation strategy in video steganography based on motion vector modification distortion analysis. Firstly, the motion vector modification distortion caused by data embedding is analyzed. Then, a rate–distortion model reflecting the residue deviation propagation in successive inter-coded frames is derived. According to this model, the residue deviation propagation weight of each inter-coded frame can be computed. Finally, an inter-frame payload allocation strategy is designed in order to restrain the residue deviation propagation. Experimental results demonstrate that the proposed payload allocation strategy can enhance existing motion vector-based video steganographic methods in terms of undetectability and video coding performance. Besides, the lower computational complexity can be achieved.

Scene-level two-pass video rate controller for H.265/HEVC standard

A scene-level two-pass video rate controller for high efficiency video coding (HEVC) standard is proposed in this paper. The proposed rate controller is suitable for many pre-recorded applications in which encoded videos are transferred later on through a communication channel with limited bandwidth and buffering constraint. This rate controller operates in two passes. In the first pass, the video sequence is compressed and relevant information is collected. In the second pass, an optimal base quantization parameter (QP) for each video scene is calculated by simplifying the λ-domain rate-distortion model. Then, using the optimal base QP and the known QP cascading technique, a QP is computed for each picture in the scene. In this way, the quality of the compressed video is kept relatively constant during a scene and thus it provides a high-level perceptual quality for the compressed video. The base QP may be slightly modified during the second pass encoding by a fuzzy controller according to the second pass encoding results and the buffer status. The proposed rate controller has been implemented on H.265/HEVC standard reference software. Experimental results show that in comparison with the constant QP encoding and three known rate control algorithms the proposed algorithm provides a high-level quality for tested video sequences while the required buffering delay is low and the available bandwidth is fully used.

Perceptually Weighted Mean Squared Error Based Rate-Distortion Optimization for HEVC

The process of rate-distortion (RD) optimization plays a key role for video coding, which aims to achieve a tradeoff between compression efficiency and video quality distortion. Although the conventional objective distortion metric mean squared error performs well in computational complexity, it is not always in accordance with the perceptual quality perceived by human visual system (HVS). Taking the characteristics of HVS into consideration, a perceptually weighted mean squared error (PWMSE) based RD model is proposed in this work. First, a low-pass filter is employed to process the original distortion information in order to simulate visual signal processing and obtain the perceptual distortion. Then, masking modulation in both temporal and spatial domains is introduced into distortion model, and a novel Lagrange multiplier is derived accordingly. The proposed PWMSE based RD model is applied to the high efficiency video coding standard, and comparative experimental results demonstrate its effectiveness. The project page can be found in https://mic.tongji.edu.cn/ab/40/c9778a174912/page.htm.