A neural network-based video bit-rate control algorithm for variable bit-rate applications of versatile video coding standard

Scalable High Efficiency Video Coding (SHVC) is the scalable extension of the latest video coding standard H.265/HEVC. Video rate control algorithm is out of the scope of video coding standards. Appropriate rate control algorithms are designed for various applications to overcome practical constraints such as bandwidth and buffering constraints. In most of the scalable video applications, such as video on demand (VoD) and broadcasting applications, encoded bitstreams with variable bit rates are preferred to bitstreams with constant bit rates. In variable bit rate (VBR) applications, the tolerable delay is relatively high. Therefore, we utilize a larger buffer to allow more variations in bitrate to provide smooth and high visual quality of output video. In this paper, we propose a fuzzy video rate controller appropriate for VBR applications of SHVC. A fuzzy controller is used for each layer of scalable video to minimize the fluctuation of QP at the frame level while the buffering constraint is obeyed for any number of layers received by a decoder. The proposed rate controller utilizes the well-known structural similarity index (SSIM) as a quality metric to increase the visual quality of the output video. The proposed rate control algorithm is implemented in HEVC reference software and comprehensive experiments are executed to tune the fuzzy controllers and also to evaluate the performance of the algorithm. Experimental results show a high performance for the proposed algorithm in terms of rate control, visual quality, and rate-distortion performance.

A novel semi-fuzzy (SF) rate control algorithm (RCA) for variable bit rate (VBR) video applications is proposed. The proposed RCA is optimized to provide high quality compressed video bit streams in a wide operating range from constant quality to nearly constant bit rate. Thanks to a low degree of computational complexity, it is suitable for real-time applications of VBR video. The proposed RCA operates under given buffer size, delay and quality constraints. It provides a VBR video bit stream by controlling the quantization parameter (QP) on a picture basis. The QP is mainly controlled by a fuzzy rate controller and a deterministic quality controller, which are optimized such that they minimize the variation of quality to provide encoded video with high and stable visual quality. The proposed RCA has been implemented in an H.264/AVC video codec and the experimental results show that it provides a high-level average quality for encoded video while strictly obeying the buffering delay and quality constraints.

Two-level sliding-window VBR control algorithm for video on demand streaming

Rate Control plays an important role in video coding, which performs successful transmission of all encoded bits with available limited bandwidth. This paper aims to provide an adaptive rate controller for H.264 scalable video coding. The Quantization Parameter (QP) plays an important role in any rate controller for encoding demanded bits. The Quantization parameter estimation and the rate controller algorithm follows two pass 1) Initial Quantization Parameter value estimated by simplified Rate Distortion model using Cauchy Density based PDF and Buffer status 2) Rate Distortion Optimization method to determine minimum cost for the mode and motion vector using the estimated Quantization Parameter. The experimental result shows that our rate control algorithm is better than the FixedQPencoder of JSVM 9.19.15 in terms of PSNR and bit rate. Moreover our scheme performs in single iteration to encode the frame with the desired QP and Buffer status calculated to prevent from overflow and underflow.

In this work, we propose a neural network based rate control algorithm for Versatile Video Coding (VVC). The proposed method relies on the modeling of the Rate-Quantization (R-Q) and Distortion-Quantization (D-Q) relationships in a data driven manner based upon the characteristics of prediction residuals. In particular, a pre-analysis framework is adopted, in an effort to obtain the prediction residuals which govern the Rate-Distortion (R-D) behaviors. By inferring from the prediction residuals with deep neural networks, the Coding Tree Unit (CTU) level R-Q and D-Q model parameters are derived, which could efficiently guide the optimal bit allocation. Subsequently, the coding parameters, including Quantization Parameter (QP) and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> , at both frame and CTU levels, are obtained according to allocated bit-rates. We implement the proposed rate control algorithm on VVC Test Model (VTM-13.0). Experimental results exhibit that the proposed rate control algorithm achieves 0.77% BD-Rate savings under Low Delay B (LDB) configurations when compared to the default rate control algorithm used in VTM-13.0. For Random Access (RA) configurations, 1.77% BD-Rate savings can be observed. Furthermore, with better bit-rate estimation, more stable buffer status can be observed, further demonstrating the advantages of the proposed rate control method.

In the last 17 years, since the finalization of the first version of the now-dominant H.264/Moving Picture Experts Group-4 (MPEG-4) Advanced Video Coding (AVC) standard in 2003, two major new generations of video coding standards have been developed. These include the standards known as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). HEVC was finalized in 2013, repeating the ten-year cycle time set by its predecessor and providing about 50% bit-rate reduction over AVC. The cycle was shortened by three years for the VVC project, which was finalized in July 2020, yet again achieving about a 50% bit-rate reduction over its predecessor (HEVC). This article summarizes these developments in video coding standardization after AVC. It especially focuses on providing an overview of the first version of VVC, including comparisons against HEVC. Besides further advances in hybrid video compression, as in previous development cycles, the broad versatility of the application domain that is highlighted in the title of VVC is explained. Included in VVC is the support for a wide range of applications beyond the typical standard- and high-definition camera-captured content codings, including features to support computer-generated/screen content, high dynamic range content, multilayer and multiview coding, and support for immersive media such as 360° video.

MPEG-2, the international standard of the ISO/IEC, is a key technology for video coding and is widely used in digital television etc. Its most frequently used coding rate control is the constant bit rate (CBR) mode. CBR's bandwidth requirement is relatively low, but it needs further improvement in terms of the coding efficiency. On the other hand, the variable bit rate (VBR) mode may be able to enhance the picture quality in situations that are rich in bandwidth or in storage media such as DVD. We have already developed a one-chip MPEG-2 video encoder LSI (SuperENC) to cope with the broader range of digital image communication applications. This paper discusses a new one-pass VBR coding control algorithm based on mathematical relations among coding parameters such as quantization parameter, bit rate and decoded image distortion. Our aim is to solve the problems of conventional techniques and verify the algorithm's effectiveness through hardware implementation. A one-pass VBR control function was implemented in the chip's firmware. Not being a very complex algorithm, it is robust with respect to abrupt scene changes. The SNR gain in a TM-5-based CBR mode was 1.5 to 1.8 dB, and the bit rate reduction under the same subjective quality was 10-30 %.

Rate control plays a very important role in constant bit rate (CBR) coding. AVC standard is jointly developed by ISO and ITU-T, which contains several inter and intra prediction modes. Rate distortion optimization (RDO) based on prerequisite quantization parameters determines the optimal prediction of each macroblock. This makes the current AVC software difficult to adopt the existing rate control techniques. This paper proposes an efficient rate control algorithm at macroblock level for AVC standard by considering both rate control and optimal prediction selection. Firstly, a quantization parameter estimated from neighboring macroblock is used in selecting an initial prediction and calculating the activity. Secondly, the estimated quantization parameter is refined according to the activity and virtual buffer occupancy. At last, the prediction mode is determined with the refined quantization parameter. Experimental results show that the proposed rate control algorithm can accurately achieve the target bit rate. Furthermore, the coding efficiency is similar to or even better than that of variable bit rate (VBR) coding.

As the upcoming video coding standard, Versatile Video Coding (i.e., VVC) achieves up to 30% Bjøntegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (H.265/HEVC). To eliminate or alleviate different kinds of compression artifacts like blocking, ringing, blurring and contouring effects, three in-loop filters, i.e. de-blocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF), have been involved in VVC. Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in many tasks in image processing. In this work, we design a CNN-based in-loop filter as an integrated single-model solution which is adaptive to almost any scenarios in video coding. An architecture named as ADCNN (i.e., Attention based Dual-scale CNN) with an attention based processing block is proposed to reduce artifacts of I frames and B frames, which take advantage of informative priors such as the quantization parameter (QP) and partitioning information. Different from existing CNN-based filtering methods, which are mainly designed for the luma component and may need to train different models for different QPs, the proposed filter is adapted to different QPs and different frame types, and all the components (i.e., both luma and chroma) are processed simultaneously with feature exchange and fusion between components for information supplementary. Experimental results show that the proposed ADCNN filter can achieve 6.54%, 13.27%, 15.72% BD-rate savings for Y, U, V respectively under the all intra configuration and 2.81%, 7.86%, 8.60% BD-rate savings under the random access configuration. It can be used to replace all the conventional in-loop filters and also outperforms them without increase in encoding time.

Variable bit rate encoded video bit streams are suitable for a wide range of high delay applications such as video streaming applications. In these applications, the visual quality of encoded video and the buffering constraint are concerned during encoding at the same time. In this paper, a fuzzy video rate controller for variable bit rate applications of the new high efficiency video coding (HEVC) standard is proposed. The proposed controller considers a given buffer size and a long-term target bit rate as controlling constraints. It provides a variable bit rate video bit stream by controlling the quantization parameter (QP) at frame-level. The bitrate of each frame is controlled by a fuzzy controller to minimize the fluctuations of QP and PSNR that leads to high visual quality. The proposed algorithm is implemented on the HEVC standard reference software (HM) and experimental results show that it can provide an average quality, in terms of PSNR, similar to the HM rate controller and constant QP case while the buffer constraint is completely obeyed. Also in comparison with the HM rate controller, the fluctuation of QP and PSNR are less in the proposed algorithm that means a higher visual quality.

In almost all video applications, a video rate control algorithm (RCA) is used by the encoder. The RCA tunes the quantization parameter (QP) to match the encoded bit rate to the available capacity of the communication channel or storage media. Conventional RCAs usually utilize a rate-quantization (R-Q) or a rate-distortion (R-D) model for rate control. A content-based R-Q model for intra coding tree units (CTUs) of the high-efficiency video coding standard is proposed. The model is a convolutional neural network that observes pixels of a CTU and its intraprediction reference pixels and it estimates required bit counts for intracoding the CTU for all QP values simultaneously. The proposed model can be easily used by any video RCA. A given RCA just selects a proper QP for which the estimated bit counts are closer to the allocated bit budget. The evaluation results show a high accuracy for the model. According to simulation results, the mean absolute normalized bit error at CTU level is 19.66% and it decreases to 6.85% at the frame level. Compared with similar networks, the proposed structure has a very low computational complexity.

The high -efficiency video coding (HEVC) standard was introduced for high-resolution video contents suitable for many high-delay applications. The bit rate of compressed video is controlled in almost all digital video applications according to the practical constraints such as available channel bandwidth and allowed delay or buffering constraint. In high-delay video applications such as video broadcasting and video streaming applications, variable-bit-rate videos can provide a higher visual quality than constant-bit-rate videos. In this paper, a rate control algorithm (RCA) for high-delay applications of the HEVC standard with buffering constraint is proposed. A fuzzy controller and a virtual buffer are used in the proposed RCA. The fuzzy controller is designed to minimize the fluctuations of quantization parameter (QP) while the buffering constraint is obeyed. It computes a base QP for each group of pictures (GOP) to prevent unnecessary fluctuations of QP at GOP level and thereafter provide a higher visual quality. Experimental results show that not only the bit rate and but also buffering constraints are fully maintained but also the objective quality of compressed video is well preserved. Moreover, the proposed RCA provides smooth QP and peak signal-to-noise ratio close to constant QP case for encoded videos that means high subjective quality.

Versatile Video Coding (H.266/VVC) standard achieves up to 30% bit-rate reduction while keeping the same quality compared with H.265/HEVC. To eliminate various coding artifacts like blocking, blurring, ringing, and contouring effects, etc., three in-loop filters have been incorporated in H.266/VVC. Recently, convolutional neural network (CNN) has attracted tremendous attention and achieved great success in many image processing tasks. In this paper, we focus on CNN-based filtering in video coding, where a single model solution for post-loop filtering is designed to replace the current in-loop filters. An architecture is proposed to reduce the artifacts of video intra frames, which take advantage of useful information such as partitioning modes and quantization parameters (QP). Different from existing CNN-based approaches, which generally need to train different models for different QP and only suitable for luma component, the proposed filter can well adapt to different QP, i.e. various levels of degradation of frames, and all components (i.e., luma and chroma) are jointly processed. Experiment results show that the proposed CNN post-loop filter not only can replace the de-blocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF) in H.266/VVC, and also outperforms them, leading to 6.46%, 10.40%, 12.79% BD-rate savings for Y, Cb and Cr, respectively, under all intra configuration.

The rapid development of virtual reality applications continues to urge better compression of 360° videos owing to the large volume of content. These videos are typically converted to 2-D formats using various projection techniques in order to benefit from ad-hoc coding tools designed to support conventional 2-D video compression. Although recently emerged video coding standard, Versatile Video Coding (VVC) introduces 360° video specific coding tools, it fails to prioritize the user observed regions in 360° videos, represented by the rectilinear images called the viewports. This leads to the encoding of redundant regions in the video frames, escalating the bit rate cost of the videos. In response to this issue, this paper proposes a novel 360° video coding framework for VVC which exploits user observed viewport information to alleviate pixel redundancy in 360° videos. In this regard, bidirectional optical flow, Gaussian filter and Spherical Convolutional Neural Networks (Spherical CNN) are deployed to extract perceptual features and predict user observed viewports. By appropriately fusing the predicted viewports on the 2-D projected 360° video frames, a novel Regions of Interest (ROI) aware weightmap is developed which can be used to mask the source video and introduce adaptive changes to the Lagrange and quantization parameters in VVC. Comprehensive experiments conducted in the context of VVC Test Model (VTM) 7.0 show that the proposed framework can improve bitrate reduction, achieving an average bitrate saving of 5.85% and up to 17.15% at the same perceptual quality which is measured using Viewport Peak Signal-To-Noise Ratio (VPSNR).

A fuzzy bit allocation method, applicable to many rate control algorithms targeted for variable rate video applications, is proposed in this paper. In low-delay video communication applications, a constant short-term average bit rate is required. However, in variable bit rate applications, such as streaming and broadcast applications, a constant long-term average bit rate is sufficient and a higher short-term variation in bit rate is acceptable. A variable bit rate video can provide better visual quality and coding efficiency in comparison with a constant bit rate video. The proposed fuzzy bit allocation algorithm is employed for encoding special P frames. To take advantage of variable bit rate video, the proposed method is implemented independently of the rate control algorithm. Two fuzzy systems are used to determine the location and the quantization parameter of special P frames. The proposed method is implemented on a JM H.264/AVC video encoder. Experimental results show that it can improve the quality of compressed video while having a low degree of computational complexity.