Rate-distortion Optimization Research Articles

Joint Rate-Distortion Optimization for Video Coding and Learning-Based In-Loop Filtering

Joint Rate-Distortion Optimization for Video Coding and Learning-Based In-Loop Filtering

Research on the Innovation and Implementation of Interactive Teaching Content of Civic and Political Education in Colleges and Universities under the Digital Transformation of the Academic Style Building

Abstract The construction of ideological and political education and the construction of academic style are complementary to each other and jointly committed to the good development of education and teaching in colleges and universities. This paper starts from the relationship between the two, discusses the interaction between ideological and political education and the construction of academic style in colleges and universities, and draws out the methodological path of ideological and political education in colleges and universities to lead the construction of academic style. It proposes interactive teaching methods to strengthen the construction of ideological and political education by utilizing the interactive teaching process between teachers and students. It points out that the digital video coding technology (H.265/HEVC intra-frame coding technology and H.265/HEVC rate-distortion optimization technology) and the video encryption mechanism based on H.265/HEVC optimize the processing of interactive video to enhance the interactive performance of ideological and political academic style. By comparing the coding performance of H.264 and H.265, as well as the PNSR and SSIM evaluation indices, the rationality of the proposed digital video coding and encryption technology is verified. A survey on the academic style construction of different majors and grades, as well as an analysis of the attitude of Civics academic style construction, is conducted in the sample schools to find out what can be optimized in the academic style construction of Civics education in colleges and universities. In all five schools, 4.76% of students believe that the academic culture situation is not conducive to the current learning culture. However, 58.96% and 14.46% of the groups think that learning is something that has to be done and something that is not interesting, indicating that there is still a lack of interest in the teaching interaction of Civics and Political Science style, and schools should make full use of digital technology to attract students to carry out teaching interaction.

Learning Adaptive Quantization Parameter for Consistent Quality Oriented Video Coding

In the industry 4.0 era, video applications such as surveillance visual systems, video conferencing, or video broadcasting have been playing a vital role. In these applications, for manipulating and tracking objects in decoded video, the quality of decoded video should be consistent because it largely affects the performance of the machine analysis. To cope with this problem, we propose a novel perceptual video coding (PVC) solution in which a full reference quality metric named video multimethod assessment fusion (VMAF) is employed together with a deep convolutional neural network (CNN) to obtain consistent quality while still achieving high compression performance. First of all, in order to achieve the consistent quality requirement, we propose a CNN model with an expected VMAF as input to adaptively adjust the quantization parameters (QP) for each coding block. Afterwards, to increase the compression performance, a Lagrange coefficient of rate-distortion optimization (RDO) mechanism is adaptively computed according to rate-QP and quality-QP models. The experimental results show that the proposed PVC solution has achieved two targets simultaneously: the quality of video sequence is kept consistently with an expected quality level and the bit rate saving of the proposed method is higher than traditional video coding standards and the relevant benchmark, notably with around 10% bitrate saving on average.

Paired decision trees for fast intra decision in H.266/VVC

The Versatile Video Coding (H.266/VVC) has demonstrated significant improvements to its predecessor High-Efficiency Video Coding (H.265/HEVC) in terms of compression efficiency. One of its advancements is attributed to the introduction of the QuadTree with nested Multi-type Tree (QTMT). However, the accompanying computing complexity challenges practical applications. To this end, this paper proposes a fast encoding algorithm to accelerate the encoding process of H.266/VVC while maintaining its compression efficiency as much as possible. Essentially, the Coding Unit (CU) partition can be modeled as a classification problem, and thus conventional machine learning methods such as the decision tree can be used to solve it. Previous decision tree-based works generally use one decision tree to dyadically select one partition from a set of candidate partition modes, easily leading to a local optimum. In contrast, this work models the classification from a new viewpoint and devises Paired Decision Trees (PDT) to tackle it. Specifically, the first decision tree determines whether to divide the current CU or not, and the second decision tree further determines whether to divide the CU horizontally or vertically. However, inaccurate results in the first decision tree can be propagated to the second one, resulting in error accumulation. To address this issue, we introduce the Indeterminate Space to both decision trees and perform Rate-Distortion Optimization (RDO) on samples falling within this space. Experimental results show that compared with H.266/VVC reference software VTM-15.0, the proposed PDT method saves 52.86% encoding time on average while maintaining a small BDBR increase of only 1.36%, which significantly outperforms previous works.

Deep Multi-Task Learning Based Fast Intra-Mode Decision for Versatile Video Coding

The latest Versatile Video Coding (VVC) standard has significantly coding efficiency improvement compared with its ancestor High Efficiency Video Coding (HEVC) standard, but at the expense of over-high complexity. As measured by the VVC test model (VTM), the intra-mode comparison and selection in the rate-distortion optimization (RDO) search consume most of the encoding time. In this paper, we propose a deep multi-task learning based fast intra-mode decision approach via adaptively pruning off most redundant modes. First, we create a large-scale intra-mode database for VVC, including both normal angular modes and the newly introduced tools, i.e., intra sub-partition (ISP) and matrix-based intra prediction (MIP). Next, we propose a multi-task intra-mode decision network (MID-Net) model to effectively predict the most probable angular modes and whether to skip ISP and MIP modes. Then, a fast intra-coding workflow is designed accordingly, involving rough mode decision (RMD) acceleration and candidate mode list (CML) pruning. For the workflow output, the learning-oriented probability and the statistics-oriented probability are synthesized together to further improve the prediction accuracy, ensuring that only unnecessary intra-modes are skipped. Finally, experimental results show that our approach can significantly reduce 40.48% of encoding time of VVC intra-coding with negligible rate-distortion degradation, outperforming other state-of-the-art approaches.

Fast CU Partition Algorithm for Intra Frame Coding Based on Joint Texture Classification and CNN.

High-efficiency video coding (HEVC/H.265) is one of the most widely used video coding standards. HEVC introduces a quad-tree coding unit (CU) partition structure to improve video compression efficiency. The determination of the optimal CU partition is achieved through the brute-force search rate-distortion optimization method, which may result in high encoding complexity and hardware implementation challenges. To address this problem, this paper proposes a method that combines convolutional neural networks (CNN) with joint texture recognition to reduce encoding complexity. First, a classification decision method based on the global and local texture features of the CU is proposed, efficiently dividing the CU into smooth and complex texture regions. Second, for the CUs in smooth texture regions, the partition is determined by terminating early. For the CUs in complex texture regions, a proposed CNN is used for predictive partitioning, thus avoiding the traditional recursive approach. Finally, combined with texture classification, the proposed CNN achieves a good balance between the coding complexity and the coding performance. The experimental results demonstrate that the proposed algorithm reduces computational complexity by 61.23%, while only increasing BD-BR by 1.86% and decreasing BD-PSNR by just 0.09 dB.

Reducing Video Coding Complexity Based on CNN-CBAM in HEVC

High-efficiency video coding (HEVC) outperforms H.264 in coding efficiency. However, the rate–distortion optimization (RDO) process in coding tree unit (CTU) partitioning requires an exhaustive exploration of all possible quad-tree partitions, resulting in high encoding complexity. To simplify this process, this paper proposed a convolution neural network (CNN) based optimization algorithm combined with a hybrid attention mechanism module. Firstly, we designed a CNN compatible with the current coding unit (CU) size to accurately predict the CU partitions. In addition, we also designed a convolution block to enhance the information interaction between CU blocks. Then, we introduced the convolution block attention module (CBAM) into CNN, called CNN-CBAM. This module concentrates on important regions in the image and attends to the target object correctly. Finally, we integrated the CNN-CBAM into the HEVC coding framework for CU partition prediction in advance. The proposed network was trained, validated, and tested using a large scale dataset covering various scenes and objects, which provides extensive samples for intra-frame CU partition prediction in HEVC. The experimental findings demonstrate that our scheme can reduce the coding time by up to 64.05% on average compared to a traditional HM16.5 encoder, with only 0.09 dB degradation in BD-PSNR and a 1.94% increase in BD-BR.

FDD Massive MIMO Channel Training: Optimal Rate-Distortion Bounds and the Spectral Efficiency of “One-Shot” Schemes

We study the problem of providing channel state information (CSI) at the transmitter in multi-user “massive” MIMO systems operating in frequency division duplexing (FDD). The wideband MIMO channel is a vector-valued random process correlated in time, space (antennas), and frequency (subcarriers). The base station (BS) broadcasts periodically β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tr</sub> pilot symbols from its <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</i> antenna ports to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> single-antenna users (UEs). Correspondingly, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> UEs send feedback messages about their channel state using β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fb</sub> symbols in the uplink (UL). Using results from remote rate-distortion theory, we show that, as snr → ∞, the optimal feedback strategy achieves a channel state estimation mean squared error (MSE) that behaves as Θ(1) if β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tr</sub> < <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> and as Θ(snr <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">–α</sup> ) when β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tr</sub> ⩾ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> , where α = min(β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fb</sub> / <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> , 1), where <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> is the rank of the channel covariance matrix. The MSE-optimal rate-distortion strategy implies encoding of long sequences of channel states, which would yield completely stale CSI and therefore poor multiuser precoding performance. Hence, we consider three practical “one-shot” CSI strategies with minimum one-slot delay and analyze their large-SNR channel estimation MSE behavior. These are: (1) digital feedback via entropy-coded scalar quantization (ECSQ), (2) analog feedback (AF), and (3) local channel estimation at the UEs via compressed sensing and digital feedback. These schemes have different requirements in terms of knowledge of the channel statistics at the UE and at the BS. In particular, the latter strategy requires no statistical knowledge and is closely inspired by a CSI feedback scheme currently proposed in 3GPP standardization. It is shown that ECSQ achieves optimal MSE at the price of a slight increase in feedback rate which vanishes for large SNR. AF achieves the optimal MSE decay rate of Θ(snr <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">–1</sup> ) whenever β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tr</sub> , β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fb</sub> ⩾ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> but is sub-optimal if β⩾ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> and β <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fb</sub> < <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</i> . The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3GPP-inspired</i> scheme is shown, via numerical simulations, to achieves performance similar to ECSQ and AF when the multipath channel is sufficiently sparse in the angle-delay domain, but suffers from a large performance gap if this requirement is not met.

Compressing of Medium- to Low-Rate Transform Residuals With Semi-Extreme Sparse Coding as an Alternate Transform in Video Coding

Modern hybrid video codecs split their video content into blocks. These blocks are predicted and the difference between original and predicted block is calculated. This residual block is transformed by a discrete cosine transform (DCT) from the pixel domain into the frequency domain and quantized by a dead-zone quantizer (DZQ), which removes high frequency signals, depending on the quantization parameter (QP). DZQ has an extended zone around zero, which acts as a noise-gate, removing noise as intended but also removing useful signals. DCT and DZQ are not the optimal solution, they create distortions and side effects at lower bitrates. In contrast, sparse coding does not have these problems at lower bitrates, but it compresses inefficient at higher bit rates. We propose a method to add sparse coding as an alternate transform, which is controlled by a rate-distortion optimization (RDO) decision, in a semi-extreme sparse coding (SESC) setup and uses a semi-extreme dictionary training (SEDT) process. It is integrated into the High Efficiency Video Coding (HEVC) test model HM-16.18 and screen content coding (HEVCSCC) test model HM-16.18+SCM-8.7. Experimental results demonstrate that the proposed method achieves a Bjontegaard rate difference (BD-rate) of up to 5.5% compared to the standard.

A Task-Driven Scene-Aware LiDAR Point Cloud Coding Framework for Autonomous Vehicles

LiDAR sensors are almost indispensable for autonomous robots to perceive the surrounding environment. However, the transmission of large-scale LiDAR point clouds is highly bandwidth-intensive, which can easily lead to transmission problems, especially for unstable communication networks. Meanwhile, existing LiDAR data compression is mainly based on rate-distortion optimization, which ignores the semantic information of ordered point clouds and the task requirements of autonomous robots. To address these challenges, this article presents a task-driven <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</u> cene- <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</u> ware <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</u> iDAR <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P</u> oint <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> louds <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> oding (SA-LPCC) framework for autonomous vehicles. Specifically, a semantic segmentation model is developed based on multi-dimension information, in which both 2D texture and 3D topology information are fully utilized to segment movable objects. Further, a prediction-based deep network is explored to remove the spatial-temporal redundancy. The experimental results on the benchmark semantic KITTI dataset validate that our SA-LPCC achieves state-of-the-art performance in terms of the reconstruction quality and storage space for downstream tasks. We believe that SA-LPCC jointly considers the scene-aware characteristics of movable objects and removes the spatial-temporal redundancy from an end-to-end learning mechanism, which will boost the related applications from algorithm optimization to industrial products.

The disparity between optimal and practical Lagrangian multiplier estimation in video encoders

With video streaming making up 80% of the global internet bandwidth, the need to deliver high-quality video at low bitrate, combined with the high complexity of modern codecs, has led to the idea of a per-clip optimisation approach in transcoding. In this paper, we revisit the Lagrangian multiplier parameter, which is at the core of rate-distortion optimisation. Currently, video encoders use prediction models to set this parameter but these models are agnostic to the video at hand. We explore the gains that could be achieved using aper-clipdirect-search optimisation of the Lagrangian multiplier parameter. We evaluate this optimisation framework on a much larger corpus of videos than that has been attempted by previous research. Our results show that per-clip optimisation of the Lagrangian multiplier leads to BD-Rate average improvements of 1.87% for x265 across a 10 k clip corpus of modern videos, and up to 25% in a single clip. Average improvements of 0.69% are reported for libaom-av1 on a subset of 100 clips. However, we show that a per-clip, per-frame-type optimisation ofλfor libaom-av1 can increase these average gains to 2.5% and up to 14.9% on a single clip. Our optimisation scheme requires about 50–250 additional encodes per-clip but we show that significant speed-up can be made using proxy videos in the optimisation. These computational gains (of up to ×200) incur a slight loss to BD-Rate improvement because the optimisation is conducted at lower resolutions. Overall, this paper highlights the value of re-examining the estimation of the Lagrangian multiplier in modern codecs as there are significant gains still available without changing the tools used in the standards.

Rate distortion optimization with adaptive content modeling for random-access versatile video coding

In this paper, we study the capability of improving the rate-distortion (RD) performance based on the adaptive content modeling in the Versatile Video Coding (VVC) standard. In particular, the frame-level dependent relationships and inherent RD relationship are explored. As such, the rate dependency, distortion dependency and inherent RD characteristics are fully utilized in the global rate distortion optimization (RDO) process, and the quantization parameter (QP) for each frame could be adaptively solved. To facilitate the adaptive QP calculation, a two-pass coding strategy is proposed. In the first-pass coding, the video statistics are sufficiently collected with the proposed Dual Motion Compensation and Residual Coding (DMCRC) method to generate the parameters for the content-aware models. During the second-pass coding, the optimal QP at the frame level is obtained by optimizing the global RD performance with the dependent and inherent models. The proposed algorithm is implemented on VVC test model (VTM-4.0) and achieves significant performance gain for test sequences with constant and varying scenes.

Learned Progressive Image Compression With Dead-Zone Quantizers

Progressive coding is essential to the practical deployment of learned image compression over heterogeneous networks and clients. Existing methods for learned progressive image compression require complex and empirical design to achieve near-optimal rate-distortion performance over a wide range of bit-rates. However, these methods are limited by the implicit learned mechanism based on neural networks and introduction of uniform quantizers. In this paper, we propose generalized learned progressive image compression with analytic rate-distortion optimization using dead-zone quantizers on the latent representation. Specifically, we reveal that dead-zone quantizers, as a general case of uniform quantizers, are equivalent to uniform quantizers in fixed-rate nonlinear transform coding and can prevent extra redundancy in embedded quantization for progressive coding. Consequently, we propose rate-distortion optimized learned progressive coding by approximating the optimal quantizer in the source spaces using dead-zone quantizers in an analytic manner on the Laplacian source. To our best knowledge, this paper is the first to achieve general learned progressive coding from the perspective of optimal quantizers. The proposed method achieves theoretically sound and practically efficient embedded quantization and learned progressive coding of latent representations with improved rate-distortion performance. It can also enable embedded quantization with diverse assignments of truncation points and support flexible configuration of quality layers of varying numbers and at varying target bit-rates. Furthermore, we successfully incorporate the proposed method into existing pre-trained fixed-rate models to realize progressive learned image compression without re-training. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performance in learned progressive image compression compared with traditional codecs and recent learned methods.

Rate-Distortion Optimization-Based Learned Fractional Interpolation Filter Design for HEVC

Fractional motion compensation (FMC) has been widely adopted in modern video coding standards such as high-efficiency video coding (HEVC). The fractional pixels in FMC are generated by the interpolation filters, and the coefficients of the filters are designed using a Fourier decomposition of the discrete cosine transform (DCT) in HEVC. To overcome the compression deficiency caused by these handcrafted filters, deep learning-based interpolation filter approaches have been introduced recently. In this paper, we analyze the drawbacks of the mean square error (MSE) only loss function which is widely adopted in existing learned-based approaches, and introduce the rate-distortion optimization (RDO) theory into network training to formulate a joint loss function to overcome this issue. Then, a new training scheme is proposed for the detailed implementation of the joint loss function. Besides the calculated MSE loss, the residual will be sent into a rate estimation model which includes three stages: transform/quantization/rate modeling, and the output rate cost will be used as the second part of the loss function. Finally, we propose a novel network structure that utilizes the deformable convolution layer and the residual dense block to enhance the feature extraction and transmission ability to adapt to complex video contents. The proposed RDO-based interpolation filter (RDOIF) has been integrated into HEVC reference software HM-16.7, and extensive experiments are conducted to verify the efficiency of the proposed method. Experimental results show that the proposed RDOIF method achieves 3.1%, 2.4%, and 1.4% BD-rate reduction on average under the low-delay-P, low-delay-B, and random access configurations, respectively. When implemented in the versatile video coding (VVC) test model VTM-10.0, the proposed method can achieve 2.3%, 1.2%, and 0.5% BD-rate savings on average under the low-delay-P, low-delay-B, and random access configurations, respectively.

Overview of Versatile Video Coding (H.266/VVC) and Its Coding Performance Analysis

Versatile Video Coding (H.266/VVC) is the newest video coding standard jointly developed by the Joint Video Experts Team (JVET), which is organized by the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). H.266/VVC provides about 40% bitrate reduction compared with High Efficiency Video Coding (H.265/HEVC) for the same visual quality. This paper introduces in detail the core structure of H.266/VVC by highlighting its features within block partitioning structure, intra/inter prediction, transform, quantization, and in-loop filtering, compared to its predecessor (H.265/HEVC). H.266/VVC yields significantly improved the coding performance, but it increased the computational complexity, particularly for the encoder side, which remains a problem to be tackled for successful commercialization in the future. This paper examined the statistical performance of H.266/VVC coding tools from the bitstreams encoded by the VVC test model (VTM12.0) encoder through rate-distortion optimization under the JVET common test conditions. In addition, the complexity and performance analyses are conducted on the block partitioning structure and group of picture structure. It is expected that an optimized H.266/VVC encoder can be designed and developed by minimizing the coding loss based on the analysis data.

Pre-encoding based temporal dependent rate–distortion optimization for HEVC

Rate–distortion optimization (RDO) is a crucial technique in block-based hybrid video encoders, which determines the coding option for a certain coding unit to achieve optimal rate–distortion (R–D) performance. However, the current RDO method, which is applied in High Efficiency Video Coding (HEVC), is far from being globally optimal as it ignores the interaction among coding options. In fact, the current coding option not only determines the R–D point of the coded unit but also influences the achievable R–D​ performance of the subsequent coding units due to the spatial–temporal dependencies among the block/frame optimizations. In this paper, we first investigate the R–D characteristics, including operational R–D curves and R–D dependencies, in HEVC through several simulation experiments. It is shown that under the hierarchical fixed QP setting in the HEVC encoder, the temporal R–D dependence dominates the coding process, and the distortion of the reference frame has a strong impact on that of coded frames while less influence on rate. We then introduce distortion effect factors to quantitatively measure the temporal R–D​ dependency. The distortion effect factor at frame level reflects the correlation degree between the current coding frame and the previous coded frames, and the one at block level reflects the influence degree that the distortion of a block to that of subsequent blocks which refer to it in temporal domain. Finally, a pre-encoding based temporal dependent RDO is proposed, which adaptively adjusts quantization parameters and Lagrange multipliers according to the distortion effect factors. Experimental results show that under configurations of low-delay B and P frames, the proposed method can achieve 5.4% and 5.5% bit rate savings on average, respectively, with around 7% encoding complexity increment, compared with the original HEVC encoder.

Machine Learning Based Fast QTMTT Partitioning Strategy for VVenC Encoder in Intra Coding

The newest video compression standard, Versatile Video Coding (VVC), was finalized in July 2020 by the Joint Video Experts Team (JVET). Its main goal is to reduce the bitrate by 50% over its predecessor video coding standard, the High Efficiency Video Coding (HEVC). Due to the new advanced tools and features included in VVC, it actually provides high coding performances—for instance, the Quad Tree with nested Multi-Type Tree (QTMTT) involved in the partitioning block. Furthermore, VVC introduces various techniques that allow for superior performance compared to HEVC, but with an increase in the computational complexity. To tackle this complexity, a fast Coding Unit partition algorithm based on machine learning for the intra configuration in VVC is proposed in this work. The proposed algorithm is formed by five binary Light Gradient Boosting Machine (LightGBM) classifiers, which can directly predict the most probable split mode for each coding unit without passing through the exhaustive process known as Rate Distortion Optimization (RDO). These LightGBM classifiers were offline trained on a large dataset; then, they were embedded on the optimized implementation of VVC known as VVenC. The results of our experiment show that our proposed approach has good trade-offs in terms of time-saving and coding efficiency. Depending on the preset chosen, our approach achieves an average time savings of 30.21% to 82.46% compared to the VVenC encoder anchor, and a Bjøntegaard Delta Bitrate (BDBR) increase of 0.67% to 3.01%, respectively.

A Survey on Perceptually Optimized Video Coding

To provide users with more realistic visual experiences, videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG), and high clarity. However, the data amount of videos increases exponentially, which requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to maximize compression efficiency by exploiting visual redundancies. In this article, we present a broad and systematic survey on perceptually optimized video coding. Firstly, we present problem formulation and framework of the perceptually optimized video coding, which includes visual perception modeling, visual quality assessment, and perceptual video coding optimization. Secondly, recent advances on visual factors, computational perceptual models, and quality assessment models are presented. Thirdly, we review perceptual video coding optimizations from four key aspects, including perceptually optimized bit allocation, rate-distortion optimization, transform and quantization, and filtering and enhancement. In each part, problem formulation, working flow, recent advances, advantages, and challenges are presented. Fourthly, perceptual coding performances of the latest coding standards and tools are experimentally analyzed. Finally, challenging issues and future opportunities are identified.

Fast Rate-Distortion Optimization for Depth Maps in 3-D Video Coding

To ensure the fidelity of virtual views, rate-distortion optimization (RDO) criterion for the 3D extension of the High Efficiency Video Coding (3D-HEVC) is well designed, in which the synthesized view distortion (SVD) is introduced to derive the rate-distortion (RD) cost. To obtain accurate SVDs, the rendering operation is employed which demands a fairly high computational complexity. To address this problem, a fast RDO method for depth maps is proposed, which checks the RD cost during its calculation process. Specifically, given a coding mode, the RD cost is composed of several cumulative items. If the accumulated RD cost is equal to or exceeds the minimum RD cost of previously coded modes, it will not be necessary to continue the RD cost calculation for the mode. To reduce the encoding complexity, existing methods usually aim at reducing the number of tested modes or block partitions. To the best of our knowledge, it is the first time that the latent redundant complexity in the RD cost calculation is investigated and removed. Experimental results demonstrate that, compared with the 3D-HEVC reference software, the proposed method can save 28.1% of depth coding time with a small coding gain (0.04% BD-rate saving). An additional test is designed to evaluate four typical fast coding methods with/without the proposed method. Extensive results verify that the proposed method can be seamlessly combined with the state-of-the-art methods.

3D-Distortion Based Rate Distortion Optimization for Video-Based Point Cloud Compression

3D-Distortion Based Rate Distortion Optimization for Video-Based Point Cloud Compression