Rate-distortion Optimization Research Articles

Rate-distortion optimized quantization for geometry-based point cloud compression

Limited by the network bandwidth, three-dimensional (3D) point cloud needs to be efficiently compressed before transmission. As one of the three attribute coding methods adopted in the geometry-based point cloud compression (G-PCC) standard developed by MPEG, predicting transform (PT) has received increasing attention. To further improve the coding efficiency of PT, we propose a rate-distortion optimized quantization (RDOQ) in which an additional option for quantization results is added by setting the quantized residuals to zero forcedly. Rate-distortion optimization is then used to determine whether the final quantized residual should be zero or nonzero. Experimental results show that average BD rates of −0.5 % , −3.0 % , and −3.0 % can be achieved for Luma, Chroma Cb, and Chroma Cr components, respectively, with negligible increment of time complexity.

Deep Learning-Based Intra Mode Derivation for Versatile Video Coding

In intra coding, Rate Distortion Optimization (RDO) is performed to achieve the optimal intra mode from a pre-defined candidate list. The optimal intra mode is also required to be encoded and transmitted to the decoder side besides the residual signal, where lots of coding bits are consumed. To further improve the performance of intra coding in Versatile Video Coding (VVC) , an intelligent intra mode derivation method is proposed in this paper, termed as Deep Learning based Intra Mode Derivation (DLIMD) . In specific, the process of intra mode derivation is formulated as a multi-class classification task, which aims to skip the module of intra mode signaling for coding bits reduction. The architecture of DLIMD is developed to adapt to different quantization parameter settings and variable coding blocks including non-square ones, where only one single trained model is required. Different from the existing deep learning based classification problems, the hand-crafted features are also fed into intra mode derivation network besides the learned features from feature learning network. To compete with traditional methods, one additional binary flag is utilized in the video codec to indicate the selected scheme with RDO. Extensive experimental results reveal that the proposed method can achieve 2.28%, 1.74%, and 2.18% bit rate reduction on average for Y, U, and V components on the platform of VVC test model, which outperforms the state-of-the-art works.

Multi-Exposure Image Compression Considering Rate-Distortion Optimization in Rendered High Dynamic Range Image

High-dynamic range (HDR) images, which provide realistic visual perception, have attracted considerable attention in various applications. A simple method for obtaining HDR images is to fuse multiple low-dynamic range (LDR) images captured under varying exposures. We proposed a method of efficiently encoding multi-exposure images using multi-view High Efficiency Video Coding (MV-HEVC) along with intensity mapping function (IMF) to create high-quality HDR images. The inter-view prediction in MV-HEVC eliminates redundancy between multi-exposure images. To achieve high-efficiency MV-HEVC-based multi-exposure image coding, two modifications to the rate-distortion optimization (RDO) process performed during encoding were proposed. First, the distortion in the rendered HDR image is used to replace the original distortion, which depends on the LDR image. Second, to balance distortion and rate in RDO, a new Lagrange multiplier is derived from the derivative of the new distortion with respect to the rate. A model of the distortion in the HDR domain was constructed and expressed as a function of the distortion in the LDR domain. The proposed scheme utilizes the conventional 8-bit codec to compress multi-exposure LDR images and generate an HDR image in the decoder. The experimental results indicate that the proposed technique outperforms HDR image coding schemes in which HEVC Range Extension and JPEG-XT coding standards are used.

Plenoptic Point Cloud Compression Using Multiview Extension of High Efficiency Video Coding

Plenoptic point clouds are more complete representations of three-dimensional (3-D) objects than single-color point clouds, as they can have multiple colors per spatial point, representing colors of each point as seen from different view angles. They are more realistic but also involve a larger volume of data in need of compression. Therefore, in this paper, a multiview-video-based framework is proposed to better exploit correlations in color across different viewpoints. To the best of the authors' knowledge, this is the first work to exploit correlations in color across different viewpoints using a multiview-video-based framework. In addition, it is observed that some unoccupied pixels, which do not have corresponding points in plenoptic point clouds and are of no use to the quality of the reconstructed plenoptic point cloud colors, may cost many bits. To address this problem, a block-based group smoothing and a combined occupancy-map-based rate distortion optimization and four-neighbor average residual padding are further proposed to reduce the bit cost of unoccupied color pixels. The proposed algorithms are implemented in the moving pictures experts group (MPEG) video-based point cloud compression (V-PCC) and multiview extension of High Efficiency Video Coding (MV-HEVC) reference software. The experimental results show that the proposed algorithms can lead to a Bjontegaard Delta bitrate (BD-rate) reduction of 40% compared with the RAHT-KLT. Compared with the V-PCC independently applied to each view direction, the proposed algorithms can provide a BD-rate reduction of over 70%.

Textural and Directional Information Based Offset In-Loop Filtering in AVS3

In this paper, we propose a novel low-complexity in-loop filtering approach named textural and directional information based offset (TDIO) for the emerging video coding standard AVS3 (the third generation of Audio Video-coding Standard). Different from conventional offset-based filtering methods which partially use contextual samples, the key contribution of TDIO is that it fully utilizes the textural and edge directional features of each sample to comprehensively determine which type of texture characteristics each sample belongs to. The corresponding offsets are generated and signaled to decoder such that sample-level distortion is reduced by correcting the quantization errors. Specifically, the multi-directionality and sample-intensity pattern based classifiers are first proposed to extract the directional and textural features, respectively. The classification results are obtained by incorporating these features, and the optimal offset values for each class are derived based on rate-distortion optimization. Since sample-level offset signalling may cause heavy burden to the overhead of TDIO, we subsequently propose a filtering offset sharing mechanism based on historical information between available temporal-adjacent compressed frames. In addition, an iteration-based filter adaptation method is designed to improve the local adaptivity of TDIO for better compression efficiency. Experimental results show that the proposed TDIO achieves 0.64%, 1.29%, 1.86%, and 2.20% bit rate savings for all intra, random access, low delay B, and low delay P configurations, respectively. Moreover, TDIO is helpful to improve subjective quality by leveraging the fine-grained local texture characteristics. It can be observed that the blurring and ringing artifacts could be significantly suppressed by using the proposed method, yielding higher subjective quality.

Rate-Distortion Optimized Geometry Compression for Spinning LiDAR Point Cloud

Point cloud is a major representation format of 3D objects and scenes. It has been increasingly applied in various applications due to the rapid advances in 3D sensing and rendering technologies. In the field of autonomous driving, point clouds captured by spinning Light Detection And Ranging (LiDAR) devices have become an informative data source for road environment perception and intelligent vehicle control. On the other hand, the massive data volume of point clouds also brings huge challenges to point cloud transmission and storage. Therefore, establishing compression frameworks and algorithms that conform to the characteristics of point cloud data has become an important research topic for both academia and industry. In this paper, a geometry compression method dedicated to spinning LiDAR point cloud was proposed taking advantage of the prior information of the LiDAR acquisition procedure. Rate-distortion optimizations were further integrated into the coding pipeline according to the characteristics of the prediction residuals. Experimental results obtained on different datasets show that the proposed method consistently outperforms the state-of-the-art G-PCC predictive geometry coding method with reduced runtime at both the encoder and decoder sides.

Hyperspectral Image Compression via Cross-Channel Contrastive Learning

In recent years, advances in deep learning have greatly promoted the development of hyperspectral image (HSI) compression algorithms. However, most existing compression approaches directly rely on rate-distortion optimization without other guidance during model learning. Therefore, this brings challenges to distinguishing similar features or objects that are widely available in HSIs, especially in remote sensing scenes, since quantification in lossy compression can cause informative attribute (e.g., category) collapse and loss problems at high compression ratios. In this paper, we propose a novel hyperspectral compression network via contrastive learning (HCCNet) to help generate discriminative representations and preserve informative attributes as much as possible. Specifically, we design a contrastive informative feature encoding (CIFE) to extract and organize discriminative attributes from the original HSIs by enlarging the discrimination over the learned latents in different channel indexes to relieve attribute collapses. In the case of attribute losses, we define a contrastive invariant feature recovery (CIFR) to discover the lost attributes via contrastive feature refinement. Experiments on five different HSI datasets illustrate that the proposed HCCNet can achieve impressive compression performance, such as improvement of the peak signal-to-noise ratio (PSNR) from 28.86 dB (at 0.2284 bpppb) to 30.30 dB (at 0.1960 bpppb) on the Chikusei dataset.

Efficient Geometry Surface Coding in V-PCC

In recent video-based point cloud compression (V-PCC), 3D point clouds are projected onto 2D images and compressed by High-Efficiency Video Coding (HEVC). However, HEVC was originally designed for natural visual signals, which is a suboptimal framework for point clouds. Therefore, there are still problems in geometry information compression in V-PCC: (1) The distortion based on the sum of squared error (SSE) in the existing rate-distortion optimization (RDO) is inconsistent with the geometric quality measurement; (2) The existing prediction cannot explore the fixed relationship between the corresponding far layer and near layer depth, which means that the far layer depth can be always not less than the corresponding near layer depth. In this paper, we present an efficient geometry surface coding (EGSC) method for V-PCC to address the problems. Firstly, an error projection (EP) model is designed to establish the relationship between the SSE-based distortion and the geometry quality metric. Secondly, an EP-based RDO is employed to improve the geometry information compression by estimating the point normals with gradients. Finally, an occupancy-map driven scheme is proposed to improve the prediction accuracy of merge modes. Experimental results show that the proposed method achieves an average of over 10% bit-rate saving compared with the V-PCC reference software.

Perceptually Weighted Rate Distortion Optimization for Video-Based Point Cloud Compression.

Dynamic point cloud is a volumetric visual data representing realistic 3D scenes for virtual reality and augmented reality applications. However, its large data volume has been the bottleneck of data processing, transmission, and storage, which requires effective compression. In this paper, we propose a Perceptually Weighted Rate-Distortion Optimization (PWRDO) scheme for Video-based Point Cloud Compression (V-PCC), which aims to minimize the perceptual distortion of reconstructed point cloud at the given bit rate. Firstly, we propose a general framework of perceptually optimized V-PCC to exploit visual redundancies in point clouds. Secondly, a multi-scale Projection based Point Cloud quality Metric (PPCM) is proposed to measure the perceptual quality of 3D point cloud. The PPCM model comprises 3D-to-2D patch projection, multi-scale structural distortion measurement, and fusion model. Approximations and simplifications of the proposed PPCM are also presented for both V-PCC integration and low complexity. Thirdly, based on the simplified PPCM model, we propose a PWRDO scheme with Lagrange multiplier adaptation, which is incorporated into the V-PCC to enhance the coding efficiency. Experimental results show that the proposed PPCM models can be used as standalone quality metrics, and they are able to achieve higher consistency with the human subjective scores than the state-of-the-art objective visual quality metrics. Also, compared with the latest V-PCC reference model, the proposed PWRDO-based V-PCC scheme achieves an average bit rate reduction of 13.52%, 8.16%, 10.56% and 9.54%, respectively, in terms of four objective visual quality metrics for point clouds. It is significantly superior to the state-of-the-art coding algorithms. The computational complexity of the proposed PWRDO increases by 1.71% and 0.05% on average to the V-PCC encoder and decoder, respectively, which is negligible. The source codes of the PPCM and PWRDO schemes are available at https://github.com/VVCodec/PPCM-PWRDO.

Improving Image Compression With Adjacent Attention and Refinement Block

Recently, learned image compression algorithms have shown incredible performance compared to classic hand-crafted image codecs. Despite its considerable achievements, the fundamental disadvantage is not optimized for retaining local redundancies, particularly non-repetitive patterns, which have a detrimental influence on the reconstruction quality. This paper introduces the autoencoder-style network-based efficient image compression method, which contains three novel blocks, i.e., adjacent attention block, Gaussian merge block, and decoded image refinement block, to improve the overall image compression performance. The adjacent attention block allocates the additional bits required to capture spatial correlations (both vertical and horizontal) and effectively remove worthless information. The Gaussian merge block assists the rate-distortion optimization performance, while the decoded image refinement block improves the defects in low-resolution reconstructed images. A comprehensive ablation study analyzes and evaluates the qualitative and quantitative capabilities of the proposed model. Experimental results on two publicly available datasets reveal that our method outperforms the state-of-the-art methods on the KODAK dataset (by around 4dB and 5dB) and CLIC dataset (by about 4dB and 3dB) in terms of PSNR and MS-SSIM.

CBANet: Toward Complexity and Bitrate Adaptive Deep Image Compression Using a Single Network.

In this work, we propose a new deep image compression framework called Complexity and Bitrate Adaptive Network (CBANet) that aims to learn one single network to support variable bitrate coding under various computational complexity levels. In contrast to the existing state-of-the-art learning-based image compression frameworks that only consider the rate-distortion trade-off without introducing any constraint related to the computational complexity, our CBANet considers the complex rate-distortion-complexity trade-off when learning a single network to support multiple computational complexity levels and variable bitrates. Since it is a non-trivial task to solve such a rate-distortion-complexity related optimization problem, we propose a two-step approach to decouple this complex optimization task into a complexity-distortion optimization sub-task and a rate-distortion optimization sub-task, and additionally propose a new network design strategy by introducing a Complexity Adaptive Module (CAM) and a Bitrate Adaptive Module (BAM) to respectively achieve the complexity-distortion and rate-distortion trade-offs. As a general approach, our network design strategy can be readily incorporated into different deep image compression methods to achieve complexity and bitrate adaptive image compression by using a single network. Comprehensive experiments on two benchmark datasets demonstrate the effectiveness of our CBANet for deep image compression. Code is released at https://github.com/JinyangGuo/CBANet-release.

Auto-Weighted Layer Representation Based View Synthesis Distortion Estimation for 3-D Video Coding

Recently, various view synthesis distortion estimation models have been studied to better serve 3-D video coding. However, they can hardly model the relationship quantitatively among different levels of depth changes, texture degeneration, and view synthesis distortion (VSD), which is crucial for rate-distortion optimization and rate allocation. In this paper, an auto-weighted layer representation based view synthesis distortion estimation model is developed. Firstly, sub-VSD (S-VSD) is defined according to the level of depth changes and their associated texture degeneration. After that, a set of theoretical derivations demonstrate that the VSD can be approximately decomposed into the S-VSDs multiplied by their associated weights. To obtain the S-VSDs efficiently, a layer-based representation method is developed, where all the pixels with the same level of depth changes are represented with a layer. It enables the S-VSD calculation at the layer level. Meanwhile, a nonlinear mapping function is learnt to accurately represent the relationship between the VSD and S-VSDs, automatically providing weights for the S-VSDs during VSD estimation. To learn such a function, a dataset of the VSD and its associated S-VSDs are built, termed as VSDSet. Experimental results show that the VSD can be accurately estimated with the weights learnt by the nonlinear mapping function once its associated S-VSDs are available. The proposed method outperforms the relevant state-of-the-art methods in both accuracy and efficiency. The VSDSet and source code of the proposed method will be available at <uri>https://github.com/jianjin008/</uri>.

Optimized video compression with residual split attention and swin-block artifact contraction

Research in video compression has seen significant advancement in the last several years. However, the existing deep learning-based algorithms continue to be plagued by erroneous motion compression and ineffective motion compensation architectures, resulting in compression errors with a lower rate–distortion trade-off. To overcome these challenges, we present an end-to-end purely deep learning-based video compression method through a set of primary operations (e.g., motion estimation, motion compression, motion compensation, residual compression, and artifact contraction) differently. A deep residual attention split (DRAS) block is introduced for motion compression networks to pay more attention to certain image regions to create more effective features for the decoder while boosting the rate–distortion optimization (RDO) efficiency. A channel residual block (CRB) is proposed in motion compensation to yield a more accurate predicted frame, potentially improving the residual frame. To mitigate the compression errors, an artifact contraction module (ACM) by residual swin convolution UNet block is included in this model to improve the reconstruction quality. To improve the final frame, a buffer is added to fine-tune the previous reference frames. These modules combine with a loss function by assessing the trade-off and enhancing the decoded video quality. A comprehensive ablation study demonstrates the effectiveness of the proposed blocks and modules for video compression. Experimental results show the competitive performance of the proposed method on four benchmark datasets.

Reference picture selection with decreased temporal dependency for HEVC error resilience

The high level of compression efficiency achieved by the High-Efficiency Video Coding standard (HEVC) decreases the robustness of the encoded bitstreams. This increased susceptibility to network errors leads to end video quality degradation. Moreover, due to the high computational complexity of HEVC, high-resolution video transmission with time constraints over hostile channels such as wireless networks becomes more challenging. This paper proposes a reference picture selection-based error-resilient method to reduce the temporal error propagation due to high-trip delay and frame-copy concealment error. First, the encoder selects the reference pictures based on the error status received from the feedback channel, taking into consideration the Rate-Distortion-Optimization (RDO). Second, the temporal information mismatch prediction resulting from the error concealment is reduced by decreasing the temporal dependency between adjacent frames based on new motion-estimation tools. Results show a PSNR gain of about 6.13 dB, 5.20 dB and 4.72 dB for 1080p, 720p and 480p resolutions respectively.

Rate Control Technology for Next Generation Video Coding Overview and Future Perspective

Video data have become the main data traffic on the Internet, and their traffic is increasing explosively every year, thus increasing the pressure of video transmission. Video coding technology has become the key to compressing original videos. As an indispensable technology, rate control plays an important role in stabilizing video stream transmission. Rate control (RC) is part of rate distortion optimization (RDO) whose job is to find the optimal solution based on balancing rate and distortion. It not only needs to consider the buffer and network status but also adjust the corresponding bit rate according to the video content. This paper reviews the related technologies of rate control under high efficiency video coding (HEVC) and versatile video coding (VVC) standards so that subsequent researchers can quickly understand the field and promote the development of rate control algorithms. Firstly, the paper summarizes the various aspects of RC, including basic principles, rate-distortion models, major processes, and performance criteria. Secondly, the paper surveys, in detail, the research progress in the field of rate control and analyzes several mainstream research directions. Thirdly, we carry out relevant experiments on the standard reference software and analyze and discuss the experimental results of the existing studies. Finally, we look ahead to the future trends of rate control and provide feasible improvement suggestions.

Texture-Aware Spherical Rotation for High Efficiency Omnidirectional Intra Video Coding

To adapt to the existing video coding standards, omnidirectional videos are usually projected from Three-Dimensional (3D) sphere to Two-Dimensional (2D) plane. However, this projection will cause geometrical stretching distortion and boundary discontinuity, which may degrade coding efficiency. In this paper, we present a Spherical Rotation based Omnidirectional Video Coding (SROVC) method, which exploits the textural properties of omnidirectional videos with spherical rotation. Firstly, SROVC framework is presented and Full-traversal Spherical Rotation (FSR) is developed to derive the optimal rotation angle with frame-level Rate Distortion Optimization (RDO). Secondly, to achieve comparable coding gains and lower computational complexity when compared with FSR, a Texture-aware Spherical Rotation (TSR) method is proposed to predict the rotation angle. Finally, to further reduce complexity and maintain coding efficiency, a Group-oriented TSR (G-TSR) approach is presented, in which the group length is statistically determined. Extensive experiments demonstrate that the proposed TSR and G-TSR schemes can achieve bit rate reductions up to 4.38%, 0.94% and 0.89% on average for CubeMap Projection (CMP) based high efficiency omnidirectional video coding. Additionally, the TSR scheme achieves bit rate saving from 0.91% to 1.19% on average under three more CMP-based projection formats, and 1.89% for joint rotation of X, Y, and Z axes.

Hypergraph-Based Source Codes for Function Computation Under Maximal Distortion

This work investigates functional source coding problems with maximal distortion, motivated by approximate function computation in many modern applications. The maximal distortion treats imprecise reconstruction of a function value as good as perfect computation if it deviates less than a tolerance level, while treating reconstruction that differs by more than that level as a failure. Using a geometric understanding of the maximal distortion, we propose a hypergraph-based source coding scheme for function computation that is constructive in the sense that it gives an explicit procedure for finding optimal or good auxiliary random variables. Moreover, we find that the hypergraph-based coding scheme achieves the optimal rate-distortion function in the setting of coding for computing with side information and achieves the Berger-Tung sum-rate inner bound in the setting of distributed source coding for computing. It also achieves the El Gamal-Cover inner bound for multiple description coding for computing and is optimal for successive refinement and cascade multiple description problems for computing. Lastly, the benefit of complexity reduction of finding a forward test channel is shown for a class of Markov sources.

DECISION ALGORITHM FOR INTRA PREDICTION IN HIGH-EFFICIENCY VIDEO CODING (HEVC)

Prediction in HEVC exploits the redundant information in the fame to improve compression efficiency. The computational complexity of prediction is comparatively high as it recursively calculates the depth by comparing the rate-distortion optimization cost (RDO) exhaustively. The deep learning technology has shown a good mark in this area compared to traditional signal processing because of its content-based analysis and learning ability. This paper proposes a deep depth decision algorithm to predict the depth of the coding tree unit (CTU) and store it as a 16-element vector, and this model is pipelined to the HEVC encoder to compare the time taken and bit rate of encoding. The comparison chart clearly shows the reduction in computational time and enhancement in bitrate while encoding. The dataset used here is generated for the model with 110000 frames of the various resolutions, split into test, training, and validation, and trained on a depth decision model. The trained model interfaced with the HEVC encoder is compared with the normal encoder. The evaluation is done for quality check for the proposed model with BD-PSNR and BD-Bitrate shows a dip of 0.6 in BD-PSNR and increment of 6.7 in BD-Bitrate. When pipelined with the original HEVC, the RDO cost shows an improvement over existing techniques. The average encoding time is reduced by about 72% by the pipelined deep depth decision algorithm that points to the reduction in computational complexity. An average time saving of 88.49% is achieved with a deep depth decision algorithm-based encoder compared to the existing techniques.

Fast CU Partition Decision Algorithm for VVC Intra Coding Using an MET-CNN

The newest video coding standard, the versatile video coding standard (VVC/H.266), came into effect in November 2020. Different from the previous generation standard—high-efficiency video coding (HEVC/H.265)—VVC adopts a more flexible block division structure, the quad-tree with nested multi-type tree (QTMT) structure, which improves its coding performance by 24%. However, it also causes a substantial increase in computational complexity. Therefore, this paper first proposes the concept of a stage grid map, which divides the overall division of a 32 × 32 coding unit (CU) into four stages and represents it as a structured output. Second, a multi-stage early termination convolutional neural network (MET-CNN) model is devised to predict the full partition information of a CU with a size of 32 × 32. Finally, a fast CU partition decision algorithm for VVC intra coding based on an MET-CNN is proposed. The algorithm can predict all partition information of a CU with a size of 32 × 32 and its sub-CUs in one run, completely replacing the complex rate-distortion optimization (RDO) process. It also has an early exit mechanism, thereby greatly reducing the encoding time. The experimental results illustrate that the scheme proposed in this paper reduces the encoding time by 49.24% on average, while the Bjøntegaard Delta Bit Rate (BDBR) only increases by 0.97%.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.