Rate-distortion Optimization Research Articles

Fast CU decision method based on texture characteristics and decision tree for depth map intra-coding

As the demand for higher quality 3D videos continues to grow, the 3D extensions of the Efficient Video Coding (3D-HEVC) standard are gradually unable to meet the needs of users. Versatile Video Coding Standard (H. 266/VVC), as an advanced video coding standard, adopts the nested Multi-Type Tree quadtree (QTMT) partitioning structure that current fast depth map coding unit (CU) partitioning methods cannot apply. Therefore, we have designed a fast intra-frame CU partitioning algorithm for VVC 3D video depth maps. Our proposed algorithm in this article consists of two steps, including two sub-algorithms. First, by analyzing the relationship between image entropy and variance and depth map CU division, we establish a bi-criterion decision algorithm to determine whether the texture complexity of the current CU is low enough to terminate its partitioning process. Then, for CUs that have been determined by the first algorithm to need further partitioning, we use a decision tree model based on Light Gradient Boosting Machine (LGBM) to predict which direction of Rate-Distortion Optimization (RDO) calculation can be skipped, which can avoid some unnecessary RDO calculations in a certain direction. The final experiment demonstrated the effect of the proposed algorithm, which can reduce 47.65% complexity of VVC 3D video intra-coding with negligible 0.23% Bjøntegaard Delta Bitrate (BDBR) increase, superior to other advanced methods.

Compressed Point Cloud Quality Index by Combining Global Appearance and Local Details

In recent years, many standardized algorithms for point cloud compression (PCC) has been developed and achieved remarkable compression ratios. To provide guidance for rate-distortion optimization and codec evaluation, point cloud quality assessment (PCQA) has become a critical problem for PCC. Therefore, in order to achieve a more consistent correlation with human visual perception of a compressed point cloud, we propose a full-reference PCQA algorithm tailored for static point clouds in this article, which can jointly measure geometry and attribute deformations. Specifically, we assume that the quality decision of compressed point clouds is determined by both global appearance (e.g., density, contrast, and complexity) and local details (e.g., gradient and hole). Motivated by the nature of compression distortions and the properties of the human visual system, we derive perceptually effective features for the above two categories, such as content complexity, luminance/geometry gradient, and hole probability. Through systematically incorporating measurements of variations in the local and global characteristics, we derive an effective quality index for the input compressed point clouds. Extensive experiments and analyses conducted on popular PCQA databases show the superiority of the proposed method in evaluating compression distortions. Subsequent investigations validate the efficacy of different components within the model design.

Hardware-friendly fast rate-distortion optimization algorithm for AV1 encoder

Hardware-friendly fast rate-distortion optimization algorithm for AV1 encoder

Syntax-Guided Content-Adaptive Transform for Image Compression.

The surge in image data has significantly increased the pressure on storage and transmission, posing new challenges for image compression technology. The structural texture of an image implies its statistical characteristics, which is effective for image encoding and decoding. Consequently, content-adaptive compression methods based on learning can better capture the content attributes of images, thereby enhancing encoding performance. However, learned image compression methods do not comprehensively account for both the global and local correlations among the pixels within an image. Moreover, they are constrained by rate-distortion optimization, which prevents the attainment of a compact representation of image attributes. To address these issues, we propose a syntax-guided content-adaptive transform framework that efficiently captures image attributes and enhances encoding efficiency. Firstly, we propose a syntax-refined side information module that fully leverages syntax and side information to guide the adaptive transformation of image attributes. Moreover, to more thoroughly exploit the global and local correlations in image space, we designed global-local modules, local-global modules, and upsampling/downsampling modules in codecs, further eliminating local and global redundancies. The experimental findings indicate that our proposed syntax-guided content-adaptive image compression model successfully adapts to the diverse complexities of different images, which enhances the efficiency of image compression. Concurrently, the method proposed has demonstrated outstanding performance across three benchmark datasets.

A fast H.266/QTMT intra coding scheme based on predictions of learned models

ABSTRACT H.266/VVC adopts a QuadTree-plus-MultiType tree (QTMT) coding-unit (CU) split structure to improve efficiency at the cost of high time complexity. Speeding up VVC coding while minimizing quality degradation is critical for practical applications. We propose predicting the coding depth and split type of an optimally coded 32 × 32 CU (CU32 × 32) to perform only a subset of exhaustive rate-distortion optimization (RDO) operations: (1) To predict the depth of an optimally coded CU32 × 32, we train a convolutional neural network (CNNdepth). CNNdepth outputs a label specifying a depth range subset by which the controller can execute EarlySkip or EarlyTerminate to reduce time complexity. (2) To predict the split type, we train random forest classifiers (RFCtype). The corresponding RDO operations can be omitted if the RFCtype classifies one CU32 × 32 as not of a specified split type. Experiments show that CNNdepth and RFCtype work seamlessly, reducing execution time by up to 69% and 39.16% on average, with a 0.7% increase in BDBR compared to the default VTM-7.0. Additionally, the proposed method yields the highest balanced time reduction rate of 61.5%.

Video Coding for Machines: Compact Visual Representation Compression for Intelligent Collaborative Analytics.

As an emerging research practice leveraging recent advanced AI techniques, e.g. deep models based prediction and generation, Video Coding for Machines (VCM) is committed to bridging to an extent separate research tracks of video/image compression and feature compression, and attempts to optimize compactness and efficiency jointly from a unified perspective of high accuracy machine vision and full fidelity human vision. With the rapid advances of deep feature representation and visual data compression in mind, in this paper, we summarize VCM methodology and philosophy based on existing academia and industrial efforts. The development of VCM follows a general rate-distortion optimization, and the categorization of key modules or techniques is established including feature-assisted coding, scalable coding, intermediate feature compression/optimization, and machine vision targeted codec, from broader perspectives of vision tasks, analytics resources, etc. From previous works, it is demonstrated that, although existing works attempt to reveal the nature of scalable representation in bits when dealing with machine and human vision tasks, there remains a rare study in the generality of low bit rate representation, and accordingly how to support a variety of visual analytic tasks. Therefore, we investigate a novel visual information compression for the analytics taxonomy problem to strengthen the capability of compact visual representations extracted from multiple tasks for visual analytics. A new perspective of task relationships versus compression is revisited. By keeping in mind the transferability among different machine vision tasks (e.g. high-level semantic and mid-level geometry-related), we aim to support multiple tasks jointly at low bit rates. In particular, to narrow the dimensionality gap between neural network generated features extracted from pixels and a variety of machine vision features/labels (e.g. scene class, segmentation labels), a codebook hyperprior is designed to compress the neural network-generated features. As demonstrated in our experiments, this new hyperprior model is expected to improve feature compression efficiency by estimating the signal entropy more accurately, which enables further investigation of the granularity of abstracting compact features among different tasks.

Fast Depth Map Coding Algorithm for 3D-HEVC Based on Gradient Boosting Machine

Three-Dimensional High-Efficiency Video Coding (3D-HEVC) has been extensively researched due to its efficient compression and deep image representation, but encoding complexity continues to pose a difficulty. This is mainly attributed to redundancy in the coding unit (CU) recursive partitioning process and rate–distortion (RD) cost calculation, resulting in a complex encoding process. Therefore, enhancing encoding efficiency and reducing redundant computations are key objectives for optimizing 3D-HEVC. This paper introduces a fast-encoding method for 3D-HEVC, comprising an adaptive CU partitioning algorithm and a rapid rate–distortion-optimization (RDO) algorithm. Based on the ALV features extracted from each coding unit, a Gradient Boosting Machine (GBM) model is constructed to obtain the corresponding CU thresholds. These thresholds are compared with the ALV to further decide whether to continue dividing the coding unit. The RDO algorithm is used to optimize the RD cost calculation process, selecting the optimal prediction mode as much as possible. The simulation results show that this method saves 52.49% of complexity while ensuring good video quality.

JND-based multi-module cooperative perceptual optimization for HEVC

Since the just noticeable distortion (JND) reflects the tolerance limit of human visual system (HVS) to coding distortion directly, JND-based perceptual video coding (PVC) plays an increasingly significant role in video compression with the developing explosion of video data. In this paper, we focus on the coupling effect among coding modules and provide a JND-based multi-module cooperative perceptual optimization (JMCPO) scheme for HEVC. The main contribution of the proposed JMCPO scheme includes the following three aspects. (1) Based on quantization distortion estimation, an adaptive perceptual quantization scheme is proposed using the binary search approach on the premise of that the quantization distortion is infinitely close to the estimated JND threshold. (2) The coupling effect among coding modules is first analyzed and a novel perceptual residual filtering scheme is presented based on statistic analysis of coupling strength. (3) The JMCPO scheme is finally developed through collaborative optimization of residual filtering, quantization and rate–distortion optimization. Experimental results show that the proposed JMCPO scheme saves more bitrate with better subjective and objective qualities.

On the rate–distortion–perception–semantics tradeoff in low-rate regime for lossy compression

This paper formulates a multi-letter information theoretical optimization problem to characterize the tradeoff between rate, distortion, perception, and semantics. It turns out that our formulation is related to existing works, such as the rate–distortion theory, utility-privacy tradeoff, and rate–distortion–perception tradeoff. In this paper, we solve this problem in the low-rate regime by applying a local geometric approach, which allows us to express the optimal solutions analytically. By using such an approach, we obtain the analytical solution of the multi-letter optimization problem, which demonstrates the optimal rate–distortion–perception–semantics tradeoff, and characterizes the gain from increasing the number of letters as well as the limiting performance. Moreover, our result also validates the single-letter optimality of the scenarios above, which reveals the information theoretical insights of system and algorithm designs.

Adaptive HEVC video steganograhpy based on PU partition modes

High EfficiencyVideoCoding (HEVC) −based steganography has gained attention as a prominent research focus. Especially, block structure based HEVC video steganography has received increasing attention due to commendable performance. However, current block structure- based steganography algorithms confront with challenges such as reduced coding efficiency and limited capacity. To avoid these problems, an adaptive video steganography algorithm based on Prediction Unit (PU) partition mode in I-frames is proposed. This is done through the analysis of the block division process and the visual distortion resulting from the modification of the PU partition mode in HEVC. The PU block structure is utilized as steganographic covers, and the Rate Distortion Optimization (RDO) technique is introduced to establish an adaptive distortion function for Syndrome-trellis code (STC). Further comparison is performed between the proposed method and the state-of-the-art steganography algorithms, confirming its advantages in embedding capacity, compression efficiency, visual quality, and resistance to video steganalysis.

EFFICIENT VIDEO COMPRESSION USING HEVC AND NON-LINEAR CONVOLUTIONAL MOBILEVNET BASED RATE-DISTORTION OPTIMIZATION

With the surge in demand for high-quality video content over various platforms, efficient video compression techniques have become indispensable. High-Efficiency Video Coding (HEVC) has been a cornerstone, yet further enhancements are essential for optimal compression. Despite HEVC’s advancements, achieving optimal compression while maintaining video quality remains challenging. Additionally, existing methods often overlook the computational complexity, hindering real-time applications. We propose a novel approach integrating HEVC with Non-Linear Convolutional MobileNet (NLCM) for enhanced compression efficiency. Our method employs a rate-distortion optimization framework, leveraging the capabilities of both HEVC and NLCM to achieve superior compression performance. NLCM provides adaptive filtering, enhancing spatial and temporal correlations, while HEVC ensures high compression efficiency. Through experimentation on standard video datasets, our method demonstrates significant improvements over existing techniques. Compared to HEVC alone, our approach achieves up to 30% reduction in bitrate at equivalent perceptual quality levels. Moreover, computational complexity is reduced by 15%, enabling real- time applications without compromising performance. The proposed method exhibits competitive results across various resolutions and frame rates, making it versatile for diverse video compression scenarios.

Towards Hybrid-Optimization Video Coding

Video coding that pursues the highest compression efficiency is the art of computing for rate-distortion optimization. The optimization has been approached in different ways, exemplified by two typical frameworks: block-based hybrid video coding and end-to-end learned video coding. The block-based hybrid framework encompasses more and more coding modes that are available at the decoder side; an encoder tries to search for the optimal coding mode for each block to be coded. This is an online, discrete, search-based optimization strategy. The end-to-end learned framework embraces more and more sophisticated neural networks; the network parameters are learned from a collection of videos, typically using gradient descent-based methods. This is an offline, continuous, numerical optimization strategy. Having analyzed these two strategies, both conceptually and with concrete schemes, this paper suggests investigating hybrid -optimization video coding, that is to combine online and offline, discrete and continuous, search-based and numerical optimization. For instance, we propose a hybrid-optimization video coding scheme, where the decoder consists of trained neural networks and supports several coding modes, and the encoder adopts both numerical and search-based algorithms for the online optimization. Our scheme achieves promising compression efficiency on par with H.265/HM for the random-access configuration.

Efficient Computation of the Quantum Rate-Distortion Function

The quantum rate-distortion function plays a fundamental role in quantum information theory, however there is currently no practical algorithm which can efficiently compute this function to high accuracy for moderate channel dimensions. In this paper, we show how symmetry reduction can significantly simplify common instances of the entanglement-assisted quantum rate-distortion problems. This allows us to better understand the properties of the quantum channels which obtain the optimal rate-distortion trade-off, while also allowing for more efficient computation of the quantum rate-distortion function regardless of the numerical algorithm being used. Additionally, we propose an inexact variant of the mirror descent algorithm to compute the quantum rate-distortion function with provable sublinear convergence rates. We show how this mirror descent algorithm is related to Blahut-Arimoto and expectation-maximization methods previously used to solve similar problems in information theory. Using these techniques, we present the first numerical experiments to compute a multi-qubit quantum rate-distortion function, and show that our proposed algorithm solves faster and to higher accuracy when compared to existing methods.

High Speed Coding Unit Depth Identification Based on Texture Image Information Using SVM

High-Efficiency Video Coding (HEVC/H.265) is a new video coding standard with half the bit rate of its predecessor, Advanced Video Coding (AVC/H.264). AVC/H.264 uses macroblocks, processing units between 4×4 and 16×16 pixels in size. H.265 uses Coding Tree Units (CTUs), a more complicated block structure that lets images be as large as 64×64 pixels. However, changing from macroblocks to coding tree units is essential for H.265 to become more efficient. Using the quadtree structure to divide the Coding Unit (CU) makes it harder for HEVC to find the optimal rate distortion. This paper presents a Support Vector Machine (SVM)-based method for finding the fastest coding unit division in intra-prediction HEVC without compromising compression efficiency. All partitions of CTU are assessed using five characteristics: Standard Deviation (SD), Root Mean Square Error (RMSE), Sub CU Complexity Difference (SCCD), Directional Complexity (DC), and Quantization Parameter (QP) to optimize the intra-prediction of HEVC in all intra-configurations. Simulations have been carried out to estimate the performance of the proposed machine learning-based algorithm using test sequences with different resolutions. Simulation results have shown that combining directional complexity and standard deviation gives a more accurate classification. SVM has been used to separate split-unsplit samples, and the standard rate-distortion optimization technique has been used to separate samples that are hard to separate. The results have shown a reduction of 67.44% in encoding time with a slight increase in bit rate.

Cracks-suppression perceptual geometry coding for dynamic point clouds

Dynamic point clouds can effectively describe 3D objects and natural scenes, providing users with an immersive visual experience, but their huge amount of data requires efficient compression tools. To this end, the video-based point cloud compression (V-PCC) standard was developed, however, it may lead to cracks in the compressed point cloud at low bitrates. To solve the problem, this paper proposes a cracks-suppression perceptual geometry coding method for dynamic point cloud. Firstly, considering that the edge regions of 2D geometry patches in V-PCC are prone to cracks, a prior knowledge based crack region detection and preprocessing scheme is designed to reduce cracks. Secondly, considering the unique perceptual geometry characteristics of point cloud, a projected geometry curvature based structural similarity (PGC-SSIM) is proposed to evaluate geometry quality of point clouds, which contains the geometric perception information of point cloud, and is more consistent with the human visual perception. Finally, based on the PGC-SSIM, an adaptive quantization parameter adjustment strategy is designed for rate-distortion optimization in geometry coding of dynamic point clouds. The experimental results show that the proposed method can effectively reduce cracks in the compressed point cloud without reducing the coding performance compared to the V-PCC anchor. Moreover, the presented PGC-SSIM can be used to improve the visual quality of the compressed point cloud.

Research on Convolutional Neural Network-Based Compression Methods for Multispectral Images

Considering the traits of multispectral images, which include numerous bands, high spatial and spectral redundancy, and a large data volume, there is a proposed investigation on compression methods based on convolutional neural networks to reduce the storage space consumed by an individual image and enhance compression effectiveness. This paper first introduces the development of image compression algorithms and deep learning in recent years. Based on these two structures, a framework for lossy compression of multispectral images utilizing an end-to-end convolutional neural network is proposed. A self-encoding structure is used to process three-dimensional hyperspectral images, extracting local spectral features and fusing spectral information using large convolutional kernels. Residual layers are employed to preserve spectral information. Rate-distortion optimization is performed to jointly optimize image distortion and compression bitrate. Finally, a comparison with the traditional JPEG method is conducted experiments to assess the efficacy of the proposed algorithm. The MS-SSIM is improved by nearly 0.08, and the compressed images exhibit no noticeable distortion.

A convolutional neural network-based rate control algorithm for VVC intra coding

The Versatile Video Coding (VVC) has shown significant improvements in Rate-Distortion (R-D) performance compared to its predecessor, High Efficiency Video Coding (HEVC). However, it still encounters several challenges. One of these challenges is the efficient allocation of bits among all Coding Tree Units (CTUs). Additionally, there is a lack of prior information for intra-frame coding, particularly for the first frame. After CTU-level bit allocation, only fixed parameters can be used to determine the λ for CTUs, which does not result in optimal rate–distortion performance. To tackle above challenges, we propose a rate control solution based on Convolutional Neural Network (CNN). This approach utilizes CNN to predict the key parameters α and β in the R-D model, addressing the problem of lacking prior information in intra-frame coding. Subsequently, the predicted α and β values are used to adaptively allocate bits for each CTU. Our proposed algorithm is implemented in VTM-16.0 under Common Test Conditions (CTC). Experimental results show that, compared to the default rate control algorithm in VTM-16.0, our proposed algorithm enhances R-D performance by 0.96% while maintaining rate control accuracy.

Near-Lossless Compression of Point Cloud Attribute Using Quantization Parameter Cascading and Rate-Distortion Optimization

Near-Lossless Compression of Point Cloud Attribute Using Quantization Parameter Cascading and Rate-Distortion Optimization

Adaptive HEVC Video Steganography With High Performance Based on Attention-Net and PU Partition Modes

With the increasing popularity of digital video, video steganography has become a hot research topic in the field of covert communication and privacy protection. The existing prediction unit (PU) based video steganography often tends to result in large bit rate increase, which is also easily noticeable to the steganography analyst. To solve this problem, an adaptive steganography for HEVC video based on attention-net and PU partition modes is proposed. First, the distortion of modified PUs is analyzed from the perspective of rate distortion optimization at the group of pictures (GOP) level, and we find that modifying PU will lead to distortion accumulation and abnormal bitrate increase. Therefore, an adaptive distortion function based on the improved rate distortion cost is designed, and the embedding distortion is minimized by using Syndrome-Trellis Code (STC) steganography coding. Meanwhile, a super-resolution convolutional neural network with non-local sparse attention-net filter is proposed to replace the in-loop filter in HEVC to reconstruct the reference frame, thereby reducing the bitrate cost and improving the visual quality of stego-video. Experimental results show that the proposed algorithm can achieve superior perceptual quality and bitrate performance comparing with the sate-of-the-art works.

Latitude-Redundancy-Aware All-Zero Block Detection for Fast 360-Degree Video Coding.

The sphere-to-plane projection of 360-degree video introduces substantial stretched redundant data, which is discarded when reprojected to the 3D sphere for display. Consequently, encoding and transmitting such redundant data is unnecessary. Highly redundant blocks can be referred to as all-zero blocks (AZBs). Detecting these AZBs in advance can reduce computational and transmission resource consumption. However, this cannot be achieved by existing AZB detection techniques due to the unawareness of the stretching redundancy. In this paper, we first derive a latitude-adaptive redundancy detection (LARD) approach to adaptively detect coefficients carrying redundancy in transformed blocks by modeling the dependency between valid frequency range and the stretching degree based on spectrum analysis. Utilizing LARD, a latitude-redundancy-aware AZB detection scheme tailored for fast 360-degree video coding (LRAS) is proposed to accelerate the encoding process. LRAS consists of three sequential stages: latitude-adaptive AZB (L-AZB) detection, latitude-adaptive genuine-AZB (LG-AZB) detection and latitude-adaptive pseudo-AZB (LP-AZB) detection. Specifically, L-AZB refers to the AZB introduced by projection. LARD is used to detect L-AZB directly. LG-AZB refers to the AZB after hard-decision quantization and zeroing redundant coefficients. A novel latitude-adaptive sum of absolute difference estimation model is built to derive the threshold for LG-AZB detection. LP-AZB refers to the AZB in terms of rate-distortion optimization considering redundancy. A latitude-adaptive rate-distortion model is established for LP-AZB detection. Experimental results show that LRAS can achieve an average total encoding time reduction of 25.85% and 20.38% under low-delay and random access configurations compared to the original HEVC encoder, with only 0.16% and 0.13% BDBR increases and 0.01dB BDPSNR loss, respectively. The transform and quantization time savings are 60.13% and 59.94% on average.