Fast Coding Unit Size Research Articles

Lightweight fully connected network-based fast CU size decision for video-based point cloud compression

Video-based point cloud compression (V-PCC) utilizes high efficiency video coding (HEVC) to compress geometry and attribute videos generated from dynamic point cloud projection. However, the HEVC exhaustive coding unit (CU) size decision process is complex and hinders the real-time application of V-PCC. To reduce the coding complexity of V-PCC, this paper proposes a method that combined hand-crafted features and lightweight neural network to accurately predict the best CU partition in advance. First, we extract hand-crafted features, including direct features (DFs) and indirect feature (IF), as mixed features. DFs are simple and require no additional calculation, while IF is obtained indirectly by transforming the global and local distortions of the CU extracted before size decision determination. Second, we propose a lightweight fully connected network (LFCN) as the backbone network, two feature types are used as inputs to the LFCN to predict whether the CU should be split into sub-CUs, and the LFCN can be fully integrated into the encoder with only about 1.58KB of additional parameters. Experimental results show that the proposed method reduces coding complexity by an average of 51.2% while Luma’s BD-TotalRate only increases by 0.1% on average under the All Intra (AI) configuration.

Low calculation cost of HEVC coding unit size based on spatial homogeneity detection

For the same video quality, HEVC gives 25% to 50% bitrate savings, compared to its predecessor the Advanced Video Coding H.264 and thus supports resolutions up to 8 K UHD. However, the reduction in bitrates provided by the HEVC leads to an increase in the computational cost of the encoding operation. This complexity can become a true handicap especially for real-time video streaming and also for VANET (Vehicular Ad-Hoc Network) applications such as traffic safety and Video surveillance. The improvement in the bitrates and also the increase in the calculation cost are due to the use of large and multi-sized coding, prediction and transform blocks. Indeed, the H264 coder is based on structure macroblocks with sizes 4 × 4, 8 × 8 and 16 × 16, while H.265 depends on Coding Tree Units (CTUs), CTUs select sizes 4 × 4, 8 × 8, 16 × 16, 32 × 32 and 64 × 64 blocks. This paper proposes a fast CU (Coding Unit) size decision method to reduce the HEVC calculation cost based on spatial homogeneity. Compared with the HM16.13 benchmark test model, the average coding time is reduced by around 40% for CIF / QCIF video sequences, 35% to 43% for class A, B and C test sequences. These important reductions in coding time are obtained with negligible loss of quality and an average increase in bitrates which does not exceed 0.89% for the three configuration modes (All intra, Random Access and Low Delay).

A Deep Learning Approach in Scalable High Efficiency Video Coding for Fast Coding Unit Size Decision

The High Efficiency Video Coding (HEVC) with scalable extension known as SHVC allows encoding the same video with different resolutions in a single bitstream. However, this process increases the encoding complexity. The complexity is increased mainly due to the motion estimation process and the Rate-Distortion Optimization (RDO) search process. The Test Zonal (TZ) algorithm with fast search patterns helps to accelerate the motion estimation process. However, the search patterns may get trapped to local minima, leading to inaccurate motion vectors. Moreover, the RDO search process used to determine the Coding Unit (CU) size increases the complexity. We proposed the Horizontal Subsampling Motion Estimation (HSME) method to find the accurate motion vectors with reduced complexity. The experimental results prove that the HSME method saves the encoding time by 53.03% with a 6.45% increase in Bjøntegaard delta bit rate and 0.28 dB loss in Bjøntegaard Delta Peak Signal-to-Noise Ratio (BD-PSNR) compared to standard SHM-12.1. In addition, we designed the Early Terminated Long- and Short-Term Memory (ET-LSTM) network that predicts the CU partition by taking the output features of the Early Terminated Convolutional Neural Network (ET-CNN). The ET-CNN learns the CU partitions from the residual Coding Tree Unit (CTU) using the deep learning approach. Our proposed method (HSME + ET-CNN + ET-LSTM) achieves 53% savings in encoding time, which is significantly higher than state-of-the-art methods.

Fast coding unit size decision based on deep reinforcement learning for versatile video coding

Fast coding unit size decision based on deep reinforcement learning for versatile video coding

Fast CU Size Decision Method Based on Just Noticeable Distortion and Deep Learning

With the development of broadband networks and high-definition displays, people have higher expectations for the quality of video images, which also brings new requirements and challenges to video coding technology. Compared with H.265/High Efficiency Video Coding (HEVC), the latest video coding standard, Versatile Video Coding (VVC), can save 50%-bit rate while maintaining the same subjective quality, but it leads to extremely high encoding complexity. To decrease the complexity, a fast coding unit (CU) size decision method based on Just Noticeable Distortion (JND) and deep learning is proposed in this paper. Specifically, the hybrid JND threshold model is first designed to distinguish smooth, normal, or complex region. Then, if CU belongs to complex area, the Ultra-Spherical SVM (US-SVM) classifiers are trained for forecasting the best splitting mode. Experimental results illustrate that the proposed method can save about 52.35% coding runtime, which can realize a trade-off between the reduction of computational burden and coding efficiency compared with the latest methods.

Machine learning-based fast CU size decision algorithm for 3D-HEVC inter-coding

3D-high efficiency video coding (3D-HEVC) is an extension of the high efficiency video coding (HEVC) standard for the compression of the texture videos and depth maps. In 3D-HEVC inter-coding, the coding unit (CU) is recursively performed on variable sizes, namely, depth levels. The CU size decision process is conducted using all the possible depth levels to obtain the one with the least rate-distortion (RD) cost using the Lagrange multiplier. These tools achieve the highest coding efficiency but incur a very high computational complexity. In this paper, a fast CU size decision algorithm is proposed to reduce the complexity caused by the CU size splitting process. The proposed algorithm is based on the CU homogeneity classification using machine learning technology. First, the tensor feature is extracted to characterize the homogeneity of CU, which has a strong relationship with CU sizes. Then, a boosted decision stump algorithm is employed to analyze and construct a binary classification model from the extracted features and find suitable thresholds for the proposed method. Finally, an efficient early termination of CU splitting is released based on adaptive thresholds for texture videos and depth maps. The experimental results show that the proposed algorithm reduces a significant encoding time, while the loss in coding efficiency is negligible.

Prediction mode grouping and coding bits grouping based on texture complexity for Fast HEVC intra-coding

HEVC (High Efficiency Video Coding) as one of the newest international video coding standard, can achieve about 50% bit rate reduction compared with H.264/AVC (Advanced Video Coding) with the same perceptual quality by the use of flexible CTU (coding tree unit) structure, but at the same time, it also dramatically adds its computational complexity for HEVC. To reduce the computational complexity, a fast intra-prediction mode and CU (Coding Unit) size decision algorithm based on prediction mode and coding bits grouping is presented for HEVC intra-encoding in this paper. The contribution of this paper lies in the fact that we successfully use the prediction mode grouping and coding bits grouping technologies to rapidly realize the optimal prediction mode and size decision for the current CU, thus saving much computation complexity for HEVC. Specifically, in our scheme, first, we use grouping technology to group 35 intra-prediction modes into 5 subsets of candidate modes list according to the texture complexity of current PU (Prediction Unit), and each subset only contains 11 intra-prediction modes, which can greatly reduce the traversing and calculating number of candidate mode in RMD (Rough Mode Decision); second, we use coding bits grouping technology to quickly judge whether the current CU needs to be further divided on the basis of the studying of texture complexity in the current CU, which can reduce many unnecessary prediction and partition operations for the current CU; at last we use the fast intra-mode prediction and CU size decision algorithm above to quickly realize the optimal encoding for the current CU in HEVC. As a result, the high computational complexity in HEVC intra-encoding can be efficiently reduced by our proposed scheme. And the simulation results of our experiments show that our proposed fast intra-coding algorithm based on prediction mode and coding bit grouping in this paper can reduce about 49.10% computational complexity on average only at a cost of 0.92% bit rate increase and 0.065 db PSNR decline compared with the standard reference HM16.1algorithm under all-intra-configuration.

A fast CU size decision algorithm for VVC intra prediction based on support vector machine

The latest generation of coding standard, Versatile Video Coding (VVC), has achieved more bitrate reduction compared with high efficiency video coding. However, the introduction of quadtree with nested Multi-Type Tree (MTT) coding structure greatly increases the computational complexity. To reduce the complexity of VVC, a Support Vector Machine (SVM) based Coding Unit (CU) size decision algorithm is presented. Firstly, effective features, derived from entropy, texture contrast, and Haar wavelet efficient of current CU, are select to distinguish the splitting directions. Then, the six SVM classifying models are on-line trained at different CU sizes. Finally, the models are utilized to prediction the direction of CU splitting in the quadtree with nested MTT coding structure. Experimental results show that the proposed algorithm can significantly save the encoding time by 51.01% with slight increase of Bjontegaard delta bit rate.

Fast CU size and mode decision algorithm for 3D-HEVC intercoding

The 3D High Efficiency Video Coding (3D-HEVC) is an extension of the last 2D video coding generation (HEVC) using the Multi-view Video plus Depth map (MVD) format. In inter-coding process, the 3D-HEVC uses multiples inter prediction modes for each Coding Unit (CU) size. These prediction modes are performed to use all the possible CU sizes to select the one with the least Rate Distortion (RD) cost as the best prediction mode using Lagrange multiplier. It achieved high coding efficiency but it consumes an extremely time coding which limits 3D-HEVC from real-time applications. In this paper, a fast inter prediction mode and CU size decision algorithm is proposed to reduce the complexity of 3D-HEVC inter-coding. This algorithm is based on the homogeneity of CU using structure tensor. An experimental analysis is first performed to study the correlation of the CU complexity with depth levels and prediction modes, then the structure tensor of each CU is extracted to characterize the CU homogeneity, which has strong relationship with coding levels and inter prediction modes. Finally, a fast algorithm is adopted to accomplish an early termination of CU split and skip unnecessary prediction modes. The experimental results demonstrate that the proposed fast algorithm can provide a considerable encoding time-saving with negligible rate-distortion performance loss compared to the original 3D-HEVC encoder.

A fast temporal and hybrid SHVC encoder

In the last few years new devices with different computational capabilities and network connections has emerged, and this leads to the need for more adaptable video streams. The last two video coding standards, namely H.264/Advanced Video Coding and High Efficiency Video Coding, have defined scalable versions of themselves that provide this adaptability by the use of several layers with different levels of quality, frame rates, and resolutions. Moreover, HEVC introduced hybrid scalability, which allows a non-HEVC base layer (e.g., H.264/AVC) in combination with HEVC enhancement layers, what provides backwards compatibility with older devices. Nevertheless, scalable video coding is very computationally expensive, making it necessary to accelerate the video encoding. This paper presents a fast Coding Unit size algorithm for the combination of hybrid and temporal scalabilities that is based on machine learning. Experimental results show that an acceleration of 54% is achieved at the cost of a slight increment in the bit rate.

Online-Learning-Based Bayesian Decision Rule for Fast Intra Mode and CU Partitioning Algorithm in HEVC Screen Content Coding

Screen content coding (SCC) is an extension of high efficiency video coding by adopting new coding modes to improve the coding efficiency of SCC at the expense of increased complexity. This paper proposes an online-learning approach for fast mode decision and coding unit (CU) size decision in SCC. To make a fast mode decision, the corner point is first extracted as a unique feature in screen content, which is an essential pre-processing step to guide Bayesian decision modeling. Second, the distinct color number in a CU is derived as another unique feature in screen content to build the precise model using online-learning for skipping unnecessary modes. Third, the correlation of the modes among spatial neighboring CUs is analyzed to further eliminate unnecessary mode candidates. Finally, the Bayesian decision rule using online-learning is applied again to make a fast CU size decision. To ensure the accuracy of the Bayesian decision models, new scene change detection is designed to update the models. Results show that the proposed algorithm achieves 36.69% encoding time reduction with 1.08% Bjøntegaard delta bitrate (BDBR) increment under all intra configuration. By integrating into the existing fast SCC approach, the proposed algorithm reduces 48.83% encoding time with a 1.78% increase in BDBR.

Fast algorithm for HEVC intra‐coding implemented by preprocessing

Compared to previous video compression standards H.264/AVC, high-efficiency video coding (HEVC) introduces a recursive quad-tree structure in intra-coding and adds intra-mode from 9 to 35 to achieve higher coding efficiency. The update significantly improves the performance of intra-coding. However, it greatly increases the amount of computation, and this will increase the need for the fast algorithm and hardware design that can satisfy the real-time encoding of HEVC encoder. In this study, the authors propose a fast intra-coding algorithm based on analysis of original pixel gradient texture. The algorithm consists of two steps, fast coding unit size decision and reduction of candidate modes. The experimental results show that compared with the HM16.7 encoder, the optimisation algorithm can obtain 50.7% time reduction on average with 1.32% Bjontegaard distortion (BD)-rate increase and 0.07 dB BD-peak-signal-to-noise ratio loss. Meanwhile, this study designs a pre-processing hardware structure based on original pixels. With TSMC 90 nm complementary metal oxide semiconductor technology, the proposed structure can achieve 625 MHz working frequency at the cost of 2094 gates, which can fulfil the throughput requirement of 8K × 4K@46 fps real-time encoding.

Run-Time Deep Learning Enhanced Fast Coding Unit Decision for High Efficiency Video Coding

The state-of-the-art high efficiency video coding (HEVC/H.265) adopts the hierarchical quadtree-structured coding unit (CU) to enhance the coding efficiency. However, the computational complexity significantly increases because of the exhaustive rate-distortion (RD) optimization process to obtain the optimal coding tree unit (CTU) partition. In this paper, we propose a fast CU size decision algorithm to reduce the heavy computational burden in the encoding process. In order to achieve this, the CU splitting process is modeled as a three-stage binary classification problem according to the CU size from [Formula: see text], [Formula: see text] to [Formula: see text]. In each CU partition stage, a deep learning approach is applied. Appropriate and efficient features for training the deep learning models are extracted from spatial and pixel domains to eliminate the dependency on video content as well as on encoding configurations. Furthermore, the deep learning framework is built as a third-party library and embedded into the HEVC simulator to speed up the process. The experiment results show the proposed algorithm can achieve significant complexity reduction and it can reduce the encoding time by 49.65%(Low Delay) and 48.81% (Random Access) on average compared with the traditional HEVC encoders with a negligible degradation (2.78% loss in BDBR, 0.145[Formula: see text]dB loss in BDPSNR for Low Delay, and 2.68% loss in BDBR, 0.128[Formula: see text]dB loss in BDPSNR for Random Access) in the coding efficiency.

An Adaptive CU Size Decision Algorithm for HEVC Intra Prediction Based on Complexity Classification Using Machine Learning

High efficiency video coding (HEVC), which is the newest video coding standard currently, achieves the best coding efficiency compared with all the other existing video coding standards. However, the computational complexity of the typical HEVC encoder dramatically increases because of the recursive searching scheme for finding the best coding unit (CU) partitions. In this paper, an adaptive fast CU size decision algorithm for HEVC Intra prediction is proposed based on CU complexity classification (CC) by using machine learning (ML) technology. Firstly, certain image features are extracted to characterize the CU complexity, which has a strong relationship with CU partitions, and then, the support vector machine is employed to analyze and construct the classification model according to the CU complexity. Finally, the proposed adaptive fast CU size decision algorithm, named as CCML, is released based on the complexity classification. The experimental results show that the proposed algorithm could achieve around 60% encoding time reduction for various test video sequences on average with only 1.26% Bjontegaard delta bit rate increase compared with the reference test model HM15.0 of HEVC.

Effective Data Driven Coding Unit Size Decision Approaches for HEVC INTRA Coding

High Efficiency Video Coding (HEVC) INTRA coding improves compression efficiency by adopting advanced coding technologies, such as multi-level quad-tree block partitioning and up to 35-mode INTRA prediction. However, it significantly increases the coding complexity, memory access, and power consumption, which goes against its widely applications, especially for ultra-high definition and/or mobile video applications. To tackle this problem, we propose effective data driven coding unit (CU) size decision approaches for HEVC INTRA coding, which consists of two stages of support vector machine-based fast INTRA CU size decision schemes at four CU decision layers. At the first stage classification, a three output classifier with offline learning is developed to early terminate the CU size decision or early skip checking the current CU depth. As for the samples that neither early skipped nor early terminated, the second stage of binary classification, which learns online from previous coded frames, is proposed to further refine the CU size decision. Representative features for the CU size decision are explored at different decision layers and stages of classifications. Finally, the optimal parameters derived from the training data are achieved to reasonably allocate complexity among different CU layers at given total rate-distortion degradation constraint. Extensive experiments show that the proposed overall algorithm can achieve 27.95%–80.53% and 52.48% on average complexity reduction for the CU size decision as compared with the original HM16.7 model. Meanwhile, the average Bjonteggard delta peak-signal-to-noise ratio degradation is only −0.08 dB, which is negligible. The overall performance of the proposed algorithm outperforms the state-of-the-art benchmark schemes.

Exploiting Pseudo-Quadtree Structure for Accelerating HEVC Spatial Resolution Downscaling Transcoder

In this paper, a novel method to accelerate the spatial resolution downscaling transcoding operation for high efficiency video coding (HEVC) is proposed. The proposed transcoder first extracts the information about the coding unit (CU) structure during the decoding process and analyzes it to construct a pseudo-quadtree of the target resolution. By utilizing the constructed pseudo-quadtree, the encoder process is accelerated by searching only the optimal depth in the quadtree structure. To evaluate the performance of the proposed transcoding method, its transcoding time and bitrate loss are measured and compared with those of the HEVC reference encoder and decoder. Also, a comparison with state-of-the-art fast CU size decision methods and spatial resolution downscaling transcoding methods for HEVC is presented. The proposed method achieves an average speedup of 1.65 with an average bitrate loss of 2.06%.

Fast CU size decision and PU mode decision algorithm for quality SHVC inter coding

As a scalable extension of the High Efficiency Video Coding (HEVC), the Scalable High Efficiency Video Coding (SHVC) encoder needs to encode multiple HEVC layers with Inter-layer predictions, which causes the significant increase in coding complexity. In this paper, we proposed a novel Inter prediction scheme to effectively reduce computational complexity in Quality SHVC. The new features of the proposed algorithm include: First, spatial and Inter-layer depth correlations are used to predict the most possible coding unit (CU) depth level candidates. Second, a statistical test method on the current CU depth level is introduced to examine whether the residual coefficients within its block present similar distribution to terminate depth selection early. Finally, during Inter prediction selection from 8 Prediction Unit (PU) sizes, spatial and Inter-layer correlations are combined with residual coefficients distribution to determine the PU partitioning mode is Symmetric Motion Partitioning (SMP) or Asymmetric Motion Partitioning (AMP). Experimental results demonstrate that the proposed algorithm can save an average of 65.33% coding time of enhancement layer (EL) while achieving a better rate distortion (RD) performance over other state-of-the-art work.

Key-point Detection based Fast CU Decision for HEVC Intra Encoding

As the most recent video coding standard, High Efficiency Video Coding (HEVC) adopts various novel techniques, including a quad-tree based coding unit (CU) structure and additional angular modes used for intra encoding. These new techniques achieve a notable improvement in coding efficiency at the penalty of significant computational complexity increase. Thus, a fast HEVC coding algorithm is highly desirable. In this paper, we propose a fast intra CU decision algorithm for HEVC to reduce the coding complexity, mainly based on a key-point detection. A CU block is considered to have multiple gradients and is early split if corner points are detected inside the block. On the other hand, a CU block without corner points is treated to be terminated when its RD cost is also small according to statistics of the previous frames. The proposed fast algorithm achieves over 62% encoding time reduction with 3.66%, 2.82%, and 2.53% BD-Rate loss for Y, U, and V components, averagely. The experimental results show that the proposed method is efficient to fast decide CU size in HEVC intra coding, even though only static parameters are applied to all test sequences.

Perceptually motivated fast coding unit size decision for HEVC intracoding based on visual regularity

Although the flexible quad-tree structure in high efficiency video coding (HEVC) has achieved significant performance improvement over H.264/AVC, it remarkably increases computation complexity due to the search for the optimal coding unit (CU) size in a very large space. We propose perceptually motivated fast CU decision for HEVC intracoding based on visual regularity. Since visual regularity represents visual masking effect in the human visual system (HVS) on structure information, i.e., irregularity concealment effect, we use it for early CU termination. We predict irregular CUs by exploiting the information of neighboring CUs, which would not introduce too much distortion. Also, the distortion cannot be perceived by HVS due to the visual masking effect in irregular regions. The experimental results demonstrate that the proposed method saves 24% computational cost on average as compared to anchor with only 0.52% increase in Bjontegaard delta bit rate and 0.02 dB loss in Bjontegaard delta PSNR.

Fast CU size decision algorithm using machine learning for HEVC intra coding

High Efficiency Video Coding (HEVC) is a state-of-the-art video compression standard which improves coding efficiency significantly compared with the previous coding standard, H.264/AVC. In the HEVC standard, novel technologies consuming massive computational power are adopted, such as quad-tree-based coding unit (CU) partitioning. Although an HEVC encoder can efficiently compress various video sequences, the computational complexity of an exhaustive search has become a critical problem in HEVC encoder implementation. In this paper, we propose a fast algorithm for the CU partitioning process of the HEVC encoder using machine learning methods. A complexity measure based on the Sobel operator and rate-distortion costs are defined as features for our algorithm. A CU size can be determined early by employing Fisher’s linear discriminant analysis and the k-nearest neighbors classifier. The statistical data used for the proposed algorithm is updated by adaptive online learning phase. The experimental results show that the proposed algorithm can reduce encoding time by approximately 54.0% with a 0.68% Bjøntegaard-Delta bit-rate increase.