Coding Unit Size Research Articles

Online-Learning-Based Bayesian Decision Rule for Fast Intra Mode and CU Partitioning Algorithm in HEVC Screen Content Coding

Screen content coding (SCC) is an extension of high efficiency video coding by adopting new coding modes to improve the coding efficiency of SCC at the expense of increased complexity. This paper proposes an online-learning approach for fast mode decision and coding unit (CU) size decision in SCC. To make a fast mode decision, the corner point is first extracted as a unique feature in screen content, which is an essential pre-processing step to guide Bayesian decision modeling. Second, the distinct color number in a CU is derived as another unique feature in screen content to build the precise model using online-learning for skipping unnecessary modes. Third, the correlation of the modes among spatial neighboring CUs is analyzed to further eliminate unnecessary mode candidates. Finally, the Bayesian decision rule using online-learning is applied again to make a fast CU size decision. To ensure the accuracy of the Bayesian decision models, new scene change detection is designed to update the models. Results show that the proposed algorithm achieves 36.69% encoding time reduction with 1.08% Bjøntegaard delta bitrate (BDBR) increment under all intra configuration. By integrating into the existing fast SCC approach, the proposed algorithm reduces 48.83% encoding time with a 1.78% increase in BDBR.

Fast algorithm for HEVC intra‐coding implemented by preprocessing

Compared to previous video compression standards H.264/AVC, high-efficiency video coding (HEVC) introduces a recursive quad-tree structure in intra-coding and adds intra-mode from 9 to 35 to achieve higher coding efficiency. The update significantly improves the performance of intra-coding. However, it greatly increases the amount of computation, and this will increase the need for the fast algorithm and hardware design that can satisfy the real-time encoding of HEVC encoder. In this study, the authors propose a fast intra-coding algorithm based on analysis of original pixel gradient texture. The algorithm consists of two steps, fast coding unit size decision and reduction of candidate modes. The experimental results show that compared with the HM16.7 encoder, the optimisation algorithm can obtain 50.7% time reduction on average with 1.32% Bjontegaard distortion (BD)-rate increase and 0.07 dB BD-peak-signal-to-noise ratio loss. Meanwhile, this study designs a pre-processing hardware structure based on original pixels. With TSMC 90 nm complementary metal oxide semiconductor technology, the proposed structure can achieve 625 MHz working frequency at the cost of 2094 gates, which can fulfil the throughput requirement of 8K × 4K@46 fps real-time encoding.

Run-Time Deep Learning Enhanced Fast Coding Unit Decision for High Efficiency Video Coding

The state-of-the-art high efficiency video coding (HEVC/H.265) adopts the hierarchical quadtree-structured coding unit (CU) to enhance the coding efficiency. However, the computational complexity significantly increases because of the exhaustive rate-distortion (RD) optimization process to obtain the optimal coding tree unit (CTU) partition. In this paper, we propose a fast CU size decision algorithm to reduce the heavy computational burden in the encoding process. In order to achieve this, the CU splitting process is modeled as a three-stage binary classification problem according to the CU size from [Formula: see text], [Formula: see text] to [Formula: see text]. In each CU partition stage, a deep learning approach is applied. Appropriate and efficient features for training the deep learning models are extracted from spatial and pixel domains to eliminate the dependency on video content as well as on encoding configurations. Furthermore, the deep learning framework is built as a third-party library and embedded into the HEVC simulator to speed up the process. The experiment results show the proposed algorithm can achieve significant complexity reduction and it can reduce the encoding time by 49.65%(Low Delay) and 48.81% (Random Access) on average compared with the traditional HEVC encoders with a negligible degradation (2.78% loss in BDBR, 0.145[Formula: see text]dB loss in BDPSNR for Low Delay, and 2.68% loss in BDBR, 0.128[Formula: see text]dB loss in BDPSNR for Random Access) in the coding efficiency.

Fast 3D-HEVC PU size decision algorithm for depth map intra-video coding

High-Efficiency Video Coding (HEVC)-based 3D video coding (3D-HEVC) is the most recent standard and last exertion of ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) for 3D video coding using a new data video format called Multi-View Video plus Depth map (MVD). This new standard achieves a high coding improvement. In any case, one of the most critical difficulties in 3D-HEVC is time computational complexity. The depth map intra-prediction is a critical factor in 3D-HEVC intra-coding, in which, the 3D-HEVC uses a highly adaptable Coding Unit (CU) structure with a specific end goal to expand the coding efficiency of all depth map characteristics. However, it results in an enormous Rate Distortion Optimization Cost (RDO-Cost) because of the broad recursive search for the best CU size from $$64\times 64$$ down to $$4\times 4$$ . This computational complexity excludes the 3D-HEVC from true and real-time application. Hence, it is imperative to build up an algorithm to diminish the complexity of the size decision in depth map intra-coding. To determine the previously mentioned issue, this paper proposes an effective 3D-HEVC PU size decision algorithm for depth map intra-video coding based on tensor features and statistical data analyses. The experimental results demonstrate that the proposed model diminishes the complexity of depth map size decision significantly with low rate distortion increase.

Low-Complexity Scalable Extension of the High-Efficiency Video Coding (SHVC) Encoding System

The scalable extension of the high-efficiency video coding (SHVC) system adopts a hierarchical quadtree-based coding unit (CU) that is suitable for various texture and motion properties of videos. Currently, the test model of SHVC identifies the optimal CU size by performing an exhaustive quadtree depth-level search, which achieves a high compression efficiency at a heavy cost in terms of the computational complexity. However, many interactive multimedia applications, such as remote monitoring and video surveillance, which are sensitive to time delays, have insufficient computational power for coding high-definition (HD) and ultra-high-definition (UHD) videos. Therefore, it is important, yet challenging, to optimize the SHVC coding procedure and accelerate video coding. In this article, we propose a fast CU quadtree depth-level decision algorithm for inter-frames on enhancement layers that is based on an analysis of inter-layer, spatial, and temporal correlations. When motion/texture properties of coding regions can be identified early, a fast algorithm can be designed for adapting CU depth-level decision procedures to video contents and avoiding unnecessary computations during CU depth-level traversal. The proposed algorithm determines the motion activity level at the treeblock size of the hierarchical quadtree by utilizing motion vectors from its corresponding blocks at the base layer. Based on the motion activity level, neighboring encoded CUs that have larger correlations are preferentially selected to predict the optimal depth level of the current treeblock. Finally, two parameters, namely, the motion activity level and the predicted CU depth level, are used to identify a subset of candidate CU depth levels and adaptively optimize CU depth-level decision processes. The experimental results demonstrate that the proposed scheme can run approximately three times faster than the most recent SHVC reference software, with a negligible loss of compression efficiency. The proposed scheme is efficient for all types of scalable video sequences under various coding conditions and outperforms state-of-the-art fast SHVC and HEVC algorithms. Our scheme is a suitable candidate for interactive HD/UHD video applications that are expected to operate in real-time and power-constrained scenarios.

A Fast Intra Mode Selection Algorithm Based on CU Size for Virtual Reality 360∘ Video

Virtual Reality (VR) 360∘ video is spherical, and has ultra-high definition. It is time-consuming to code it using High Efficient Video Coding (HEVC) directly. In this paper, a fast intra mode selection algorithm based on Coding Unit (CU) size is proposed for VR 360∘ video in Cubemap projection (CMP) format. A redefined mode selection strategy for two-stage Rough mode decision (RMD) is designed based on different depth CUs to reduce coding time during intra prediction process. The experimental results show that the proposed algorithm brings 27.70% saving in average coding time at the cost of only 0.8% Bjontegaard delta rate increase in All-Intra mode.

Hybrid stopping model-based fast PU and CU decision for 3D-HEVC texture coding

As an extension of High-Efficiency Video Coding (HEVC) standard, 3D-HEVC needs to encode multiple texture views and depth maps, which further increases the computational complexity. To reduce the complexity of dependent texture view coding, a fast prediction unit (PU) and coding unit (CU) decision method is proposed for 3D-HEVC based on hybrid stopping model. The inter-view correlation is used as a priori information to roughly predict the possible optimal PU and CU sizes. Then, by exploiting the encoded posterior information, the rate distortion cost correlation and the code block flag, the optimal PU and CU are further examined as being optimal or not. Experimental results show that the proposed fast PU and CU decision method achieves 52.7% encoding time saving on average with negligible loss of coding efficiency for 3D-HEVC-dependent texture view coding.

Homogeneity-based fast CU partitioning algorithm for HEVC intra coding

High Efficiency Video Coding (HEVC) is a new video coding standard released as a successor for H.264/AVC. It expected to reduce by 50% the bitrate for the same perceptual quality. One of the major contributors to the higher compression performance of HEVC is the introduction of larger Coding Units (CU) with recursive partitioning mechanisms. This encoding performance is accompanied by a high computational complexity, which makes it very difficult to achieve real-time encoding especially if we aim to implement this encoder on embedded platforms. In this context, this paper suggests a fast CU partitioning algorithm for Intra-only (All Intra) configuration. The proposal aims to early terminate CU partitioning for homogeneous regions in the video frame, or skip some depths for high textured regions. The decision of Split/Non-split is based on homogeneity classification algorithm, which allows us to avoid the test of the all depths in order to determine the best CU size. Experimental results confirm that the proposed approach can reduce up to 41% of encoding time in average for different video classes and can reach up to 58% for high homogenous texture video sequences.

An Adaptive CU Size Decision Algorithm for HEVC Intra Prediction Based on Complexity Classification Using Machine Learning

High efficiency video coding (HEVC), which is the newest video coding standard currently, achieves the best coding efficiency compared with all the other existing video coding standards. However, the computational complexity of the typical HEVC encoder dramatically increases because of the recursive searching scheme for finding the best coding unit (CU) partitions. In this paper, an adaptive fast CU size decision algorithm for HEVC Intra prediction is proposed based on CU complexity classification (CC) by using machine learning (ML) technology. Firstly, certain image features are extracted to characterize the CU complexity, which has a strong relationship with CU partitions, and then, the support vector machine is employed to analyze and construct the classification model according to the CU complexity. Finally, the proposed adaptive fast CU size decision algorithm, named as CCML, is released based on the complexity classification. The experimental results show that the proposed algorithm could achieve around 60% encoding time reduction for various test video sequences on average with only 1.26% Bjontegaard delta bit rate increase compared with the reference test model HM15.0 of HEVC.

Fast prediction for quality scalability of High Efficiency Video Coding Scalable Extension

In response to the increased demand for high-resolution video, the new generation of video standards, High Efficiency Video Coding (HEVC) and its scalable extension (SHVC) have been finalized. The compression of HEVC/SHVC is efficiently improved and supports ultra-high resolution (UHD). Therefore, the coding complexity of HEVC/SHVC is much higher than those of previous standards. The framework of SHVC is based on HEVC and is divided into several types of scalable video. SHVC can be decoded into various video resolutions, frame rates and qualities, and only needs to be encoded once, but with higher complexity than HEVC. Thus, how to reduce the coding complexity of SHVC is the purpose of this paper. Our proposed algorithm accelerates the enhancement layer (EL) prediction by utilizing encoded Coding Unit (CU) sizes, prediction modes, motion vectors and Rate-Distortion Costs (RD-Costs) of the base layer (BL) and encoded CU sizes of the enhancement layer for quality scalability of SHVC. Experimental results show that the proposed algorithm can save lots of time while maintaining good video quality, and the performance is better than those of previous works.

Efficient early termination and bypass scheme for CU size decision in spatially scalable SHVC intra coding

As an extension of the High Efficiency Video Coding standard, Scalable High-efficiency Video Coding (SHVC) has been developed to satisfy the challenges posed by heterogeneous environments. A layer-based scheme is employed to support various forms of scalability, but this results in significantly increased computational complexity. A fast Coding Unit (CU) size decision algorithm is proposed to alleviate the additional computational requirement of the Enhancement Layers (ELs). The inter-layer dependency and the spatial correlation are jointly used to skip and early terminate the CU size decision process. In addition, the quantitive relationship between the texture homogeneity and the AC coefficient energy is investigated to further narrow the depth range to be evaluated. Experimental results show that the proposed scheme saves encoding time by up to 72% compared with the SHM 12.0 implementation, with negligible degradation in Rate-Distortion performance.

Effective Data Driven Coding Unit Size Decision Approaches for HEVC INTRA Coding

High Efficiency Video Coding (HEVC) INTRA coding improves compression efficiency by adopting advanced coding technologies, such as multi-level quad-tree block partitioning and up to 35-mode INTRA prediction. However, it significantly increases the coding complexity, memory access, and power consumption, which goes against its widely applications, especially for ultra-high definition and/or mobile video applications. To tackle this problem, we propose effective data driven coding unit (CU) size decision approaches for HEVC INTRA coding, which consists of two stages of support vector machine-based fast INTRA CU size decision schemes at four CU decision layers. At the first stage classification, a three output classifier with offline learning is developed to early terminate the CU size decision or early skip checking the current CU depth. As for the samples that neither early skipped nor early terminated, the second stage of binary classification, which learns online from previous coded frames, is proposed to further refine the CU size decision. Representative features for the CU size decision are explored at different decision layers and stages of classifications. Finally, the optimal parameters derived from the training data are achieved to reasonably allocate complexity among different CU layers at given total rate-distortion degradation constraint. Extensive experiments show that the proposed overall algorithm can achieve 27.95%–80.53% and 52.48% on average complexity reduction for the CU size decision as compared with the original HM16.7 model. Meanwhile, the average Bjonteggard delta peak-signal-to-noise ratio degradation is only −0.08 dB, which is negligible. The overall performance of the proposed algorithm outperforms the state-of-the-art benchmark schemes.

Exploiting Pseudo-Quadtree Structure for Accelerating HEVC Spatial Resolution Downscaling Transcoder

In this paper, a novel method to accelerate the spatial resolution downscaling transcoding operation for high efficiency video coding (HEVC) is proposed. The proposed transcoder first extracts the information about the coding unit (CU) structure during the decoding process and analyzes it to construct a pseudo-quadtree of the target resolution. By utilizing the constructed pseudo-quadtree, the encoder process is accelerated by searching only the optimal depth in the quadtree structure. To evaluate the performance of the proposed transcoding method, its transcoding time and bitrate loss are measured and compared with those of the HEVC reference encoder and decoder. Also, a comparison with state-of-the-art fast CU size decision methods and spatial resolution downscaling transcoding methods for HEVC is presented. The proposed method achieves an average speedup of 1.65 with an average bitrate loss of 2.06%.

Fast CU size decision and PU mode decision algorithm for quality SHVC inter coding

As a scalable extension of the High Efficiency Video Coding (HEVC), the Scalable High Efficiency Video Coding (SHVC) encoder needs to encode multiple HEVC layers with Inter-layer predictions, which causes the significant increase in coding complexity. In this paper, we proposed a novel Inter prediction scheme to effectively reduce computational complexity in Quality SHVC. The new features of the proposed algorithm include: First, spatial and Inter-layer depth correlations are used to predict the most possible coding unit (CU) depth level candidates. Second, a statistical test method on the current CU depth level is introduced to examine whether the residual coefficients within its block present similar distribution to terminate depth selection early. Finally, during Inter prediction selection from 8 Prediction Unit (PU) sizes, spatial and Inter-layer correlations are combined with residual coefficients distribution to determine the PU partitioning mode is Symmetric Motion Partitioning (SMP) or Asymmetric Motion Partitioning (AMP). Experimental results demonstrate that the proposed algorithm can save an average of 65.33% coding time of enhancement layer (EL) while achieving a better rate distortion (RD) performance over other state-of-the-art work.

Key-point Detection based Fast CU Decision for HEVC Intra Encoding

As the most recent video coding standard, High Efficiency Video Coding (HEVC) adopts various novel techniques, including a quad-tree based coding unit (CU) structure and additional angular modes used for intra encoding. These new techniques achieve a notable improvement in coding efficiency at the penalty of significant computational complexity increase. Thus, a fast HEVC coding algorithm is highly desirable. In this paper, we propose a fast intra CU decision algorithm for HEVC to reduce the coding complexity, mainly based on a key-point detection. A CU block is considered to have multiple gradients and is early split if corner points are detected inside the block. On the other hand, a CU block without corner points is treated to be terminated when its RD cost is also small according to statistics of the previous frames. The proposed fast algorithm achieves over 62% encoding time reduction with 3.66%, 2.82%, and 2.53% BD-Rate loss for Y, U, and V components, averagely. The experimental results show that the proposed method is efficient to fast decide CU size in HEVC intra coding, even though only static parameters are applied to all test sequences.

Real-Time Moving Object Segmentation and Classification From HEVC Compressed Surveillance Video

Moving object segmentation and classification from compressed video plays an important role in intelligent video surveillance. Compared with H.264/AVC, High Efficiency Video Coding (HEVC) introduces a host of new coding features that can be further exploited for moving object segmentation and classification. In this paper, we present a real-time approach to segment and classify moving objects using unique features directly extracted from the HEVC compressed domain for video surveillance. In the proposed method, first, motion vector (MV) interpolation for intra-coded prediction unit (PU) and MV outlier removal are employed for preprocessing. Second, blocks with nonzero MVs are clustered into the connected foreground regions using the four-connectivity component labeling algorithm. Third, object region tracking based on temporal consistency is applied to the connected foreground regions to remove the noise regions. The boundary of moving object region is further refined by the coding unit size and PU size. Finally, a person–vehicle classification model using bag of spatial–temporal HEVC syntax words is trained to classify the moving objects, either persons or vehicles. The experimental results demonstrate that the proposed method provides solid performance and can classify moving persons and vehicles accurately.

Perceptually motivated fast coding unit size decision for HEVC intracoding based on visual regularity

Although the flexible quad-tree structure in high efficiency video coding (HEVC) has achieved significant performance improvement over H.264/AVC, it remarkably increases computation complexity due to the search for the optimal coding unit (CU) size in a very large space. We propose perceptually motivated fast CU decision for HEVC intracoding based on visual regularity. Since visual regularity represents visual masking effect in the human visual system (HVS) on structure information, i.e., irregularity concealment effect, we use it for early CU termination. We predict irregular CUs by exploiting the information of neighboring CUs, which would not introduce too much distortion. Also, the distortion cannot be perceived by HVS due to the visual masking effect in irregular regions. The experimental results demonstrate that the proposed method saves 24% computational cost on average as compared to anchor with only 0.52% increase in Bjontegaard delta bit rate and 0.02 dB loss in Bjontegaard delta PSNR.

Separable authentication in encrypted HEVC video

A joint encryption and authentication scheme for HEVC compressed video is proposed in this work. It produces a HEVC format compliant video stream that permits the authentication process to be carried out irregardless of the video being in the encrypted or plaintext (i.e., decrypted) form. To achieve this separable property, one set of syntax elements within the HEVC standard is utilized to achieve authentication, while another set is exploited for encryption. Specifically, for authentication, information such as coding unit size in each video slice is utilized to generate the authentication code, which is then embedded into each video slice to detect the tampered regions. A two-level authentication method is implemented to facilitate the legitimacy verification process. On the other hand, for video encryption, syntax elements in the HEVC standard including Sign Bins, Transform Skip Bins and Suffix Bins are randomized to perceptually distort the video. Experiment results show that by using the proposed encryption modules, the perceptual quality of the plaintext video can be distorted, and the distorted video can be restored to the original HEVC compressed video. Furthermore, the proposed joint scheme is verified to be viable in detecting and localizing the tampered regions. Finally, a pragmatic comparison among the proposed and conventional joint schemes is performed.

Efficient CU and PU Decision Based on Neural Network and Gray Level Co-Occurrence Matrix for Intra Prediction of Screen Content Coding

Screen content coding (SCC) significantly improves the screen content compression efficiency over the high efficient video coding but at the cost of extremely huge computational complexity. The flexible quad-tree coding unit (CU) partitioning structure and the newly-introduced SCC intra modes are largely responsible for the high computational complexity. In order to meet the challenge of huge computational complexity of SCC, we propose an efficient CU and prediction unit (PU) decision based on neural network (NN) and gray level co-occurrence matrix (GLCM) for SCC intra prediction. The proposed efficient SCC intra prediction algorithm contains three stages, including NN-based CU classification model, efficient PU mode decision based on classification (EPMD), and efficient CU size decision based on GLCM and spatiotemporal correlations (ECSD). Consequently, the computational complexity of SCC intra prediction can be drastically reduced by replacing the brute-force search with EPMD and ECSD to decide the optimal combination of CU size and PU mode. In addition, an online updating method of weighted factor is introduced to cope with different characteristics of test sequences. In order to achieve a good tradeoff between complexity reduction and rate distortion (RD) performance, extensive experiments are conducted to select the optimal threshold for ECSD. The experimental results show that in comparison with the original SCC reference software, the proposed algorithm can reduce 49.33% intra coding time with 1.36% BDBR increase and 0.13 dB BDPSNR decrease. Meanwhile, the proposed algorithm outperforms eight state-of-the-art algorithms in terms of computational complexity reduction and RD performance.

Fast CU size decision algorithm using machine learning for HEVC intra coding

High Efficiency Video Coding (HEVC) is a state-of-the-art video compression standard which improves coding efficiency significantly compared with the previous coding standard, H.264/AVC. In the HEVC standard, novel technologies consuming massive computational power are adopted, such as quad-tree-based coding unit (CU) partitioning. Although an HEVC encoder can efficiently compress various video sequences, the computational complexity of an exhaustive search has become a critical problem in HEVC encoder implementation. In this paper, we propose a fast algorithm for the CU partitioning process of the HEVC encoder using machine learning methods. A complexity measure based on the Sobel operator and rate-distortion costs are defined as features for our algorithm. A CU size can be determined early by employing Fisher’s linear discriminant analysis and the k-nearest neighbors classifier. The statistical data used for the proposed algorithm is updated by adaptive online learning phase. The experimental results show that the proposed algorithm can reduce encoding time by approximately 54.0% with a 0.68% Bjøntegaard-Delta bit-rate increase.