Prediction Unit Research Articles

Fast intra- and inter-coding algorithms for the spatially scalable extension of H.265/HEVC

High Efficiency Video Coding (HEVC) is the latest international video coding standard. As an extension of HEVC, scalable HEVC (SHVC) achieves high compression efficiency at the cost of dramatically increased computational complexity. In this paper, a fast Coding Unit (CU) size decision algorithm is suggested for spatial SHVC intra-coding to early terminate the CU size decision process by jointly utilising the inter-layer and spatiotemporal correlations. Furthermore, in order to realise a fast Prediction Unit (PU) intra-mode decision, the mode dependency between the Base Layer (BL) and the Enhancement Layer (EL) is effectively used to remove unlikely coding modes. A fast PU inter-mode decision method is also developed for spatial SHVC inter-coding, in which the degree of motion activity and the PU mode dependency are exploited. Simulation results show that the proposed fast intra-coding algorithm produces a saving in encoding time of up to 72% and the proposed fast inter-coding scheme reduces the encoding time by up to 50% compared with the original SHVC encoder, and the loss in Rate-Distortion (RD) performance is negligible.

Algorithm optimisation and hardware implementation of interprediction mode decision

High efficiency video coding is the most widely used video coding standard. It has higher coding performance compared with its predecessor, H.264, but it also has higher computational complexity. Interprediction is the most computationally intensive part of the entire video encoding process. Selecting the optimal interprediction mode by the rate–distortion cost calculation function requires substantial complex calculation and memory access, thus greatly increasing the difficulty of real-time hardware encoding. This study proposes to replace the traditional complex error square sum calculation with an estimation method for distortion and rate. The estimation of distortion uses the Hadamard-transformed sum of absolute transformation difference instead of the complex calculation of the sum of squared difference, whereas the estimation of rate is obtained by weighting the number of prediction units (PUs). The experiment proves that the proposed interprediction rate–distortion cost calculation model can greatly reduce computational complexity when BD-rate is increased by 3.02%. In hardware implementation, the value of rate can be obtained by indexing the number of PUs, and the resource expenditure is small.

Occupancy-Map-Based Rate Distortion Optimization and Partition for Video-Based Point Cloud Compression

The video-based point cloud compression (V-PCC) is the state-of-the-art dynamic point cloud compression method developed by the Moving Pictures Experts Group (MPEG). It projects the point cloud patch by patch to its bounding box and organizes all the patches into a video to utilize the efficient video coding framework. However, the unoccupied pixels among different patches will lead to inefficiency of the video compression. First, the unoccupied pixels are treated equal to the occupied pixels in the rate distortion optimization process. However, the unoccupied pixels are useless for the reconstructed quality of the point cloud. Second, the edges of the occupied and unoccupied pixels can divide a coding unit into arbitrary shapes. Consequently, they are not well-characterized by the regular partitions supported in the video coding framework. Therefore, we propose using occupancy-map-based rate distortion optimization and partition to deal with these two problems. First, the occupancy map is used as a mask to ignore the distortions of the unoccupied pixels when calculating the rate distortion cost of a specified block. This strategy is applied to both intra prediction, inter prediction, and the sample adaptive offset to boost the performance. Second, we propose an occupancy-map-based partition to divide the occupied pixels from different patches and the unoccupied pixels into different prediction units (PUs). The motions of occupied PUs are then predicted using the auxiliary information. The unoccupied PU is finally padded using the approach that generates the geometry frames. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding reference software. The experimental results show that the algorithms can individually contribute significant performance improvements compared with the V-PCC. Additionally, their combination can achieve even more bitrate savings than the sum of the individual algorithms.

Hierarchical Classification for Complexity Reduction in HEVC Inter Coding

In order to adapt to various real-time applications, fast coding algorithms for high efficiency video coding (HEVC) standard maintain a hot research topic in recent years. In this paper, a complexity reduction algorithm based on hierarchical classification for HEVC inter coding is proposed. It consists of five fast algorithms which is accomplished by hierarchical classification trees at coding unit (CU) level, prediction unit (PU) level and transformation unit (TU) level respectively. At the beginning of proposed algorithm, intra features and inter features which describe the texture and context properties of CU, PU and TU are extracted from the training set. Then the classification trees for CU, PU and TU are generated by carefully selecting features and designing the classification criteria. By analyzing the spatiotemporal correlation, two strategies including early termination and early split are applied to fast coding by referring to these classification trees. The objective evaluation demonstrates that the proposed algorithm can significantly reduce coding complexity with little compression loss. Particularly the subjective evaluation based on similarity measurement for color histogram approves that decoded video quality between the original HM16.9 algorithm and the proposed algorithm is nearly identical.

Deep Video Prediction Network-Based Inter-Frame Coding in HEVC

In this paper, we propose a novel Convolutional Neural Network (CNN) based video coding technique using a video prediction network (VPN) to support enhanced motion prediction in High Efficiency Video Coding (HEVC). Specifically, we design a CNN VPN to generate a virtual reference frame (VRF), which is synthesized using previously coded frames, to improve coding efficiency. The proposed VPN uses two sub-VPN architectures in cascade to predict the current frame in the same time instance. The VRF is expected to have higher temporal correlation than a conventional reference frame, and, thus it is substituted for a conventional reference frame. The proposed technique is incorporated into the HEVC inter-coding framework. Particularly, the VRF is managed in a HEVC reference picture list, so that each prediction unit (PU) can choose a better prediction signal through Rate-Distortion optimization without any additional side information. Furthermore, we modify the HEVC inter-prediction mechanisms of Advanced Motion Vector Prediction and Merge modes adaptively when the current PU uses the VRF as a reference frame. In this manner, the proposed technique can exploit the PU-wise multi-hypothesis prediction techniques in HEVC. Since the proposed VPN can perform both the video interpolation and extrapolation, it can be used for Random Access (RA) and Low Delay B (LD) coding configurations. It is shown in experimental results that the proposed technique provides -2.9% and -5.7% coding gains, respectively, in RA and LD coding configurations as compared to the HEVC reference software, HM 16.6 version.

Bit Rate Transcoding for High Efficiency Video Coding

High efficiency video coding (HEVC) has demonstrated a notable increase in compression performance and is taken as a successor to H.264/AVC. Efficient bit rate adaptation algorithms are required to contain the HEVC standard between real life community facilities. A present issue of bit rate transcoding is its high computational complexity which is related with the encoder of a cascaded pixel domain transcoder. This paper gives Top to Bottom (T2B) approach to reduce complexity by using different complexity schemes. Proposed approach is effective in reducing complexity in Coding Unit (CU) optimization level. Coding Unit has been analyzed in T2B Approach. While examining the coding unit information of the input video is turned to account for decreasing the number of evaluation and early terminate the process. For the Prediction Unit (PU) level the units are powerfully chosen contingent upon likelihood of Prediction Unit sizes and co-found input prediction partitioning. By utilizing this approach, complexity scalable bit rate transcoding has achieved. Machine learning approach can be used to control computational complexity. Additionally, the T2B strategy is able to gain a spread on trade-offs in transrating complexity and coding performance. Using T2B approach 15% encoding time saving is accomplished. From this scheme, for the less resolution video 27% time saving has achieved.

Fast Hardware-Based IME With an Idle Cycle and Computational Redundancy Reduction

Extensive efforts have been made to design hardware-based integer motion estimation (IME) that is much faster than software-based IME but suffers from the degradation in the coding efficiency. This is because the strategy for previous efforts was a simple algorithmic modification of the fast IME to facilitate the given hardware design at the expense of coding efficiency. This paper proposes a novel hardware design of the IME that not only offers real-time processing capability but also provides a flexible tradeoff between computational complexity and coding efficiency. First, a prediction unit (PU) loop unrolling scheme is proposed to solve the pipeline stall problem owing to the nature of fast IME algorithms such as the test zone search (TZS). It reduces idle cycles by 89.24%. Next, to further reduce the computational complexity of the TZS algorithm, a computational redundancy among PUs within a coding unit is reduced through a search step synchronization and search point sharing scheme. Thus, the computational complexity is reduced by 72.25%. The proposed schemes eliminate the inefficiency of hardware design; thus, they do not suffer from serious degradation in the coding efficiency. Consequently, the proposed hardware-based IME processes $7680\times4320$ videos at 30 frames per second while increasing the Bjontegaard delta bitrate by only 0.90% on average. The hardware design is synthesized using a 65 nm general purpose CMOS technology, and its gate count is 268.5K at an operating clock frequency of 500 MHz.

Detection of Transcoding from H.264/AVC to HEVC Based on CU and PU Partition Types

High Efficiency Video Coding (HEVC) is a worldwide popular video coding standard due to its high coding efficiency. To make profits, forgers prefer to transcode videos from previous standards such as H.264/AVC to HEVC. To deal with this issue, an efficient method is proposed to expose such transcoded HEVC videos based on coding unit (CU) and prediction unit (PU) partition types. CU and PU partitioning are two unique syntactic units of HEVC that can reflect a video’s compression history. In this paper, CU and PU partition types of I pictures and P pictures are firstly extracted. Then, their mean frequencies are calculated and concatenated as distinguishing features, which are further sent to a support vector machine (SVM) for classification. Experimental results show that the proposed method can identify transcoded HEVC videos with high accuracy and has strong robustness against frame-deletion and shifted Group of Pictures (GOP) structure attacks.

A VVC Proposal With Quaternary Tree Plus Binary-Ternary Tree Coding Block Structure and Advanced Coding Techniques

This paper describes a video coding technology proposal submitted by MediaTek in April 2018 in response to a joint call for proposals (CfP) issued by ITU-T VCEG and ISO/IEC MPEG. The proposal uses the conventional block-based hybrid coding approach with a breakthrough design of block structure, substantial improvements of coding tools, and new inventions of loop filtering techniques. First, the block structure is designed using a highly flexible partitioning scheme, where each coding tree unit in a picture is split into multiple coding units (CUs) by a recursive quaternary tree followed by a recursive binary-ternary tree. The prediction unit (PU) and transform unit (TU) concepts are unified with the CU concept, i.e., PU and TU are basically the same as CU. Moreover, inter prediction, intra prediction, transform, quantization, in-loop filtering, and entropy coding are all improved. Finally, new loop filtering techniques are invented, where the proposed convolutional neural network loop filtering is of particular interest. The proposed video codec achieves 43.81% average bit rate savings based on objective measures compared to the High Efficiency Video Coding (HEVC) anchors and is among the best-performing CfP responses both objectively and subjectively. It has been partially adopted into the working draft of the Versatile Video Coding (VVC) standard.

Content-Based Light Field Image Compression Method With Gaussian Process Regression

Light field (LF) imaging enables new possibilities for digital imaging, such as digital refocusing, changing of focus plane, changing of viewpoint, scene-depth estimation, and 3D scene reconstruction, by capturing both spatial and angular information of light rays. However, one main problem in dealing with LF data is its sheer volume. In this context, efficient compression methods are needed for such a particular type of content. In this paper, we propose a content-based LF image-compression method with Gaussian process regression to improve the compression efficiency and accelerate the prediction procedure. First, the LF image is fed to the intra-frame codec of HEVC. In the prediction procedure, the prediction units (PUs) are classified as non-homogenous texture units, homogenous texture units, and visually flat units, based on the content property of the LF image. For each category, we design a corresponding Gaussian process regression (GPR)-based prediction method. Moreover, we propose a classification mechanism to exactly decide to which category the current PU belongs, so as to adjust the trade-off between the computational burden and the LF image coding efficiency. Experimental results demonstrate that the proposed LF image compression method is superior to several other state-of-the-art compression methods in terms of different quality metrics. Furthermore, the proposed method can also achieve a good visual quality of views rendered from decoded LF contents.

High Capacity HEVC Video Hiding Algorithm Based on EMD Coded PU Partition Modes

Data hiding in videos has been a big concern as their rich redundancy can be used for embedding a lot of secret information. Further, as high efficiency video coding (HEVC) introduces many innovative technologies compared with the previous standard, H.264, it has gradually become the mainstream. Therefore, it is valuable to develop new information hiding algorithms by using novel features of HEVC. A HEVC video data hiding algorithm based on prediction unit (PU) partition modes from inter prediction process is proposed in this paper. Firstly, code units (CUs) in two sizes of 8 × 8 and 16 × 16 are selected for embedding, then the PU partition modes in these CUs are coded by a spatial coding method. After that, two specific hiding algorithms by modifying coded PU partition modes in CUs of 8 × 8 and 16 × 16 are proposed, respectively. Experimental results show that the proposed algorithm has achieved excellent performance with high visual quality, and high embedding capacity and low bitrate increase in both high- and low-resolution videos compressed with different quantization parameters (QPs). Compared with the state-of-the-art work, the proposed algorithm achieves a much higher capacity while keeping quite high visual quality with little increase of bitrate.

Advanced 3D Motion Prediction for Video Based Dynamic Point Cloud Compression.

Point cloud based immersive media representation format has provided many opportunities for extended reality applications and has become widely used in volumetric content capturing scenarios. The high data rate of the point cloud is one of the key problems preventing the adoption of this media format. MPEG Immersive media working group (MPEG-I) aims to create a point cloud compression methodology relying on the existing video coding hardware implementations to solve this problem. However, in the scope of the state-of-the-art video-based dynamic point cloud compression (V-PCC) standard under MPEG-I, the intrinsic 3D object's motion continuity is destroyed by the 2D projections resulting in a significant loss of inter prediction coding efficiency. In this paper, we first propose a general model utilizing the 3D motion and 3D to 2D correspondence to calculate the 2D motion vector (MV). Then under the V-PCC, we propose a geometry-based method using the accurate 3D reconstructed geometry from the 2D geometry video to estimate the 2D MV in the 2D attribute video. In addition, we propose an auxiliary-information-based method using the coarse 3D reconstructed geometry provided by the auxiliary information to estimate the 2D MV in both the 2D geometry and attribute videos. Furthermore, we provide the following two ways to use the estimated 2D MV to improve the coding efficiency. The first one is normative. We propose adding the estimated MV into the advanced motion vector candidate list and find a better motion vector predictor for each prediction unit (PU). The second one is non-normative. We propose applying the estimated MV as an additional candidate of the centers for motion estimation. We implement the proposed algorithms in the V-PCC reference software. The experimental results show that the proposed methods present significant coding gains compared with the current state-of-the-art motion prediction algorithm.

HEVC double compression detection under different bitrates based on TU partition type

During the process of authenticating the integrity of digital videos, double compression is an important evidence. High-efficiency video coding (HEVC) is the latest coding standard proposed in 2013 and has shown superior performance over its predecessors. It brings in several novel syntactic units, such as coding tree unit (CTU), prediction unit (PU), and transform unit (TU). Few methods have been reported to detect HEVC double compression by utilizing such new characteristics. In this paper, a novel scheme based on TU is proposed to detect double compression of HEVC videos. The histogram of each TU partition type in the first I/P frame of all GOPs is calculated. The feature set is fed into SVM to classify single and double compressed videos. Experimental results have demonstrated the effectiveness of TU-based feature (accuracy in [0.84, 0.97]) that bears very low dimension (10-D). And it is also shown that TU-based feature set can be combined with other feature sets to boost their performance.

Hybrid stopping model-based fast PU and CU decision for 3D-HEVC texture coding

As an extension of High-Efficiency Video Coding (HEVC) standard, 3D-HEVC needs to encode multiple texture views and depth maps, which further increases the computational complexity. To reduce the complexity of dependent texture view coding, a fast prediction unit (PU) and coding unit (CU) decision method is proposed for 3D-HEVC based on hybrid stopping model. The inter-view correlation is used as a priori information to roughly predict the possible optimal PU and CU sizes. Then, by exploiting the encoded posterior information, the rate distortion cost correlation and the code block flag, the optimal PU and CU are further examined as being optimal or not. Experimental results show that the proposed fast PU and CU decision method achieves 52.7% encoding time saving on average with negligible loss of coding efficiency for 3D-HEVC-dependent texture view coding.

A Motion-Based Partitioning Algorithm for HEVC Using a Pre-Analysis Stage

The High Efficiency Video Coding (HEVC) standard has shown large improvements in coding efficiency compared with previous standards. In particular, HEVC outperforms H.264/MPEG-4 Advanced Video Coding by up to 50% in terms of bitrate reduction for similar perceptual quality. This improvement is the result of the introduction of new coding tools, which enable the representation of data using fewer bits, but at the cost of long computation times. One of the most significant tools introduced by HEVC is the novel quadtree-based structure called the coding tree unit (CTU), which can be subsequently split into coding units (CUs), prediction units (PUs), and transform units (TUs), providing huge flexibility in the encoding. However, selecting the optimal tree partitioning requires the evaluation of a huge number of possibilities, which constitutes the most complex operation for the encoder. In order to tackle this task, this paper proposes a CTU/CU partitioning algorithm based on a pre-analysis stage. This stage performs a fast motion estimation that provides preliminary information to the encoder, including estimate distortion costs, which enables the building of the least-cost quadtree. On the basis of this tree, the evaluation performed by the encoder is restricted to a subset of PUs. After a thorough statistical analysis of numerous CU/PU combinations, we propose two different configurations that prioritize each of the target variables: coding efficiency and time reduction. In the former case, 58.09% of the encoding time can be saved at the cost of a 2.51% increase in Bjontegaard delta (BD)-rate, while a larger reduction of 63.08% is achieved with a 3.43% increase in BD-rate in the later case.

GPU-Based Hierarchical Motion Estimation for High Efficiency Video Coding

Motion estimation (ME) plays a crucial role in removing the temporal redundancy for video compression. However, during the encoding process a substantial computational burden is imposed by ME due to the exhaustive evaluations of possible candidates within the searching window. In view of the increasing computing capacity of GPU, we propose a GPU-based low delay parallel ME scheme for high efficiency video coding (HEVC). In particular, considering the quadtree coding structure of HEVC, we achieve the parallelization in a hierarchical way by optimizing the ME process in a coding tree unit (CTU), prediction unit (PU), and motion vector (MV) layers. Specifically, in the CTU layer, a novel motion vector predictor determination scheme is proposed to alleviate the side effects of inaccurate MV prediction due to the removal of the CTU-level dependency. In the PU layer, a novel indexing table is particularly designed to realize an efficient cost derivation strategy. As such, the cost of each PU can be computed in a convenient and efficient manner. In an MV layer, we propose a compact descriptor to represent MV and its corresponding cost as a whole, such that the redundant branches can be further avoided in the searching process. With such an optimization strategy, the proposed scheme can completely save the encoding time for ME on CPU. Experimental results demonstrate that the proposed scheme can achieve 41% encoding time savings with the ME acceleration up to 12.7 times, and the incurred BD-BR loss is only 0.52% on average. Moreover, further experimental results show that the proposed GPU-based ME can achieve up to 200 times acceleration compared to the full search ME on CPU.

A New Rate-Complexity-Distortion Model for Fast Motion Estimation Algorithm in HEVC

In the high efficiency video coding (HEVC) standard, motion estimation (ME) adopts a quadtree coding structure and a larger search range to improve the coding performance. These advanced coding tools, however, dramatically increase the computational complexity. To accelerate ME, fast methods have been proposed that reduce ME complexity at the expense of rate-distortion (R-D) performance. However, none of these methods can claim their tradeoff to be optimal. In this paper, we propose an optimal fast motion estimation (FME) algorithm based on an analytical model of rate-complexity-distortion (R-C-D). We extend the traditional R-D model by introducing the ME complexity into it, which enables us to explicitly express the R-D performance under different complexity budgets. Based on the R-C-D model, the proposed FME finds the R-C-D optimized search ranges for some representative prediction units (PUs), which are then extended or refined dynamically according to motion characteristics for neighboring PUs. The proposed FME enables an R-D performance close to that of full search. When compared with the default FME method in the reference software of HEVC, the proposed fast algorithm can reduce the complexity by over $80\%$ while improving the R-D performance. Furthermore, our proposed FME algorithm is hardware friendly, as regular data flow enables high data reuse efficiency.

Complexity Control in the HEVC Intracoding for Industrial Video Applications

A large number of industrial video applications are expected to work in real-time and power-constrained scenarios. For such applications, the coding complexity has a great impact on their performance. In this paper, we propose a complexity control method in the high-efficiency video coding intracoding to facilitate these video applications. The proposed method is performed on the coding tree unit (CTU) level, which consists of three steps, namely complexity estimation, complexity allocation, and prediction unit (PU) adaption. In the first step, a complexity estimation model is proposed to estimate the coding complexity of each CTU based on the sum of absolute transformed difference. Then, the complexity budget is allocated to each CTU proportionally to its estimated coding complexity. In PU adaption, only a subset of PU sizes are selected for the CTU according to its allocated complexity and prediction performance. A feedback-based error elimination scheme removes the complexity error during the encoding process. Experimental results show that the proposed method is able to adjust the complexity ratio from 100% to 20%. Meanwhile, the rate-distortion performance and complexity control accuracy of the proposed method are superior to those of the state-of-the-art methods.

Exposing Video Compression History by Detecting Transcoded HEVC Videos from AVC Coding

The analysis of video compression history is one of the important issues in video forensics. It can assist forensics analysts in many ways, e.g., to determine whether a video is original or potentially tampered with, or to evaluate the real quality of a re-encoded video, etc. In the existing literature, however, there are very few works targeting videos in HEVC format (the most recent standard), especially for the issue of the detection of transcoded videos. In this paper, we propose a novel method based on the statistics of Prediction Units (PUs) to detect transcoded HEVC videos from AVC format. According to the analysis of the footprints of HEVC videos, the frequencies of PUs (whether in symmetric patterns or not) are distinguishable between original HEVC videos and transcoded ones. The reason is that previous AVC encoding disturbs the PU partition scheme of HEVC. Based on this observation, a 5D and a 25D feature set are extracted from I frames and P frames, respectively, and are combined to form the proposed 30D feature set, which is finally fed to an SVM classifier. To validate the proposed method, extensive experiments are conducted on a dataset consisting of CIF ( 352 × 288 ) and HD 720p videos with a diversity of bitrates and different encoding parameters. Experimental results show that the proposed method is very effective at detecting transcoded HEVC videos and outperforms the most recent work.

Detection of Double Compressed HEVC Videos Using GOP-Based PU Type Statistics

Double compression detection is of great significance to recover the compression history of suspicious videos and help analyzers locate tampering points in forged videos. However, the lack of efficient methods still haunts researchers when it comes to double HEVC compression detection with a shifted group of pictures (GOP) structures. To tackle this problem, we first analyze the properties of re-encoded frames in double compressed HEVC videos. It is found that the abnormal statistics of prediction unit (PU) types in relocated I-frames can be used as the clue to expose double compression. Relocated I-frames are re-encoded P-frames, which were I-frames in the first compression. Different from the existing works, we designed the GOP-based features instead of frame-based features, enabling more efficient utilization of temporal variation patterns of PU-type statistics within each GOP unit. In the proposed method, the ratio of I-PUs and S-Pus is first computed to construct PU sequences for each GOP unit. The three-tap median filter is applied to PU sequences in order to mitigate the influence of various video contents. Several first-order statistics, including average, variation, and maximum/minimum, are selected to generate a 6-D detection feature. Then, multi-layer perceptron (MLP) is used to classify GOP-based features. Finally, the score fusion strategy is applied to obtain the detection result of the input video. Several publicly available YUV sequences are used to construct the double compression dataset with various coding parameters. In experiments, the proposed method outperforms several state-of-the-art methods under different settings of bitrates and GOP structures. The proposed method has more robust detection capability even when tampered videos are generated by combining two video clips with different GOP sizes.