Prediction Mode Research Articles

Fast CU size and mode decision algorithm for 3D-HEVC intercoding

The 3D High Efficiency Video Coding (3D-HEVC) is an extension of the last 2D video coding generation (HEVC) using the Multi-view Video plus Depth map (MVD) format. In inter-coding process, the 3D-HEVC uses multiples inter prediction modes for each Coding Unit (CU) size. These prediction modes are performed to use all the possible CU sizes to select the one with the least Rate Distortion (RD) cost as the best prediction mode using Lagrange multiplier. It achieved high coding efficiency but it consumes an extremely time coding which limits 3D-HEVC from real-time applications. In this paper, a fast inter prediction mode and CU size decision algorithm is proposed to reduce the complexity of 3D-HEVC inter-coding. This algorithm is based on the homogeneity of CU using structure tensor. An experimental analysis is first performed to study the correlation of the CU complexity with depth levels and prediction modes, then the structure tensor of each CU is extracted to characterize the CU homogeneity, which has strong relationship with coding levels and inter prediction modes. Finally, a fast algorithm is adopted to accomplish an early termination of CU split and skip unnecessary prediction modes. The experimental results demonstrate that the proposed fast algorithm can provide a considerable encoding time-saving with negligible rate-distortion performance loss compared to the original 3D-HEVC encoder.

Efficient drone hijacking detection using two-step GA-XGBoost

Abstract With the fast growth of civilian drones, their security problems meet significant challenges. A commercial drone may be hijacked by Global Positioning System (GPS)-spoofing attacks for illegal activities, such as terrorist attacks. Ideally, comparing positions respectively estimated by GPS and Inertial Navigation System (INS) can detect such attacks, while the results may always get fault because of the accumulated errors over time in INS. Therefore, in this paper, we propose a two-step GA-XGBoost method to detect GPS-spoofing attacks that just uses GPS and Inertial Measurement Unit (IMU) data. However, tunning the proper values of XGBoost parameters directly on the drone to achieve high prediction results consumes lots of resources which would influence the real-time performance of the drone. The proposed method separates the training phase into offboard step and onboard step. In offboard step, model is first trained by flight logs, and the training parameter values are automatically tuned by Genetic Algorithm (GA). Once the offboard model is trained, it could be uploaded to drones. To adapt our method to drones with different types of sensors and improve the correctness of prediction results, in onboard step, the model is further trained when a drone starts a mission. After onboard training finishes, the proposed method switches to the prediction mode. Besides, our method does not require any extra onboard hardware. The experiments with a real quadrotor drone also show the detection correctness is 96.3% and 100% in hijacked and non-hijacked cases at each sampling time respectively. Moreover, our method can achieve 100% detection correctness just within 1 s just after the attacks start.

A Semi-fragile Video Watermarking Algorithm Based On Chromatic Residual DCT

In this paper, a semi fragile video watermarking scheme based on chromatic DCT is proposed, which can protect copyright and tamper detection simultaneously. Firstly, through the experiment, it is found that the chroma block has a more stable prediction mode relative to the luminance block, which can effectively reduce the asynchronous probability brought by the change of the prediction mode. At the same time, blocks contains more the number of nonzero (NNZ) coefficients, the less the prediction mode changes. Therefore, for each macroblock, the algorithm sort its 4 × 4 sub macroblock according to the NNZ of chroma DCT coefficients, then select the sub block which has most NNZ residual coefficients. According to the secret key K and medium frequency stability of Intra 4 × 4 coefficients, modulate the relationship of three DCT coefficients near medium frequency. Secondly, the sensitivity of the prediction mode to malicious attacks and recompression operations is different. Therefore, the algorithm classifies the prediction mode of the macroblock and generates the authentication code using the prediction mode. The experimental results show that the video quality is negligible affected after watermark embedding, and the change of video bit rate is basically constant. At the same time, the chromatic DCT algorithm can also use the authentication code of the prediction mode to detect and locate the tampering at the 4 × 4 sub block level with good robustness.

Long-Term Background Redundancy Reduction for Earth Observatory Video Coding

Huge earth observatory video data (EOVD) and the limited transmission bandwidth from satellites to terrestrial devices pose serious challenges to compression efficiency for satellite video. In this work, we deeply explore long-term background redundancy caused by periodical satellite revisit, and make utilization of long-term background reference (LTBR) to design a high-efficiency coding method specific for EOVD. Firstly, we use data of Google Earth as prior background knowledge and construct LTBR from it. Then, two novel prediction methods are proposed to make full use of prior information of LTBR, including color and definition correction based inter prediction with LTBR (CDIP-LTBR), structure and texture constrained intra prediction with LTBR (STIP-LTBR). Lastly, the proposed two new prediction schemes are integrated into one unified coding framework along with HEVC to achieve improved coding performance, and an improved RDO method is designed for additional prediction modes selection problem in the framework to obtain a higher prediction efficiency. Extensive experiments on real-world EOVD show that the proposed coding scheme exhibit significant improvement over HEVC and H.264, in terms of BD-Rate, BD-PSNR and rate distortion comparison.

SHVC CU Processing Aided by a Feedforward Neural Network

The development of multimedia and hardware technologies has led to a great number of industrial video applications, such as virtual reality, high-definition video surveillance, and remote monitoring. As complex communication environments and heterogeneous networks are common in industrial applications, industrial videos are required to support a diverse range of display resolutions and transmission channel capacities. Scalable high-efficiency video coding (SHVC) standards provide the tools to meet this requirement. However, it is highly computationally expensive. Coding complexity has a great impact on SHVC performance in industrial applications. Many of these applications are sensitive to time delay and have limited power. Thus, improvements are required to ensure the practical usability of SHVC encoders. In SHVC encoders, intra/interprediction of variable coding unit (CU) sizes is independently performed for the base and enhancement layers (ELs). There are many interlayer similarities that can be exploited to speed up the procedure for EL coding. In this paper, we propose a feedforward neural network aided model for CU size and mode decisions for SHVC, which utilizes base layer coding information and the coding data of spatiotemporal neighboring CUs to decide which CU sizes or prediction modes can be bypassed for certain EL CUs. Two feedforward neural network based learning models are built for CU classification, which are introduced in the procedures for CU size and mode decisions, respectively. According to the analysis from a large number of video sequences, the representative features are directly extracted from the coding information of previously coded neighboring CUs to avoid computational overheads. After the training is finished, these two models are designed and integrated to build classifiers. Then, two online classification approaches are designed for the CU size and mode decision procedures to classify each CU's type. Finally, different candidate CU sizes and prediction modes are adaptively assigned for each type of CU. This approach outperforms the state-of-the-art fast SHVC/high-efficiency video coding (HEVC) algorithms with approximately 19–42% coding time savings or better compression efficiency, which will be beneficial for the realization of real-time scalable video coding.

A Fast Intra Mode Prediction Decision Algorithm Based on Neighborhood Grouping for HEVC

Abstract High Efficiency Video Coding (HEVC) employs quadtree coding tree unit (CTU) structure to improve its coding efficiency, but at the same time, it requires a very high computational complexity due to its exhaustive calculating process to find the optimal prediction mode for the current CU (Coding Unit). Aiming to solve the problem, a fast intra mode prediction decision algorithm based on neighborhood grouping is presented for HEVC in this article. The contribution of this article lies in the fact that the authors successfully use the neighborhood grouping technology to rapidly find the optimal prediction mode for the current CU, which can significantly reduce computation complexity for HEVC. Specifically, they use the correlative information of adjacent angle prediction mode in their first scheme to quickly reduce the number of Rough Mode Decision (RMD); then they use the correlation between Most Probable Mode (MPM) and the first candidate mode selected as the optimal prediction mode to quickly determine the number of candidate modes; at last, they quickly find the optimal prediction mode with the minimum Rate Distortion Cost (RDCost) by using neighborhood grouping technology or direct calculation approach based on the number of candidate modes. As a result, their proposed algorithm can efficiently solve the problem above. The simulation results show that our proposed intra mode prediction decision algorithm based on neighborhood grouping in this article can reduce about 22.3% computational complexity on average only at a cost of 0.03% bit rate increase and 0.26% PSNR decline compared with the standard reference HM16.1 algorithm.

Efficient Algorithm Adaptations and Fully Parallel Hardware Architecture of H.265/HEVC Intra Encoder

The growing demand for high-performance ultra-high-definition video coding leads to H.265/high-efficiency video coding (HEVC), where the increased computational complexity and data/timing dependence hinder its coding throughput. To address these challenges, this paper presents four algorithm adaptations and a fully parallel hardware architecture for an H.265/HEVC intra encoder. To the best of our knowledge, this is the first fully parallel H.265/HEVC intra encoder. This design supports 35 prediction modes and all coding tree unit partitions. All PUs are independently processed in four prediction engines for high parallelism. An appropriate set of intra prediction modes, RDO candidates, and CABAC rate estimate instances is assigned to each prediction engine, where internal computational tasks are pipelined and scheduled to maximize the processing throughput. Compared with the HM-15.0 software, the proposed algorithm adaptations lead to a reduction of 27% in computational workload, while the average BD-rate and BD-PSNR are 4.39% and -0.21 dB, respectively. This BD-rate is lower than the existing designs with the same video resolution. FPGA implementation of the proposed design shows that it operates at 120 MHz and supports 45 fps of 1080P video sequences using 201-K logic elements and 120-KB on-chip SRAM. ASIC implementation of the proposed design in TSMC 90-nm technology shows that its clock frequency reaches 320 MHz with a hardware gate count of 2288 K, and that it supports real-time encoding of 30 fps of 4-K video sequences. Compared with the state-of-the-art designs, our proposed design demonstrates advantages in computational complexity, bit rate, video quality, throughput, reliability, and flexibility.

Video Abstraction in H.264/AVC Compressed Domain

Video abstraction allows searching, browsing and evaluating videos only by accessing the useful contents. Most of the studies are using pixel domain, which requires the decoding process and needs more time and process consuming than compressed domain video abstraction. In this paper, we present a new video abstraction method in H.264/AVC compressed domain, AVAIF. The method is based on the normalized histogram of extracted I-frame prediction modes in H.264 standard. The frames’ similarity is calculated by intersecting their I-frame prediction modes’ histogram. Moreover, fuzzy c-means clustering is employed to categorize similar frames and extract key frames. The results show that the proposed method achieves on average 85% accuracy and 22% error rate in compressed domain video abstraction, which is higher than the other tested methods in the pixel domain. Moreover, on average, it generates video key frames that are closer to human summaries and it shows robustness to coding parameters.

Content-Based Adaptive SHVC Mode Decision Algorithm

The scalable video coding extensions of the High Efficient Video Coding (HEVC) standard (SHVC) have adopted a new quadtree-structured coding unit (CU). The SHVC test model (SHM) needs to test seven intermode sizes and one intramode size at depth levels of “0,” “1,” “2,” and four intermode sizes and two intramode sizes at a depth level of “3” for interframe CUs. It checks all possible depth levels and prediction modes to find the one with the lowest rate distortion cost using the Lagrange multiplier method in the mode decision procedure to achieve high coding efficiency at the expense of computational complexity. Furthermore, it utilizes the conventional approach for the base layer (BL) and enhancement layer (EL) coding to support SNR/spatial scalable coding. Both the intralayer and interlayer predictions should be performed for each EL CU. Although there is a large amount of interlayer redundancy that can be exploited to speed up the EL encoding, the mode decision procedure is independently performed for the BL and the ELs. In this paper, we propose a content-adaptive mode decision algorithm to reduce the SHVC complexity at the ELs. When the major characteristics of the CUs, such as mode complexity and motion activity, can be estimated early and used for adjusting the mode decision procedure, unnecessary mode and CU size searches can be avoided. First, an experimental analysis is performed to study the interlayer and spatiotemporal correlations in the coding information and the interlevel correlations among the quadtree structures. Based on these correlations, three parameters, including the conditional probability of a SKIP/Merge mode, motion activity, and mode complexity, are defined to describe the video content and are further utilized to adaptively adjust the EL mode decision procedure. The experimental results show that the proposed algorithm can reduce the coding time for ELs by 62%–67% with less than a 1.5% Bjontegaard rate increase compared to the original SHVC encoder.

Fast intra-mode decision for depth map coding in 3D-HEVC

3D-high efficiency video coding (3D-HEVC) contains more encoding viewpoints than traditional HEVC, resulting in a significant increase of coding complexity. In this paper, we propose a low complexity intra mode decision algorithm to reduce the number of intra modes by detecting the flat area and texture direction of the depth map. The corresponding intra prediction modes are skipped when the flat region condition is satisfied. Otherwise, the direction of the edge is detected to decrease the number of angle modes in rough mode decision, which can reduce the intra-coding complexity and coding time cost. Experimental results demonstrate that the proposed algorithm achieves on average 36.48% time saving with negligible degradation of coding performance.

Fast depth intra coding based on spatial correlation and rate distortion cost in 3D-HEVC

3D-HEVC video coding standard adopts many sophisticated algorithms to improve the coding performance, which results in heavy calculation complexity. To reduce the computation burden, a complexity reduction scheme for depth map based on spatial correlation and the rate–distortion (RD) cost is proposed. Firstly, A maximum depth layer decision (MDLD) algorithm is proposed to predict the maximum depth layer (MDL) of each coding tree unit (CTU) accurately. Using the frequency distribution histogram theory in the statistics, the frequency distribution characteristics (FDC) of the RDcost J0 of the CTU level in depth layer 0 is investigated under different MDL. The initial threshold for different MDL is designed based on the FDC of the J0 under allowable false rate (FR). An exponential relationship between the threshold and QP is discovered. Secondly, A fast depth intra mode decision (FDIMD) algorithm is proposed to skip unnecessary prediction modes. Considering the spatial correlation, the FDC of the RDcost JORM of the optimal rough mode (ORM) in rough mode decision (RMD) is analyzed under different optimal prediction mode. Based on the FDC of the JORM, the mode-skipping threshold THst is designed to skip angular modes and depth modeling modes (DMMs). An exponential model is built to represent the relationship between the THst and QP. Finally, a fast wedgelet pattern decision method based on K-Means is developed to reduce the complexity of DMM1. Experimental results show that the proposed algorithm achieves an average reduction of 58.9% on intra coding time, with negligible degradation of coding performance. The proposed algorithm performs better in terms of complexity and performance than other approaches.

Video Compression Using Generalized Binary Partitioning, Trellis Coded Quantization, Perceptually Optimized Encoding, and Advanced Prediction and Transform Coding

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

Adaptive lossless compression of skeleton sequences

The increasing importance of skeleton information in surveillance big data feature analysis demands significant storage space. The development of an effective and efficient solution for storage is still a challenging task. In this paper, we propose a new framework for the lossless compression of skeleton sequences by exploiting both spatial and temporal prediction and coding redundancies. Firstly, we propose a set of skeleton prediction modes, namely, spatial differential-based, motion vector-based, relative motion vector-based, and trajectory-based skeleton prediction mode. These modes can effectively handle both spatial and temporal redundancies present in the skeleton sequences. Secondly, we further enhance performance by introducing a novel approach to handle coding redundancy. Our proposed scheme is able to significantly reduce the size of skeleton data while maintaining exactly the same skeleton quality due to lossless compression approach. Experiments are conducted on standard surveillance and Posetrack action datasets containing challenging test skeleton sequences. Our method obviously outperforms the traditional direct coding methods by providing an average of 73% and 66% bit-savings on the two datasets.

A Low-Cost Complexity Intra Prediction Hardware Architecture for HEVC Decoder

High Efficiency Video Coding (HEVC) adopts new techniques to reduce bit-rate by 50% over a previous video compression standard. The number of intra prediction modes in HEVC is 35 modes and increased compared with the compression. Therefore, hardware architecture with common equation and a fast filter coefficient generation algorithm is proposed for low complexity intra prediction hardware. The proposed architecture performs a smoothing filter, interpolation filter, generation of predicted pixels with only Common Operation Unit (COU). Various equations in intra prediction for smoothing filter of reference samples, calculating the average of the reference samples, generating predicted pixels and filtering predicted pixels is modified to one common equation. The common operation unit using a common equation in intra prediction hardware architecture reduces hardware area and the number of computational operators to perform various equations. COU uses 2 multipliers, 9 adders, 3 shifters and generates 1 predicted sample in planar mode and 2 predicted samples in the other mode. Also, COU generates 2 filtered reference samples in filtering operation of reference samples and the average of 4x4 PU in DC mode. The fast filter coefficient generation algorithm reduces processing time by using only Look-Up Table (LUT) and adders, instead of multiplying operation and the number of computational operators. The number of gates of the architecture is 45.6k. The number of gates in the proposed intra prediction hardware is 36.7% less than previous architecture.

An Efficient Multiple Description Coding for Multi-View Video Based on the Correlation of Spatial Polyphase Transformed Subsequences

Abstract For a robust three-dimensional video transmission through error prone channels, an efficient multiple description coding for multi-view video based on the correlation of spatial polyphase transformed subsequences (CSPT_MDC_MVC) is proposed in this article. The input multi-view video sequence is first separated into four subsequences by spatial polyphase transform and then grouped into two descriptions. With the correlation of macroblocks in corresponding subsequence positions, these subsequences should not be coded in completely the same way. In each description, one subsequence is directly coded by the Joint Multi-view Video Coding (JMVC) encoder and the other subsequence is classified into four sets. According to the classification, the indirectly coding subsequence selectively employed the prediction mode and the prediction vector of the counter directly coding subsequence, which reduces the bitrate consumption and the coding complexity of multiple description coding for multi-view video. On the decoder side, the gradient-based directional interpolation is employed to improve the side reconstructed quality. The effectiveness and robustness of the proposed algorithm is verified by experiments in the JMVC coding platform.

A Novel Fast Partition Algorithm Based on EDGE Information in HEVC

High efficiency video coding (HEVC) introduces a well-designed quad-tree structure to improve coding efficiency. During quad-tree building, the encoder tries all possible coding unit (CU) partition schemes to determine the optimal one with the minimum rate distortion cost, which introduces high computation complexity. To deal with this problem, a fast algorithm based on edge information is proposed to accelerate CU splitting in this paper. Firstly, the depth and prediction mode of adjacent blocks are obtained and used to facilitate rough partition. Secondly, the edge pixel values of this CU are counted and analyzed to fine-tune the partition. Experimental results show that the proposed algorithm can reduce coding time by 35.7% on average, with an acceptable loss of Bjontegaard delta rate of 0.6%.

Fast Depth and Inter Mode Prediction for Quality Scalable High Efficiency Video Coding

The scalable high efficiency video coding (SHVC) is an extension of high efficiency video coding (HEVC). It introduces multiple layers and inter-layer prediction, thus significantly increases the coding complexity on top of the already complicated HEVC encoder. In inter prediction for quality SHVC, in order to determine the best possible mode at each depth level, a coding tree unit can be recursively split into four depth levels, including merge mode, inter2N×2N, inter2N×N, interN×2N, interN×N, inter2N×nU, inter2N×nD, internL×2N and internRx×2N, intra modes and inter-layer reference (ILR) mode. This can obtain the highest coding efficiency, but also result in very high coding complexity. Therefore, it is crucial to improve coding speed while maintaining coding efficiency. In this research, we have proposed a new depth level and inter mode prediction algorithm for quality SHVC. First, the depth level candidates are predicted based on inter-layer correlation, spatial correlation and its correlation degree. Second, for a given depth candidate, we divide mode prediction into square and non-square mode predictions respectively. Third, in the square mode prediction, ILR and merge modes are predicted according to depth correlation, and early terminated whether residual distribution follows a Gaussian distribution. Moreover, ILR mode, merge mode and inter2N×2N are early terminated based on significant differences in Rate Distortion (RD) costs. Fourth, if the early termination condition cannot be satisfied, non-square modes are further predicted based on significant differences in expected values of residual coefficients. Finally, inter-layer and spatial correlations are combined with residual distribution to examine whether to early terminate depth selection. Experimental results have demonstrated that, on average, the proposed algorithm can achieve a time saving of 71.14%, with a bit rate increase of 1.27%.

Detection of fake high definition for HEVC videos based on prediction mode feature

A high definition (HD) video usually means a good visual quality, and the video with low quantization parameter (QP) or high bit rate has a good video definition. However, forgers prefer to directly re-encode lower definition videos using lower QP or higher bit rate without video quality improvement to pretend to be HD videos. Therefore, fake HD video detection is necessary in video forensics. In this paper, a novel method is proposed to detect the fake HD for high efficient video coding (HEVC) videos based on prediction mode feature (PMF). According to our analysis, the HEVC prediction mode scheme of the subsequent encoding could be influenced by the previous lower quality encoding, hence the frequencies of prediction unit with all types of prediction modes can be used to detect the fake HD videos. Firstly, a 4-D features are extracted from Planar, DC, H0 direction and V0 direction intra prediction modes. Secondly, a 6-D features are extracted from Skip, Merge and AMVP inter prediction modes. Finally, these two feature sets are combined into the PMF to detect fake HD videos and further estimate their original QPs and bit rates. Experimental results show that the performance of the proposed method outperforms state-of-the-art works.

Context-adaptive neural network based prediction for image compression.

This paper describes a set of neural network architectures, called Prediction Neural Networks Set (PNNS), based on both fully-connected and convolutional neural networks, for intra image prediction. The choice of neural network for predicting a given image block depends on the block size, hence does not need to be signalled to the decoder. It is shown that, while fully-connected neural networks give good performance for small block sizes, convolutional neural networks provide better predictions in large blocks with complex textures. Thanks to the use of masks of random sizes during training, the neural networks of PNNS well adapt to the available context that may vary, depending on the position of the image block to be predicted. When integrating PNNS into a H.265 codec, PSNRrate performance gains going from 1:46% to 5:20% are obtained. These gains are on average 0:99% larger than those of prior neural network based methods. Unlike the H.265 intra prediction modes, which are each specialized in predicting a specific texture, the proposed PNNS can model a large set of complex textures.

Statistical Early Termination and Early Skip Models for Fast Mode Decision in HEVC INTRA Coding

In this article, statistical Early Termination (ET) and Early Skip (ES) models are proposed for fast Coding Unit (CU) and prediction mode decision in HEVC INTRA coding, in which three categories of ET and ES sub-algorithms are included. First, the CU ranges of the current CU are recursively predicted based on the texture and CU depth of the spatial neighboring CUs. Second, the statistical model based ET and ES schemes are proposed and applied to optimize the CU and INTRA prediction mode decision, in which the coding complexities over different decision layers are jointly minimized subject to acceptable rate-distortion degradation. Third, the mode correlations among the INTRA prediction modes are exploited to early terminate the full rate-distortion optimization in each CU decision layer. Extensive experiments are performed to evaluate the coding performance of each sub-algorithm and the overall algorithm. Experimental results reveal that the overall proposed algorithm can achieve 45.47% to 74.77%, and 58.09% on average complexity reduction, while the overall Bjøntegaard delta bit rate increase and Bjøntegaard delta peak signal-to-noise ratio degradation are 2.29% and −0.11 dB, respectively.