HEVC Decoder Research Articles

On the Security of Selectively Encrypted HEVC Video Bitstreams

With the growing applications of video, ensuring its security has become of utmost importance. Selective encryption (SE) has gained significant attention in the field of video content protection due to its compatibility with video codecs, favorable visual distortion, and low time complexity. However, few studies consider SE security under cryptographic attacks. To fill this gap, we analyze the security concerns of encrypted bitstreams by SE schemes and propose two known plaintext attacks (KPAs). Then the corresponding defense is presented against the KPAs. To validate the effectiveness of the KPA, it is applied to attack two existing SE schemes with superior visual degradation in HEVC videos. Firstly, the video sequences are encoded using H.265/HEVC. During encoding, the selected syntax elements are recorded. Secondly, the recorded syntax elements are imported into the HEVC decoder. By utilizing the encryption parameters and the imported data in the decoder, it becomes possible to reconstruct a significant portion of the original syntax elements before encryption. Finally, the reconstructed syntax elements are compared with the encrypted syntax elements in the decoder, allowing the design of a pseudo-key stream (PKS) through the inverse of the encryption operations. The PKS is used to decrypt the existing SE scheme, and the experimental results provide evidence that the two existing SE schemes are vulnerable to the proposed KPAs. In the case of single bitstream estimation (SBE), the average correct rate of key stream estimation exceeds 93%. Moreover, with multi-bitstream complementation (MBC), the average estimation accuracy can be further improved to 99%.

Performance comparison of throughput between AVC, HEVC and VVC hardware CABAC decoder

Performance comparison of throughput between AVC, HEVC and VVC hardware CABAC decoder

Joint DBF and SAO Parallel Filtering Based on Multithread Load Balancing

This paper presents a joint parallel loop filtering algorithm based on multi-thread load balancing in HEVC decoding, which implements the parallel processing of deblocking filtering (DBF) and sample adaptive compensation (SAO). Because of the diversity of video, the texture of different regions in an image is also different, which leads to various CTU partition methods. Therefore, the number of the boundary to be filtered is greatly different, resulting the computation load among multiple threads unbalanced in parallel processing. To solve this problem, an area division scheme is proposed, which divides the image into multiple areas, and the number of boundaries to be filtered in each area is similar. Then, the mapping relationship table is used to allocate these areas to multiple threads for parallel processing, so as to achieve the load balancing among the filtering threads. Finally, the cache technology is used to combine DBF and SAO to reduce the delay between them and improve the overall parallelism of the loop filter. Experimental results show that the performance of the proposed load balancing joint filtering algorithm is 8.15% higher than the previous scheme.

An efficient model for quantifying the interaction between structural properties of software and hardware in the ARM big.LITTLE architecture

SummaryHeterogeneous architectures offer the opportunity to achieve high performance and energy efficiency by selecting appropriate cores for the execution of ever‐changing software applications. Appropriate core selection depends on the interaction between the structural properties of the software and the hardware that influences the performance of the software. We propose a model for efficient core selection when executing software on ARM's big.LITTLE heterogeneous architecture. It features a metric based on the correlation between the performance and the number of last‐level data cache (LLC) misses on a big and a LITTLE core. Additionally, our model defines a soft threshold in terms of the number of LLC misses, which determines efficient core selection. We verify the model using stress and variable workload benchmarks as well as two popular high‐throughput applications for mutlicore targets, namely, HEVC and LDPC decoders, profiled with X‐Mem, Linux perf, and PMCTrack dynamic tools. Results show that our model can be used for efficient core selection with a relatively small error probability.

Design and Implementation of Efficient Streaming Deblocking and SAO Filter for HEVC Decoder

This paper aims to design an efficient mixed serial five-stage pipeline processing hardware architecture of deblocking filter (DBF) and sample adaptive offset (SAO) filter for high efficiency video coding decoder. The proposed hardware is designed to increase the throughput and reduce the number of clock cycles by processing the pixels in a stream of ${4 \times 36}$ samples in which edge filters are applied vertically in a parallel fashion for processing of luma/chroma samples. Subsequently these filtered pixels are transposed and reprocessed through vertical filter for horizontal filtering in a pipeline fashion. Finally, the filtered block transposed back to the original orientation and forwarded to a three-stage pipeline SAO filter. The proposed architecture is implemented in field programmable gate array and application specific integrated circuit platform using 90-nm library. Experimental results illustrate that the proposed DBF and SAO architecture decreases the processing cycles (172) required for processing each ${64 \times 64}$ or large coding unit compared with the state-of-the-art literature with the increase of gate count (593.32K) including memory. The results show that the throughput of the proposed filter can successfully decode ultrahigh definition video sequences at 200 frames/s at 341 MHz.

Highly parallel HEVC decoding for heterogeneous systems with CPU and GPU

The High Efficiency Video Coding HEVC standard provides a higher compression efficiency than other video coding standards but at the cost of an increased computational load, which makes hard to achieve real-time encoding/decoding for ultra high-resolution and high-quality video sequences. Graphics Processing Units GPU are known to provide massive processing capability for highly parallel and regular computing kernels, but not all HEVC decoding procedures are suited for GPU execution. Furthermore, if HEVC decoding is accelerated by GPUs, energy efficiency is another concern for heterogeneous CPU+GPU decoding. In this paper, a highly parallel HEVC decoder for heterogeneous CPU+GPU system is proposed. It exploits available parallelism in HEVC decoding on the CPU, GPU, and between the CPU and GPU devices simultaneously. On top of that, different workload balancing schemes can be selected according to the devoted CPU and GPU computing resources. Furthermore, an energy optimized solution is proposed by tuning GPU clock rates. Results show that the proposed decoder achieves better performance than the state-of-the-art CPU decoder, and the best performance among the workload balancing schemes depends on the available CPU and GPU computing resources. In particular, with an NVIDIA Titan X Maxwell GPU and an Intel Xeon E5-2699v3 CPU, the proposed decoder delivers 167 frames per second (fps) for Ultra HD 4K videos, when four CPU cores are used. Compared to the state-of-the-art CPU decoder using four CPU cores, the proposed decoder gains a speedup factor of 2.2×. When decoding performance is bounded by the CPU, a system wise energy reduction up to 36% is achieved by using fixed (and lower) GPU clocks, compared to the default dynamic clock settings on the GPU.

Software pipelining with CGA and proposed intrinsics on a reconfigurable processor for HEVC decoders

This work proposes several intrinsics on a reconfigurable processor intended for HEVC decoding and software pipelining algorithms with a coarse-grained array (CGA) architecture as well as the proposed intrinsic instructions. Software pipelining algorithms are developed for the CGA acceleration of inverse transform, pixel reconstruction, de-blocking filter and sample adaptive offset modules. To enable efficient software pipelining, several very-long instruction-word-based intrinsics are designed in order to maximize the parallelization rather than the computational acceleration. We found that the HEVC decoder with the proposed intrinsics yields 2.3 times faster in running clock cycle than a decoder that does not use the intrinsics. In addition, the HEVC decoder with CGA pipelining algorithms executes 10.9 times faster than that without the CGA mode.

A New profiling and pipelining approach for HEVC Decoder on ZedBoard Platform

A New profiling and pipelining approach for HEVC Decoder on ZedBoard Platform

A novel architecture for parallel multi-view HEVC decoder on mobile device

The multi-view HEVC (MV-HEVC) extension was finalized in July of 2014 by the Moving Picture Experts Group and the Video Coding Experts Group. Recently, multi-view videos based on stereo representations are becoming widely popular. Also, a variety of multimedia contents are now available for mobile devices. A real-time multi-view video decoder is therefore needed. In mobile devices, a real-time decoding multi-view video is difficult because of the increasing number of views, spatial resolutions, and limited speed of the processors on mobile platforms. In this paper, we propose a novel architecture for a real-time decoder in mobile devices. The proposed MV-HEVC decoder uses parallel-optimized multi-view video decoding with multi-threading, using advanced reduced instruction set computer machine (ARM) Cortex multi-core processors. Moreover, it is optimized in single instruction multiple data for an ARM platform. The proposed multi-core decoding architectures enable multi-threading with minimum processing overhead. Experimental results show that the proposed multi-view video coding increased the speed by around 2.4–4.8 times in the ARM platform compared to MV-HEVC.

Performance Estimation of HEVC/h.265 Decoder in a Co-Design Flow with SADF-FSM Graphs

Multiprocessor System on Chip (MPSoC) technology presents an interesting solution to reduce the computational time of complex applications such as multimedia applications. Implementing the new High Efficiency Video Coding (HEVC/h.265) codec on the MPSoC architecture becomes an interesting research point that can reduce its algorithmic complexity and resolve the real time constraints. The implementation consists of a set of steps that compose the Co-design flow of an embedded system design process. One of the first anf key steps of a Co-design flow is the modeling phase which allows designers to make best architectural choices in order to meet user requirements and platform constraints. Multimedia applications such as HEVC decoder are complex applications that demand increasing degrees of agility and flexibility. These applications are usually modeling by dataflow techniques. Several extensions with several schedules techniques of dataflow model of computation have been proposed to support dynamic behavior changes while preserving static analyzability. In this paper, the HEVC/h.265 video decoder is modeled with SADF based FSM in order to solve problems of placing and scheduling this application on an embedded architecture. In the modeling step, a high-level performance analysis is performed to find an optimal balance between the decoding efficiency and the implementation cost, thereby reducing the complexity of the system. The case study in this case works with the HEVC/h.265 decoder that runs on the Xilinx Zedboard platform, which offers a real environment of experimentation.

Architecture-aware optimization of an HEVC decoder on asymmetric multicore processors

Low-power asymmetric multicore processors (AMPs) have attracted considerable attention due to their appealing performance/power ratio for energy-constrained environments. However, these processors pose a significant programming challenge due to the integration of cores with different performance capabilities, asking for an asymmetry-aware scheduling solution that carefully distributes the workload. The recent HEVC standard, which offers several high-level parallelization strategies, is an important application that can benefit from an implementation tailored for the low-power AMPs present in many current mobile or handheld devices. In this scenario, we present an architecture-aware implementation of an HEVC decoder that embeds a criticality-aware scheduling strategy tuned for a Samsung Exynos 5422 System-on-Chip furnished with an ARM big.LITTLE AMP. The performance and energy efficiency of our solution are further enhanced by exploiting the NEON vector engine available in the ARM big.LITTLE architecture. Our experimental results expose a 1080p real-time HEVC decoding at 24 frames/s and a reduction of energy consumption over 20 %.

Design of multicore HEVC decoders using actor-based dataflow models and OpenMP

New multimedia portable devices support increasing spatial and temporal video resolutions as well as new and more efficient video codecs. This scenario leads to the use of multicore chips in order to comply with the higher computational loads. In a context dominated by the time to market pressure, the programming of multicore chips is a challenge. In this work, a method based on a combination of dataflow models and OpenMP is proposed for quick design of HEVC decoders using multicore chips. The proposed method has been automated by its integration into an open source framework. Tests have been carried out by implementing the same HEVC decoder targeted on three different multicore chips.1

Dataflow Programs Analysis and Optimization Using Model Predictive Control Techniques

The analysis of the trace graphs generated by dataflow program executions has been shown to be an effective tool for exploring and optimizing the design space of application programs on manycore/multicore platforms. In this work a new approach aiming at finding bounded buffer size configurations for implementations generated by dataflow programs is presented. The introduced method is based on an original transformation procedure which converts the execution trace graph into an event driven linear system made up by a Petri Net. A control theoretic approach based on Model Predictive Control methodologies is then applied to the obtained Petri Net system in order to effectively explore the dataflow program design space and find nearly optimal buffer dimensioning solutions leading to a deadlock free program execution. Two real challenging design case examples, namely a JPEG and a MPEG HEVC decoder, are introduced to show the effectiveness of the introduced approach.

Fast video transcoding from HEVC to VP9

HEVC and VP9 are the current state-of-the-art in video compression, since their bit-streams were recently finalized in January and May 2013, respectively. These codecs are the generational successors of the currently most widelyused video codecs, H.264/AVC and VP8, respectively. Consequently, it is expected that in the near future these new codecs will replace their predecessors. However, the process of converting video contents compressed with one standard to those using another standard is highly computationally expensive, since a priori the video contents must be decompressed and compressed with the target video encoder. Nevertheless, it is known that some information can be extracted from the decoding process in order to accelerate the encoding process. In this paper, some techniques for transcoding from HEVC to VP9 are presented. By using some information extracted from the HEVC decoding process some coding modes will be discarded from being checked in the VP9 encoder. By applying the proposed approaches, a reduction of about 60% of the coding complexity is achieved with acceptable Rate Distortion penalties.

Efficient In-Loop Filtering Across Tile Boundaries for Multi-Core HEVC Hardware Decoders With 4 K/8 K-UHD Video Applications

HEVC is a next generation video coding standard designed with modern coding techniques to be especially efficient for coding high-resolution video such as 4 K/8 K-ultra high- definition (UHD) video. Among the advanced coding tools of HEVC, tiles and wavefront parallel processing (WPP) have been newly adopted for parallel processing of such high-resolution (4 K/8 K-UHD) video. To realize UHD video services over portable devices with limited battery power, it is essential to implement multi-core-based and dedicated HEVC hardware decoders that support the tile- and wavefront-based parallel processing. By doing so, each frame is divided into a multiple number of picture partitions which can then be processed by multiple hardware decoder cores in parallel. However, in-loop filtering (ILF) at tile boundaries cannot be easily parallelized by a multi-core HEVC hardware decoder because of the data dependency between samples in different tiles. In this paper, an efficient control method for ILF across tile boundaries is proposed for multi-core HEVC hardware decoders. The proposed method does not require additional in-loop filters for ILF across the tile boundaries and it allows a decoder core to continue to process the next coding tree unit (CTU) without waiting for other decoders until they finish their ILF processing for the neighboring CTUs in other tiles. From experiments, we show the effectiveness of our ILF control method via a quad-core HEVC decoder for 4 K-UHD video implemented on a prototyping FPGA board.

Low-Power High-Efficiency Video Decoding using General-Purpose Processors

In this article, we investigate how code optimization techniques and low-power states of general-purpose processors improve the power efficiency of HEVC decoding. The power and performance efficiency of the use of SIMD instructions, multicore architectures, and low-power active and idle states are analyzed in detail for offline video decoding. In addition, the power efficiency of techniques such as “race to idle” and “exploiting slack” with DVFS are evaluated for real-time video decoding. Results show that “exploiting slack” is more power efficient than “race to idle” for all evaluated platforms representing smartphone, tablet, laptop, and desktop computing systems.

Analysis and Evaluation of HEVC

HEVC video coding standard is the next generation of H.264 coding standard, which is able to achieve twice coding efficiency compared with H.264/AVC High Profile. This paper analyzes the key features of HEVC standard based on HEVC reference software HM12.0. Experiments cover different resolutions ranging from 240p to 1600p. The most complex parts of HEVC decoder are intra prediction, loop filter and inverse transform when video sequences are coded with all-intra configuration. When video sequences are coded in random-access configuration, HEVC decoding process is occupied by motion compensation, which takes nearly half of the decoding time. Experiment results indicate that HEVC standard can achieve 50% average bit-rate saving compared with H.264.

Data dependency reduction for parallelism enhancement of HEVC decoder

Data dependency reduction for parallelism enhancement of HEVC decoder

Design on HEVC Streaming Media Player Based for Android

The HEVC standard is mainly for high-definition and ultra-high-definition video images. The compression performance of HEVC has been further improved, but at the same time coding complexity is also higher than the existing standard. This paper firstly analyzed the architecture of HEVC decoding process and its mainly algorithm components. Then this paper designed how to use the Android NDK to transplant the HM decoding library, and then take the java native interface to resolve the HEVC stream. After decoding the data, this paper converts the YUV data into RGB format for display. The experiment results show that the design scheme is feasible and the HEVC player achieves the basic functions in the Android platform.

HEVC 복호기의 연산 복잡도 감소를 위한 화면내 예측 하드웨어 구조 설계

본 논문에서는 HEVC 복호기내 화면내 예측의 연산 복잡도를 감소시키기 위해 공유 연산기, 공통 연산기, 고속 smoothing 결정 알고리즘, 고속 필터계수 생성 알고리즘을 적용한 하드웨어 구조를 제안한다. 공유 연산기는 공통수식을 공유하여 smoothing 과정의 연산 중복성을 제거하고, DC모드의 평균값을 미리 계산하여 수행 사이클 수를 감소시킨다. 공통 연산기는 모든 예측모드의 예측픽셀 생성과 필터링 과정을 하나의 연산기로 처리하기 때문에 연산기의 개수를 감소시킨다. 고속 smoothing 결정 알고리즘은 비트 비교기만을 사용하고, 고속 필터계수 생성 알고리즘은 곱셈연산 대신 LUT를 사용하여 연산 개수, 하드웨어 면적과 처리 시간을 감소시킨다. 또한 제안하는 구조는 2개의 공유 연산기와 8개의 공통 연산기를 사용하여 병렬처리함으로써 화면내 예측의 수행 사이클 수를 감소시킨다. 제안하는 구조를 TSMC 0.13um CMOS 공정 라이브러리를 이용하여 합성한 결과 게이트 수는 40.5k, 최대 동작 주파수는 164MHz이다. HEVC 참조 소프트웨어 HM 7.1에서 추출한 데이터를 이용하여 성능을 측정한 결과 제안하는 구조의 수행 사이클 수가 기존 구조 대비 93.7% 감소하였다. In this paper, an intra prediction hardware architecture is proposed to reduce computational complexity of intra prediction in HEVC decoder. The architecture uses shared operation units and common operation units and adopts a fast smoothing decision algorithm and a fast algorithm to generate coefficients of a filter. The shared operation unit shares adders processing common equations to remove the computational redundancy. The unit computes an average value in DC mode for reducing the number of execution cycles in DC mode. In order to reduce operation units, the common operation unit uses one operation unit generating predicted pixels and filtered pixels in all prediction modes. In order to reduce processing time and operators, the decision algorithm uses only bit-comparators and the fast algorithm uses LUT instead of multiplication operators. The proposed architecture using four shared operation units and eight common operation units which can reduce execution cycles of intra prediction. The architecture is synthesized using TSMC 0.13um CMOS technology. The gate count and the maximum operating frequency are 40.5k and 164MHz, respectively. As the result of measuring the performance of the proposed architecture using the extracted data from HM 7.1, the execution cycle of the architecture is about 93.7% less than the previous design.