High Efficiency Video Coding Codec Research Articles

A fast integrated deblocking filter and sample-adaptive-offset parameter estimation architecture for HEVC

Low power hardware acceleration cores for integration into real-time High Efficiency Video Coding (HEVC) codec for smartphones, tablets, camcorders, and televisions are in great demand. This demand motivates one for an efficient approximation of important power-consuming modules of HEVC including in-loop filters. This paper presents a hardware-efficient implementation of integrated deblocking filter (DBF) and sample adaptive offset (SAO) parameter estimation architecture for 16×16, 32×32, and 64×64 coding tree units (CTU) in HEVC. When the architecture is extended to HEVC-Test-Model (HM) Software, the luminance peak-signal-to-noise-ratio gets increased by 0.02 decibel, and the execution time of DBF and SAO gets decreased by at most 35% and 38%, respectively while compared to the reference algorithm. Moreover, it delivers rate–distortion performance comparable to the HEVC standard and reports mean-squared-error, structural-similarity (SSIM) index, and multi-scale SSIM (MS-SSIM) index of values 0.15, 0.9984, and 1, respectively for 4K video sequences. The architecture consumes minimum power, area, and energy equal to 9.83 milliwatts, 162 kilo-gate-equivalents, and 44 picojoules, respectively while supporting up to forty-six 8K frames per second. Additionally, it reports 78% smaller area and requires 75% less clock-cycles-per-largest-coding-unit as compared to the separate implementation of DBF and SAO with reference to HEVC. Such designs with low power, area, and energy can be integrated into a real-time HEVC codec for portable HEVC-compliant consumer electronic devices.

Rate-distortion/complexity analysis of HEVC, VVC and AV1 video codecs

With the advent of smartphones and tablets, the amount of online video traffic has increased enormously. This, together with the growing popularity of high-definition video, motivated the development of the High Efficiency Video Coding (HEVC) standard, released in 2013, with the aim of achieving a 50% bitrate reduction with respect to its predecessor, namely H.264/MPEG-4 Advanced Video Coding (AVC). However, new contents with greater resolutions and requirements arise every day, making it necessary to reduce the bitrate to a further extent. In this regard, the efforts to become the leading video codec in the market resulted in two main contenders: the Joint Video Experts Team (JVET), which leads the development of the Versatile Video Coding (VVC) standard, and the Alliance for Open Media (AOMedia), which spearheads the AOMedia Video 1 (AV1) project. In this context, this paper presents a rate-distortion/complexity analysis of HEVC, VVC and AV1 main video codecs using objective measures of assessment in order to analyze their real capabilities. The analysis, which was done using well-defined test conditions, reveals that VVC considerably outperforms both HEVC and AV1 in terms of coding efficiency.

Hardware-Efficient 2D-DCT/IDCT Architecture for Portable HEVC-Compliant Devices

Low power hardware acceleration cores for integration into real-time High Efficiency Video Coding (HEVC) codec for smartphones, tablets, camcorders, and televisions are in great demand. This need motivates one for an efficient realization of Discrete Cosine Transform (DCT) and Inverse-DCT (IDCT) for HEVC. This paper presents an algorithm to calculate the required minimum number of low-frequency DCT-output/IDCT-input coefficients for 4, 8, 16, and 32-point DCT/IDCT in HEVC, such that there is a slight decrease in peak-signal-to-noise-ratio (<; 0.15 decibel) and a minor increment in bitrate (<; 1.5%) as compared to the reference HEVC-Test-Model (HM) Software. However, the encoding time gets reduced at most by 17.95% for Class-A type sequences, while reporting mean-squared-error and structural-similarity of 1.42 and 0.9913, respectively for 4K ultra-high-definition videos. Moreover, HEVC-compliant computationally efficient architectures are introduced for n-point DCT/IDCT. The presented flexible Transpose Memory architecture uses only sixteen random-access-memories to support all transform-unit sizes in HEVC. The proposed two-dimensional DCT/IDCT architecture can process up to 288@$4K$ frames-per-second, and it consumes the minimum power, energy, and area of 11.23 milliwatts, 2.34 picojoules, and 120 kilo-gate-equivalents, respectively. Such design with low power, area, and energy features can be included in a real-time HEVC codec for HEVC-compliant consumer electronic devices.

High Bit-Depth Seismic Data Compression: A Novel Codec Under the Framework of HEVC

Motivated by the superior performance of High Efficiency Video Coding (HEVC), and driven by the rapid growth in data volume produced by seismic surveys, in this work we explore a 32 bits per pixel (b/p) extension of the HEVC codec for compression of seismic data. We propose to reassemble seismic slices in a format that corresponds to video signal and benefit from the coding gain achieved by HEVC inter mode, besides the possible advantages of the (still image) HEVC intra mode. To this end, we modify almost all components of the original HEVC codec to cater for high bit-depth coding of seismic data: Lagrange multiplier used in optimization of the coding parameters has been adapted to the new data statistics, core transform and quantization have been reimplemented to handle the increased bit-depth range, and modified adaptive binary arithmetic coder has been employed for efficient entropy coding. Even though the new codec after implementation of the proposed modifications goes beyond the standardized HEVC, it still maintains a generic HEVC structure, and it is developed under the general HEVC framework. Thus, we tailored a specific codec design which, when compared to the JPEG-XR and commercial wavelet-based codec, significantly improves the peak-signal-to-noise-ratio (PSNR) vs. compression ratio performance for 32 b/p seismic data. Depending on a configuration, PSNR gain goes from 3.39 dB up to 9.48 dB. Also, relying on the specific characteristics of seismic data, we proposed an optimized encoder that reduces encoding time by 67.17% for All-I configuration on trace image dataset, and 67.39% for All-I, 97.96% for P2-configuration and 98.64% for B-configuration on 3D wavefield dataset, with negligible coding performance losses.

Analysis of Emerging Video Codecs: Coding Tools, Compression Efficiency

The video compression community, from technologists to customers, is affected by the competition between two lines of development for future video compression technology beyond high-efficiency video coding (HEVC): AV1, as developed by the industry-driven Alliance for Open Media (AOM), and Versatile Video Coding (VVC), as developed and studied by the Joint Video Experts Team (JVET) on Future Video Coding of ITU-T Video Coding Expert Group (VCEG) and International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Motion Picture Experts Group (MPEG). Both groups have asserted that their respective codecs, AV1 and VVC, outperform the state-of-the-art HEVC. However, in many cases, related studies are evaluating video codec performances based on different test conditions, scenarios, or codec implementations, which makes comparisons sometimes unfair and at least difficult to interpret. This paper provides a neutral and well-balanced evaluation of compression and runtime performance for both AV1 and VVC codecs, using their respective reference models, in comparison to two HEVC codec implementations: HEVC Test Model (HM) as the HEVC standard reference model and x265 as a well-known open-source HEVC codec implementation. Comparison of differentiating features of AV1 and VVC over HEVC is also reported. Our main findings show that, based on an objective analysis, AV1 allows bitrate reduction of 10% in comparison to HM for the same peak signal-to-noise ratio (PSNR) and Structural SIMilarity (SSIM) quality scores while increasing encoding runtime by 55.5x. However, we show that this objective advantage in coding efficiency does not necessarily translate into gain in terms of subjective quality assessment. Overall, objective results show VVC to be the best candidate for compression technology beyond HEVC.

Convolutional Neural Network Based Synthesized View Quality Enhancement for 3D Video Coding.

The quality of synthesized view plays an important role in the three dimensional (3D) video system. In this paper, to further improve the coding efficiency, a convolutional neural network (CNN) based synthesized view quality enhancement method for 3D High Efficiency Video Coding (HEVC) is proposed. Firstly, the distortion elimination in synthesized view is formulated as an image restoration task with the aim to reconstruct the latent distortion free synthesized image. Secondly, the learned CNN models are incorporated into 3D HEVC codec to improve the view synthesis performance for both view synthesis optimization (VSO) and the final synthesized view, where the geometric and compression distortions are considered according to the specific characteristics of synthesized view. Thirdly, a new Lagrange multiplier in the rate-distortion (RD) cost function is derived to adapt the CNN based VSO process to embrace a better 3D video coding performance. Extensive experimental results show that the proposed scheme can efficiently eliminate the artifacts in the synthesized image, and reduce 25.9% and 11.7% bit rate in terms of peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) index, which significantly outperforms the state-of-theart methods.

Understanding and Removal of False Contour in HEVC Compressed Images

A contour-like artifact called false contour is often observed in large smooth areas of decoded images and video. Without loss of generality, we focus on detection and removal of false contours resulting from the state-of-the-art High Efficiency Video Coding codec. First, we identify the cause of false contours by explaining the human perceptual experiences on them with specific experiments. Next, we propose a precise pixel-based false contour detection method based on the evolution of a false contour candidate (FCC) map. The number of points in the FCC map becomes fewer by imposing more constraints step by step. Special attention is paid to separating false contours from real contours such as edges and textures in the video source. Then, a decontour method is designed to remove false contours in the exact contour position while preserving edge/texture details. Extensive experimental results are provided to demonstrate the superior performance of the proposed false contour detection and removal method in both compressed images and videos.

Temporally Dependent Rate-Distortion Optimization for Low-Delay Hierarchical Video Coding.

Low-delay hierarchical coding structure (LD-HCS), as one of the most important components in the latest High Efficiency Video Coding (HEVC) standard, greatly improves coding performance. It groups consecutive P/B frames into different layers and encodes them with different quantization parameters (QPs) and reference mechanisms in such a way that temporal dependency among frames can be exploited. However, due to varying characteristics of video contents, temporal dependency among coding units differs significantly from each other in the same or different layers, while a fixed LD-HCS scheme cannot take full advantage of the dependency, leading to a substantial loss in coding performance. This paper addresses the temporally dependent rate distortion optimization (RDO) problem by attempting to exploit varying temporal dependency of different units. First, the temporal relationship of different frames under the LD-HCS is examined, and hierarchical temporal propagation chains are constructed to represent the temporal dependency among coding units in different frames. Then, a hierarchical temporally dependent RDO scheme is developed specifically for the LD-HCS based on a source distortion propagation model. Experimental results show that our proposed scheme can achieve 2.5% and 2.3% BD-rate gain in average compared with the HEVC codec under the same configuration of P and B frames, respectively, with a negligible increase in encoding time. Furthermore, coupled with QP adaption, our proposed method can achieve higher coding gains, e.g., with multi-QP optimization, about 5.4% and 5.0% BD-rate saving in average over the HEVC codec under the same setting of P and B frames, respectively.

New context of compression problem and approach to solution: a survey of the literature

The new generation multimedia applications such as ultra-high definition TV, 3D video and cloud storage keeps the compression research relevant even post to the advent of the state-of-art standard HEVC (High Efficiency Video Coding) codec. The continued studies on human visual system suggest that a perceptually acceptable reconstruction with reduced bandwidth requirements may be more acceptable and commercially viable solution rather than a bit accurate reconstruction for a variety of video and graphics applications. One way to achieve compression beyond that achieved by the advanced codec like HEVC is to sample the input data at a lower rate than the Nyquist rate, exploit the sparsity and perceptual redundancy in the content using advanced signal processing and computer vision tools and represents it using a spatio-temporal model. This paper contains an exhaustive review of the new context of video compression methods along with their limitations and open issues.

Parallelization and performance evaluation of open-source HEVC codecs

High Efficiency Video Coding (HEVC) was developed by the Joint Collaborative Team on Video Coding (JCT-VC) to replace the current H.264/Advanced Video Coding (AVC) standard, which has dominated digital video services in all segments of the domestic and professional markets for over 10 years. In terms of rate-distortion, HEVC roughly doubles the compression performance of H.264/AVC, but at a cost of extremely high computational complexities during encoding. Parallelizing HEVC encoding is an efficient way of fulfilling this computational requirement. Since its standardization, several open-source HEVC video codecs have been developed with parallel encoding capabilities. This paper presents a rate-distortion/complexity analysis of the open-source HEVC codecs using objective measures of assessment to analyze their real parallelization capabilities. Experimental results show that DivX265 and x265 obtain the best parallel performance on average for the high-quality encoding scenario. In the case of the high-speed encoding scenario, x265 obtains the best parallel performance. In both scenarios, the coding efficiency of the encoders remains virtually unaffected regardless of the amount of parallelism.

Exploiting Thread-Level Parallelism on HEVC by Employing a Reference Dependency Graph

This paper presents an optimized parallel algorithm for the next-generation video codec High Efficiency Video Coding (HEVC). The proposed method provides maximized parallel scalability by exploiting two levels of parallelism: 1) frame level and 2) task level. Frame-level parallelism is exploited using a graph that efficiently provides a parallel coding order of the frames with complex reference dependencies. The proposed reference dependency graph is generated at runtime by a novel construction algorithm that dynamically analyzes the configuration of the HEVC codec. Task-level parallelism is exploited to provide further scalability to frame-level parallelization. A pipelined execution is allowed for independent tasks, which are defined by dividing and categorizing a single coding process into multiple types of tasks. The proposed parallel encoder and decoder do not suffer from loss in coding efficiency because neither constraints nor modification in coding options are required. The proposed parallel methods result in an average encoding speedup of 1.75 and the aggressive method that exploits additional frame-level parallelism achieved 6.52 speedup using eight physical cores.

Study of No-Reference Video Quality Metrics for HEVC Compression

The paper proposes a No-Reference (NR) quality assessment measurement originally developed for H.264, used for High Efficiency Video Coding (HEVC). In particular, authors present an investigation of NR metrics to objectively estimate the perceptual quality of a set of processed video sequences. The authors take into account typical distortions introduced by the block-based coding approaches like HEVC codec. The underlying processing used for the quality assessment considers the blockiness caused by the boundaries of each coded block and the blurring as a lack of spatial details. The correlation between the NR quality metrics and the well-known and most widely used objective metric, the Video Quality Model (VQM), is performed to validate the quality prediction accuracy based on the provided scores. The Pearson correlation coefficients obtained stand for promising results for different types of videos.

Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard

This paper provides an overview of Scalable High efficiency Video Coding (SHVC), the scalable extensions of the High Efficiency Video Coding (HEVC) standard, published in the second version of HEVC. In addition to the temporal scalability already provided by the first version of HEVC, SHVC further provides spatial, signal-to-noise ratio, bit depth, and color gamut scalability functionalities, as well as combinations of any of these. The SHVC architecture design enables SHVC implementations to be built using multiple repurposed single-layer HEVC codec cores, with the addition of interlayer reference picture processing modules. The general multilayer high-level syntax design common to all multilayer HEVC extensions, including SHVC, MV-HEVC, and 3D HEVC, is described. The interlayer reference picture processing modules, including texture and motion resampling and color mapping, are also described. Performance comparisons are provided for SHVC versus simulcast HEVC and versus the scalable video coding extension to H.264/advanced video coding.

HEVC in wireless environments

The increasing demand for real-time applications with high and ultra-high-definition video urged the ITU-T and the ISO/IEC to join their forces to develop the next-generation video coding standard. The new coding standard that has been produced is known as High-Efficiency Video Coding (HEVC). The proposed HEVC standard fulfilled its target to achieve more than 50 % improvement in video compression over the existing H.264 Advanced Video Coding standard, keeping comparable image quality, at the expense of increased computational complexity. Advances in wireless communications and mobile networking have dramatically increased the popularity of video services for mobile users, with video delivery at their fingertips. Delivering high perceptual quality video over wireless environments is challenging due to the changing channel quality and the variations in the importance of one source packet to the next for the end user's quality of experience. The main objective of this paper is to provide an overview over the new characteristics which are likely to be used in HEVC in wireless environments and discusses several research challenges. Experimental results demonstrate that for low-delay wireless video communications, the HEVC codec is more effective compare to previous H.264 codec and shows better overall performance. Both subjective and objective visual quality comparative study has been also carried out to validate the proposed approach.

In-Loop Noise-Filtered Prediction for High Efficiency Video Coding

In this paper, we focus on optimization and enhancement of High Efficiency Video Coding (HEVC) with respect to professional applications. In most of the professional video applications, noise dominates the compression performance. We therefore theoretically and practically analyze the denoising performance of an HEVC codec and show that, especially for low to medium quantization parameters (QPs), the source noise is still present in the reconstructed frame. This motivates us to use an explicit in-loop reference frame denoising filter for this quality range. We show how the noise, which badly influences the prediction, can be modeled and estimated. Two approaches are introduced, in which either the noise is calculated from prior knowledge or estimated automatically without prior knowledge. A denoising filter that exploits the HEVC core transform is introduced. Finally, a coding mode adaptive scheme for motion compensation using the noise-filtered reference frame is described. The compression results show that especially for low to medium QP settings as well as lossless compression considerable bitrate savings can be achieved using the proposed schemes.

Adaptive transform skipping for improved coding of motion compensated residuals

New generations of video compression algorithms, such as those included in the under development High Efficiency Video Coding (HEVC) standard, provide substantially higher compression compared to their ancestors. The gain is achieved by improved prediction of pixels, both within a frame and between frames. Novel coding tools that contribute to the gain provide highly uncorrelated prediction residuals for which classical frequency decomposition methods, such as the discrete cosine transform, may not be able to supply a compact representation with few significant coefficients. To further increase the compression gains, this paper proposes transform skip modes which allow skipping one or both 1-D constituent transforms (i.e., vertical and horizontal), which is more suitable for sparse residuals. The proposed transform skip mode is tested in the HEVC codec and is able to provide bitrate reductions of up to 10% at the same objective quality when compared with the application of 2-D block transforms only. Moreover, the proposed transform skip mode outperforms the full transform skip currently investigated for possible adoption in the HEVC standard.

Comparative Rate-Distortion-Complexity Analysis of HEVC and AVC Video Codecs

This paper analyzes the rate-distortion-complexity of High Efficiency Video Coding (HEVC) reference video codec (HM) and compares the results with AVC reference codec (JM). The examined software codecs are HM 6.0 using Main Profile (MP) and JM 18.0 using High Profile (HiP). These codes are benchmarked under the all-intra (AI), random access (RA), low-delay B (LB), and low-delay P (LP) coding configurations. In order to obtain a fair comparison, JM HiP anchor codec has been configured to conform to HM MP settings and coding configurations. The rate-distortion comparisons rely on objective quality assessments, i.e., bit rate differences for equal PSNR. The complexities of HM and JM have been profiled at the cycle level with Intel VTune on Intel Core 2 Duo processor. The coding efficiency of HEVC is drastically better than that of AVC. According to our experiments, the average bit rate decrements of HM MP over JM HiP are 23%, 35%, 40%, and 35% under the AI, RA, LB, and LP configurations, respectively. However, HM achieves its coding gain with a realistic overhead in complexity. Our profiling results show that the average software complexity ratios of HM MP and JM HiP encoders are 3.2× in the AI case, 1.2× in the RA case, 1.5× in the LB case, and 1.3× in the LP case. The respective ratios with HM MP and JM HiP decoders are 2.0×, 1.6×, 1.5×, and 1.4×. This paper also reveals the bottlenecks of HM codec and provides implementation guidelines for future real-time HEVC codecs.

Complexity control of high efficiency video encoders for power-constrained devices

The emerging High Efficiency Video Coding (HEVC) standard is expected to require much more processing power than its predecessors due to the higher algorithmic complexity of new coding tools and associated data structures. This paper proposes a novel complexity control method for the near future HEVC encoders running on power-constrained devices. The proposed method is based on a decision algorithm that dynamically adjusts the depth of the Coding Units (CU) defined by quad-tree structures. New evidence about the relationship between CU depth and coding complexity is used to selectively constrain the CU depth in order to not exceed a predefined complexity target. The experimental results show that the encoder computational complexity can be downscaled by up to 60% at the cost of negligible loss of rate-distortion (RD) performance. The proposed method finds application in the near future multimedia portable devices using HEVC codecs.