3D High Efficiency Video Coding Research Articles

A Computation Complexity Reduction of the Size Decision Algorithm in 3D-HEVC Depth Map Intracoding

A new video format called a multiview video plus depth map is recently designed as the most efficient 3D video representation. 3D-high efficiency video coding (3D-HEVC) is the international standard for 3D video coding finalized in 2016. In 3D-HEVC intracoding, the depth map is an essential component, in which the depth map intraprediction occupies more than 85% of the overall intraencoding time. In 3D-HEVC design, a highly flexible quadtree coding unit partitioning is adopted, in which one coding tree unit is partitioned recursively into a prediction unit (PU) from 64 × 64 down to 4 × 4. This highly flexible partitioning provides more accurate prediction signals and thereby achieves better intradepth map compression efficiency. However, performing all depth map intraprediction modes for each PU level achieves high depth map intracoding efficiency, but it results in a substantial computational complexity increase. This paper proposes an amelioration of the previously proposed depth map PU size decision using an efficient homogeneity determination. The resulting experiences show that the proposed method can significantly save the computational complexity with a negligible loss of intracoding efficiency.

Fast coding scheme for low complexity 3D-HEVC based on video content property

3D-high efficiency video coding (3D-HEVC) is the latest new extension standard of high efficiency video coding (HEVC) for multi-view video plus depth (MVD). Several novel coding tools are employed in the 3D-HEVC for better representation of the dependent texture and depth video. However, employing these tools causes a significant increase in coding complexity. In this paper, we introduce a fast coding scheme for complexity reduction of 3D-HEVC. Since the multi-view video and depth are highly content dependent, testing all the prediction modes are not efficient. A statistically analysis is performed to study the features of 3D video contents from spatio, spatial and inter-view correlations. Based on this correlation, the proposed fast coding scheme determines to skip some treeblocks of texture and depth video at the early stage. Experimental results demonstrate that the proposed scheme can save 63.3% runtime of 3D-HEVC with only 0.41% bitrate increase.

Sparse Representations-based depth images quality assessment

The conventional 2D metrics can be used for measuring the quality of depth maps, but none of them is considered to be efficient and is not accurate when used for evaluating 3D quality. In this paper, we propose a new full reference objective metric, called Sparse Representations-Mean Squared Error (SR-MSE), which efficiently evaluates the depth maps compression distortions. It adaptively models the reference and compressed depth maps in a mixed redundant transform domain dedicated to depth features. Then, it computes the mean squared error between the sparse coefficients issued from this modeling. As a benchmark of quality assessment, we perform a subjective evaluation test for depth maps compressed using the latest 3D High Efficiency Video Coding standard at various bitrates. We compare the subjective results with the proposed and conventional objective metrics. Experimental results demonstrate that the proposed SR-MSE, compared to the conventional image quality assessment metrics, yields the highest correlated scores to the subjective ones.

TSAN: Synthesized View Quality Enhancement via Two-Stream Attention Network for 3D-HEVC

In three-dimensional video system, the texture and depth videos are jointly encoded, and then the Depth Image Based Rendering (DIBR) is utilized to realize view synthesis. However, the compression distortion of texture and depth videos, as well as the disocclusion problem in DIBR degrade the visual quality of the synthesized view. To address this problem, a Two-stream Attention Network (TSAN)-based synthesized view quality enhancement method is proposed for 3D-High Efficiency Video Coding (3D-HEVC) in this article. First, the shortcomings of the view synthesis technique and traditional convolutional neural networks are analyzed. Then, based on these analyses, a TSAN with two information extraction streams is proposed for enhancing the quality of the synthesized view, in which the global information extraction stream learns the contextual information, and the local information extraction stream extracts the texture information from the rendered image. Third, a Multi-Scale Residual Attention Block (MSRAB) is proposed, which can efficiently detect features in different scales, and adaptively refine features by considering interdependencies among spatial dimensions. Extensive experimental results show that the proposed synthesized view quality enhancement method achieves significantly better performance than the state-of-the-art methods.

Machine learning-based fast CU size decision algorithm for 3D-HEVC inter-coding

3D-high efficiency video coding (3D-HEVC) is an extension of the high efficiency video coding (HEVC) standard for the compression of the texture videos and depth maps. In 3D-HEVC inter-coding, the coding unit (CU) is recursively performed on variable sizes, namely, depth levels. The CU size decision process is conducted using all the possible depth levels to obtain the one with the least rate-distortion (RD) cost using the Lagrange multiplier. These tools achieve the highest coding efficiency but incur a very high computational complexity. In this paper, a fast CU size decision algorithm is proposed to reduce the complexity caused by the CU size splitting process. The proposed algorithm is based on the CU homogeneity classification using machine learning technology. First, the tensor feature is extracted to characterize the homogeneity of CU, which has a strong relationship with CU sizes. Then, a boosted decision stump algorithm is employed to analyze and construct a binary classification model from the extracted features and find suitable thresholds for the proposed method. Finally, an efficient early termination of CU splitting is released based on adaptive thresholds for texture videos and depth maps. The experimental results show that the proposed algorithm reduces a significant encoding time, while the loss in coding efficiency is negligible.

Adaptive CU split prediction and fast mode decision for 3D-HEVC texture coding based on just noticeable difference model

3D-high efficiency video coding (3D-HEVC) is the latest extension of HEVC standard. Through adopting novel coding tools, 3D-HEVC could improve the compression efficiency of multi-view texture video. However, with the great bitrate savings, 3D-HEVC brings the extreme complexity. To reduce the encoding time of texture video, a fast 3D-HEVC method based on just noticeable difference (JND) model is proposed in this paper. The JND model is introduced to analyze the coding tree block (CTB) characteristics of the texture image. Moreover, combining with adaptive coding unit (CU) split prediction and fast mode decision, the proposed scheme could skip some CTBs of texture video at the early stage without a normal coding procedure. Simulation results demonstrate that the proposed scheme can reduce 51.5% encoding runtime on average with negligible loss of subjective quality.

Complexity and compression efficiency assessment of 3D-HEVC encoder

This paper presents a complexity and compression efficiency assessment of the 3D-High Efficiency Video Coding (3D-HEVC) encoder. An experimental evaluation was carried out to identify the encoding tools with the highest impact on the computational complexity of 3D-HEVC. The evaluated tools were firstly divided into three categories: HEVC, multiview, and depth maps tools. Thus, incremental analyses with a set of 28 encoder configurations were performed to point out the compression efficiency and the computational complexity of each encoding tool individually. Experimental results demonstrated different compression-complexity operation points for the 3D-HTM reference software, providing relevant information for the implementation of 3D-HEVC encoders with different tradeoff possibilities between compression efficiency and complexity. The obtained results also allowed identifying encoding tools that can be further optimized in future works.

6WR: A Hardware Friendly 3D-HEVC DMM-1 Algorithm and its Energy-Aware and High-Throughput Design

This brief presents the Six Wedgelets and six Refinements (6WR), a hardware friendly algorithm targeting the Depth Modeling Mode 1 (DMM-1) encoding tool of the 3D-High Efficiency Video Coding (3D-HEVC) standard. This brief also presents the high-throughput and energy-aware hardware design for the 6WR. The 6WR algorithm reduces 98.5% of the evaluated wedge lets by exploring the edges gradients, with average coding efficiency losses between 1.2% and 2.8%. The hardware design implements the Bresenham algorithm to avoid the use of memory. The synthesis results show that the 6WR architecture can process up to nine views in 3D full HD 1080p videos at 30 frames per second, with a power dissipation of 263.7 mW. When compared with related works, the 6WR architecture reached the highest throughput and the best results of coding efficiency and energy efficiency when the same target throughput is considered.

Tile Adaptation for Workload Balancing of 3D-HEVC Encoder in Homogeneous Multicore Systems

This paper proposes a tile adaptation for workload balancing to speed up the 3D-High Efficiency Video Coding (3D-HEVC) encoder. Experiments were done to evaluate the usage of tiles when encoding texture and depth maps considering a static uniform partitioning. Based on these evaluations, we observed that the encoding workload of neighboring frames in the same view tends to be similar. Moreover, the encoding workload of dependent views can be predicted using the information of previously encoded frames of the same and base views. Therefore, the proposed solution adaptively defines the boundaries of the tiles based on inter-frame and inter-view correlations to balance the workloads of the tiles, resulting in a higher speedup when executing in homogeneous multicore systems. The proposed method was implemented in 3D-HEVC Test Model and evaluated according to the Common Test Conditions. The results demonstrated that the proposed solution speeds up texture and depth map coding from 9.94% to 18.21%, when compared to the uniform-sized tiles, without coding efficiency loss. To the best of the authors' knowledge, this is the first work proposing a complete solution for speeding up the 3D-HEVC video encoder by balancing the tiles workload dynamically.

Performance Analysis of HEVC Scalable Extension for Depth Maps

High Efficiency Video Coding (HEVC) is actually the latest video standard that enables significant compression performance improvement in comparison with its predecessors. Scalable High Efficiency Video Coding (SHVC) and 3D High-Efficiency Video Coding (3D-HEVC) are extensions of HEVC hooked on recently emerged applications and issues. 3D-HEVC targets the monolayer compression of the specific multiview plus depth 3D video representation. It consists of multiple views and their associated depth maps that are required for synthesizing intermediate views in advanced free viewpoint applications. SHVC combines the compression efficiency and visual quality of HEVC, with the possibility of encoding different versions of a same video in a single bitstream. It is conceived to cope with the heterogeneity of networks and devices used in the transmission chain of texture videos. With the ongoing progess in equipments and technologies, the scalable multilayer coding is also solicited, not to say required, for depth maps. In comparison with 3D-HEVC, this paper shows how SHVC works for quality scalable depth maps compression over different base and enhancement layers quantization scenarios. Besides, this paper analyzes the SHVC efficiency for intermediate views that are synthesized from encoded depth maps. Thus we aim to provide support for upcoming works on depth map quality scalable encoding.

A Fast and Efficient 3D-HEVC Method for Complexity Reduction Based on the Correlations of Inter-View, Spatio-Temporal, and Texture-Depth

3D High Efficiency Video Coding (3D-HEVC) is the latest High Efficiency Video Coding (HEVC) standard development to improve the compression performance of multi-view video plus depth (MVD) format. It adopts several additional coding tools for better representation of the dependent texture and depth data. These tools achieve high compression efficiency, but require high computational complexity. In this paper, we introduce a fast 3D-HEVC method for the complexity reduction of MVD. A simulation analysis is performed to study the coding information correlations of inter-view, spatio-temporal and texture-depth. The proposed fast algorithm decides to skip some treeblocks of texture and depth map at the early stage without a normal coding process, which includes fast depth level, early SKIP/Merge mode, adaptive early termination, based on the correlations of inter-view, spatio-temporal, and texture-depth. Experimental results show that the proposed fast scheme can save 70.2% encoding runtime of 3D-HEVC, with negligible loss of rate-distortion (RD) performance.

Fast CU size and mode decision algorithm for 3D-HEVC intercoding

The 3D High Efficiency Video Coding (3D-HEVC) is an extension of the last 2D video coding generation (HEVC) using the Multi-view Video plus Depth map (MVD) format. In inter-coding process, the 3D-HEVC uses multiples inter prediction modes for each Coding Unit (CU) size. These prediction modes are performed to use all the possible CU sizes to select the one with the least Rate Distortion (RD) cost as the best prediction mode using Lagrange multiplier. It achieved high coding efficiency but it consumes an extremely time coding which limits 3D-HEVC from real-time applications. In this paper, a fast inter prediction mode and CU size decision algorithm is proposed to reduce the complexity of 3D-HEVC inter-coding. This algorithm is based on the homogeneity of CU using structure tensor. An experimental analysis is first performed to study the correlation of the CU complexity with depth levels and prediction modes, then the structure tensor of each CU is extracted to characterize the CU homogeneity, which has strong relationship with coding levels and inter prediction modes. Finally, a fast algorithm is adopted to accomplish an early termination of CU split and skip unnecessary prediction modes. The experimental results demonstrate that the proposed fast algorithm can provide a considerable encoding time-saving with negligible rate-distortion performance loss compared to the original 3D-HEVC encoder.

Fast intra-mode decision for depth map coding in 3D-HEVC

3D-high efficiency video coding (3D-HEVC) contains more encoding viewpoints than traditional HEVC, resulting in a significant increase of coding complexity. In this paper, we propose a low complexity intra mode decision algorithm to reduce the number of intra modes by detecting the flat area and texture direction of the depth map. The corresponding intra prediction modes are skipped when the flat region condition is satisfied. Otherwise, the direction of the edge is detected to decrease the number of angle modes in rough mode decision, which can reduce the intra-coding complexity and coding time cost. Experimental results demonstrate that the proposed algorithm achieves on average 36.48% time saving with negligible degradation of coding performance.

3D-HEVC Bipartition Modes Encoder and Decoder Design Targeting High-Resolution Videos

This paper presents hardware designs for the encoder and decoder of the 3D-High Efficiency Video Coding (3D-HEVC) bipartition modes targeting real-time processing of high-resolution videos. These hardware designs include the depth modeling modes 1 (DMM-1) and 4 (DMM-4). The encoder was designed using a simplification in the DMM-1 algorithm with the advantage of better using the hardware resources and without coding efficiency loss. The encoder architecture was designed in a scalable structure supporting different block sizes and reaching different throughputs according to the application requirements. The decoder was designed sharing resources between DMM-1 and DMM-4 execution and supporting all available block sizes. Both architectures were synthesized for 65- and 28-nm technologies, being capable of achieving real-time processing for ${1920}\times {1080}$ videos at 30 frames/s.

Performance Analysis of Depth Intra-Coding in 3D-HEVC

The depth maps intra-frame prediction of 3D High Efficiency Video Coding (3D-HEVC) inherits all texture encoding techniques provided by HEVC and provides new coding tools for depth map predictions. These tools comprise algorithms, such as bipartition modes, intra-picture skip, and DC-only. This paper details these tools and shows how they work together with the original HEVC algorithms in the depth map intra-frame prediction for allowing high-efficiency encoding. Besides, this paper analyzes the encoding time and the encoding mode distribution of the intra-frame prediction tools over different quantization scenarios. We aim to provide support for upcoming works on depth map encoding, including complexity reduction and control, real-time embedded systems implementations, and even the development of improved tools to encode depth maps.

Multi-strategy depth intra mode decision algorithm in 3D-HEVC

The new generation of 3D video coding standard, denoted as 3D High Efficiency Video Coding (3D-HEVC), uses Multi-view Video Plus Depth (MVD) format to represent stereo videos. Due to the reason that the fringe information of depth maps has great influence on the video quality of synthesized views, Depth Modeling Modes (DMMs) are introduced to maintain the quality of depth maps, while causing computational complexity increase tremendously. Therefore, a multi-strategy depth intra mode decision algorithm is proposed in this paper. It incorporates early rough mode decision (RMD) termination strategy, candidate mode reduction strategy and fast DMM decision strategy. The experimental results indicate that the proposed algorithm can obtain an average encoding time reduction of 34.38% for depth map coding, with just 0.43% BDBR increase.

Energy-Aware Motion and Disparity Estimation System for 3D-HEVC With Run-Time Adaptive Memory Hierarchy

The popularization of multimedia services has pushed forward the development of 2D/3D video-capable embedded mobile devices. Such devices require efficient energy/memory-management strategies to deal with severe memory/processing requirements and limited energy supply. Therefore, we propose a motion and disparity estimation (ME and DE) system—the most memory/processing demanding encoding steps—for the 3D High Efficiency Video Coding (3D-HEVC) standard. It was designed for low energy consumption, featuring a run-time adaptive memory hierarchy. The processing unit employs flexible coding order and optimizations to reduce the computational effort by exploring the inter-channel and inter-view redundancies. The memory hierarchy features window-based prefetching, data reuse, subsampling, and dynamic voltage scaling controlled by our depth-based dynamic search window resizing algorithm. Memory results demonstrate an average on-chip energy reduction of 79% in comparison to the widely used Level-C solution for a 45-nm technology. The proposed energy-aware ME and DE system dissipates 7.55 W while processing three HD 1080p views (video + depth) at 30 frames per second and presents a mean energy consumption of 0.107 J per access unit. To the best of our knowledge, this is the first work that proposes a real-time ME/DE system for the 3D-HEVC standard with an adaptive memory hierarchy.

A Complexity Reduction Scheme for Depth Coding in 3D-HEVC

3D-high efficiency video coding (3D-HEVC) is the next-generation compression standard for multiview system applications, which has recently been approved by MPEG and VCEG as an extension of HEVC. To improve the compression efficiency of depth map, several compression tools have been developed for a better representation depth edges. These supplementary coding tools together with existing prediction modes can achieve high compression efficiency, but require a very high complexity that restricts the encoders from ongoing application. In this paper, we introduce a fast scheme to reduce complexity of depth coding in inter and intramode prediction procedure. A simulation analysis is performed to study intra and intermode distribution correlations in the depth compression information. Based on that correlation, we exploit two complexity reduction strategies, including early SKIP and adaptive intra prediction selection. Experimental results demonstrate that our scheme can achieve a complexity reduction up to 63.0%, without any noticeable loss of compression efficiency.

A low-latency DMM-1 encoder for 3D-HEVC

Depth modeling mode 1 (DMM-1) is vital for 3D high-efficiency video coding, since it results in low distortion and is suitable for sharp boundaries. When designing DMM-1 encoders, traditional work used serial methods to calculate all the wedgelets, leading to large latency; they may meet the requirements of 30 fps 1080P 3D video at most, but may be powerless for higher frame rate or 3D video resolution. In this paper, we propose a flexible parallel architecture for DMM-1 encoder. It can simultaneously evaluate all the wedgelets, saving encoding time without increasing distortion; it can also be configured into a partial-parallel architecture to save area. Experiments show that our method can save 33.6–94.3% encoding time for different test schemes. Synthesis results in SMIC 55 nm show that the VLSI design for proposed parallel architecture can meet the requirements of 1080P and higher-resolution video processing in real time.

Analysis of parallel encoding using tiles in 3D High Efficiency Video Coding

This paper analyzes the usage of tiles in 3D-High Efficiency Video Coding. We investigate the encoding impact of some tiles divisions using a Bjontegaard delta rate (BD-rate) on the common test conditions for 3D videos. This analysis demonstrated dividing the video into square tiles increases the encoding performance compared to non-square tiles (about 2.2%). In another analysis, we concluded the workload is highly unbalanced in the uniform distribution contributing to limit the speedup possibilities for the homogeneous multicore systems. Authors are not aware of any work focusing on balancing depth maps workload in tiles; therefore, we pointed out relevant studies and solutions for future projects in this open research area.