3D-HEVC Encoder Research Articles

Vision‐oriented algorithm for fast decision in 3D video coding

This paper designs a novel method to reduce the coding complexity of 3D-HEVC encoder by utilizing the properties of human visual perception. Two vision-oriented edge detections are proposed: for colour texture detection, the authors adopt the Just-Noticeable Distortion (JND); for depth map, the authors combine the Sample Adaptive Offset (SAO) and the Just Noticeable Depth Difference (JNDD) model. The authors also analyse the properties of colour texture and depth map to classify the coding tree unit (CTU) into various kinds of types, including complex-edge CTU, moderate-edge CTU and homogeneous CTU. Besides, fast mode decisions and early termination criteria are performed individually on each type of CTUs according to their characteristics. Especially for those CTUs with more edge information, the proposed projection-based fast mode decision and residual-based early termination preserve important colour texture while speeding up the coding at the same time. The proposed vision-oriented algorithm reduces 31.981% of the overall average coding time with only 1.580% BD-Bitrate increase. Experimental results show that the proposed algorithm can provide considerable time-saving while still maintain the video quality, which outperforms the previous researches.

Complexity and compression efficiency assessment of 3D-HEVC encoder

This paper presents a complexity and compression efficiency assessment of the 3D-High Efficiency Video Coding (3D-HEVC) encoder. An experimental evaluation was carried out to identify the encoding tools with the highest impact on the computational complexity of 3D-HEVC. The evaluated tools were firstly divided into three categories: HEVC, multiview, and depth maps tools. Thus, incremental analyses with a set of 28 encoder configurations were performed to point out the compression efficiency and the computational complexity of each encoding tool individually. Experimental results demonstrated different compression-complexity operation points for the 3D-HTM reference software, providing relevant information for the implementation of 3D-HEVC encoders with different tradeoff possibilities between compression efficiency and complexity. The obtained results also allowed identifying encoding tools that can be further optimized in future works.

Tile Adaptation for Workload Balancing of 3D-HEVC Encoder in Homogeneous Multicore Systems

This paper proposes a tile adaptation for workload balancing to speed up the 3D-High Efficiency Video Coding (3D-HEVC) encoder. Experiments were done to evaluate the usage of tiles when encoding texture and depth maps considering a static uniform partitioning. Based on these evaluations, we observed that the encoding workload of neighboring frames in the same view tends to be similar. Moreover, the encoding workload of dependent views can be predicted using the information of previously encoded frames of the same and base views. Therefore, the proposed solution adaptively defines the boundaries of the tiles based on inter-frame and inter-view correlations to balance the workloads of the tiles, resulting in a higher speedup when executing in homogeneous multicore systems. The proposed method was implemented in 3D-HEVC Test Model and evaluated according to the Common Test Conditions. The results demonstrated that the proposed solution speeds up texture and depth map coding from 9.94% to 18.21%, when compared to the uniform-sized tiles, without coding efficiency loss. To the best of the authors' knowledge, this is the first work proposing a complete solution for speeding up the 3D-HEVC video encoder by balancing the tiles workload dynamically.

Fast CU size and mode decision algorithm for 3D-HEVC intercoding

The 3D High Efficiency Video Coding (3D-HEVC) is an extension of the last 2D video coding generation (HEVC) using the Multi-view Video plus Depth map (MVD) format. In inter-coding process, the 3D-HEVC uses multiples inter prediction modes for each Coding Unit (CU) size. These prediction modes are performed to use all the possible CU sizes to select the one with the least Rate Distortion (RD) cost as the best prediction mode using Lagrange multiplier. It achieved high coding efficiency but it consumes an extremely time coding which limits 3D-HEVC from real-time applications. In this paper, a fast inter prediction mode and CU size decision algorithm is proposed to reduce the complexity of 3D-HEVC inter-coding. This algorithm is based on the homogeneity of CU using structure tensor. An experimental analysis is first performed to study the correlation of the CU complexity with depth levels and prediction modes, then the structure tensor of each CU is extracted to characterize the CU homogeneity, which has strong relationship with coding levels and inter prediction modes. Finally, a fast algorithm is adopted to accomplish an early termination of CU split and skip unnecessary prediction modes. The experimental results demonstrate that the proposed fast algorithm can provide a considerable encoding time-saving with negligible rate-distortion performance loss compared to the original 3D-HEVC encoder.

Fast 3D-HEVC Depth Map Encoding Using Machine Learning

This paper presents a fast depth map encoding for 3D-High Efficiency Video Coding (3D-HEVC) based on static decision trees. We used data mining and machine learning to correlate the encoder context attributes, building the static decision trees. Each decision tree defines that a depth map Coding Unit (CU) must be or not be split into smaller blocks, considering the encoding context through the evaluation of the encoder attributes. Specialized decision trees for I-frames, P-frames and B-frames define the partitioning of $64\times 64$ , $32\times 32$ , and $16\times 16$ CUs. We trained the decision trees using data extracted from the 3D-HEVC Test Model considering all-intra and random-access configurations, and we evaluated the proposed approach considering the common test conditions. The experimental results demonstrated that this approach can halve the 3D-HEVC encoder computational effort with less than 0.24% of BD-rate increase on the average for all-intra configuration. When running on random-access configuration, our solution is able to reduce up to 58% the complete 3D-HEVC encoder computational effort with a BD-rate drop of only 0.13%. These results surpass all related works regarding computational effort reduction and BD-rate.

Fast Depth Map Intra Coding for 3D Video Compression Based Tensor Feature Extraction and Data Analysis

3D high-efficiency video coding (3D-HEVC) is the latest standard for 3D video compression created by the ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) based on a new video format called multiview texture videos plus depth maps (MVDs). In 3D-HEVC depth map intra prediction, the test model uses the conventional HEVC intra modes and new supplementary coding tools called depth modeling modes (DMMs) to better preserve the sharp edges of depth maps. These intra prediction modes fundamentally enhance the depth map intra coding efficiency. Although this process enables high coding efficiency, it also results in a very high encoding complexity, which limits the use of the 3D-HEVC encoder in practical and real-world applications. For most cases, the conventional intra coding and DMMs can be skipped if the current region is classified as a homogenous region. Bringing this intuitive approach to practice, this paper proposes fast depth map intra coding based on tensor feature extraction and data analysis. The experimental results show that the proposed intra model decision brings a good complexity reduction with negligible loss of rate distortion performance.

Probability-Based Depth Intra-Mode Skipping Strategy and Novel VSO Metric for DMM Decision in 3D-HEVC

Multiview video plus depth format has been adopted as the emerging 3D video representation recently. It includes a limited number of textures and depth maps to synthesize additional virtual views. Since the quality of depth maps influences the view synthesis process, their sharp edges should be well preserved to avoid mixing foreground with background. To address this issue, 3D-High Efficiency Video Coding (HEVC) introduces new coding tools, a partition-based intra mode [depth modeling mode (DMM)], a residual description technique [segmentwise depth coding (SDC)], and a more complex rate-distortion (RD) evaluation with view synthesis optimization (VSO), to provide more accurate predictions and achieve higher compression rate. However, these new techniques introduce a lot of possible candidates, and each of them requires complicated RD calculation in the process of intra-mode decision. They lead to unacceptable computational burden in a 3D-HEVC encoder. Therefore, in this paper, we raise two efficient techniques for depth intra-mode decision. First, by investigating the statistical characteristics of variance distributions in the two partitions of DMM, a simple but efficient criterion based on the squared Euclidean distance of variances (SEDV) is suggested to evaluate RD costs of the DMM candidates instead of the time-consuming VSO process. Second, a probability-based early depth intra-mode decision is proposed to select only the most promising mode and make the early determination of using SDC based on the low-complexity RD cost in rough mode decision. Experimental results show that the proposed algorithm with these two new techniques provides 33%–48% time reduction with little drop in the coding performance compared with the state-of-the-art algorithms.

Efficient multiview video plus depth coding for 3D-HEVC based on complexity classification of the treeblock

High efficiency video coding standard-based 3D video (3D-HEVC) has been extended from HEVC to improve the coding efficiency of multiview video plus depth (MVD). Similar to the joint model of HEVC, a computationally expensive exhaustive mode decision is performed to find the least rate-distortion cost for each treeblock in 3D-HEVC. Furthermore, additional coding tools have been added to 3D-HEVC for improving the coding efficiency of the dependent texture video and depth map. Those tools achieve the highest possible coding efficiency, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In order to reduce computational complexity, we propose an efficient multiview video plus depth coding algorithm for 3D-HEVC that adaptively utilizes the complexity classification of the treeblock. The coding complexity model of a treeblock is first analyzed according to the prediction mode and coding mode from the corresponding treeblocks in the reference views. Based on the complexity classification model of the treeblock, we propose two efficient low-complexity approaches, including fast mode size decision and adaptive motion search range selection. Extensive experimental results demonstrate that the proposed MVD coding algorithm can achieve the average computational saving about 60.1% with negligible rate-distortion performance loss in comparison with the original 3D-HEVC encoder.

Fast depth map mode decision based on depth–texture correlation and edge classification for 3D-HEVC

The 3D extension of High Efficiency Video Coding (3D-HEVC) has been adopted as the emerging 3D video coding standard to support the multi-view video plus depth map (MVD) compression. In the joint model of 3D-HEVC design, the exhaustive mode decision is required to be checked all the possible prediction modes and coding levels to find the one with least rate distortion cost in depth map coding. Furthermore, new coding tools (such as depth-modeling mode (DMM) and segment-wise depth coding (SDC)) are exploited for the characteristics of depth map to improve the coding efficiency. These achieve the highest possible coding efficiency to code depth map, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In this paper, we propose a fast depth map mode decision algorithm for 3D-HEVC by jointly using the correlation of depth map-texture video and the edge information of depth map. Since the depth map and texture video represent the same scene at the same time instant (they have the same motion characteristics), it is not efficient to use all the prediction modes and coding levels in depth map coding. Therefore, we can skip some specific prediction modes and depth coding levels rarely used in corresponding texture video. Meanwhile, the depth map is mainly characterized by sharp object edges and large areas of nearly constant regions. By fully exploiting these characteristics, we can skip some prediction modes which are rarely used in homogeneity regions based on the edge classification. Experimental results show that the proposed algorithm achieves considerable encoding time saving while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.

Segment-based view synthesis optimization scheme in 3D-HEVC

The 3D extension of high efficiency video coding (3D-HEVC) adopts a view synthesis optimization (VSO) technique to improve the quality of synthesized views for depth map coding. The exact synthesized view distortion change (SVDC) is calculated in VSO which in turn brings huge coding complexity to the 3D-HEVC encoder due to the real view synthesis process. This work presents a scheme aimed at reducing coding complexity of the SVDC calculating process in the 3D-HEVC encoder. It skips line segments of pixels with variable lengths based on information from both of the textures and depth maps in the SVDC calculation. Experimental results demonstrate that the proposed scheme can reduce the coding complexity without any significant loss in rate distortion performance for the synthesized views.

An efficient depth modeling mode decision algorithm for 3D-HEVC depth map coding

With the purpose of augmenting immersive perception, 3D video coding extension of High Efficiency Video Coding (3D-HEVC) standard is developed. This extension employs Multiview Video plus Depth (MVD) format in which immediate virtual view texture videos can be rendered by adjacent depth maps, so that bitrates and coding runtime of multiview video are saved. A good preservation of depth map is vital to synthesize high-quality virtual view texture videos. Hence, several new depth map coding tools have been introduced into 3D-HEVC, such as Depth Modeling Mode (DMM) decision. However, the numerous DMM candidates result in a huge computational complexity, which prevents 3D-HEVC for practical use. To this end, we propose an efficient DMM decision algorithm for 3D-HEVC depth map coding, which incorporates unnecessary depth blocks skipping approach and Contour pattern early termination method. Simulated experiment results state that the proposed efficient DMM decision method can save about 31.33% coding time on average with only 0.01dB of PSNR drop and just 0.60% bitrate increase on average compared to the original 3D-HEVC encoder.

Online-Learning-Based Complexity Reduction Scheme for 3D-HEVC

3-D High Efficiency Video Coding (HEVC) is a new emerging video compression standard for multiview video applications. This standard utilizes advanced interview prediction characteristics in addition to the prediction features of the HEVC standard for efficient encoding of multiview video content. While using combined features improves the compression performance by utilizing the correlation between the views captured from slightly different angles of the same scene, they also increase coding complexity. The focus of this paper is on developing an efficient complexity reduction scheme for 3D-HEVC, with the intention to facilitate the adoption of this upcoming standard, especially for real-time applications. In this regard, first, we introduce two ways to decrease the complexity of the inter-/ intra-mode search process of the to-be-encoded blocks in the dependent texture views ( ${\mathrm {DV}}_{t}\text{s}$ ) of 3D-HEVC. Then, we propose a hybrid complexity reduction scheme that utilizes the two-mode prediction approaches, motion information of the base texture view (BV t ), and the rate distortion cost of the already encoded blocks in the BV t and DV t . The performance of our proposed scheme is tested for the case with two views (i.e., base view + dependent view). The evaluations confirm that our proposed hybrid complexity reduction scheme reduces the 3D-HEVC codec complexity by 67.70% on average for the DV t compared with the unmodified 3D-HEVC encoder, while maintaining the overall video quality. Compared with the state-of-the-art method, it reduces complexity by 25.74% on average.

Fast 3D-HEVC encoder algorithm for multiview video plus depth coding

The three-dimensional high efficiency video coding (3D-HEVC) is an extension of the HEVC standard for the compression of the multiview video plus depth (MVD) format. In 3D-HEVC, additional coding tools have been added to HEVC for improving the coding efficiency of the dependent video views and depth maps. Those tools achieve the highest possible coding efficiency, but also bring extremely high computational complexity which limits 3D-HEVC from real-time applications. In this paper, we propose a fast 3D-HEVC encoder algorithm based on the texture video and depth map correlation to reduce MVD coding computational complexity. Since the video texture and depth map represent the same scene at the same time instant (they have the same motion characteristics), it is not efficient to use all the prediction modes in 3D-HEVC mode decision procedure. The basic idea of the algorithm is to utilize the texture video and depth map property of coding information to predict the current CU prediction mode and early skip unnecessary variable-size mode decision. Experimental results demonstrate that the proposed fast 3D-HEVC encoder algorithm can save 75% computational complexity on average, while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.

Low complexity depth map intra prediction mode decision algorithm for 3D-HEVC

The presented 3D video coding extension of High Efficiency Video Coding (3D-HEVC) was developed for multiview video plus depth (MVD) format. MVD is depth-enhanced 3D video format that uses a few views to synthesize more immediate virtual views with depth map. In the original 3D-HEVC encoder, as a novel tool Depth Modeling Modes (DMM) is added into intra prediction mode candidate list, which aims at improving as much coding efficiency as possible for 3D-HEVC. However, this promotion leads to extremely complicated computation simultaneously which prevents 3D-HEVC encoder from real-time application. In this paper, we propose a low complexity depth map intra prediction mode decision algorithm, including fast angular modes decision algorithm and early skipping DMM decision algorithm. The basic idea of the proposed algorithm is to predict the current depth block intra prediction mode based on depth map texture feature and early skip DMM decision. Experimental results demonstrate that the proposed low complexity algorithm can achieve 38% coding time reduction with just 1.00% bitrate augment on average under “all-intra” case compared to the original 3D-HEVC encoder.

Fast intra mode decision for depth coding in 3D-HEVC

The emergent 3D High Efficiency Video Coding (3D-HEVC) is an extension of the High Efficiency Video Coding (HEVC) standard for the compression of the multi-view texture videos plus depth maps format. Since depth maps have different statistical properties compared to texture video, various new intra tools have been added to 3D-HEVC depth coding. In current 3D-HEVC, new intra tools are utilized together with the conventional HEVC intra prediction modes for depth coding. This technique achieves the highest possible coding efficiency, but leads to an extremely high computational complexity which limits 3D-HEVC from practical applications. In this paper, we propose a fast intra mode decision algorithm for depth coding in 3D-HEVC. The basic idea of the proposed algorithm is to utilize the depth map characteristics to predict the current depth prediction mode and skip some specific depth intra modes rarely used in 3D-HEVC depth coding. Based on this analysis, two fast intra mode decision strategies are proposed including reduction of the number of conventional intra prediction modes, and simplification of depth modeling modes (DMMs). Experimental results demonstrate that the proposed algorithm can save 30 % coding runtime on average while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.

Fast mode decision algorithm for 3D-HEVC encoding optimization based on depth information

The advanced 3D extension of the High Efficiency Video Coding standard (3D-HEVC) encodes 3D videos in the multi-view video plus depth format (MVD) and introduces new prediction modes to improve the coding performance of dependent views and depth data. Similar to HEVC, the mode decision process in 3D-HEVC is performed using a large number of prediction modes to find the one mode with the least rate distortion (RD) cost using Lagrange multiplier. These techniques achieve the highest possible coding efficiency, but also bring high computational complexity, which may limit the use of 3D-HEVC in practical applications. In this paper, we propose a fast mode decision algorithm based on the depth information to reduce 3D-HEVC computational complexity. Since the video texture and depth map are highly content-dependent, it is not efficient to use all the prediction modes in 3D-HEVC mode decision procedure. There exist high correlations of the coding information when two CUs have the similar depth values. Based on the correlations, we propose two efficient mode decision techniques, 1) an early coding level determination is developed based on the spatially neighboring CUs depth values; 2) an adaptive mode size decision is introduced based on the correlations between the motion characteristics and the depth values. Experimental results show that the proposed algorithm can save 74% computational complexity on average with only 1.47% increase on bit rate or 0.06 dB loss on peak signal-to-noise ratio (PSNR) compared with the original 3D-HEVC encoder.

Low-complexity depth map compression in HEVC-based 3D video coding

In this paper, a low-complexity algorithm is proposed to reduce the complexity of depth map compression in the high-efficiency video coding (HEVC)-based 3D video coding (3D-HEVC). Since the depth map and the corresponding texture video represent the same scene in a 3D video, there is a high correlation among the coding information from depth map and texture video. An experimental analysis is performed to study depth map and texture video correlation in the coding information such as the motion vector and prediction mode. Based on the correlation, we propose three efficient low-complexity approaches, including early termination mode decision, adaptive search range motion estimation (ME), and fast disparity estimation (DE). Experimental results show that the proposed algorithm can reduce about 66% computational complexity with negligible rate-distortion (RD) performance loss in comparison with the original 3D-HEVC encoder.

Effective early termination algorithm for depth map intra coding in 3D‐HEVC

In the current high-efficiency video coding-based three-dimensional video coding (3D-HEVC) design, new depth intra modes including depth modelling modes and region boundary chain coding are applied for depth map coding. These partition-based intra modes achieve the highest possible coding efficiency, but result in extremely large encoding time which obstructs the 3D-HEVC from practical applications. An efficient early termination algorithm for depth map coding in 3D-HEVC is proposed. It makes use of the coding information from the spatial neighbouring depth map treeblock and the co-located texture video treeblock to predict the depth map intra mode treeblock and terminate its mode decision process early. Experimental results show that the proposed algorithm can achieve an average computational saving of about 40% with negligible loss of rate distortion performance in the 3D-HEVC encoder.