3D Video Coding Research Articles

Efficient representation of disoccluded regions in 3D video coding

Efficient representation of disoccluded regions in 3D video coding

Efficient CU Decision Algorithm for VVC 3D Video Depth Map Using GLCM and Extra Trees

The new generation of 3D video is an international frontier research hotspot. However, the large amount of data and high complexity are core problems to be solved urgently in 3D video coding. The latest generation of video coding standard versatile video coding (VVC) adopts the quad-tree with nested multi-type tree (QTMT) partition structure, and the coding efficiency is much higher than other coding standards. However, the current research work undertaken for VVC is less for 3D video. In light of this context, we propose a fast coding unit (CU) decision algorithm based on the gray level co-occurrence matrix (GLCM) and Extra trees for the characteristics of the depth map in 3D video. In the first stage, we introduce an edge detection algorithm using GLCM to classify the CU in the depth map into smooth and complex edge blocks based on the extracted features. Subsequently, the extracted features from the CUs, classified as complex edge blocks in the first stage, are fed into the constructed Extra trees model to make a fast decision on the partition type of that CU and avoid calculating unnecessary rate-distortion cost. Experimental results show that the overall algorithm can effectively reduce the coding time by 36.27–51.98%, while the Bjøntegaard delta bit rate (BDBR) is only increased by 0.24% on average which is negligible, all reflecting the superior performance of our method. Moreover, our algorithm can effectively ensure video quality while saving much encoding time compared with other algorithms.

Vision‐oriented algorithm for fast decision in 3D video coding

This paper designs a novel method to reduce the coding complexity of 3D-HEVC encoder by utilizing the properties of human visual perception. Two vision-oriented edge detections are proposed: for colour texture detection, the authors adopt the Just-Noticeable Distortion (JND); for depth map, the authors combine the Sample Adaptive Offset (SAO) and the Just Noticeable Depth Difference (JNDD) model. The authors also analyse the properties of colour texture and depth map to classify the coding tree unit (CTU) into various kinds of types, including complex-edge CTU, moderate-edge CTU and homogeneous CTU. Besides, fast mode decisions and early termination criteria are performed individually on each type of CTUs according to their characteristics. Especially for those CTUs with more edge information, the proposed projection-based fast mode decision and residual-based early termination preserve important colour texture while speeding up the coding at the same time. The proposed vision-oriented algorithm reduces 31.981% of the overall average coding time with only 1.580% BD-Bitrate increase. Experimental results show that the proposed algorithm can provide considerable time-saving while still maintain the video quality, which outperforms the previous researches.

Deep Multi-Domain Prediction for 3D Video Coding

Three-dimensional (3D) video contains plentiful multi-domain correlations, including spatial, temporal, and inter-view correlations. In this paper, a deep multi-domain prediction method is proposed for 3D video coding. Different from previous methods, our proposed method utilizes not only spatial and temporal correlations but also inter-view correlation to obtain a more accurate prediction, and adopts deep convolutional neural networks to effectively fuse multi-domain references. More specifically, a hierarchical prediction mechanism, which includes a spatial-temporal prediction network and a multi-domain prediction network, is designed to overcome the fusion difficulty of multi-domain reference information. Furthermore, a progressive spatial-temporal prediction network and a multi-scale multi-domain prediction network are designed to obtain the spatial-temporal prediction result and multi-domain prediction result respectively. Experimental results show that the proposed method achieves considerable bitrate saving compared with 3D-HEVC.

Fast Texture Coding Based on Spatial, Temporal and Inter-View Correlations for 3D Video Coding

3D-HEVC (The 3D Extension of High Efficiency Video Coding), the latest international standard for 3D video coding, supports multiview plus depth 3D video format to enrich multimedia applications. For the texture coding, 3D-HEVC utilizes not only the information of temporal and spatial domains but also that of inter-view domain. However, the time consumption and complexity of 3D-HEVC also increase significantly. In this paper, fast texture coding algorithm for 3D-HEVC is proposed. We individually calculate the Pearson correlation coefficients by the rate-distortion costs (RD-costs) of coding tree units (CTUs) in the temporal, spatial and inter-view domains to analyze the correlations for independent view and dependent view. The proposed coding algorithm is based on the coding information of CTUs with higher correlations. The fast algorithm predicts and dynamically adjusts the depth range of the coding unit (CU). The prediction unit (PU) mode decision is proposed according to the complexity and partition direction of the best PU modes obtained from the highly correlated CTUs. The search range is adaptively adjusted for motion estimation. In addition, the RD-cost threshold is estimated to early terminate the CU split. Experimental results show that the proposed fast texture coding algorithm reduces the texture coding time significantly and outperforms numerous previous works.

HEVC-based three-layer texture and depth coding for lossless synthesis in 3D video coding

Efficient lossless coding of a texture image and its corresponding depth map is important to perform accurate view synthesis in 3D applications. In this paper, a novel HEVC-based three-layer texture and depth coding method is proposed for lossless synthesis in 3D video coding. The proposed method performs lossy and lossless texture coding in the first and second layers, respectively. A quantization parameter (QP) in the first lossy coding layer is adaptively selected for each block, based on a relationship between the first and second layers. In the third layer, the lossy depth coding is performed by using synthesis-based depth coding and texture-based depth intra prediction mode. The synthesis-based depth coding technique adopts the lossy coding but guarantees zero synthesis distortion. The texture-based depth intra prediction mode performs the depth prediction by using the associated texture information. Experimental results demonstrate that the proposed method obtains higher coding performance than conventional lossless coding methods.

Fast depth map intra-mode selection for 3D-HEVC intra-coding

As the last effort of ISO/IEC MPEG and ITU-T Video Coding Experts Group for 3D video coding, high-efficiency video coding (HEVC)-based 3D video coding (3D-HEVC) is the new international emerging standard to compress multi-view texture videos plus depth maps data. In the test model of 3D-HEVC, the conventional HEVC intra-prediction modes and new additional intra-coding tools called depth modeling modes (DMMs) are exploited for depth map intra-coding. These intra-modes provide important coding improvement. However, the improvement of the coding efficiency is obtained at the expense of a computational complexity increase, which obstructs 3D-HEVC from practical real-world applications. Previously, we propose a depth map intra-model decision using tensor features as homogeneity determination. In this paper, we propose an amelioration of the previously proposed depth map intra-model decision using an efficient homogeneity determination. In most cases, the angular intra-coding and DMM could be skipped if some specific conditions are satisfied. The experimental results show that the proposed intra-model decision can reduce the encoding time significantly, with negligible loss of rate distortion performance.

Overview of current development in depth map coding of 3D video and its future

3D videos have attracted attention from academia and industry after great success in the film industry. Multiview video plus depth (MVD) is the most popular 3D video format to provide vivid 3D feeling and has been adopted as an international 3D video coding standard, namely 3D extension of high efficiency video coding (HEVC). MVD includes a limited number of textures and depth maps to synthesise virtual views. In MVD, depth samples describe the distance between a camera and an actual object as a grey-level image. The characteristics of depth maps are quite different from texture images. Consequently, new coding tools are designed for depth maps in 3D-HEVC to improve the coding efficiency at the expense of high-computational complexity, which faces great challenges in coding systems. Depth map coding is also an important technique in immersive media to support three degrees of freedom 3DoF+/6DoF applications such as virtual reality/augmented reality. The study starts with an overview of what has been done over the last decade in 3D-HEVC, especially depth map coding, regarding theories, methodologies, current research and state-of-the-art fast approaches. Following this, a comprehensive comparison of the reviewed techniques is presented, and an outlook on its future trends is provided.

Fast CU size and mode decision algorithm for 3D-HEVC intercoding

The 3D High Efficiency Video Coding (3D-HEVC) is an extension of the last 2D video coding generation (HEVC) using the Multi-view Video plus Depth map (MVD) format. In inter-coding process, the 3D-HEVC uses multiples inter prediction modes for each Coding Unit (CU) size. These prediction modes are performed to use all the possible CU sizes to select the one with the least Rate Distortion (RD) cost as the best prediction mode using Lagrange multiplier. It achieved high coding efficiency but it consumes an extremely time coding which limits 3D-HEVC from real-time applications. In this paper, a fast inter prediction mode and CU size decision algorithm is proposed to reduce the complexity of 3D-HEVC inter-coding. This algorithm is based on the homogeneity of CU using structure tensor. An experimental analysis is first performed to study the correlation of the CU complexity with depth levels and prediction modes, then the structure tensor of each CU is extracted to characterize the CU homogeneity, which has strong relationship with coding levels and inter prediction modes. Finally, a fast algorithm is adopted to accomplish an early termination of CU split and skip unnecessary prediction modes. The experimental results demonstrate that the proposed fast algorithm can provide a considerable encoding time-saving with negligible rate-distortion performance loss compared to the original 3D-HEVC encoder.

PoM: Power-Efficient Multi-View Video Streaming Over Multi-Antenna Wireless Systems

Multi-view video streaming is becoming more popular, with increased potential for immersive applications. However, it poses a challenge in terms of energy consumption on the transmission devices with limited battery power due to the vast traffic volume. Previous literature has considered efficient video transmission but not exploited the full potential of joint design of 3D video coding and MIMO physical layer. In this paper, we present the first study on power consumption minimization for multi-view video streaming with a quality guarantee in 802.11 MIMO-OFDM systems. We propose an efficient algorithm to perform antenna assignment and transmission power allocation, by exploiting both the source coding characteristics of MVC and channel diversity of multiple antennas. A proof-of-concept system, namely PoM , is designed on the software-defined radio platform and is evaluated in realistic indoor environments. To the best of our knowledge, this is the first practical system for energy-efficient multi-view video streaming. Experimental results show that PoM can significantly save energy in the transmission by 88% ~ 26% on average for the PSNR requirement ranging from 35dB to 45dB, and 85% ~ 12% for SSIM from 0.80 to 0.95.

Texture-Distortion-Constrained Joint Source-Channel Coding of Multi-View Video Plus Depth-Based 3D Video

A novel joint source and channel coding scheme tailored to 3D video is proposed in this paper to minimize the end-to-end view synthesis distortion within a given total bit rate for both texture and depth as well as a maximum tolerable distortion constraint for texture. First, we formulate a joint texture and depth coding mode selection strategy for error-resilient source coding of multi-view video plus depth-based 3D video through using the Lagrange multiplier method. Then, by considering the effect of residual errors after channel coding, we evolve to a more general formulation that jointly optimizes error-resilient source coding and channel coding in an integrated manner for unequal error protection between texture and depth, for which a theoretic solution using a proposed dual-trellis is derived. Finally, we extend the general formulation by including the texture distortion constraint. We show how to optimize the view synthesis quality while simultaneously catering to the texture quality constraint. Experimental results demonstrate the proposed algorithm has much better performance than existing related work.

Fast 3D-HEVC PU size decision algorithm for depth map intra-video coding

High-Efficiency Video Coding (HEVC)-based 3D video coding (3D-HEVC) is the most recent standard and last exertion of ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) for 3D video coding using a new data video format called Multi-View Video plus Depth map (MVD). This new standard achieves a high coding improvement. In any case, one of the most critical difficulties in 3D-HEVC is time computational complexity. The depth map intra-prediction is a critical factor in 3D-HEVC intra-coding, in which, the 3D-HEVC uses a highly adaptable Coding Unit (CU) structure with a specific end goal to expand the coding efficiency of all depth map characteristics. However, it results in an enormous Rate Distortion Optimization Cost (RDO-Cost) because of the broad recursive search for the best CU size from $$64\times 64$$ down to $$4\times 4$$ . This computational complexity excludes the 3D-HEVC from true and real-time application. Hence, it is imperative to build up an algorithm to diminish the complexity of the size decision in depth map intra-coding. To determine the previously mentioned issue, this paper proposes an effective 3D-HEVC PU size decision algorithm for depth map intra-video coding based on tensor features and statistical data analyses. The experimental results demonstrate that the proposed model diminishes the complexity of depth map size decision significantly with low rate distortion increase.

Multi-strategy depth intra mode decision algorithm in 3D-HEVC

The new generation of 3D video coding standard, denoted as 3D High Efficiency Video Coding (3D-HEVC), uses Multi-view Video Plus Depth (MVD) format to represent stereo videos. Due to the reason that the fringe information of depth maps has great influence on the video quality of synthesized views, Depth Modeling Modes (DMMs) are introduced to maintain the quality of depth maps, while causing computational complexity increase tremendously. Therefore, a multi-strategy depth intra mode decision algorithm is proposed in this paper. It incorporates early rough mode decision (RMD) termination strategy, candidate mode reduction strategy and fast DMM decision strategy. The experimental results indicate that the proposed algorithm can obtain an average encoding time reduction of 34.38% for depth map coding, with just 0.43% BDBR increase.

Energy-Aware Motion and Disparity Estimation System for 3D-HEVC With Run-Time Adaptive Memory Hierarchy

The popularization of multimedia services has pushed forward the development of 2D/3D video-capable embedded mobile devices. Such devices require efficient energy/memory-management strategies to deal with severe memory/processing requirements and limited energy supply. Therefore, we propose a motion and disparity estimation (ME and DE) system—the most memory/processing demanding encoding steps—for the 3D High Efficiency Video Coding (3D-HEVC) standard. It was designed for low energy consumption, featuring a run-time adaptive memory hierarchy. The processing unit employs flexible coding order and optimizations to reduce the computational effort by exploring the inter-channel and inter-view redundancies. The memory hierarchy features window-based prefetching, data reuse, subsampling, and dynamic voltage scaling controlled by our depth-based dynamic search window resizing algorithm. Memory results demonstrate an average on-chip energy reduction of 79% in comparison to the widely used Level-C solution for a 45-nm technology. The proposed energy-aware ME and DE system dissipates 7.55 W while processing three HD 1080p views (video + depth) at 30 frames per second and presents a mean energy consumption of 0.107 J per access unit. To the best of our knowledge, this is the first work that proposes a real-time ME/DE system for the 3D-HEVC standard with an adaptive memory hierarchy.

Fast Depth Intra Mode Decision Based on Mode Analysis in 3D Video Coding

Multiview video plus depth (MVD), which consists of a texture image and its associated depth map, has been introduced as a 3D video format, and 3D video coding, such as 3D-HEVC, was developed to efficiently compress this MVD data. However, this requires high encoding complexity because of the additional depth coding. In particular, intra coding using various prediction modes is very complicated. To reduce the complexity, we propose a fast depth intra mode decision method based on mode analysis. The proposed method adaptively reduces the number of original candidate modes in a mode decision process. Experimental results show that the proposed method achieves high performance in terms of the complexity reduction.

Analysis of parallel encoding using tiles in 3D High Efficiency Video Coding

This paper analyzes the usage of tiles in 3D-High Efficiency Video Coding. We investigate the encoding impact of some tiles divisions using a Bjontegaard delta rate (BD-rate) on the common test conditions for 3D videos. This analysis demonstrated dividing the video into square tiles increases the encoding performance compared to non-square tiles (about 2.2%). In another analysis, we concluded the workload is highly unbalanced in the uniform distribution contributing to limit the speedup possibilities for the homogeneous multicore systems. Authors are not aware of any work focusing on balancing depth maps workload in tiles; therefore, we pointed out relevant studies and solutions for future projects in this open research area.

Study of 3D Video Compression Using Nonlinear Depth Representation

The extensions of Advanced Video Coding (AVC) and High-Efficiency Video Coding (HEVC) for multiview video plus depth allow for the coding of 3D video scenes. During the standardization of these extensions, it was demonstrated that compression performance can often be improved by the application of nonlinear depth transformation prior to the compression and applying the respective backward nonlinear depth transformation after decompression. Such an approach is referred to as the compression of nonlinear depth representation (NDR). In this paper, a survey of the NDR technology is provided, but a large part of this survey comprises yet unpublished results. First, the motivation and rationale behind the idea of NDR are discussed. The options for the choice of nonlinear transformation are provided with the respective formulations. Also, the standardization of 3D video coding with NDR within ISO/IEC and ITU is discussed along with the experimental results that demonstrate benefits related to the application of NDR in 3D-AVC and 3D-HEVC. Furthermore, the experimental comparisons are provided for simulcast coding of multiview plus depth video using AVC and HEVC technology with and without the NDR. The similar experimental results are also provided for simulcast coding using the current version of Versatile Video Coding technology that is under development for the forthcoming standard of ISO/IEC (MPEG-I) and ITU. All of the experimental results have been obtained for the standard JVC-3V video test sequences under common test conditions for 3D video coding.

Bit Allocation Algorithm based on SSIM for 3D Video Coding

Bit Allocation Algorithm based on SSIM for 3D Video Coding

Pixel-Level View Synthesis Distortion Estimation for 3-D Video Coding

Recently, region-based 3D video coding has been proposed. However, existing view synthesis distortion estimation (VSDE) methods are performed at the frame level. To guide the rate-distortion optimization process of region-based 3D video coding schemes, this paper proposes the first pixel-level VSDE (PL-VSDE) method. We first give the definition of the pixel-level view synthesis distortion. To estimate it, a backward prediction method is then developed, which starts from the pixels of interest (POIs) in the virtual view and finds their corresponding pixels in the reference view via a coarse-to-fine approach, denoted as coarse-to-fine backward prediction (CFBP) method. Additionally, the CFBP fully considers the details of 3D warping, the rounding operation and the warping competition in view synthesis, leading to improve accuracy of the prediction. Besides, a table-lookup method and a warping property are introduced to speed up the CFBP. After integrating the CFBP into the PL-VSDE, we can estimate the view synthesis distortion at the pixel level. Our method is carried out pixel-by-pixel independently, which is friendly for parallel processing. The experimental results demonstrate that our proposed method has significant advantages in both accuracy and efficiency compared with the state-of-the-art frame-level VSDE methods.

Parallelism exploration for 3D high-efficiency video coding depth modeling mode one

This article presents a parallelism exploration over the depth modeling mode 1 (DMM-1) encoding algorithm of the 3D high-efficiency video coding (3D-HEVC) standard and applied the proposed solutions in a multicore central processing unit (CPU) and two graphics processor unities (GPU). The article evaluates efficient parallel algorithms for DMM-1, which also take advantage of simplifications proposed in our previous works. We demonstrate that DMM-1 can obtain a scalable speedup when running in systems with several available cores even when simplifications are being applied. Experimental results for 1920 × 1088 resolution videos show that the proposed parallel algorithms achieved up to 2 frames per second (fps) in a four-cores (with eight threads) CPU and more than 30 fps in two different GPUs. Therefore, the speedup attained with GPU enables real-time 3D-video encoding applying the proposed parallelism strategies together with the DMM-1 proposed simplifications.