3D Video Coding Research Articles

Binocular Suppression-Based Stereoscopic Video Coding by Joint Rate Control With KKT Conditions for a Hybrid Video Codec System

Asymmetric video coding based on binocular suppression provides a prospect for improving coding efficiency since one view in stereoscopic video can be encoded at lower bitrates than the other view without a loss of perceptual video quality. In this paper, we propose a joint rate control scheme for asymmetric stereoscopic video coding in a hybrid video codec system, which supports binocular suppression-based asymmetric stereoscopic video coding in a theoretical basis within an optimization framework. To obtain an optimal solution to quantization steps for joint rate control, we apply the Karush-Kuhn-Tucker (KKT) conditions in minimizing the perceptual distortion of decoded video under the conditions that: 1) the sum of bits generated from two video encoders is constrained within a given bit budget; 2) the quality of a primary view is superior to that of an auxiliary view with an allowable difference between two view qualities; and 3) the distortion of an auxiliary view is less than a given threshold value for high bitrate encoding. As a result, the proposed rate control scheme effectively enhances the perceptual quality based on an optimization framework by taking into account the binocular suppression and also simultaneously controlling the rates of both the left and right views of stereoscopic video under the constraints with a given target bit budget and an allowed quality difference. Experimental results demonstrate that, compared with the independent rate control scheme, the proposed joint rate control scheme with KKT conditions for asymmetric stereoscopic video coding not only demonstrates comparable 3-D perceived visual quality with an overall gain of 0.33 dB, but also attains 2-D visual quality gain by an average of 2.49 dB while accurately satisfying the given constraints in the proposed optimization framework

Motion Hooks for the Multiview Extension of HEVC

MV-HEVC refers to the multiview extension of High Efficiency Video Coding (HEVC). At the time of writing, MV-HEVC was being developed by the Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V) of International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Moving Picture Experts Group and ITU-T VCEG. Before HEVC itself was technically finalized in January 2013, the development of MV-HEVC had already started and it was decided that MV-HEVC would only contain high-level syntax changes compared with HEVC, i.e., no changes to block-level processes, to enable the reuse of the first-generation HEVC decoder hardware as is for constructing an MV-HEVC decoder with only firmware changes corresponding to the high-level syntax part of the codec. Consequently, any block-level process that is not necessary for HEVC itself but on the other hand is useful for MV-HEVC can only be enabled through so-called hooks. Motion hooks refer to techniques that do not have a significant impact on the HEVC single-view version 1 codec and can mainly improve MV-HEVC. This paper presents techniques for efficient MV-HEVC coding by introducing hooks into the HEVC design to accommodate inter-view prediction in MV-HEVC. These hooks relate to motion prediction, hence named motion hooks. Some of the motion hooks developed by the authors have been adopted into HEVC during its finalization. Simulation results show that the proposed motion hooks provide on average 4% of bitrate reduction for the views coded with inter-view prediction.

Fast motion and disparity estimation for HEVC based 3D video coding

The emerging international standard for high efficiency video coding (HEVC) based 3D video coding (3D-HEVC) is an extension of HEVC. In the test model of 3D-HEVC, variable size motion estimation (ME) and disparity estimation (DE) are both employed to select the best coding mode for each treeblock in the encoding process. This technique achieves the highest possible coding efficiency, but it brings extremely high computational complexity which limits 3D-HEVC from practical applications. In this paper, a fast ME/DE algorithm based on inter-view and spatial correlations is proposed to reduce 3D-HEVC computational complexity. Since the multi-view videos represent the same scene with similar characteristic, there is a high correlation among the coding information from inter-view prediction. Besides, the homogeneous regions in texture video have a strong spatial correlation, and thus spatially neighboring treeblocks have similar coding information. Therefore, we can determine ME search range and skip some specific ME and DE rarely used in the previously coded view frames and spatially neighboring coding unit. Experimental results demonstrate that the proposed algorithm can significantly reduce computational complexity of 3D-HEVC encoding while maintaining almost the same rate-distortion performance.

Panorama view with spatiotemporal occlusion compensation for 3D video coding.

The future of novel 3D display technologies largely depends on the design of efficient techniques for 3D video representation and coding. Recently, multiple view plus depth video formats have attracted many research efforts since they enable intermediate view estimation and permit to efficiently represent and compress 3D video sequences. In this paper, we present spatiotemporal occlusion compensation with panorama view (STOP), a novel 3D video coding technique based on the creation of a panorama view and occlusion coding in terms of spatiotemporal offsets. The panorama picture represents the most of the visual information acquired from multiple views using a single virtual view, characterized by a larger field of view. Encoding the panorama video with state-of-the-art HECV and representing occlusions with simple spatiotemporal ancillary information STOP achieves high-compression ratio and good visual quality with competitive results with respect to competing techniques. Moreover, STOP enables free viewpoint 3D TV applications whilst allowing legacy display to get a bidimensional service using a standard video codec and simple cropping operations.

Intra Mode Selection for Depth Map Coding in 3D-HEVC

3D high efficiency video coding was recently standardized by the Joint Collaborative Team on 3D Video coding as an extension of high efficiency video coding. In the current 3D high efficiency video coding method, new depth intra modes including Depth Modeling Modes and Region Boundary Chain coding are used for depth map coding. These partition-based intra modes achieve the highest possible coding efficiency, but take an extremely long time, which hinders the practical application of the new standard. In this paper, we propose an efficient intra mode selection algorithm for depth map coding in 3D high efficiency video coding. It makes use of the coding information from a spatial neighboring depth map treeblock and the co-located texture video treeblock to predict the depth map intra mode treeblock and terminate its mode decision process early. The experimental results show that the proposed intra mode selection algorithm can achieve the average computational savings of about 40% with a negligible loss of rate distortion performance in 3D high efficiency video coding intra-only configuration.

Visual Attention Model Aided Non-Uniform Asymmetric Coding of Stereoscopic Video

Owing to its wide-interoperability, stereoscopic 3D video format in High Definition (HD) is a popular choice for 3D entertainment media distribution. However, the delivery over bandwidth constrained networks exhibits challenges in terms of intermittent congestions in the network traffic, which enforce the delivery system to perform perception-aware coding to save bandwidth. In the scope of stereoscopic 3D video, asymmetric quality adaptation has proved to be an effective method in terms of maintaining the perceived quality while reducing the required transmission bandwidth. On the other hand, Region-Of-Interest (ROI) based coding in accordance with the visual attention cues, which offers non-uniform quality assignment to regions of different saliency levels has not been widely studied in combination with asymmetric coding of stereoscopic 3D video. In this work, the effectiveness of using visual attention aided non-uniform asymmetric 3D video coding is explored. The importance of incorporating compression artefacts in the formulation of visual attention model is also revealed. The discussions in this paper are based on a comprehensive subjective test with 8 stereoscopic video sequences of different spatial and temporal characteristics at different conditions.

Quality-Efficient Upsampling Method for Asymmetric Resolution Stereoscopic Video Coding With Interview Motion Compensation and Error Compensation

In asymmetric resolution stereoscopic video coding (ARSVC), to reduce the bitrate required for bandwidth-limited channels, each downsampled right-view frame is a quarter the size of the corresponding left-view frame and will be upsampled to the original size by the decoder. In this paper, two upsampling methods for ARSVC are proposed. The first proposed method integrates the traditional Wiener filter-based method and the interview prediction scheme by incorporating the information from the similarity between the left-view and right-view frames. By compensating for the prediction errors, the second proposed method further improves the quality of the upsampled images obtained by the first proposed method, especially in the sequences with heavy irregular textures.

Low complexity mode decision for 3D-HEVC.

High efficiency video coding- (HEVC-) based 3D video coding (3D-HEVC) developed by joint collaborative team on 3D video coding (JCT-3V) for multiview video and depth map is an extension of HEVC standard. In the test model of 3D-HEVC, variable coding unit (CU) size decision and disparity estimation (DE) are introduced to achieve the highest coding efficiency with the cost of very high computational complexity. In this paper, a fast mode decision algorithm based on variable size CU and DE is proposed to reduce 3D-HEVC computational complexity. The basic idea of the method is to utilize the correlations between depth map and motion activity in prediction mode where variable size CU and DE are needed, and only in these regions variable size CU and DE are enabled. Experimental results show that the proposed algorithm can save about 43% average computational complexity of 3D-HEVC while maintaining almost the same rate-distortion (RD) performance.

A Novel Fast Intra Prediction Scheme for Depth-Map in 3D High Efficiency Video Coding

A Novel Fast Intra Prediction Scheme for Depth-Map in 3D High Efficiency Video Coding

Fast mode decision for 3D-HEVC depth intracoding.

The emerging international standard of high efficiency video coding based 3D video coding (3D-HEVC) is a successor to multiview video coding (MVC). In 3D-HEVC depth intracoding, depth modeling mode (DMM) and high efficiency video coding (HEVC) intraprediction mode are both employed to select the best coding mode for each coding unit (CU). This technique achieves the highest possible coding efficiency, but it results in extremely large encoding time which obstructs the 3D-HEVC from practical application. In this paper, a fast mode decision algorithm based on the correlation between texture video and depth map is proposed to reduce 3D-HEVC depth intracoding computational complexity. Since the texture video and its associated depth map represent the same scene, there is a high correlation among the prediction mode from texture video and depth map. Therefore, we can skip some specific depth intraprediction modes rarely used in related texture CU. Experimental results show that the proposed algorithm can significantly reduce computational complexity of 3D-HEVC depth intracoding while maintaining coding efficiency.

Low Complexity Adaptive View Synthesis Optimization in HEVC Based 3D Video Coding

In this correspondence, we explore a low-complexity adaptive view synthesis optimization (VSO) scheme in the upcoming high-efficiency video coding (HEVC)-based 3-D video coding standard. We first devise a novel zero-synthesized view difference (ZSVD) model which jointly accounts for the distortion of the synthesized view induced by the compound impact of depth-disparity mapping, texture adaptation, and occlusion in the view synthesis process. This model can efficiently estimate the maximum allowable depth distortion in synthesizing a virtual view without introducing any geometry distortion. Then, an adaptive ZSVD-aware VSO scheme is proposed by incorporating the ZSVD model into the rate-distortion optimization process, which is developed by pruning the conventional view synthesis algorithm. Extensive experimental results confirm that the proposed model is capable of accurately predicting the zero distortion of the synthesized view and exhibit that the proposed ZSVD-aware VSO scheme can remarkably reduce the coding computational complexity with negligible performance loss.

Joint Bit Allocation and Rate Control for Coding Multi-View Video Plus Depth Based 3D Video

In three-dimensional (3D) video coding, distortion in texture video and depth maps can all affect the quality of the synthesized virtual views. Therefore, under the total bitrate constraint, effective bit allocation between texture and depth information is very important for 3D video coding. In this paper, the major technical contribution is to formulate view synthesis quality for optimal resource allocation in 3D video coding, since such quality is what that matters most to the ultimate user (i.e., the viewer) of the system; to be more specific, a new joint bit allocation and rate control method for multi-view video plus depth (MVD) based 3D video coding is proposed accordingly. We firstly derive a view synthesis distortion model to characterize the effect of coding distortion of texture video and depth maps on the synthesized virtual views. Based on this model, we derive a rate-distortion model to characterize the relationship between the bitrate and the view synthesis distortion, and the optimal bitrate ratio between texture and depth is established adaptively by solving the associated optimization problem. Finally, the rate control algorithm is performed on view level, texture/depth level and frame level. Experimental results show that compared with other methods, the proposed bit allocation method obtains higher performance of view synthesis. Moreover, the proposed rate control method can accurately control the bitrate to satisfy the total bitrate constraint.

An interview with Aljosa Smolic

Dr. Aljosa Smolic joined Disney Research Zürich, Switzerland in 2009, as Senior Research Scientist and Head of the "Advanced Video Technology" group. Before he was Scientific Project Manager at the Fraunhofer Institute for Telecommunications, Heinrich- Hertz-Institut (HHI), Berlin, also heading a research group. He has been involved in several national and international research projects, where he conducted research in various fields of video processing, video coding, computer vision and computer graphics and published more than 100 refereed papers in these fields. In current projects he is responsible for research in 2D video, 3D video and free viewpoint video processing and coding.

A Hybrid Stereoscopic Video Coding Scheme Based on MPEG-2 and HEVC for 3DTV Services

Recently, 3DTV has drawn much attention as a new broadcasting service. In spite of technical advances in 3DTV broadcasting services, there are two barriers that hinder the new services from being launched for terrestrial broadcasting: the lack of available bandwidth for transmission of additional-view video via terrestrial channel and backward compatibility with the legacy 2-D HDTV services. As an alternative, a hybrid stereoscopic video coding scheme is proposed for a stereoscopic 3DTV service, where one-view video is transmitted through the legacy broadcasting systems and the other one is delivered via broadband networks to which TV terminals are connected. In this paper, the proposed hybrid stereoscopic video coding scheme incorporates an MPEG-2 codec for backward compatibility with a legacy 2-D HDTV service and an HEVC codec with inter-view prediction coding as an extension to encode additional-view video sequences with high coding efficiency. The proposed inter-view prediction coding scheme in the extended HEVC incorporates an advanced motion and disparity vector prediction (AMDVP) method for enhanced motion- and disparity-compensated coding. The experimental results demonstrate that the proposed hybrid stereoscopic video coding scheme achieves an average coding efficiency of 38.22% (32.86%) in BD-rate or 1.394 dB (1.235 dB) in BD-PSNR for the out-band (in-band) scenario.

Multiview-Video-Plus-Depth Coding Based on the Advanced Video Coding Standard

This paper presents a multiview-video-plus-depth coding scheme, which is compatible with the advanced video coding (H.264/AVC) standard and its multiview video coding (MVC) extension. This scheme introduces several encoding and in-loop coding tools for depth and texture video coding, such as depth-based texture motion vector prediction, depth-range-based weighted prediction, joint inter-view depth filtering, and gradual view refresh. The presented coding scheme is submitted to the 3D video coding (3DV) call for proposals (CfP) of the Moving Picture Experts Group standardization committee. When measured with commonly used objective metrics against the MVC anchor, the proposed scheme provides an average bitrate reduction of 26% and 35% for the 3DV CfP test scenarios with two and three views, respectively. The observed bitrate reduction is similar according to an analysis of the results obtained for the subjective tests on the 3DV CfP submissions.

High Efficiency 3D Video Coding Using New Tools Based on View Synthesis

We propose a new coding technology for 3D video represented by multiple views and the respective depth maps. The proposed technology is demonstrated as an extension of the recently developed high efficiency video coding (HEVC). One base views are compressed into a standard bitstream (like in HEVC). The remaining views and the depth maps are compressed using new coding tools that mostly rely on view synthesis. In the decoder, those views and the depth maps are derived via synthesis in the 3D space from the decoded baseview and from data corresponding to small disoccluded regions. The shapes and locations of those disoccluded regions can be derived by the decoder without any side information transmitted. To achieve high compression efficiency, we propose several new tools such as depth-based motion prediction, joint high frequency layer coding, consistent depth representation, and nonlinear depth representation. The experiments show high compression efficiency of the proposed technology. The bitrate needed for transmission of two side views with depth maps is mostly less than 50% than that of the bitrate for a single-view video.

PID-based bit allocation strategy for asymmetric stereoscopic video coding with fuzzy quality controller

Asymmetric stereoscopic video coding can take advantage of binocular suppression in human vision by representing one of the two views in lower quality. This paper proposes a bit allocation strategy for asymmetric stereoscopic video coding. In order to improve the accuracy of bit allocation and rate control in the left view, a proportionalintegral-derivative controller is adopted. Meanwhile, to control the quality fluctuation between consecutive frames of the left view, a quality controller is adopted. Besides, a fuzzy controller is proposed to control the variation in quality between the left and right views by comparing the PSNR disparity of two views with a fixed threshold, which is used to quantize the binocular psycho-visual redundancy and adjust the quantization parameter (QP) of the right view correspondingly. The proposed algorithm has been implemented in H.264/AVC video codec, and the experimental results show its effectiveness in rate control while keeping a good quality for the left view, and fewer bits are allocated for the right view so that the overall bit rate is saved by 7.2% at most without the loss of subjective visual quality for stereoscopic video.

3D High-Efficiency Video Coding for Multi-View Video and Depth Data

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

Efficient depth map compression for view rendering in 3D video

A depth map represents three-dimensional (3D) scene information, which is used to synthesise virtual views in 3D video coding. Since the quality of synthesised virtual view highly depends on the quality of depth map, efficient depth coding is crucial for the 3D video system. However, depth map is not directly used for display but to generate virtual views. Compressing depth map using existing video coding techniques yields unacceptable distortions in rendered virtual view. Thus, the depth map needs to be compressed in a way that can minimise distortions in the rendered views. To solve this problem, we propose an efficient depth map compression method for view rendering based on view rendering distortion other than depth map distortion itself. Firstly, we derive relationship between distortion in coded depth map and the rendered view. Then, a region-based video characteristic distortion model is proposed to precisely estimate distortion in view synthesis. Finally, a multilateral filtering is applied as an in-loop filter to reduce rendering artefacts. Experimental results show that the proposed method achieves 2 dB coding gain in terms of PSNR and subjective quality improvement of synthesised views.

Subjective quality assessment of asymmetric stereoscopic 3D video

In asymmetric stereoscopic video compression, the views are coded with different qualities. According to the binocular suppression theory, the perceived quality is closer to that of the higher-fidelity view. Hence, a higher compression ratio is potentially achieved through asymmetric coding. Furthermore, when mixed-resolution coding is applied, the complexity of the coding and decoding is reduced. In this paper, we study whether asymmetric stereoscopic video coding achieves the mentioned claimed benefits. Two sets of systematic subjective quality evaluation experiments are presented in the paper. In the first set of the experiments, we analyze the extent of downsampling for the lower-resolution view in mixed-resolution stereoscopic videos. We show that the lower-resolution view becomes dominant in the subjective quality rating at a certain downsampling ratio, and this is dependent on the sequence, the angular resolution, and the angular width. In the second set of the experiments, we compare symmetric stereoscopic video coding, quality-asymmetric stereoscopic video coding, and mixed-resolution coding subjectively. We show that in many cases, mixed-resolution coding achieves a similar subjective quality to that of symmetric stereoscopic video coding, while the computational complexity is significantly reduced.