Multiview High Efficiency Video Coding Research Articles

Multi-Exposure Image Compression Considering Rate-Distortion Optimization in Rendered High Dynamic Range Image

High-dynamic range (HDR) images, which provide realistic visual perception, have attracted considerable attention in various applications. A simple method for obtaining HDR images is to fuse multiple low-dynamic range (LDR) images captured under varying exposures. We proposed a method of efficiently encoding multi-exposure images using multi-view High Efficiency Video Coding (MV-HEVC) along with intensity mapping function (IMF) to create high-quality HDR images. The inter-view prediction in MV-HEVC eliminates redundancy between multi-exposure images. To achieve high-efficiency MV-HEVC-based multi-exposure image coding, two modifications to the rate-distortion optimization (RDO) process performed during encoding were proposed. First, the distortion in the rendered HDR image is used to replace the original distortion, which depends on the LDR image. Second, to balance distortion and rate in RDO, a new Lagrange multiplier is derived from the derivative of the new distortion with respect to the rate. A model of the distortion in the HDR domain was constructed and expressed as a function of the distortion in the LDR domain. The proposed scheme utilizes the conventional 8-bit codec to compress multi-exposure LDR images and generate an HDR image in the decoder. The experimental results indicate that the proposed technique outperforms HDR image coding schemes in which HEVC Range Extension and JPEG-XT coding standards are used.

Parallel multi-view HEVC for heterogeneously embedded cluster system

In this paper, we present a computer cluster with heterogeneous computing components intended to provide concurrency and parallelism with embedded processors to achieve a real-time Multi-View High-Efficiency Video Coding (MV-HEVC) encoder/decoder with a maximum resolution of 1088p. The latest MV-HEVC standard represents a significant improvement over the previous video coding standard (MVC). However, the MV-HEVC standard also has higher computational complexity. To this point, research using the MV-HEVC has had to use the Central Processing Unit (CPU) on a Personal Computer (PC) or workstation for decompression, because MV-HEVC is much more complex than High-Efficiency Video Coding (HEVC), and because decompressors need higher parallelism to decompress in real time. It is particularly difficult to encode/decode in an embedded device. Therefore, we propose a novel framework for an MV-HEVC encoder/decoder that is based on a heterogeneously distributed embedded system. To this end, we use a parallel computing method to divide the video into multiple blocks and then code the blocks independently in each sub-work node with a group of pictures and a coding tree unit level. To appropriately assign the tasks to each work node, we propose a new allocation method that makes the operation of the entire heterogeneously distributed system more efficient. Our experimental results show that, compared to the single device (3D-HTM single threading), the proposed distributed MV-HEVC decoder and encoder performance increased approximately (20.39 and 68.7) times under 20 devices (multithreading) with the CTU level of a 1088p resolution video, respectively. Further, at the proposed GOP level, the decoder and encoder performance with 20 devices (multithreading) respectively increased approximately (20.78 and 77) times for a 1088p resolution video with heterogeneously distributed computing compared to the single device (3D-HTM single threading).

Flexible Complexity Optimization in Multiview Video Coding

Flexible optimization on video coding computational complexity is imperative to adapt to diverse terminals in video streaming, especially with high definition videos and increasingly comprehensive encoders. Until now, few efforts have been contributed to the encoder of multiview videos, which complicates its encoding structure with duplex frame display. To address this issue, we propose a flexible complexity optimization framework in this paper. In the first place, the proposed algorithm flexibly reduces the encoder complexity at different external-defined complexity constraints; furthermore, it is also committed to compression efficiency optimization at different complexity levels. The framework is achieved by a hybrid approach: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity allocation</i> , which allocates the external-defined complexity constraint to all views, hierarchical layers and frames based on linear programming; and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity regulation</i> , which dynamically adjusts local candidate partitions to fulfil the targeted local complexity constraint, with a probability-driven Alternate Partition Cost (APC) minimization. The overall algorithm is implemented on the popular multiview encoder, Multiview High Efficiency Video Coding (MV-HEVC), with promising Rate-Distortion (RD) performances at different complexity constraints, which are also superior to two recent complexity optimization algorithms.

Highly Efficient Multiview Depth Coding Based on Histogram Projection and Allowable Depth Distortion.

Mismatches between the precisions of representing the disparity, depth value and rendering position in 3D video systems cause redundancies in depth map representations. In this paper, we propose a highly efficient multiview depth coding scheme based on Depth Histogram Projection (DHP) and Allowable Depth Distortion (ADD) in view synthesis. Firstly, DHP exploits the sparse representation of depth maps generated from stereo matching to reduce the residual error from INTER and INTRA predictions in depth coding. We provide a mathematical foundation for DHP-based lossless depth coding by theoretically analyzing its rate-distortion cost. Then, due to the mismatch between depth value and rendering position, there is a many-to-one mapping relationship between them in view synthesis, which induces the ADD model. Based on this ADD model and DHP, depth coding with lossless view synthesis quality is proposed to further improve the compression performance of depth coding while maintaining the same synthesized video quality. Experimental results reveal that the proposed DHP based depth coding can achieve an average bit rate saving of 20.66% to 19.52% for lossless coding on Multiview High Efficiency Video Coding (MV-HEVC) with different groups of pictures. In addition, our depth coding based on DHP and ADD achieves an average depth bit rate reduction of 46.69%, 34.12% and 28.68% for lossless view synthesis quality when the rendering precision varies from integer, half to quarter pixels, respectively. We obtain similar gains for lossless depth coding on the 3D-HEVC, HEVC Intra coding and JPEG2000 platforms.

High-efficiency parallelism solution for a Multiview High-Efficiency Video Coding decoder

Thus far, Multiview High-Efficiency Video Coding (MV-HEVC) can only use a central processing unit (CPU) to perform decompression on a personal computer (PC) or workstation. Because MV-HEVC is much more complex than High-Efficiency Video Coding (HEVC), decompressors need higher parallelism to decompress in real time. Therefore, this study presents a parallel method based on MV-HEVC. Inter-view complete parallelism is realized according to the dependent relationship between other MV-HEVC views, and a search range is not required to avoid the data dependence between frames. Based on the dependencies of each task in MV-HEVC, an advanced wavefront parallel processing method is proposed to achieve higher intra-frame parallelism. The parallel structure of the proposed method is compatible with that of the single-instruction multiple-data acceleration method. The results showed that the proposed method can decompress MV-HEVC with 20 threads in real time for 1088p video with three views.

Random access prediction structures for light field video coding with MV-HEVC

Computational imaging and light field technology promise to deliver the required six-degrees-of-freedom for natural scenes in virtual reality. Already existing extensions of standardized video coding formats, such as multi-view coding and multi-view plus depth, are the most conventional light field video coding solutions at the moment. The latest multi-view coding format, which is a direct extension of the high efficiency video coding (HEVC) standard, is called multi-view HEVC (or MV-HEVC). MV-HEVC treats each light field view as a separate video sequence, and uses syntax elements similar to standard HEVC for exploiting redundancies between neighboring views. To achieve this, inter-view and temporal prediction schemes are deployed with the aim to find the most optimal trade-off between coding performance and reconstruction quality. The number of possible prediction structures is unlimited and many of them are proposed in the literature. Although some of them are efficient in terms of compression ratio, they complicate random access due to the dependencies on previously decoded pixels or frames. Random access is an important feature in video delivery, and a crucial requirement in multi-view video coding. In this work, we propose and compare different prediction structures for coding light field video using MV-HEVC with a focus on both compression efficiency and random accessibility. Experiments on three different short-baseline light field video sequences show the trade-off between bit-rate and distortion, as well as the average number of decoded views/frames, necessary for displaying any random frame at any time instance. The findings of this work indicate the most appropriate prediction structure depending on the available bandwidth and the required degree of random access.

A Novel Rate Control Algorithm Based on ρ Model for Multiview High Efficiency Video Coding

Most existing rate control algorithms are based on the rate-quantization (R-Q) model. However, with video coding schemes becoming more flexible, it is very difficult to accurately model the R-Q relationship. Therefore, in this study we propose a novel ρ domain rate control algorithm for multiview high efficiency video coding (MV-HEVC). Firstly, in order to further improve the efficiency of MV-HEVC, this paper uses our previous research algorithm to optimize the MV-HEVC prediction structure. Then, we established the ρ domain rate control model based on multi-objective optimization. Finally, it used image similarity to analyze the correlation between viewpoints, using encoded information and frame complexity to proceed in bit allocation and bit rate control of the inter-view, frame lay, and base unit. The experimental simulation results show that the algorithm can simultaneously maintain high coding efficiency, where the average error of the actual bit rate and the target bit rate is only 0.9%.

CTU Layer Rate Control Algorithm in Scene Change Video for Free-Viewpoint Video

At present, the rate control algorithm for multiview high-efficiency video coding (MV-HEVC) does not have the capability of efficient coding tree unit(CTU) layer bit allocation, and the video quality varies greatly for sequences with sudden scene changes or large motions. To overcome this limitation, this paper proposes a rate control algorithm for MV-HEVC based on scene detection. Firstly, we established ρ domain rate control model based on multi-objective optimization. Then, it uses image similarity to make reasonable bit allocation among viewpoints. If the video scene is switched, the image similarity is recalculated, and then the correlation between the weights of the interview point rates and the correlation between the viewpoints are analyzed. Finally, the frame layer rate control considers the layer B-frame and other factors in allocating the code rate, and the basic unit layer rate control adopts different quantization methods according to the content complexity of the CTU. Experimental results show that the proposed rate control algorithm can maintain good coding efficiency and decrease the average video quality variation by 25.29%.

Improved Bit Allocation Algorithm for Multiview High Efficiency Video Coding

Improved Bit Allocation Algorithm for Multiview High Efficiency Video Coding

Efficient Inter-View Prediction Structure for Multi-View High Efficiency Video Coding

Efficient Inter-View Prediction Structure for Multi-View High Efficiency Video Coding

Improved Bit Allocation Algorithm for Multiview High Efficiency Video Coding

Improved Bit Allocation Algorithm for Multiview High Efficiency Video Coding

Fast inter-frame prediction in multi-view video coding based on perceptual distortion threshold model

Multi-view video coding (MVC) utilizes hierarchical B picture prediction structure and adopts many coding techniques to remove spatiotemporal and inter-view redundancies at the cost of high computational complexity. In this paper, a novel perceptual distortion threshold model (PDTM) is proposed to reveal the relationship between the mode selection of inter-frame prediction and coding distortion threshold. Based on the proposed PDTM, a new fast inter-frame prediction algorithm in MVC is developed aimed at minimizing computational complexity for dependent view coding. Then the fast MVC algorithm is incorporated into the multi-view High Efficiency Video Coding (MV-HEVC) software to improve MVC coding efficiency. In practical coding, the mode selection for inter-frame prediction of dependent views may be early terminated based on the thresholds derived from the PDTM, thereby reducing the coding time complexity. Experimental results demonstrate that the proposed algorithm can reduce the computational complexity of the dependent views by 52.9% compared with the HTM14.1 algorithm under the coding structure of hierarchical B pictures. Moreover, the bitrate is increased by 0.9% under the same subjective quality and only increased by 1.0% under the same objective quality peak signal-to-noise ratio (PSNR). Compared with the state-of-the-art fast algorithm, the proposed algorithm can save more coding time, while the bitrate under the same PSNR increases slightly.

Classified-edge guided depth resampling for multi-view coding

A new depth resampling for multi-view coding is proposed in this paper. At first, the depth video is downsampled by median filtering before encoding. After decoding, the classified edges, including credible edge and probable edge from the aligned texture image and the depth image, are interpolated by the selected diagonal pair, whose intensity difference is the minimum among four diagonal pairs around edge pixel. According to different category of edge, the intensity difference is measured by either real depth or percentage depth without any parameter setting. Finally, the resampled depth video and the decoded full-resolution texture video are synthesized into virtual views for the performance evaluation. Experiments on the platform of multi-view high efficiency video coding (HEVC) demonstrate that the proposed method is superior to the contrastive methods in terms of visual quality and rate distortion (RD) performance.

4K Real-Time and Parallel Software Video Decoder for Multilayer HEVC Extensions

Two High Efficiency Video Coding (HEVC) extensions, namely, the scalable HEVC (SHVC) extension and multiview HEVC (MV-HEVC) extension, have been finalized in July 2014 by the Moving Picture Experts Group and Video Coding Experts Group. These two extensions enable additional features not covered in the first version of the HEVC standard such as spatial, fidelity, bitdepth, and color gamut scalability, as well as stereoscopic and multiview representations. In this paper, we propose a software parallel decoder architecture for the HEVC standard and its multilayer extensions, including SHVC and MV-HEVC extensions. The decoder consists of multiple instances of the OpenHEVC decoder, one instance to decode each layer with a communication between dependent layers to perform inter-layer predictions. The proposed multilayer HEVC decoder is parallel friendly and supports both wavefront parallelism to simultaneously process adjacent rows of the frame and frame-based parallelism to decode a set of temporal and spatial frames in parallel. Moreover, the most time-consuming operation introduced in the SHVC extension, namely, the resampling of the inter-layer reference picture in spatial scalability, is optimized in single instruction multiple data for x86 platform. We assess the complexity of the multilayer HEVC decoder with respect to the simulcast configuration. The multilayer decoder decoding two SHVC layers introduces in average 40%–71% additional complexity compared with the single layer HEVC decoder. Moreover, the low level optimizations with a hybrid parallel processing solution enable a real-time decoding of 4Kp60 enhancement layer on a 6-core Intel i7 processor running at 3.4 GHz.