Multi-view Coding Research Articles

Temporal–Spatial Symmetric Distributed Multi-View Video Coding Scheme

To improve the rate stability and make a balance for different viewpoints in distributed multi-view video coding (DMVC) system, a novel symmetric DMVC (SDMVC) scheme is proposed in this paper. In the proposed scheme, every frame from all views adopts the same encoding mode and stable output rates are achieved, which are significant to improve the transmission efficiency in the channel. Both temporal and spatial correlations are exploited, in addition, a novel side information (SI) generation algorithm aiming at better exploring the correlations of proposed scheme has been proposed to obtain better performance. The simulation results show that the proposed SDMVC scheme gets a much more stable rate than the asymmetric scheme, only with neglectable bit-rate increasing. Meanwhile, the proposed SI generation algorithm significantly improves the coding performance.

Efficient multiview video plus depth coding for 3D-HEVC based on complexity classification of the treeblock

High efficiency video coding standard-based 3D video (3D-HEVC) has been extended from HEVC to improve the coding efficiency of multiview video plus depth (MVD). Similar to the joint model of HEVC, a computationally expensive exhaustive mode decision is performed to find the least rate-distortion cost for each treeblock in 3D-HEVC. Furthermore, additional coding tools have been added to 3D-HEVC for improving the coding efficiency of the dependent texture video and depth map. Those tools achieve the highest possible coding efficiency, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In order to reduce computational complexity, we propose an efficient multiview video plus depth coding algorithm for 3D-HEVC that adaptively utilizes the complexity classification of the treeblock. The coding complexity model of a treeblock is first analyzed according to the prediction mode and coding mode from the corresponding treeblocks in the reference views. Based on the complexity classification model of the treeblock, we propose two efficient low-complexity approaches, including fast mode size decision and adaptive motion search range selection. Extensive experimental results demonstrate that the proposed MVD coding algorithm can achieve the average computational saving about 60.1% with negligible rate-distortion performance loss in comparison with the original 3D-HEVC encoder.

Cross-layer optimized authentication and error control for wireless 3D medical video streaming over LTE

3D video for tele-medicine applications is gradually gaining momentum since the 3D technology can provide precise location information. However, the weak link for 3D video streaming is the necessary wireless link of the communication system. Neglecting the wireless impairments can severely degrade the performance of 3D video streaming that communicates complex critical medical data. In this paper, we propose systematic methodology for ensuring high performance of the 3D medical video streaming system. First, we present a recursive end-to-end distortion estimation approach for MVC (multiview video coding)-based 3D video streaming over error-prone networks by considering the 3D inter-view prediction. Then, based on the previous model, we develop a cross-layer optimization scheme that considers the LTE wireless physical layer (PHY). In this optimization, the authentication requirements of 3D medical video are also taken into account. The proposed cross-layer optimization approach jointly controls and manages the authentication, video coding quantization of 3D video, and the modulation and channel coding scheme (MCS) of the LTE wireless PHY to minimize the end-to-end video distortion. Experimental results show that the proposed approach can provide superior 3D medical video streaming performance in terms of peak signal-to-noise ratio (PSNR) when compared to state-of-the-art approaches that include joint source-channel optimized streaming with multi-path hash-chaining based-authentication, and also conventional video streaming with single path hash-chaining-based authentication.

Prediction architecture based on block matching statistics for mixed spatial-resolution multi-view video coding

The use of mixed spatial resolutions in multi-view video coding is a promising approach for coding videos efficiently at low bitrates. It can achieve a perceived quality, which is close to the view with the highest quality, according to the suppression theory of binocular vision. The aim of the work reported in this paper is to develop a new multi-view video coding technique suitable for low bitrate applications in terms of coding efficiency, computational and memory complexity, when coding videos, which contain either a single or multiple scenes. The paper proposes a new prediction architecture that addresses deficiencies of prediction architectures for multi-view video coding based on H.264/AVC. The prediction architectures which are used in mixed spatial-resolution multi-view video coding (MSR-MVC) are afflicted with significant computational complexity and require significant memory size, with regards to coding time and to the minimum number of reference frames. The architecture proposed herein is based on a set of investigations, which explore the effect of different inter-view prediction directions on the coding efficiency of multi-view video coding, conduct a comparative study of different decimation and interpolation methods, in addition to analyzing block matching statistics. The proposed prediction architecture has been integrated with an adaptive reference frame ordering algorithm, to provide an efficient coding solution for multi-view videos with hard scene changes. The paper includes a comparative performance assessment of the proposed architecture against an extended architecture based on the 3D digital multimedia broadcast (3D-DMB) and the Hierarchical B-Picture (HBP) architecture, which are two most widely used architectures for MSR-MVC. The assessment experiments show that the proposed architecture needs less bitrate by on average 13.1 Kbps, less coding time by 14% and less memory consumption by 31.6%, compared to a corresponding codec, which deploys the extended 3D-DMB architecture when coding single-scene videos. Furthermore, the codec, which deploys the proposed architecture, accelerates coding by on average 57% and requires 52% less memory, compared to a corresponding codec, which uses the HBP architecture. On the other hand, multi-view video coding which uses the proposed architecture needs more bitrate by on average 24.9 Kbps compared to a corresponding codec that uses the HBP architecture. For coding a multi-view video which has hard scene changes, the proposed architecture yields less bitrate (by on average 28.7 to 35.4 Kbps), and accelerates coding time (by on average 64 and 33%), compared to the HBP and extended 3D-DMB architectures, respectively. The proposed architecture will thus be most beneficial in low bitrate applications, which require multi-view video coding for video content depicting hard scene changes.

Improving view random access via increasing hierarchical levels for multi-view video coding

Multi-view video has attracted considerable interest due to its wide use in a growing market including 3D television, free viewpoint video and intelligent surveillance. Besides the efficiency of compression methods, a vital requirement for multi-view video coding procedures is the view random access which is described as the capability to navigate quickly to any arbitrary view at any given time. In this paper, a new prediction structure based on the increase of the hierarchical level of B-views is proposed. This approach involves reducing the number of images needed for the prediction of B pictures for specific B-views. To show the effectiveness of the proposed prediction structure, a new metric for evaluating the view random access is described. In contrast to the metric proposed by the JVT group which is limited to consider only the image having the maximum number of frames needed for decoding, the key basis of the proposed metric is to consider evaluating all the images contained within a Group of Group Of Pictures. Experimental results have shown that compared with the IBP prediction structure of the reference model JMVM, the proposed algorithm improves the view random access by up to 33.5% with significant improvement in terms of bit rate.

Adaptive Bit Allocation for 3D Video Coding

In the multi-view video plus depth 3D video coding, texture image and depth map are coded jointly. The texture image is utilized for displaying and synthesizing the virtual view as reference image. The depth map provides the scene geometry information and is utilized to synthesize the virtual view at the terminal through Depth-Image Based Rendering technique. The distortion of the compressed texture image and depth map will be propagated to the synthesized virtual view. Besides the coding efficiency of texture image and depth map, bit allocation between texture image and depth map also has a great effect on the synthesized virtual view quality. Several methods are proposed for bit allocation between texture image and depth map, but most of them attempt to allocate a fixed target bitrate based on virtual view distortion model to achieve optimal synthesized virtual view quality, and the modeling process brings extra complexity. In practical application, the video sequence has different contents and fixed bit ratio cannot achieve optimal performance. In this paper, we propose an adaptive bit allocation algorithm for 3D video coding. First, we present a model to estimate the synthesized virtual view distortion, and then adjust the bit ratio between adjacent views and between texture image and depth map at Group of Picture level based on the virtual view quality fluctuation. We adjust the bit ratio to achieve the optimal virtual view quality for different video contents. Experimental results demonstrate that the proposed algorithm can optimally allocate bits to achieve optimal virtual view quality under different target bitrates and for different video contents, and the computational complexity of the proposed algorithm is extremely low.

Early DIRECT Mode Decision for MVC Using MB Mode Homogeneity and RD Cost Correlation

Multi-view video coding (MVC) adopts variable size mode decision to achieve high coding efficiency. However, its high computational complexity is a bottleneck of enabling MVC into practical real-time applications. In this paper, an early termination strategy is proposed for DIRECT mode decision of MVC by exploiting mode homogeneity and rate distortion (RD) cost correlation. By comparing the RD cost between DIRECT mode and Inter$16\times 16$ mode, an adaptive threshold is defined based on the MB's mode homogeneity and RD cost so as to early terminate the remaining inter and intra modes. Experimental results show that compared with the original JMVC model, the proposed approach can reduce the total encoding time from 65.08% to 91.45% (80.43% on average). Meanwhile, the Bjontegaard delta peak signal-to-noise ratio only decreases 0.031 dB and Bjontegaard delta bit rate increases 0.97% on average, which is a negligible loss of coding efficiency and superior to the performance of state-of-the-art methods.

Light Field Multi-View Video Coding With Two-Directional Parallel Inter-View Prediction.

Light field (LF) technology has been popularly adopted by a wide range of conventional industries. However, one problem when dealing with LFs is the sheer size of data volume. There have been many multi-view video coding (MVC)-based LF video coding methods reported in the literature, aiming at finding the best prediction structure for LF video coding. It is clear that the number of possible prediction structures is unlimited, and it is also observed that the coding bit-rate can be reduced by increasing the number of bi-directionally encoded views in the prediction structure. However, none work has been conducted to analyze the relationship of the prediction structure with its coding performance. In light of this observation, we first design a new LF-MVC prediction structure by extending the inter-view prediction into a two-directional parallel structure. Analytical models for source coding rate and encoding time are developed to analyze their relationships with the prediction structure, and are proven to be well-matched to our experimental results. Experimental evaluation of two LF video sequences demonstrates that the proposed LF-MVC prediction structure can achieve a factor of 26% bit-rate reduction against the conventional MVC prediction structure for an LF video with 5×5 views, and a further 34% bit-rate reduction for an LF video with a larger 10×10 views. Compared with the state-of-the-art MVC-based LF video coding prediction structures in the literature, LF-MVC can achieve the best coding performance, and with its high encoding efficiency, is well suited for deployment in practical LF-based 3D systems.

Parallel Multiview Video Coding Exploiting Group of Pictures Level Parallelism

This paper presents the use of a computer cluster with heterogeneous computing components to provide concurrency and multi-level parallelism at coarse grain and massive fine-grain for multiview video coding (MVC) applications. MVC involves coding of multiple video sequences that are taken from the same scene but different perspective. In addition to motion estimation (ME) used in conventional video coding for single view video for exploiting inter-frame temporal similarities, MVC adopts disparity estimation (DE) to further increase compression. To overcome the huge computational cost associated with ME and by extension with DE, attention has been mainly focused on developing fast ME/DE algorithms. Although fast ME/DE algorithms bring substantial speedup, to achieve realtime MVC encoding, it requires further acceleration of the coding process at higher levels. Towards this end, this paper proposes a multiple-view-parallel, multiple-interleaved group of pictures (multiple-IGOP) scheduling scheme for MVC. When evaluated over eight views, with no loss in rate distortion (RD) performance, the proposed scheme outperforms view-sequential coding by a factor of up to 12.4 and 12.3, respectively, for two popular prediction structures, IBP and IPP.

A Modified Multiview Video Streaming System Using 3-Tier Architecture

In this paper, we present a modified inter-view prediction Multiview Video Coding (MVC) scheme from the perspective of viewer's interactivity. When a viewer requests some view(s), our scheme leads to lower transmission bit-rate. We develop an interactive multiview video streaming system exploiting that modified MVC scheme. Conventional interactive multiview video systems require high bandwidth due to redundant data being transferred. With real data test sequences, clear improvements are shown using the proposed interactive multiview video system compared to competing ones in terms of the average transmission bit-rate and storage size of the decoded (i.e., transferred) data with comparable rate-distortion.

Hierarchical modulation for client-driven selective streaming of multi-view video over AWGN channels

We consider a client-driven selective streaming system for multi-view video, which is supported by scalable multi-view video coding (SMVC). The system is used to reduce bit-rates for efficient transmission of multi-view video. The transmit source is partitioned into three layers: a base and two enhancement layers. The base layer contains all views encoded by MVC at a low-quality, while each enhancement layer contains high-quality part of the selected left and right views, respectively. The base layer is more important than the enhancement layers to create 3D perception, albeit of low-quality, and should be more protected than the enhancement layers. In this paper, two-level and three-level hierarchical 64QAM are used to provide unequal error protection (UEP) for the client-driven selective streaming system. First we find the suitable hierarchical values, which is the ratio of the distances in hierarchical 64QAM, for the client-driven selective streaming system. Furthermore, we analyze numerically how the hierarchical 64QAM impacts peak signal-to-noise ratio (PSNR) performance of fast and slow-motion sequences. Simulation results show that the three-level hierarchical 64QAM outperforms both conventional 64QAM with Gray mapping and the two-level hierarchical 64QAM in terms of achieving the target average PSNR performance at a low SNR environment.

Texture-Aware Depth Prediction in 3D Video Coding

3D video has raised great interest in the last decade and currently a new 3D video coding standard, known as 3D video coding extension of High Efficiency Video Coding (3D-HEVC), has been developed. The standard investigates the coding of multiview video plus depth, which consists of texture videos and depth videos of multiple views. Depth video, as a description of geometry information of a scene, is generally composed of large flat regions separated by sharp edges. The conventional video coding may fail to generate an accurate prediction for units with sharp edges due to its block-based prediction which cannot compensate (minor) boundary changes well. In order to attack the problem, a new texture-aware depth inter-prediction method is proposed, which incorporates pixel-oriented weighting in the bi-prediction process by exploiting motion and structure similarities between texture and depth videos. Furthermore, such pixel-oriented weighting scheme can be extended to the uni-prediction process by considering more prediction blocks with small motion vector displacements. Experimental results demonstrate that the adapted 3D-HEVC codec with the proposed method can achieve better rate-distortion performance compared to the original 3D-HEVC standard codec.

Big video data for light-field-based 3D telemedicine

Big data and 3D technologies have been successfully leveraged in a variety of industries to improve their efficiency and quality. The healthcare sector has lagged in the uptake of these new technologies. In this article, we propose a novel light field (LF)-based 3D telemedicine system. The proposed system is able to provide a life-like tele-consultation experience that provides a quality of experience far beyond conventional 2D telemedicine systems. In addition, its embedded 3D data in light field video (LFV) format can also facilitate a higher level of big data analysis, so-called big LFV data analysis. To solve the challenges in storage and analysis of LFV, we extend the standard multi-view video coding (MVC) approach to LF-MVC, which is able to achieve up to a 23 percent higher compression rate when compared to standard MVC. Furthermore, a big data analysis framework is proposed to integrate LFV into conventional telemedicine analysis, which can achieve improved classification, statistics gathering, prediction, and cognitive analysis for healthcare applications.

Enhanced view random access ability for multiview video coding

Apart from the efficient compression, reducing the complexity of the view random access is one of the most important requirements that should be considered in multiview video coding. In order to obtain an efficient compression, both temporal and inter-view correlations are exploited in the multiview video coding schemes, introducing higher complexity in the temporal and view random access. We propose an inter-view prediction structure that aims to lower the cost of randomly accessing any picture at any position and instant, with respect to the multiview reference model JMVM and other recent relevant works. The proposed scheme is mainly based on the use of two base views (I-views) in the structure with selected positions instead of a single reference view as in the standard structures. This will, therefore, provide a direct inter-view prediction for all the remaining views and will ensure a low-delay view random access ability while maintaining a very competitive bit-rate performance with a similar video quality measured in peak signal-to-noise ratio. In addition to a new evaluation method of the random access ability, the obtained results show a significant improvement in the view random accessibility with respect to other reported works.

Adaptive early termination mode decision for 3D-HEVC using inter-view and spatio-temporal correlations

3D extension of high efficiency video coding (3D-HEVC) supports the coding of multi-view video and depth map. It adds new coding tools to the high efficiency video coding (HEVC) design, which improve the HEVC compression capabilities for 3D video. These coding tools achieve high coding efficiency, but it brings relatively high computational complexity which limits 3D-HEVC from practical applications. In this paper, we propose an adaptive early termination mode decision algorithm to reduce 3D-HEVC computational complexity. An experimental analysis is performed to study inter-view and spatio-temporal correlations in the coding information. Based on the analysis results, we can determine to skip some specific prediction mode at the early termination stage without a normal 3D-HEVC encoding process. The proposed algorithm uses the correlation between inter-view and spatio-temporal blocks. Experimental results demonstrate that the proposed algorithm can significantly reduce computational complexity of 3D-HEVC encoding while maintaining almost the same rate-distortion (RD) performance.

Error‐Resilient Multi‐view Video Coding Based on End‐to‐End Rate‐Distortion Optimization

Video transmission over packet-switched networks usually suffers from packet losses. The use of the prediction loop in video coding will cause these errors to propagate to subsequent frames, and thus significantly impacts on the received video quality. With the increasing number of cameras to capture the scene, robustly delivering multi-view video over error-prone channels becomes a rather challenging task. A rate-distortion optimization algorithm is proposed to improve error resilience for multi-view video transmission. A recursive model to estimate the end-to-end distortion is developed for multiview video coding, in which the distortion model explicitly takes into consideration the inherent error resilience property of the hierarchical bi-prediction structure. Based on the proposed distortion model, end-to-end rate-distortion optimization criterion is employed to perform coding mode switching. Extensive experimental results demonstrate significant performance gains can be achieved for multi-view video communication against transmission errors.

Optimal reference view selection algorithm for low complexity disparity estimation

Multi-view video coding (MVC) is composed of multiple video sequences captured simultaneously by multiple closely spaced cameras. Such a system demands a higher computational complexity due to the increased number of cameras. Disparity estimation (DE) is one of the compression techniques used to remove inter-view redundancy in MVC, which causes high complexity. In order to overcome the problem of high computational complexity in real-time MVC, an optimal reference view selection algorithm is proposed. The proposed scheme decreases the coding complexity by reducing one-directional DE operation and minimizing the decrease in coding performance in MVC.

A Scalable Massively Parallel Motion and Disparity Estimation Scheme for Multiview Video Coding

Multiview video coding (MVC) has recently received considerable attention. It is proposed as an extension of H.264/Advanced Video Coding (AVC) standard for multiple video source compression. To resolve the extremely high computational complexity of MVC (and in fact other AVC techniques), suitable parallel algorithms need to be developed that are amenable to implementation on low-cost massively parallel architecture, platforms that have found a common place due to recent advances in the parallel computer architecture. The high complexity of MVC is due to its prediction structure, where motion estimation (ME) between the frames and disparity estimation (DE) between the views contribute to more than 99% of overall complexity of the coder. This paper presents the development and implementation of a scalable massively parallel fast search algorithm to significantly reduce the computational cost of ME/DE over the current best available full block matching, and suboptimal fast search algorithms. The proposed massively parallel fast search algorithm (DZfast), when evaluated over eight views, outperforms the existing full search and fast search MVC algorithms by a factor of up to 245.8 and 8.4, respectively. This speedup comes at no or minute loss in rate-distortion performance.

Reference View Selection in DIBR-Based Multiview Coding.

Augmented reality, interactive navigation in 3D scenes, multiview video, and other emerging multimedia applications require large sets of images, hence larger data volumes and increased resources compared with traditional video services. The significant increase in the number of images in multiview systems leads to new challenging problems in data representation and data transmission to provide high quality of experience on resource-constrained environments. In order to reduce the size of the data, different multiview video compression strategies have been proposed recently. Most of them use the concept of reference or key views that are used to estimate other images when there is high correlation in the data set. In such coding schemes, the two following questions become fundamental: 1) how many reference views have to be chosen for keeping a good reconstruction quality under coding cost constraints? And 2) where to place these key views in the multiview data set? As these questions are largely overlooked in the literature, we study the reference view selection problem and propose an algorithm for the optimal selection of reference views in multiview coding systems. Based on a novel metric that measures the similarity between the views, we formulate an optimization problem for the positioning of the reference views, such that both the distortion of the view reconstruction and the coding rate cost are minimized. We solve this new problem with a shortest path algorithm that determines both the optimal number of reference views and their positions in the image set. We experimentally validate our solution in a practical multiview distributed coding system and in the standardized 3D-HEVC multiview coding scheme. We show that considering the 3D scene geometry in the reference view, positioning problem brings significant rate-distortion improvements and outperforms the traditional coding strategy that simply selects key frames based on the distance between cameras.

Early DIRECT mode decision based on all‐zero block and rate distortion cost for multiview video coding

The exhaustive variable-block-size mode decision can efficiently remove the redundancies among the multiview videos, while it also leads to significant increase of computational complexity in the multiview video coding (MVC) encoder, and the high encoding complexity becomes a bottleneck for the MVC encoder to achieve real-time multimedia applications. To address this bottleneck, many fast mode decision methods have been proposed. However, most of them are only suitable for optimising the encoding complexity of the odd views of the MVC encoder. In this study, based on the property of the all-zero block and rate distortion (RD) cost of the DIRECT mode as well as the correlations between the current macroblock (MB) and its spatial–temporal nearby MBs, an early DIRECT mode decision method is proposed for reducing the encoding complexity of the MVC. Experimental results show that the proposed method achieves 48.25 and 55.64% on average encoding time saving for the even and odd views, respectively, whereas the RD performance degradation is quite acceptable. In summary, the proposed method efficiently reduces the encoding complexity for the MVC encoder.