Multiview Video Streaming Research Articles

High Quality Multi-View Video Streaming over Multiple Transmission Paths

High Quality Multi-View Video Streaming over Multiple Transmission Paths

A Modified Multiview Video Streaming System Using 3-Tier Architecture

In this paper, we present a modified inter-view prediction Multiview Video Coding (MVC) scheme from the perspective of viewer's interactivity. When a viewer requests some view(s), our scheme leads to lower transmission bit-rate. We develop an interactive multiview video streaming system exploiting that modified MVC scheme. Conventional interactive multiview video systems require high bandwidth due to redundant data being transferred. With real data test sequences, clear improvements are shown using the proposed interactive multiview video system compared to competing ones in terms of the average transmission bit-rate and storage size of the decoded (i.e., transferred) data with comparable rate-distortion.

Hierarchical modulation for client-driven selective streaming of multi-view video over AWGN channels

We consider a client-driven selective streaming system for multi-view video, which is supported by scalable multi-view video coding (SMVC). The system is used to reduce bit-rates for efficient transmission of multi-view video. The transmit source is partitioned into three layers: a base and two enhancement layers. The base layer contains all views encoded by MVC at a low-quality, while each enhancement layer contains high-quality part of the selected left and right views, respectively. The base layer is more important than the enhancement layers to create 3D perception, albeit of low-quality, and should be more protected than the enhancement layers. In this paper, two-level and three-level hierarchical 64QAM are used to provide unequal error protection (UEP) for the client-driven selective streaming system. First we find the suitable hierarchical values, which is the ratio of the distances in hierarchical 64QAM, for the client-driven selective streaming system. Furthermore, we analyze numerically how the hierarchical 64QAM impacts peak signal-to-noise ratio (PSNR) performance of fast and slow-motion sequences. Simulation results show that the three-level hierarchical 64QAM outperforms both conventional 64QAM with Gray mapping and the two-level hierarchical 64QAM in terms of achieving the target average PSNR performance at a low SNR environment.

Adaptive delivery of immersive 3D multi-view video over the Internet

The increase in Internet bandwidth and the developments in 3D video technology have paved the way for the delivery of 3D Multi-View Video (MVV) over the Internet. However, large amounts of data and dynamic network conditions result in frequent network congestion, which may prevent video packets from being delivered on time. As a consequence, the 3D video experience may well be degraded unless content-aware precautionary mechanisms and adaptation methods are deployed. In this work, a novel adaptive MVV streaming method is introduced which addresses the future generation 3D immersive MVV experiences with multi-view displays. When the user experiences network congestion, making it necessary to perform adaptation, the rate-distortion optimum set of views that are pre-determined by the server, are truncated from the delivered MVV streams. In order to maintain high Quality of Experience (QoE) service during the frequent network congestion, the proposed method involves the calculation of low-overhead additional metadata that is delivered to the client. The proposed adaptive 3D MVV streaming solution is tested using the MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) standard. Both extensive objective and subjective evaluations are presented, showing that the proposed method provides significant quality enhancement under the adverse network conditions.

Rate-distortion-optimized multi-view streaming in wireless environment using network coding

Multi-view video streaming is an emerging video paradigm that enables new interactive services, such as 3D video, free viewpoint television, and immersive teleconferencing. Because of the high bandwidth cost they come with, multi-view streaming applications can greatly benefit from the use of network coding, in particular in transmission scenarios such as wireless network, where the channels have limited capacity and are affected by losses. In this paper, we address the topic of cooperative streaming of multi-view video content, wherein users who recently acquired the content can contribute parts of it to their neighbors by providing linear combinations of the video packets. We propose a novel method for selection and network encoding of the transmitted frames based on the users’ preferences for the different views and the rate-distortion properties of the stream. Using network coding enables the users to retrieve the content in a faster and more reliable manner and without the need for coordination among the senders. Our experimental results prove that our preference-based approach provides a high-quality decoding even when the uplink capacity of each node is only a small fraction of the rate of the stream.

Wireless Multi-View Video Streaming with Subcarrier Allocation

Wireless Multi-View Video Streaming with Subcarrier Allocation

Automatic 3D tracking system for large swarm of moving objects

Natural systems such as bird flocks, fish schools and insect swarms consist of a large group of moving individuals. For many years, scientists have been interested in the complex 3D motion patterns and dynamics they exhibit, trying to discover enlightening rules and causes behind them. Unfortunately, the lack of effective techniques to accurately measure the real 3D trajectories of the individuals had limited the quantitative study on these systems. We propose in this paper an automatic tracking system which is able to track a large number of tiny animals in a 3D volume with multiple cameras. Most visual details of such targets are lost in the captured images because of limited image resolution, and the remainder can be easily corrupted due to frequent occlusion or motion blur, which makes it difficult to establish cross-view and cross-frame correspondences. We formulate the problem as a repeated process of hypothesis generation and verification. Hypotheses are generated when cross-view matching ambiguities occur and are verified at an efficient 3D tracking stage where targets are modeled in 3D space and weak yet existing visual information from multi-view video streams are furthest collected. The whole system is fully automatic in dealing with variable number of targets and robust against detection and matching errors.

Optimal layered representation for adaptive interactive multiview video streaming

We consider an interactive multiview video streaming (IMVS) system where clients select their preferred viewpoint in a given navigation window. To provide high quality IMVS, many high quality views should be transmitted to the clients. However, this is not always possible due to the limited and heterogeneous capabilities of the clients. In this paper, we propose a novel adaptive IMVS solution based on a layered multiview representation where camera views are organized into layered subsets to match the different clients constraints. We formulate an optimization problem for the joint selection of the views subsets and their encoding rates. Then, we propose an optimal and a reduced computational complexity greedy algorithms, both based on dynamic-programming. Simulation results show the good performance of our novel algorithms compared to a baseline algorithm, proving that an effective IMVS adaptive solution should consider the scene content and the client capabilities and their preferences in navigation.

A DASH-based HEVC multi-view video streaming system

Recent advancement in cameras and image processing technology has generated a paradigm shift from traditional 2D and 3D video to multi-view video (MVV) technology, while at the same time improving video quality and compression through standards such as high efficiency video coding (HEVC). In multi-view, cameras are placed in predetermined positions to capture the video from various views. Delivering such views with high quality over the Internet is a challenging prospect, as MVV traffic is several times larger than traditional video, since it consists of multiple video sequences, each captured from a different angle, requiring more bandwidth than single-view video to transmit MVV. In addition, the Internet is known to be prone to packet loss, delay, and bandwidth variation, which adversely affect MVV transmission. Another challenge is that end users' devices have different capabilities in terms of computing power, display, and access link capacity, requiring MVV to be adapted to each user's context. In this paper, we propose an HEVC multi-view system using Dynamic Adaptive Streaming over HTTP to overcome the above-mentioned challenges. Our system uses an adaptive mechanism to adjust the video bit rate to the variations of bandwidth in best effort networks. We also propose a novel scalable way for the multi-view video and depth content for 3D video in terms of the number of transmitted views. Our objective measurements show that our method of transmitting MVV content can maximize the perceptual quality of virtual views after the rendering and hence increase the user's quality of experience.

Optimizing Multiview Video Plus Depth Prediction Structures for Interactive Multiview Video Streaming

Several multiview video coding standards have been developed to efficiently compress images from different camera views capturing the same scene by exploiting the spatial, the temporal and the interview correlations. However, the compressed texture and depth data have typically many interview coding dependencies, which may not suit interactive multiview video streaming (IMVS) systems, where the user requests only one view at a time. In this context, this paper proposes an algorithm for the effective selection of the interview prediction structures (PSs) and associated texture and depth quantization parameters (QPs) for IMVS under relevant constraints. These PSs and QPs are selected such that the visual distortion is minimized, given some storage and point-to-point transmission rate constraints, and a user interaction behavior model. Simulation results show that the novel algorithm has near-optimal compression efficiency with low computational complexity, so that it offers an effective encoding solution for IMVS applications.

Coding Structure and Replication Optimization for Interactive Multiview Video Streaming

Multiview video refers to videos of the same dynamic 3-D scene captured simultaneously by multiple closely spaced cameras from different viewpoints. We study interactive streaming of pre-encoded multiview videos, where, at any time, a client can request any one of many captured views for playback. Moreover, the client can periodically freeze the video in time and switch to neighboring views for a compelling look-around visual effect. We consider distributed content servers to support large-scale interactive multiview video service. These servers collaboratively replicate and access video contents. We study two challenges in this setting: what is an efficient coding structure that supports interactive view switching and, given that, what to replicate in each server in order to minimize the cost incurred by interactive temporal and view switches? We first propose a redundant coding structure that facilitates interactive view-switching, trading off storage with transmission rate. Using the coding structure, we next propose a content replication strategy that takes advantage of indirect hit to lower view-switching cost: in the event that the exact requested view is not available locally, the local server can fetch a different but correlated view from the other servers, so that the remote repository only needs to supply the pre-encoded view differential. We formulate the video content replication problem to minimize the switching cost as an integer linear programming (ILP) problem and show that it is NP-hard. We first propose an LP relaxation and rounding algorithm (termed Minimum Eviction) with bounded approximation error. We then study a more scalable solution based on dynamic programming and Lagrangian optimization (DPLO) with little sacrifice in performance. Simulation results show that our replication algorithms achieve substantially lower switching cost compared to other content replication schemes.

A Pyramidal Layered HMM for Multiview Human Behavior Recognition in Asynchronous Video Streams

Extracted features which are obtained from a multiview video stream form a special case of a multi-sensor observation sequence. If the sensors are not synchronous, the observed features of views are not aligned together and this makes some difficulties in classification applications. A new architecture for hidden Markov model, namely pyramidal layered hidden Markov model, is proposed in this paper to handle this situation. This is accomplished by means of separate decoding in each view stream in bottom layer and then fusion of the aligned decoded symbols in top layer. Structure and algorithms of the new structure are introduced and are then used for human behaviour recognition in multiview video sequences. Considering collected information from all views of a multiview human action recognition system, one expects the recognition rate to increase and some problems like occlusion to be rectified. Several experiments have been performed in this paper. The experimental results show high performance, about 93.8% in average, in multiview human behavior recognition, as well as accuracy improvement compared to similar methods. The results are also compared with other contributions on three different multiview behavior datasets. General Terms Video Processing, Classification, Recognition, Hidden Markov Models

Incentive analysis for cooperative interactive multiview video streaming

In interactive multiview video streaming (IMVS), users can periodically select one out of many captured views available for observation. In single-view video streaming, cooperative strategies where peers share received packets of the same video have proven to be effective in reducing server׳s upload burden, and incentive mechanisms are designed to stimulate user cooperation. However, exploiting user cooperation in higher dimensional IMVS is difficult, since users watching different views makes it difficult to establish partnership, and users switching views frequently and independently makes it difficult to maintain partnership over time. In this paper, we use a multiview video frame structure for IMVS to support cooperative view-switching, where peers may help each other even if they are observing different views. We then model peers׳ interaction as an indirect reciprocity game, where each user is assigned a reputation level. To gain a higher reputation level, users help others, which in turn leads to a higher likelihood to receive others׳ help later. In this work, we focus on how view switching, the key feature of IMVS, affects user cooperation. By modeling users׳ decision making as a Markov decision process, our analysis shows that users tend to cooperate at some views but not others: given peers can predict their future view navigation paths probabilistically, for a peer who is likely to enter a view-switching path not requiring others׳ help, he also has less incentive to cooperate. Furthermore, we observe that the game may have multiple Nash Equilibria corresponding to different cooperation levels, e.g., users cooperate at all views in the full cooperation equilibrium, while users only cooperate at certain views in the partial cooperation equilibrium. The particular equilibrium the game will converge to depends on the initial cooperation level of the game. To stimulate user cooperation at all views, we propose a Pay-for-Cooperation (PfC) scheme at the beginning of the game to drive the game to the full cooperation equilibrium to improve system efficiency. Our simulation results show the effectiveness of PfC.

Wireless Streaming of Interactive Multi-View Video via Network Compression and Path Diversity

We formulate a system framework for network compression of interactive multi-view streaming video. The setup comprises a media server that delivers the content over two independent network paths to a client. Our system features a proxy-server located at the junction of the wired and wireless portions of each path. The proxy dynamically adapts the content data sent over the wireless links, in response to channel quality feedback from the client, such that video distortion at the client is minimized. We analyze the performance of our system and contrast its characteristics with dynamic content adaptation at the source and conventional streaming architectures, including scalable video. We numerically simulate the operation of all streaming systems under comparison and establish a close agreement between our analysis and the experimental findings. The proposed system delivers superior video quality over the reference competitors, while enabling notable transmission rate savings, at the same time.

Viewpoint Switching Prediction Model for Multi-view Video Based on Viewing Logs

Multi-view video consists of multiple views which are shot simultaneously by multiple cameras from different viewpoints. Users can watch the video while choosing a viewpoint from them freely. In the multi-view video streaming, we need to deal with a problem that the multi-view videos put an enormous burden on the network for delivering because of transmitting many videos at a same time. Hierarchical multi-view streaming(HMVS) has been proposed as one of the methods of reducing the transmission quantity of multi-view video. The prediction of switching viewpoints is necessary for bandwidth allocation in HMVS. In this paper, we propose a viewpoint switching prediction model based on viewing logs for HMVS. The proposed model divides the sequence of multi-view video into some scenes and learns the probability of switching viewpoints for each scene from viewing logs. Through the evaluation experiment, the proposed model achieved effective viewpoint switching predictions.

UMSM: A Traffic Reduction Method on Multi-View Video Streaming for Multiple Users

Multi-view video consists of multiple video sequences captured simultaneously by multiple closely spaced cameras and enables users to freely change their viewpoints by playing different video sequences. Therefore, the transmission bitrate of multi-view video requires more bandwidth than conventional multimedia. In order to reduce the bandwidth requirement, User Dependent Multi-view Video Transmission (UDMVT), which is based on Multi-view Video Coding (MVC), has been proposed for single users. In UDMVT, the same frames are encoded into different versions for each user, which increases the transmission bitrate for multiple users because of the redundant transmission of overlapping frames. In order to address this problem, we proposed User dependent Multi-view video Streaming for Multi-users (UMSM). UMSM possesses two characteristics: overlapping frames required by multiple users are transmitted only once by multicast while un-overlapping frames required by each user are transmitted by unicast and offset of video requests between multiple users is aligned so as to maximize the area of overlapping frames. UMSM achieves a low transmission bitrate of multi-view video for multiple users by means of combining these characteristics. To investigate the effect of user's view switching on traffic reduction, we design three types of view-switching models, namely, scanning model, random model, and watching model. Simulation results obtained using benchmark test sequences provided by MERL reveal that UMSM decreases the transmission bitrate by 80.6% on average for five users, as compared to MVC, when users gaze a viewpoint for a while after switching views in order to search an attractive viewpoint.

Optimizing Distributed Source Coding for Interactive Multiview Video Streaming Over Lossy Networks

In interactive multiview video streaming (IMVS), a user observes one view at a time, but can periodically switch to a desired neighboring captured view as the video is played back in time. Previous IMVS works focus on efficient compression techniques that facilitate interactive view switching. In this paper, in addition to the loss-resilient aspect during network streaming we address how to design efficient coding tools and optimize frame structure for transmission to facilitate view switching and contain error propagation in differentially coded video due to packet losses. We first design a new unified distributed source coding (uDSC) frame-a new coding tool that simultaneously offers view switching and loss-resilient capabilities-for periodic insertion into the multiview frame structure. After inserting uDSC-frames into the coding structure, we schedule packets for network transmission in a rate-distortion optimal manner for both wireless multicast and wired unicast streaming scenarios. For wireless multicast over a Gilbert-Elliott loss model, frames in a group of pictures are packetized and reordered, so that uDSC frames are correctly decoded with high probability, mitigating error propagation. For wired unicast, we use a Markov decision process to optimize packet transmission to minimize expected distortion given a bandwidth constraint. Experimental results show that systems that insert uDSC frames and optimize packet transmission can outperform other competing coding schemes by up to 2.8 and 11.6 dB in wireless multicast and wired unicast streaming scenarios, respectively.

Optimized MVC Prediction Structures for Interactive Multiview Video Streaming

The Multiview Video Coding (MVC) standard efficiently compresses multiview video by considering spatial, temporal and interview correlations. This letter studies the impact of the MVC interview prediction structure on both the transmission and the overall coding rates for an interactive multiview video streaming system, considering both unicast and multicast scenarios, with the user interactive behavior represented by some view-popularity model. We propose a method to identify the optimal prediction structure minimizing the visual distortion, given some storage and link capacities constraints. Simulation results confirm that the optimal prediction structure results from a non-trivial tradeoff between the system constraints, the transmission model and the views' popularity.

Adaptive multiview video streaming: challenges and opportunities

Delivering multiview video content over present packet networks poses multiple challenges. First, the best effort nature of the Internet exposes media packets to variable bandwidth, loss, and delay as they traverse the network. Second, the prediction dependencies employed to maximize compression efficiency make the reconstruction process at the client extremely vulnerable to missing data. Third, the heterogeneity of client devices in terms of computing power, display capabilities, and access link capacity necessitates customizing the streaming process per user. My article reviews existing opportunities for addressing these challenges from within each of the three main stages of the content delivery pipeline (i.e., encoding, transmission, and reconstruction). Concretely, I first describe adaptive source coding techniques that construct a compressed representation of the multiview video source that exhibits resilience to network bandwidth variations and client view selection uncertainty. Then I discuss intelligent methods for error protection, caching, and packet scheduling that organize the transmission of multiview data in a bandwidth-effective way. Here, I also review prospective multipath and cloud-assisted techniques for multiview video streaming. Finally, I identify robust client-side content reconstruction schemes and adaptive media playout methods that can minimize the impact of missing data and enhance the user's interactive experience. Then I proceed to describe community-driven streaming techniques for delivering interactive multiview content over a population of social peers. The article concludes with an outline of approaches for synergistic exploitation of the techniques I will present theretofore, jointly across the different layers of the network protocol stack at which they individually operate. Here, I also highlight the main deployment challenges for some of these techniques, and how their design should be addressed accordingly, to overcome them.

H.264/MVC interleaving for real-time multiview video streaming

The multiview video streaming demands high bandwidth even after coding. Any loss during transmission will have effect on the real-time quality of experience (QoE) of the end user due to the prediction structure used in H.264/MVC encoder. This paper proposes an interleaving scheme by considering the prediction structure used in the encoder. The priority for every picture in the prediction structure is computed using proposed mathematical model in order to satisfy any GOP size and any number of views. The validation and verification of the proposed work is done using H.264/MVC reference software JMVC 8.5. The test results show that the proposed interleaving scheme significantly outperforms the non-interleaved scheme.