Interactive Multiview Video Streaming Research Articles

A packetization strategy for interactive multiview video streaming over lossy networks

Interactive multiview video streaming (IMVS) applications permit to freely navigate within 3D scenes. This endows end-users the immersive and virtual reality experiences. Any additional viewpoint of interest can be virtually synthesized by received texture views with associated depth maps. In this paper, we consider the setup where both the encoded texture and depth bitstreams experience packet losses during network streaming. We investigate two packetization strategies and develop a novel strategy to improve error-resilience capabilities for multiview video plus depth transmission, where texture data and the corresponding depth go into the same packet. We examine the resulting distortion of different packetization schemes and analyze the real difference between them in diverse scenarios. In particular, we take into account: (i) various packet loss patterns, where the effect of a burst loss is identified as an important characteristic; (ii) two packet loss models that characterize different probabilities for each scenario. Experimental results demonstrate that, compared with the competing approach, our proposed packetization scheme increases the visual quality by up to 2.32 dB and 1.55 dB for texture views and synthesized views, respectively.

Optimal Representations for Adaptive Streaming in Interactive Multiview Video Systems

Interactive multiview video streaming (IMVS) services permit to remotely navigate within a 3D scene with an immersive experience. This is possible by transmitting a set of reference camera views (anchor views), which are used by the clients to freely navigate in the scene and possibly synthesize additional viewpoints of interest. From a networking perspective, the big challenge in IMVS systems is to deliver to each client the best set of anchor views that maximizes the navigation quality, minimizes the view-switching delay and yet satisfies the network constraints. Integrating adaptive streaming solutions in free-viewpoint systems offers a promising solution to deploy IMVS in large and heterogeneous scenarios, as long as the multiview video representations on the server are properly selected. Therefore, we propose to optimize the multiview data at the server by minimizing the overall resource requirements while offering a good navigation quality to the different users. We propose a representation set optimization problem for multiview adaptive streaming systems, and we show that it is NP-hard. Therefore, we introduce the concept of multiview navigation segment that permits to cast the video representation set selection as an integer linear programming problem with a bounded computational complexity. We then show that the proposed solution reduces the computational complexity, while preserving optimality in most of the 3D scenes. We finally provide simulation results for different classes of users and show the gain offered by an optimal multiview video representation selection compared to recommended representation sets (e.g., Netflix and Apple ones) or to a baseline representation selection algorithm, where the encoding parameters are decided a priori for all the camera views.

QoE-driven optimization for cloud-assisted DASH-based scalable interactive multiview video streaming over wireless network

In interactive multiview video streaming (IMVS), the viewers can periodically switch viewpoints. If the captured view is not available at the desired viewpoint, virtual views can be rendered from neighboring coded views using view synthesis techniques. Dynamic adaptive streaming over HTTP (DASH) is a new standard that allows to adjust the quality of video streaming based on the network condition. In this paper, an improved DASH-based IMVS framework is proposed. It has the following characteristics. First, virtual views could be adaptively generated at either the cloud-based server or the client in our scheme, depending on the network condition and the cost of the cloud. Second, scalable video coding (SVC) is used to improve the flexibility in our system. To optimize the Quality of Experience (QoE) of multiple clients in wireless scenario, we develop a cross-layer optimization scheme. We first propose a new cache management method to selectively store the video data according to SVC structure and the clients’ requesting condition. Next, a cross-layer scheduling scheme is proposed by considering the video rate adaptation and the wireless resource allocation. The optimization problem is shown to be equivalent to the Multiple Choice Knapsack (MCKP) problem. A dynamic programming method and a low-cost greedy method are developed to solve the problem. Simulations with the NS3 tool demonstrate the advantage of our proposed scheme over the existing approach that always uses client-based view synthesis and single-layer video coding.

Dynamic bitrate allocation of interactive real-time streamed multi-view video with view-switch prediction

In multi-view video (MVV), the real-world scene is usually captured by more than two cameras positioned in an array. A viewer can consume MVV using either a non-interactive or an interactive transmission method. In the context of interactive MVV streaming, view switching may cause a long delay due to the frequent requests by the viewer. In this paper, we consider the use case of real-time interactive MVV (IMVV) streaming, where the view switching delay problem has a significant user experience impact. Our proposed method compress and send all the captured views using a dynamic bitrate allocation method. Also, a novel prediction algorithm has been used to choose possible views that the user might switch to. The predicted view switching is mapped to a hidden Markov model, and the transition probability is solved using Zipf distribution. The experimental results of the proposed method show a superior performance on an objective metric and view-switching delay for better viewing quality over the existing method.

A Modified Multiview Video Streaming System Using 3-Tier Architecture

In this paper, we present a modified inter-view prediction Multiview Video Coding (MVC) scheme from the perspective of viewer's interactivity. When a viewer requests some view(s), our scheme leads to lower transmission bit-rate. We develop an interactive multiview video streaming system exploiting that modified MVC scheme. Conventional interactive multiview video systems require high bandwidth due to redundant data being transferred. With real data test sequences, clear improvements are shown using the proposed interactive multiview video system compared to competing ones in terms of the average transmission bit-rate and storage size of the decoded (i.e., transferred) data with comparable rate-distortion.

Optimal layered representation for adaptive interactive multiview video streaming

We consider an interactive multiview video streaming (IMVS) system where clients select their preferred viewpoint in a given navigation window. To provide high quality IMVS, many high quality views should be transmitted to the clients. However, this is not always possible due to the limited and heterogeneous capabilities of the clients. In this paper, we propose a novel adaptive IMVS solution based on a layered multiview representation where camera views are organized into layered subsets to match the different clients constraints. We formulate an optimization problem for the joint selection of the views subsets and their encoding rates. Then, we propose an optimal and a reduced computational complexity greedy algorithms, both based on dynamic-programming. Simulation results show the good performance of our novel algorithms compared to a baseline algorithm, proving that an effective IMVS adaptive solution should consider the scene content and the client capabilities and their preferences in navigation.

Optimizing Multiview Video Plus Depth Prediction Structures for Interactive Multiview Video Streaming

Several multiview video coding standards have been developed to efficiently compress images from different camera views capturing the same scene by exploiting the spatial, the temporal and the interview correlations. However, the compressed texture and depth data have typically many interview coding dependencies, which may not suit interactive multiview video streaming (IMVS) systems, where the user requests only one view at a time. In this context, this paper proposes an algorithm for the effective selection of the interview prediction structures (PSs) and associated texture and depth quantization parameters (QPs) for IMVS under relevant constraints. These PSs and QPs are selected such that the visual distortion is minimized, given some storage and point-to-point transmission rate constraints, and a user interaction behavior model. Simulation results show that the novel algorithm has near-optimal compression efficiency with low computational complexity, so that it offers an effective encoding solution for IMVS applications.

Coding Structure and Replication Optimization for Interactive Multiview Video Streaming

Multiview video refers to videos of the same dynamic 3-D scene captured simultaneously by multiple closely spaced cameras from different viewpoints. We study interactive streaming of pre-encoded multiview videos, where, at any time, a client can request any one of many captured views for playback. Moreover, the client can periodically freeze the video in time and switch to neighboring views for a compelling look-around visual effect. We consider distributed content servers to support large-scale interactive multiview video service. These servers collaboratively replicate and access video contents. We study two challenges in this setting: what is an efficient coding structure that supports interactive view switching and, given that, what to replicate in each server in order to minimize the cost incurred by interactive temporal and view switches? We first propose a redundant coding structure that facilitates interactive view-switching, trading off storage with transmission rate. Using the coding structure, we next propose a content replication strategy that takes advantage of indirect hit to lower view-switching cost: in the event that the exact requested view is not available locally, the local server can fetch a different but correlated view from the other servers, so that the remote repository only needs to supply the pre-encoded view differential. We formulate the video content replication problem to minimize the switching cost as an integer linear programming (ILP) problem and show that it is NP-hard. We first propose an LP relaxation and rounding algorithm (termed Minimum Eviction) with bounded approximation error. We then study a more scalable solution based on dynamic programming and Lagrangian optimization (DPLO) with little sacrifice in performance. Simulation results show that our replication algorithms achieve substantially lower switching cost compared to other content replication schemes.

Incentive analysis for cooperative interactive multiview video streaming

In interactive multiview video streaming (IMVS), users can periodically select one out of many captured views available for observation. In single-view video streaming, cooperative strategies where peers share received packets of the same video have proven to be effective in reducing server׳s upload burden, and incentive mechanisms are designed to stimulate user cooperation. However, exploiting user cooperation in higher dimensional IMVS is difficult, since users watching different views makes it difficult to establish partnership, and users switching views frequently and independently makes it difficult to maintain partnership over time. In this paper, we use a multiview video frame structure for IMVS to support cooperative view-switching, where peers may help each other even if they are observing different views. We then model peers׳ interaction as an indirect reciprocity game, where each user is assigned a reputation level. To gain a higher reputation level, users help others, which in turn leads to a higher likelihood to receive others׳ help later. In this work, we focus on how view switching, the key feature of IMVS, affects user cooperation. By modeling users׳ decision making as a Markov decision process, our analysis shows that users tend to cooperate at some views but not others: given peers can predict their future view navigation paths probabilistically, for a peer who is likely to enter a view-switching path not requiring others׳ help, he also has less incentive to cooperate. Furthermore, we observe that the game may have multiple Nash Equilibria corresponding to different cooperation levels, e.g., users cooperate at all views in the full cooperation equilibrium, while users only cooperate at certain views in the partial cooperation equilibrium. The particular equilibrium the game will converge to depends on the initial cooperation level of the game. To stimulate user cooperation at all views, we propose a Pay-for-Cooperation (PfC) scheme at the beginning of the game to drive the game to the full cooperation equilibrium to improve system efficiency. Our simulation results show the effectiveness of PfC.

Optimizing Distributed Source Coding for Interactive Multiview Video Streaming Over Lossy Networks

In interactive multiview video streaming (IMVS), a user observes one view at a time, but can periodically switch to a desired neighboring captured view as the video is played back in time. Previous IMVS works focus on efficient compression techniques that facilitate interactive view switching. In this paper, in addition to the loss-resilient aspect during network streaming we address how to design efficient coding tools and optimize frame structure for transmission to facilitate view switching and contain error propagation in differentially coded video due to packet losses. We first design a new unified distributed source coding (uDSC) frame-a new coding tool that simultaneously offers view switching and loss-resilient capabilities-for periodic insertion into the multiview frame structure. After inserting uDSC-frames into the coding structure, we schedule packets for network transmission in a rate-distortion optimal manner for both wireless multicast and wired unicast streaming scenarios. For wireless multicast over a Gilbert-Elliott loss model, frames in a group of pictures are packetized and reordered, so that uDSC frames are correctly decoded with high probability, mitigating error propagation. For wired unicast, we use a Markov decision process to optimize packet transmission to minimize expected distortion given a bandwidth constraint. Experimental results show that systems that insert uDSC frames and optimize packet transmission can outperform other competing coding schemes by up to 2.8 and 11.6 dB in wireless multicast and wired unicast streaming scenarios, respectively.

Optimized MVC Prediction Structures for Interactive Multiview Video Streaming

The Multiview Video Coding (MVC) standard efficiently compresses multiview video by considering spatial, temporal and interview correlations. This letter studies the impact of the MVC interview prediction structure on both the transmission and the overall coding rates for an interactive multiview video streaming system, considering both unicast and multicast scenarios, with the user interactive behavior represented by some view-popularity model. We propose a method to identify the optimal prediction structure minimizing the visual distortion, given some storage and link capacities constraints. Simulation results confirm that the optimal prediction structure results from a non-trivial tradeoff between the system constraints, the transmission model and the views' popularity.

Delay-Cognizant Interactive Streaming of Multiview Video With Free Viewpoint Synthesis

In interactive multiview video streaming (IMVS), a client receives and observes one of many available viewpoints of the same scene and periodically requests from the server view switches to neighboring views, as the video is played back in time uninterruptedly. One key technical challenge is to design a frame coding structure that facilitates periodic view switching and achieves an optimal tradeoff between storage cost and expected transmission rate. In this paper, we first propose three significant improvements over existing IMVS systems and then study the corresponding frame structure optimization. First, using depth-image-based rendering, the new IMVS system enables free viewpoint switching, i.e., by encoding and transmitting both texture and depth maps of captured views, a client can select and synthesize any virtual view from an almost continuum of viewpoints between the left-most and right-most captured views. Second, the IMVS system adopts a more realistic Markovian view-switching model with memory that more accurately captures user behaviors than previous memoryless models . A view-switching model is used in predicting client's future view-switching patterns. Third, assuming that the round-trip-time (RTT) delay during server-client communication is nonnegligible, during an IMVS session, the IMVS system additionally transmits redundant frames RTT into future playback, so that zero-delay view switching can be achieved. Given these improvements, we formalize a new joint optimization of the frame coding structure, transmission schedule, and quantization parameters of the texture and depth maps of multiple camera views. We propose an iterative algorithm to achieve fast and near-optimal solutions. The convergence of the algorithm is also demonstrated. Experimental results show that the proposed optimized rate-allocation method requires 38% lower transmission rate than the fixed rate-allocation scheme. In addition, with the same storage, the transmission rate of the optimized frame structure can be up to 55% lower than that of an I-frame-only structure and 27% lower than that of the structure without distributed source coding frames.