Multiview Video Streaming Research Articles

PADELVIC: Multicamera videos and motion capture data of an amateur padel match

Recent advances in computer vision and deep learning techniques have opened new possibilities regarding the automatic labeling of sport videos. However, an essential requirement for supervised techniques is the availability of accurately labeled training datasets. In this paper we present PadelVic, an annotated dataset of an amateur padel match which consists of multi-view video streams, accurate motion capture data of one of the players, as well as synthetic videos specifically designed to serve as training sets for convolutional neural networks estimating positional data from videos. As a demonstration of one of the applications of the dataset, we present a system for the accurate prediction of the center-of-mass of the players projected onto the court plane, from a single-view video of the match.

QoE Optimization in DASH-Based Multiview Video Streaming

QoE Optimization in DASH-Based Multiview Video Streaming

Design of new energy vehicle exhibition space based on virtual reality technology

At present, although the new energy vehicle experience center presents a dynamic development trend of science, technology, diversity and innovation, the display and dissemination of the new energy vehicle industry and culture cannot be satisfied, resulting in the display function cannot be fully exerted. VR (Virtual Reality) display design is a new topic of audio–visual​ design with the popularization of VR technology and related computer application technology. This paper focuses on the realization and application of VR exhibition space design for new energy vehicles, and discusses the design principles that should be followed in VR exhibition space design for new energy vehicles and its practical significance to design communication. In this paper, a parallel 3D reconstruction method based on multi-view video stream is proposed. In this method, the mobile terminal and the server execute asynchronously, the terminal decodes the multi-view video stream in parallel, and the server performs feature extraction and stereo matching on the returned key frames. Finally, the calculated parallax is combined with the projection matrix to restore the 3D scene.

Towards real-time 3D visualization with multiview RGB camera array.

A real-time 3D visualization (RT3DV) system using a multiview RGB camera array is presented. RT3DV can process multiple synchronized video streams to produce a stereo video of a dynamic scene from a chosen view angle. Its design objective is to facilitate 3D visualization at the video frame rate with good viewing quality. To facilitate 3D vision, RT3DV estimates and updates a surface mesh model formed directly from a set of sparse key points. The 3D coordinates of these key points are estimated from matching 2D key points across multiview video streams with the aid of epipolar geometry and trifocal tensor. To capture the scene dynamics, 2D key points in individual video streams are tracked between successive frames. We implemented a proof of concept RT3DV system tasked to process five synchronous video streams acquired by an RGB camera array. It achieves a processing speed of 44 milliseconds per frame and a peak signal to noise ratio (PSNR) of 15.9 dB from a viewpoint coinciding with a reference view. As a comparison, an image-based MVS algorithm utilizing a dense point cloud model and frame by frame feature detection and matching will require 7 seconds to render a frame and yield a reference view PSNR of 16.3 dB.

A Survey on Mobile Edge Computing for Video Streaming: Opportunities and Challenges

5G communication brings substantial improvements in the quality of service provided to various applications by achieving higher throughput and lower latency. However, interactive multimedia applications (e.g., ultra high definition video conferencing, 3D and multiview video streaming, crowd-sourced video streaming, cloud gaming, virtual and augmented reality) are becoming more ambitious with high volume and low latency video streams putting strict demands on the already congested networks. Mobile Edge Computing (MEC) is an emerging paradigm that extends cloud computing capabilities to the edge of the network i.e., at the base station level. To meet the latency requirements and avoid the end-to-end communication with remote cloud data centers, MEC allows to store and process video content (e.g., caching, transcoding, pre-processing) at the base stations. Both video on demand and live video streaming can utilize MEC to improve existing services and develop novel use cases, such as video analytics, and targeted advertisements. MEC is expected to reshape the future of video streaming by providing ultra-reliable and low latency streaming (e.g., in augmented reality, virtual reality, and autonomous vehicles), pervasive computing (e.g., in real-time video analytics), and blockchain-enabled architecture for secure live streaming. This paper presents a comprehensive survey of recent developments in MEC-enabled video streaming bringing unprecedented improvement to enable novel use cases. A detailed review of the state-of-the-art is presented covering novel caching schemes, optimal computation offloading, cooperative caching and offloading and the use of artificial intelligence (i.e., machine learning, deep learning, and reinforcement learning) in MEC-assisted video streaming services.

Optimal adaptive streaming of a scalable multi-view video via rate splitting and SIC

Optimal adaptive streaming of a scalable multi-view video via rate splitting and SIC

Location-proximity-based clustering method for peer-to-peer multimedia streaming services with multiple sources.

Social distancing to reduce the spread of coronavirus disease 2019 (COVID-19) made a huge increase in the global OTT market, and OTT service providers get millions of new subscribers. Recently OTT service providers are extending their service to video broadcasting. As a one type of video broadcasting, this paper covers multimedia streaming with multiple sources. Multimedia streaming with multiple sources has multiple sources, and receivers can select one specific source to watch the video from the source. Sources include cameras capturing different angles of same event or location, cameras in geographical locations, etc. For delivering video to rapidly increasing number of users, multimedia streaming with multiple sources system needs efficient and scalable delivery method. Tree-based Peer-to-peer (P2P) networking has been investigated as the delivery solution of multimedia streaming with multiple sources, and set-top boxes or mobile apps of OTT service can be used as peers connecting the subscriber of OTT service. However, the scalability of the tree-based P2P networking is limited by the out-degree of a tree that branches linearly with the number of users. Hence, this study proposes clustering peers based on the location proximity of the peers to enhance the scalability of the P2P multimedia streaming with multiple sources. By clustering peers, one or more peers can be grouped into a virtual peer with an aggregated uplink/downlink capacity. This paper describes P2P multimedia streaming with multiple sources and algorithms for the proposed clustering method. Two applications which are one-view multiparty video conferencing and multi-view video streaming are introduced, and considerations for applying the proposed method to the applications are also discussed. The experimental results show that location-proximity-based clustering is effective in achieving a scalable P2P multimedia streaming with multiple sources by reducing the out-degree of a tree for the introduced applications. The proposed clustering leads improvement in the maximum achievable video bit rate, the average viewing video bit rate, and perceived delay.

Soft Video Uploading for Low-Power Crowdsourced Multi-view Video Streaming

Soft Video Uploading for Low-Power Crowdsourced Multi-view Video Streaming

PoM: Power-Efficient Multi-View Video Streaming Over Multi-Antenna Wireless Systems

Multi-view video streaming is becoming more popular, with increased potential for immersive applications. However, it poses a challenge in terms of energy consumption on the transmission devices with limited battery power due to the vast traffic volume. Previous literature has considered efficient video transmission but not exploited the full potential of joint design of 3D video coding and MIMO physical layer. In this paper, we present the first study on power consumption minimization for multi-view video streaming with a quality guarantee in 802.11 MIMO-OFDM systems. We propose an efficient algorithm to perform antenna assignment and transmission power allocation, by exploiting both the source coding characteristics of MVC and channel diversity of multiple antennas. A proof-of-concept system, namely PoM , is designed on the software-defined radio platform and is evaluated in realistic indoor environments. To the best of our knowledge, this is the first practical system for energy-efficient multi-view video streaming. Experimental results show that PoM can significantly save energy in the transmission by 88% ~ 26% on average for the PSNR requirement ranging from 35dB to 45dB, and 85% ~ 12% for SSIM from 0.80 to 0.95.

MMVS/COE: mobile multi-view video streaming with constant order encoding

Multi-view video systems are designed to allow users to watch 3D videos or a scene recorded by multiple cameras from multiple viewpoints. They are actually used by crowd sourced journalism services or to cover events using a set of wireless drones/sensors filming the same scene, etc. Multi-view videos are captured by multiple cameras at different positions with significant correlations between neighboring views. Owing to the increased data volume of multi-view video, highly efficient encoding techniques are needed. The common idea for Multi-View Video Coding (MVC) is to further exploit the redundancy between adjacent views. In this paper, we focus on the acquisition phase of the multi-view video system. We propose a Mobile Multi-view Video Streaming scheme with Constant Order Encoding (MMVS/COE). It encodes by exploiting the inter/intra-view dependency to reduce redundancy and optimize the tradeoff between traffic volume (bite rate) and video quality. Evaluations’ results show that MMVS/COE reduces traffic, compared to existing methods, mainly MVC/MC, by decreasing redundancies among video streams while maintaining video quality.

Multiview Video Transmission Over Underwater Acoustic Path

Existing studies on multiview video streaming are proposed for terrestrial environments. They can be classified into request-reply and all camera transmission models, while both models suffer from a long switching delay and low video quality in an underwater acoustic path. The implementation of multiview video transmission in an underwater acoustic path requires reduction of the switching delay and the maintenance of high video quality. To this end, we propose a video transmission scheme, namely, Dolphin. Dolphin comprises time-shifted slot assignment, stochastic transmission, correlation-based compensation, and greedy transmission. Dolphin assigns asymmetric and time-shifted transmission slots to an encoder node and a user node according to a propagation delay. Stochastic transmission sends a potential camera's video that will be the most likely to be played back by the user before playback. If the stochastic transmission fails, the video frames of the user's desired camera are encoded with the mispredicted video frames to prevent switching delay from increasing. Moreover, greedy transmission sends all potential video frames, which are potentially displayed by a user, and determines rate allocation of the video frames based on the display probability. Evaluations using Mitsubishi Electric Research Laboratories' benchmark test sequences demonstrated that Dolphin achieved a short switching delay with a slight degradation in video quality irrespective of communication distance and acoustic path quality. For example, Dolphin decreases the switching delay by 92.7% compared with an existing request-reply model with 0.4-dB quality degradation at a communication distance of 300 m when a user tends to gaze at one camera.

DASH-Based Multi-View Video Streaming System

With the rapid development of electronic and computing technology, multi-view video transmission has attracted significant attention from the research and industrial communities. A multi-view video service provides a user with various viewpoints according to the user’s input. Alongside the recent trend of shifting video streaming service platforms to HTTP adaptive streaming, dynamic adaptive streaming over HTTP (DASH)-based multi-view video services has been widely studied. However, DASH-based multi-view video streaming suffers from high view-switching delay. The delay between a request for view switching and the rendering of the target view adversely affects the quality of experience. This paper provides an analysis of the view-switching delay of DASH-based multi-view video streaming. Based on the analysis, a novel DASH-based multi-view video streaming system is proposed. It can minimize the view-switching delay by employing a buffer control, parallel streaming, and server push schemes. The results obtained from the implementation of this solution show that the proposed system significantly reduces the view-switching delay while guaranteeing the quality of multi-view video.

User-perceived quality aware adaptive streaming of 3D multi-view video plus depth over the internet

Video streaming is a foremost and growing contributor in the ever increasing Internet traffic. Since last two decades, due to the enhancement in cameras and image processing technology, we have seen a shift towards multi-view plus depth (MVD) technology from traditional 2D and 3D video technology. This growth comes with deep changes in the Internet bandwidth, video coding and network technologies, which smoothed the mode for delivery of MVD content to end-users over the Internet. Since, MVD contains large amounts of data than single view video, it requires more bandwidth. It is a challenging task for network service provider to deliver such views with the best user’s Quality of Experience(QoE) in dynamic network condition. Also, Internet is known to be prone to packet loss, bandwidth variation, delay and network congestion, which may prevent video packets from being delivered on time. Besides that, different capabilities of end user’s devices in terms of computing power, display, and access link capacity are other challenges. As consequences, the viewing experiences of 3D videos may well degrade, if the quality-aware adaptation techniques are not deployed. In this article, our work concentrates to present a comprehensive analysis of a dynamic network environment for streaming of 3D MVD over Internet (HTTP). We analyzed the effect of different adaptation of decision strategies and formulated a new quality-aware adaptation technique. The proposed technique is promoting from layer based video coding in terms of transmitted views scalability. The results of MVD streaming experiment, using the proposed approach have shown that the video quality of perceptual 3D improves significantly, as an effect of proposed quality aware adaptation even in adverse network conditions.

Convolutional Neural Network for Intermediate View Enhancement in Multiview Streaming

Multiview video streaming continues to gain popularity due to the great viewing experience it offers, as well as its availability that has been enabled by increased network throughput and other recent technical developments. User demand for interactive multiview video streaming that provides seamless view switching upon request is also increasing. However, it is a highly challenging task to stream stable and high quality videos that allow real-time scene navigation within the bandwidth constraint. In this paper, a convolutional neural network (ConvNet)-assisted seamless multiview video streaming system is proposed to tackle the challenge. The proposed method solves the problem from two perspectives. First, a ConvNet-assisted multiview representation method is proposed, which provides flexible interactivity without compromising on multiview video compression efficiency. Second, a bit allocation mechanism guided by a navigation model is developed to provide seamless navigation and adapt to network bandwidth fluctuations at the same time. These two blocks work closely to provide an optimized viewing experience to users. They can be integrated into any existing multiview video streaming framework to enhance overall performance. Experimental results demonstrate the effectiveness of the proposed method for seamless multiview streaming.

A packetization strategy for interactive multiview video streaming over lossy networks

Interactive multiview video streaming (IMVS) applications permit to freely navigate within 3D scenes. This endows end-users the immersive and virtual reality experiences. Any additional viewpoint of interest can be virtually synthesized by received texture views with associated depth maps. In this paper, we consider the setup where both the encoded texture and depth bitstreams experience packet losses during network streaming. We investigate two packetization strategies and develop a novel strategy to improve error-resilience capabilities for multiview video plus depth transmission, where texture data and the corresponding depth go into the same packet. We examine the resulting distortion of different packetization schemes and analyze the real difference between them in diverse scenarios. In particular, we take into account: (i) various packet loss patterns, where the effect of a burst loss is identified as an important characteristic; (ii) two packet loss models that characterize different probabilities for each scenario. Experimental results demonstrate that, compared with the competing approach, our proposed packetization scheme increases the visual quality by up to 2.32 dB and 1.55 dB for texture views and synthesized views, respectively.

Optimal Representations for Adaptive Streaming in Interactive Multiview Video Systems

Interactive multiview video streaming (IMVS) services permit to remotely navigate within a 3D scene with an immersive experience. This is possible by transmitting a set of reference camera views (anchor views), which are used by the clients to freely navigate in the scene and possibly synthesize additional viewpoints of interest. From a networking perspective, the big challenge in IMVS systems is to deliver to each client the best set of anchor views that maximizes the navigation quality, minimizes the view-switching delay and yet satisfies the network constraints. Integrating adaptive streaming solutions in free-viewpoint systems offers a promising solution to deploy IMVS in large and heterogeneous scenarios, as long as the multiview video representations on the server are properly selected. Therefore, we propose to optimize the multiview data at the server by minimizing the overall resource requirements while offering a good navigation quality to the different users. We propose a representation set optimization problem for multiview adaptive streaming systems, and we show that it is NP-hard. Therefore, we introduce the concept of multiview navigation segment that permits to cast the video representation set selection as an integer linear programming problem with a bounded computational complexity. We then show that the proposed solution reduces the computational complexity, while preserving optimality in most of the 3D scenes. We finally provide simulation results for different classes of users and show the gain offered by an optimal multiview video representation selection compared to recommended representation sets (e.g., Netflix and Apple ones) or to a baseline representation selection algorithm, where the encoding parameters are decided a priori for all the camera views.

QoE-driven optimization for cloud-assisted DASH-based scalable interactive multiview video streaming over wireless network

In interactive multiview video streaming (IMVS), the viewers can periodically switch viewpoints. If the captured view is not available at the desired viewpoint, virtual views can be rendered from neighboring coded views using view synthesis techniques. Dynamic adaptive streaming over HTTP (DASH) is a new standard that allows to adjust the quality of video streaming based on the network condition. In this paper, an improved DASH-based IMVS framework is proposed. It has the following characteristics. First, virtual views could be adaptively generated at either the cloud-based server or the client in our scheme, depending on the network condition and the cost of the cloud. Second, scalable video coding (SVC) is used to improve the flexibility in our system. To optimize the Quality of Experience (QoE) of multiple clients in wireless scenario, we develop a cross-layer optimization scheme. We first propose a new cache management method to selectively store the video data according to SVC structure and the clients’ requesting condition. Next, a cross-layer scheduling scheme is proposed by considering the video rate adaptation and the wireless resource allocation. The optimization problem is shown to be equivalent to the Multiple Choice Knapsack (MCKP) problem. A dynamic programming method and a low-cost greedy method are developed to solve the problem. Simulations with the NS3 tool demonstrate the advantage of our proposed scheme over the existing approach that always uses client-based view synthesis and single-layer video coding.

Interview Motion Compensated Joint Decoding for Compressively Sampled Multiview Video Streams

In this paper, we design a novel multiview video encoding/decoding architecture for wirelessly multiview video streaming applications, e.g., 360 degrees video, Internet of Things (IoT) multimedia sensing, among others, based on distributed video coding and compressed sensing principles. Specifically, we focus on joint decoding of independently encoded compressively sampled multiview video streams. We first propose a novel side-information (SI) generation method based on a new interview motion compensation algorithm for multiview video joint reconstruction at the decoder end. Then, we propose a technique to fuse the received measurements with resampled measurements from the generated SI to perform the final recovery. Based on the proposed joint reconstruction method, we also derive a blind video quality estimation technique that can be used to adapt online the video encoding rate at the sensors to guarantee desired quality levels in multiview video streaming. Extensive simulation results of real multiview video traces show the effectiveness of the proposed fusion reconstruction method with the assistance of SI generated by an interview motion compensation method. Moreover, they also illustrate that the blind quality estimation algorithm can accurately estimate the reconstruction quality.

Dynamic bitrate allocation of interactive real-time streamed multi-view video with view-switch prediction

In multi-view video (MVV), the real-world scene is usually captured by more than two cameras positioned in an array. A viewer can consume MVV using either a non-interactive or an interactive transmission method. In the context of interactive MVV streaming, view switching may cause a long delay due to the frequent requests by the viewer. In this paper, we consider the use case of real-time interactive MVV (IMVV) streaming, where the view switching delay problem has a significant user experience impact. Our proposed method compress and send all the captured views using a dynamic bitrate allocation method. Also, a novel prediction algorithm has been used to choose possible views that the user might switch to. The predicted view switching is mapped to a hidden Markov model, and the transition probability is solved using Zipf distribution. The experimental results of the proposed method show a superior performance on an objective metric and view-switching delay for better viewing quality over the existing method.

Optimized Adaptive Streaming of Multi-video Stream Bundles

In contrast to traditional video, multi-view video streaming allows viewers to interactively switch among multiple perspectives provided by different cameras. One approach to achieve such a service is to encode the video from all of the cameras into a single stream, but this has the disadvantage that only a portion of the received video data will be used, namely that required for the selected view at each point in time. In this paper, we introduce the concept of a “multi-video stream bundle” that consists of multiple parallel video streams that are synchronized in time, each providing the video from a different camera capturing the same event or movie. For delivery we leverage the adaptive features and time-based chunking of HTTP-based adaptive streaming, but now employing adaptation in both content and rate. Users are able to change their viewpoint on-demand and the client player adapts the rate at which data are retrieved from each stream based on the user's current view, the probabilities of switching to other views, and the user's current bandwidth conditions. A crucial component of such a system is the prefetching policy. For this we present an optimization model as well as a simpler heuristic that can balance the playback quality and the probability of playback interruptions. After analytically and numerically characterizing the optimal solution, we present a prototype implementation and sample results. Our prefetching and buffer management solution is shown to provide close to seamless playback switching when there is sufficient bandwidth to prefetch the parallel streams.