360-degree Video Streaming Research Articles

Online tile dispatching framework with guarantees for 360-degree video streaming in wireless networks

Online tile dispatching framework with guarantees for 360-degree video streaming in wireless networks

MNCATM: A Multi-Layer Non-Uniform Coding-Based Adaptive Transmission Method for 360° Video

With the rapid development of multimedia services and smart devices, 360-degree video has enhanced the user viewing experience, ushering in a new era of immersive human–computer interaction. These technologies are increasingly integrating everyday life, including gaming, education, and healthcare. However, the uneven spatiotemporal distribution of wireless resources presents significant challenges for the transmission of ultra-high-definition 360-degree video streaming. To address this issue, this paper proposes a multi-layer non-uniform coding-based adaptive transmission method for 360° video (MNCATM). This method optimizes video caching and transmission by dividing non-uniform tiles and leveraging users’ dynamic field of view (FoV) information and the multi-bitrate characteristics of video content. First, the video transmission process is formalized and modeled, and an adaptive transmission optimization framework for a non-uniform video is proposed. Based on this, the optimization problem required by the paper is summarized, and an algorithm is proposed to solve the problem. Simulation experiments demonstrate that the proposed method, MNCATM, outperforms existing transmission schemes in terms of bandwidth utilization and user quality of experience (QoE). MNCATM can effectively utilize network bandwidth, reduce latency, improve transmission efficiency, and maximize user experience quality.

CA-Live360: Crowd-assisted transcoding and delivery for live 360-degree video streaming

In the burgeoning field of digital media, the popularity of live 360-degree video has witnessed an upward trajectory, underpinned by its immersive experience. In live 360-degree video streaming services, transcoding enormous video tiles and providing low-latency services for global viewers require significantly more computing and communication resources than traditional live services. Inspired by the huge crowdsourcing computational resources available at crowd, this paper designs CA-Live360, a Crowd-Assisted Live 360-degree video streaming solution, in which transcoding and delivery are supported by crowdsourcing devices. Due to the dynamic availability of resources, it is challenging to achieve low-latency live services, while guaranteeing fair transcoding. Therefore, firstly an innovative fairness-guaranteed transcoding task assignment scheme is proposed. This scheme employs a novel Fair Bandit algorithm based on the combinatorial multi-armed bandit approach to achieve low-latency transcoding while considering time-varying available resources and fairness constraints. Secondly, to reduce delivery latency, we design a transcoding-aware delivery scheme, in which a provider–requester matching algorithm is introduced to realize effective requester scheduling. Real-world trace-driven experiments demonstrate the improved performance of the proposed CA-Live360 solution in terms of system delay and fairness.

MADRL-Based Joint Edge Caching and Bitrate Selection for Multicategory 360° Video Streaming

360-degree video streaming has gained increasing attraction in the current popular VR, AR, and MR applications, which can provide users with an immersive experience. In tile-based 360-degree video streaming, edge caching and bitrate selection strategies are jointly designed to improve users’ Quality of Experience (QoE), which incorporates video quality and rebuffer. However, the existing QoE-driven approaches use a unified QoE function to guide the decisions of edge caching and bitrate selection, which neglect the impact of video quality and rebuffer on different categories of 360-degree videos, thus failing to provide high average QoE for users. In this paper, we propose a MADRL-based joint edge caching and bitrate selection strategy for multicategory 360-degree video streaming to improve users’ average QoE. The key idea is to employ different edge caching and bitrate selection strategies for different video categories to enable fine-grained performance optimization. Based on multicategory 360-degree video streaming, we first model a joint edge caching and bitrate selection problem as a multi-agent cooperative Markov decision process with the goal of maximizing users’ average QoE. Next, an FoV-aware multi-agent soft actor-critic (FA-MASAC) algorithm is designed to help agents collaboratively learn optimal edge caching and bitrate selection decisions in a distributed way, in which each video category is treated as an agent. Finally, experimental results on real-world datasets show that our proposed strategy can greatly benefit users’ average QoE compared to existing strategies.

Local and Global Viewport History Sampling for Improved User Quality of Experience in Viewport-Aware Tile-Based 360-Degree Video Streaming

Local and Global Viewport History Sampling for Improved User Quality of Experience in Viewport-Aware Tile-Based 360-Degree Video Streaming

Virtual Reality Telepresence: 360-Degree Video Streaming with Edge-Compute Assisted Static Foveated Compression.

Real-time communication with immersive 360° video can enable users to be telepresent within a remotely streamed environment. Increasingly, users are shifting to mobile devices and connecting to the Internet via mobile-cellular networks. As the ideal media for 360° videos, some VR headsets now also come with cellular capacity, giving them potential for mobile applications. However, streaming high-quality 360° live video poses challenges for network bandwidth, particularly on cellular connections. To reduce bandwidth requirements, videos can be compressed using viewport-adaptive streaming or foveated rendering techniques. Such approaches require very low latency in order to be effective, which has previously limited their applications on traditional cellular networks. In this work, we demonstrate an end-to-end virtual reality telepresence system that streams $\sim$6K 360° video over 5G millimeter-wave (mmW) radio. Our use of 5G technologies, in conjunction with mobile edge compute nodes, substantially reduces latency when compared with existing 4G networks, enabling high-efficiency foveated compression over modern cellular networks on par with WiFi. We performed a technical evaluation of our system's visual quality post-compression with peak signal-to-noise ratio (PSNR) and FOVVideoVDP. We also conducted a user study to evaluate users' sensitivity to compressed video. Our findings demonstrate that our system achieves visually indistinguishable video streams while using up to 80% less data when compared with un-foveated video. We demonstrate our video compression system in the context of an immersive, telepresent video calling application.

Masked360: Enabling Robust 360-degree Video Streaming with Ultra Low Bandwidth Consumption.

360-degree video streaming has gained tremendous growth over the past years. However, the delivery of 360-degree videos over the Internet still suffers from the scarcity of network bandwidth and adverse network conditions (e.g., packet loss, delay). In this paper, we propose a practical neural-enhanced 360-degree video streaming framework called Masked360, which can significantly reduce bandwidth consumption and achieve robustness against packet loss. In Masked360, instead of transmitting the complete video frame, the video server only transmits a masked low-resolution version of each video frame to reduce bandwidth significantly. When delivering masked video frames, the video server also sends a lightweight neural network model called MaskedEncoder to clients. Upon receiving masked frames, the client can reconstruct the original 360-degree video frames and start playback. To further improve the quality of video streaming, we also propose a set of optimization techniques, such as complexity-based patch selection, quarter masking strategy, redundant patch transmission and enhanced model training methods. In addition to bandwidth savings, Masked360 is also robust to packet loss during the transmission, because packet losses can be concealed by the reconstruction operation performed by the MaskedEncoder. Finally, we implement the whole Masked360 framework and evaluate its performance using real datasets. The experimental results show that Masked360 can achieve 4K 360-degree video streaming with bandwidth as low as 2.4 Mbps. Besides, video quality of Masked360 is also improved significantly, with an improvement of 5.24-16.61% in terms of PSNR and 4.74-16.15% in terms of SSIM compared to other baselines.

Predicting Long-Term Field of View in 360-Degree Video Streaming

Many virtual reality (VR) and augmented reality (AR) applications are closely associated with the streaming of 360-degree videos. The user only views a portion of the 360- degree video within his field of view (FoV). Thus, long-term prediction of the user's future FoV can save bandwidth resources in on-demand video streaming, minimize video rebuffering time caused by significant bandwidth fluctuations in the network and reduce unnecessary delays between the server and the client. In this article, we predict the target user's FoV based on the user's past FoV trajectory as well as the other users' future FoV trajectory. Firstly, we propose an eye-gaze-based heatmap to describe the historical movement trajectory of the user's viewport more accurately. Then, we use a method based on other user's eye gaze positions to select similar users. Finally, we introduce the squeeze-and-excitation (SE) network and U-Net (SE-Unet) to combine the historical trajectory of the target user with the future trajectory of other users to predict long-term FoV. Experimental results show that the prediction model proposed in our research is more accurate and durable.

Toward Low-Latency and High-Quality Adaptive 360$^\circ$ Streaming

Advanced 360-degree video streaming is essential for high-quality communication and industrial video applications, supporting novel and interactive visual experiences that promote consumption. However, due to the inherent fluctuations in communication networks, hardware resources, and power costs, a trade-off needs to be achieved between occupying network bandwidth and guaranteeing 360-degree visual quality. To address this issue, this paper proposes a quality-aware global optimization solution that combines different types of sensory characteristics to further enhance the 360-degree streaming efficiency in cellular networks. First, considering the characteristics of 360-degree video, each frame is divided into three different regions based on a proposed field-of-view (FoV) prediction method. Second, a new region-based rate-distortion model reflecting the bitrate-quality relationship is proposed. The divided regions are represented by different rate-distortion models. In addition, a rate-distortion model parameter update strategy with robustness to region changes is proposed to further guarantee transmission performance. Finally, we propose a globally optimized adaptive bitrate allocation algorithm to optimize 360-degree mobile streaming, which uses both the rate-distortion models and viewpoint prediction results. Evaluation results indicate that the proposed method outperforms state-of-the-art approaches in terms of several quality of experience (QoE) objectives under various network conditions.

Rate-Splitting for Intelligent Reflecting Surface-Aided Multiuser VR Streaming

The growing demand for virtual reality (VR) applications requires wireless systems to provide a high transmission rate to support 360-degree video streaming to multiple users simultaneously. In this paper, we propose an intelligent reflecting surface (IRS)-aided rate-splitting (RS) VR streaming system. In the proposed system, RS facilitates the exploitation of the shared interests of the users in VR streaming, and IRS creates additional propagation channels to support the transmission of high-resolution 360-degree videos. IRS also enhances the capability to mitigate the performance bottleneck caused by the requirement that all RS users have to be able to decode the common message. We formulate an optimization problem for maximization of the achievable bitrate of the 360-degree video subject to the quality-of-service (QoS) constraints of the users. We propose a deep deterministic policy gradient with imitation learning (Deep-GRAIL) algorithm, in which we leverage deep reinforcement learning (DRL) and the hidden convexity of the formulated problem to optimize the IRS phase shifts, RS parameters, beamforming vectors, and bitrate selection of the 360-degree video tiles. We also propose RavNet, which is a deep neural network customized for the policy learning in our Deep-GRAIL algorithm. Performance evaluation based on a real-world VR streaming dataset shows that the proposed IRS-aided RS VR streaming system outperforms several baseline schemes in terms of system sum-rate, achievable bitrate of the 360-degree videos, and online execution runtime. Our results also reveal the respective performance gains obtained from RS and IRS for improving the QoS in multiuser VR streaming systems.

Edge Intelligence-Empowered Immersive Media

Recent years have witnessed many immersive media services and applications, ranging from 360-degree video streaming, to augmented and virtual reality (AR and VR) and the recent metaverse experiences. These new applications usually have common features including high fidelity, immersive interaction, and open data exchange between people and the environment. As an emerging paradigm, edge computing has become increasingly ready to support these features. We first show that a key to unleashing the power of edge computing for immersive multimedia is handling AI models and data. Then, we present a framework that enables joint accuracy- and latency-aware edge intelligence, with adaptive deep learning model deployment and data streaming. We show that not only conventional mechanisms such as content placement and rate adaptation, but also the emerging 360-degree and virtual reality streaming can benefit from such edge intelligence.

Live360: Viewport-Aware Transmission Optimization in Live 360-Degree Video Streaming

Live 360-degree video streaming is emerging as the next disruptive innovation in the coming era. However, compared to conventional live video streaming, it is much more challenging to provide high-quality live 360-degree video streaming due to high bandwidth demand, stringent latency requirement, and diverse user viewports. In this paper, we propose a viewport-aware live 360-degree video streaming framework called Live360 to optimize end-to-end video stream transmission. To make the problem tractable, we decompose the transmission optimization problem into two sub-problems, and optimize upstream and downstream transmission in an asynchronous way. We prioritize 360-degree cameras according to users’ real-time viewing interests and upload attractive contents with a higher bitrate. We also re-define the metric of QoE (Quality-of-Experience) for live 360-degree video viewers, and solve the optimization problems efficiently using dynamic programming. Extensive experiments show that Live360 is lightweight and significantly outperforms other state-of-the-art methods in terms of various QoE metrics. The average QoE of Live360 is twice that of other baseline methods.

Robust Saliency-Driven Quality Adaptation for Mobile 360-Degree Video Streaming

Mobile 360-degree video streaming has grown significantly in popularity but the quality of experience (QoE) suffers from insufficient and variable wireless network bandwidth. Recently, saliency-driven 360-degree streaming overcomes the buffer size limitation of head movement trajectory (HMT)-driven solutions and thus strikes a better balance between video quality and rebuffering. However, inaccurate network estimations and intrinsic saliency bias still challenge saliency-based streaming approaches, limiting further QoE improvement. To address these challenges, we design a robust saliency-driven quality adaptation algorithm for 360-degree video streaming, RoSal360. Specifically, we present a practical, tile-size-aware deep neural network (DNN) model with a decoupled self-attention architecture to accurately and efficiently predict the transmission time of video tiles. Moreover, we design a reinforcement learning (RL)-driven online correction algorithm to robustly compensate the improper quality allocations due to saliency bias. Through extensive prototype evaluations over real wireless network environments including commodity WiFi, 4G/LTE, and 5G links in the wild, RoSal360 significantly enhances the video quality and reduces the rebuffering ratio, thereby improving the viewer QoE, compared to the state-of-the-art algorithms.

Seer: Learning-Based $360^{\circ }$ Video Streaming for MEC-Equipped Cellular Networks

The increasing popularity of virtual reality and 360-degree video streaming has created a demand for streaming high-quality video content over cellular networks. However, streaming 360-degree videos over cellular networks poses a challenge due to the need for ultra-low latency and high bandwidth. Although viewport adaptive streaming presents a promising solution to reduce the bandwidth requirement of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> video, it remains challenging to predict the Field of View (FoV) due to the adverse impact of prediction errors on the user's Quality of Experience (QoE). Moreover, bitrate adaptation gets complicated due to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> videos and the dynamic nature of the cellular network. Therefore, in this paper, we propose an intelligent framework, termed Seer, to fully realize <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> video streaming over cellular networks by leveraging Multi-access Edge Computing (MEC). It incorporates an FoV prediction scheme and a bitrate adaptation policy into the MEC platform. In particular, the FoV prediction scheme uses perceived video features coupled with time-series based viewport prediction to achieve higher prediction accuracy. For an effective bitrate selection in cellular network conditions, a Reinforcement Learning (RL) based bitrate adaptation policy is further employed. The results demonstrate that Seer's FoV prediction scheme delivers higher prediction accuracy, while the streaming framework significantly reduces the bandwidth requirements and improves the user's QoE.

Live 360 Degree Video Delivery Based on User Collaboration in a Streaming Flock

Streaming of live 360-degree video allows users to follow a live event from any view point and has already been deployed on some commercial platforms. However, the current systems can only stream the video at relatively low-quality because the entire 360-degree video is delivered to the users under limited bandwidth. Streaming video falling into user field of view (FoV) can improve bandwidth efficiency of 360-degree video delivery. In this paper, we propose to use the idea of flocking to simultaneously improve the accuracy of user FoV prediction and video delivery efficiency for live 360-degree video streaming. By assigning variable playback latencies to users in a streaming session based on their network conditions, a streaming flock is formed and led by strong users with low playback latencies in the front of the flock. We propose a long short-term memory (LSTM) based collaborative FoV prediction scheme where the FoV traces of users in the front of the flock are utilized to predict the FoV of users behind them. Given a predicted FoV, we develop an optimal rate allocation strategy to maximize the perceptual quality. By conducting experiments using real-world user FoV traces and LTE/5G network bandwidth traces, we evaluate the gains of the proposed strategies over several benchmarks. Our experimental results demonstrate that the proposed streaming system can increase the overall quality dramatically by about 10 dB compared with heuristic FoV prediction strategy. In addition, the network-aware flocking formation can further reduce the video freeze without influencing video quality.

A Semantic-Aware Transmission With Adaptive Control Scheme for Volumetric Video Service

Volumetric video provides a more immersive holographic virtual experience than conventional video services such as 360-degree and virtual reality (VR) videos. However, due to ultra-high bandwidth requirements, existing compression and transmission technology cannot handle the delivery of real-time volumetric video. Unlike traditional compression methods and the approaches that extend 360-degree video streaming, we propose AITransfer, an AI-powered compression and semantic-aware transmission method for point cloud video data (a popular volumetric data format). AITransfer targets the semantic-level communication beyond transmitting raw point cloud video or compressed video with two outstanding contributions: (1) designing an integrated end-to-end architecture with two fundamental contents of feature extraction and reconstruction to reduce the bandwidth consumption and alleviate the computational pressure; and (2) incorporating the dynamic network condition into end-to-end architecture design and employing a deep reinforcement learning-based adaptive control scheme to provide robust transmission. We conduct extensive experiments on the typical datasets and develop a case study to demonstrate the efficiency and effectiveness. The results show that AITransfer can provide extremely efficient point cloud transmission while maintaining considerable user experience with more than 30.72x compression ratio under the existing network environments.

360-Degree Video Bandwidth Reduction: Technique and Approaches Comprehensive Review

Recently, the usage of 360-degree videos has prevailed in various sectors such as education, real estate, medical, entertainment and more. The development of the Virtual World “Metaverse” demanded a Virtual Reality (VR) environment with high immersion and a smooth user experience. However, various challenges are faced to provide real-time streaming due to the nature of high-resolution 360-degree videos such as high bandwidth requirement, high computing power and low delay tolerance. To overcome these challenges, streaming methods such as Dynamic Adaptive Streaming over HTTP (DASH), Tiling, Viewport-Adaptive and Machine Learning (ML) are discussed. Moreover, the superiorities of the development of 5G and 6G networks, Mobile Edge Computing (MEC) and Caching and the Information-Centric Network (ICN) approaches to optimize the 360-degree video streaming are elaborated. All of these methods strike to improve the Quality of Experience (QoE) and Quality of Service (QoS) of VR services. Next, the challenges faced in QoE modeling and the existing objective and subjective QoE assessment methods of 360-degree video are presented. Lastly, potential future research that utilizes and further improves the existing methods substantially is discussed. With the efforts of various research studies and industries and the gradual development of the network in recent years, a deep fake virtual world, “Metaverse” with high immersion and conducive for daily life working, learning and socializing are around the corner.

RAPT360: Reinforcement Learning-Based Rate Adaptation for 360-Degree Video Streaming With Adaptive Prediction and Tiling

Tile-based rate adaption can improve the quality of experience (QoE) for adaptive 360-degree video streaming under constrained network conditions, which, however, is a challenging problem due to the requirements of accurate prediction for users&#x2019; viewports and optimal bitrate allocation for tiles. In this paper, we propose a strategy that deploys reinforcement learning-based Rate Adaptation with adaptive Prediction and Tiling for 360-degree video streaming, named RAPT360, to address these challenges. Specifically, to improve the accuracy of the state-of-the-art viewport prediction approaches, we fit the time-varying Laplace distribution-based probability density function of the prediction error for different prediction lengths. On the basis of that, we develop a viewport identification method to determine the viewport area of a user depending on the buffer occupancy, where the obtained viewport can cover the real viewport with any given probability confidence level. We then propose a viewport-aware adaptive tiling scheme to improve the bandwidth efficiency, where three types of tile granularities are allocated according to the shape and position of the 2-D projection of that viewport. By establishing an adaptive streaming model and QoE metric specific to 360-degree videos, we finally formulate the rate adaptation problem for tile-based 360-degree video streaming as a non-linear discrete optimization problem that targets at maximizing the long-term user QoE under a bandwidth-constrained network. To efficiently solve this problem, we model the rate adaptation logic as a Markov decision process (MDP) and employ the deep reinforcement learning (DRL)-based algorithm to dynamically learn the optimal bitrate allocation of tiles. Extensive experimental results show that RAPT360 achieves a performance gain of at least 1.47 dB on average chunk QoE, including a video quality improvement of at least 1.33 dB, in comparison to the existing strategies for tile-based adaptive 360-degree video streaming.

Tile-Based Wireless Streaming of 360-Degree Video With Rate Adaptation Using Viewport Estimation

This paper presents tile-based wireless streaming of 360-degree videos with rate adaptation using viewport estimation. We propose a probabilistic model for viewport location to design importance weights of tiles. Based on the tile weights, we improve the streaming performance by dynamically adjusting quantization parameters and forward error correction rates across tiles.

Bandit Learning-Based Edge Caching for 360-Degree Video Streaming With Switching Cost

Virtual Reality (VR) and Augmented Reality (AR) applications are expected to be a key driver for the 5G network. The need of the MEC support for caching and transcoding of omnidirectional content has been identified as a major topic of research. In this paper, we study the optimal caching problem in an MEC-enabled VR system for tile-based viewport-adaptive 360-degree video streaming. For content caching, the replacement of the cached content inevitably incurs content switching cost, which is largely ignored in caching policy design. In this work, we aim to find the optimal caching policy to optimize the long-term transmission quality by considering both transmission latency and content switching cost. We apply the combinatorial multi-armed bandit (CMAB) theory to solve the problem with no a-priori knowledge on content popularity. Moreover, the CUCB with switching cost (CUCBSC) algorithm is adopted in this scenario. A transformation mechanism is designed to transform the generated single period optimization (SPO) problem into a multiple choice knapsack problem (MCKP). Rigorous theoretical analysis on the performance of the proposed algorithm is also provided. Finally, the effectiveness of the proposed learning based caching policy is confirmed by simulation results in terms of learning, hit-ratio, transmission and content switching delay.respondence) papers with a suitable choice of class options.