Omnidirectional Video Streaming Research Articles

Quality-Constrained Encoding Optimization for Omnidirectional Video Streaming

Omnidirectional video streaming is usually implemented based on the representations of tiles, where the tiles are obtained by splitting the video frame into several rectangular areas and each tile is converted into multiple representations with different resolutions and encoded at different bitrates. One key issue in omnidirectional video streaming is how to choose the optimal representations for each tile at the server to save the overall transmission bitrate to all users while offering them satisfactory quality. This is different from the adaptive bitrate-based method that optimizes the downloading procedure of individual users, where the given video representations are stored on the server. In this work, we focus on optimization for the encoding of omnidirectional video streaming by using the optimal combination of tile representations. To achieve our goal, we formulate the selection of the representations into an optimization problem in which the transmission bitrate of all the representations is minimized with a quality constraint. By using this constraint, we can improve the average quality of omnidirectional videos for users. More specifically, we first construct the tile-level rate-distortion (R-D) model and determine the available tile bandwidth based on the previous viewers’ statistics. Then, we formulate the representation selection problem based on the obtained R-D model and tile bandwidth. Finally, we solve this problem to obtain the optimal combination of tile representations so that we can transmit the omnidirectional video to users with satisfactory quality but low bitrate. The experimental results demonstrate the effectiveness of our proposed approach when it is applied to omnidirectional video streaming.

A Quality-of-Experience Database for Adaptive Omnidirectional Video Streaming

A Quality-of-Experience Database for Adaptive Omnidirectional Video Streaming

Metadata-Based Quality-Estimation Model for Tile-Based Omnidirectional Video Streaming

In this paper, a metadata-based quality-estimation model is proposed for tile-based omnidirectional video streaming services, aiming to realize quality monitoring during service provision. In the tile-based omnidirectional video (ODV) streaming services, the ODV is divided into tiles, and the high-quality tiles and the low-quality tiles are distributed in accordance with the user's viewing direction. When the user changes the viewing direction, the user temporarily watches video with the low-quality tiles. In addition, the longer the time (delay time) until the high-quality tile for the new viewing direction is downloaded, the longer the viewing time of video with the low-quality tile, and thus the delay time affects quality. From the above, the video quality of the low-quality tiles and the delay time significantly impact quality, and these factors need to be considered in the quality-estimation model. We develop quality-estimation models by extending the conventional quality-estimation models for 2D adaptive streaming. We also show that the quality-estimation model using the bitrate, resolution, and frame rate of high- and low-quality tiles and that the delay time has sufficient estimation accuracy based on the results of subjective quality evaluation experiments.

Extension of ITU-T P.1203 model to Tile-based Omnidirectional Video Streaming

Omnidirectional video (ODV) streaming has become widespread. Since the data size of ODV is extremely large, tilebased streaming has been developed to compress the data. In this coding technology, high-quality tiles encoded at a higher bitrate for the users’ viewing direction and low-quality tiles encoded at a lower bitrate for the whole environment are sent to the client, and a player decodes these tiles. As a result, quality degrades due to coding, streaming, and the client’s buffer. Therefore, to provide high-quality tile-based ODV streaming services, qualityof- experience needs to be monitored by comprehensively evaluating the quality degradations. By taking into account the quality degradation due to the low-quality tiles, the ITU-T Recommendation P.1203 model, which can be used for monitoring the quality of 2D video streaming services, is extended to tile-based ODV streaming services. Our model is demonstrated to estimate quality with sufficiently high accuracy.

Modeling the Perceptual Quality for Viewport-Adaptive Omnidirectional Video Streaming Considering Dynamic Quality Boundary Artifact

Instead of streaming the entire omnidirectional video that induces a waste of bandwidth, the tile-based viewport adaptive streaming is preferred in practice. Tiles within users’ viewport are transmitted in high quality, while tiles elsewhere are delivered in lower quality to reduce the network bandwidth consumption. In this setup, when users change their viewport, they may potentially be disturbed by the exposure of low-quality tiles and the subsequent quality refinement in a period of delay. We defined this phenomenon in the viewport as a Dynamic Quality Boundary (DQB) artifact, as there exists a clear boundary between low-quality tiles and high-quality tiles which appears, moves, and disappears along with the change of users’ viewport. The DQB artifact is a specific quality degradation in the viewport-adaptive omnidirectional video streaming whose impact on users’ perceptual quality has not been adequately studied. In this paper, we focused on investigating and modeling its impact on the quality perception of users when streaming omnidirectional videos. More specifically, systematic subjective quality evaluation experiments towards DQB artifact were designed and conducted. Accordingly, a perceptual quality metric was proposed considering the quality of tiles, the proportion of low-quality tiles, and the refinement duration. Experimental results, measured in various aspects, showed that the proposed metric can accurately predict users’ perceptual quality of the tile-based viewport-adaptive omnidirectional video streaming service.

Subjective Evaluation of User Quality of Experience for Omnidirectional Video Streaming

This paper reports on the results of subjective testing of user Quality of Experience (QoE) for omnidirectional video (ODV) streaming quality. The test was conducted among 20 test subjects who watched three ODVs using a Head Mounted Display (HMD) system. The length of the videos was between two and three minutes. The first video was used for training purposes and contained no quality degradations. The quality of the other two ODVs was degraded by manipulating the resolution or by introducing different frame drop patterns. While watching the pre-prepared videos the subjects indicated if they noticed the changes in the quality and then rated it. After watching each video, the subjects completed a separate questionnaire, which evaluated their level of enjoyment and discomfort with the video. The results showed that the degradation of both objective parameters (video resolution and frame rate) impacted the subjects’ perception of quality; however, the impact was somewhat alleviated in ODV which contained dynamic scenes and fast camera movements.

An Innovative Machine-Learning-Based Scheduling Solution for Improving Live UHD Video Streaming Quality in Highly Dynamic Network Environments

The latest advances in terms of network technologies open up new opportunities for high-end applications, including using the next generation video streaming technologies. As mobile devices become more affordable and powerful, an increasing range of rich media applications could offer a highly realistic and immersive experience to mobile users. However, this comes at the cost of very stringent Quality of Service (QoS) requirements, putting significant pressure on the underlying networks. In order to accommodate these new rich media applications and overcome their associated challenges, this paper proposes an innovative Machine Learning-based scheduling solution which supports increased quality for live omnidirectional (360°) video streaming. The proposed solution is deployed in a highly dynamic Unmanned Aerial Vehicle (UAV)-based environment to support immersive live omnidirectional video streaming to mobile users. The effectiveness of the proposed method is demonstrated through simulations and compared against three state-of-the-art scheduling solutions, such as: static Prioritization (SP), Required Activity Detection Scheduler (RADS) and Frame Level Scheduler (FLS). The results show that the proposed solution outperforms the other schemes involved in terms of PSNR, throughput and packet loss rate.

Perceptually Optimized Quality Adaptation of Viewport-Dependent Omnidirectional Video Streaming

Viewport-Dependent Streaming (VDS) is a preferred way in practice to deliver the omnidirectional videos, of which a High-Quality (HQ) scale is applied for the content in current viewport but a Low-Quality (LQ) scale elsewhere. Quality adaptation or refinement happens after users stabilize their fixations to a new viewport. In this article, we formulate this as a perceptually optimized quality adaptation problem to maximize the Quality of Experience (QoE) for the refinement from a LQ scale to another HQ level within a specific duration under the given network constraint. With our developed perceptual model considering the adaptation quality for VDS of omnidirectional videos, we first provide baseline solutions numerically, demonstrating the noticeable subjective improvements of model-driven solution against the heuristic selection based approach. We also propose a novel viewport prediction algorithm based on the Hidden Markov Model (HMM), and experimental results show that it significantly outperforms the relevant methods with better prediction accuracy. We then improve the adaptation strategy with proposed viewport prediction-based data prefetching, leading to better visual perception than the baseline system at the same bandwidth constraint. Generally, prefetching the content of predicted next viewport one second ahead of its playback time, would lead to more than 8% Bjontegaard Delta Rate (BD-Rate) gain.

Shared Coded Picture Technique for Tile-Based Viewport-Adaptive Streaming of Omnidirectional Video

Tile-based viewport-adaptive streaming methods have been used in delivering omnidirectional video for virtual reality applications. In these methods, the 360° video is encoded in multiple quality versions by using the motion constrained tile set (MCTS) technique. A set of high-quality and low-quality tiles, corresponding to viewport and non-viewport areas, respectively, are selected and transmitted to the user. However, these methods require frequent intra random access points to ensure seamless viewport switching capability, very high decoding complexity, or a multi-layer coding scheme. The frequent intra random access points include very high bitrate in viewport switching points. The high decoding complexity and multi-layer decoder requirements are not aligned with the omnidirectional media format (OMAF) standard. Such requirements make these methods sub-optimal or impractical for streaming the omnidirectional video. This paper studies the current tile-based solutions for delivering the omnidirectional content. Moreover, the OMAF-compliant shared coded picture (SCP)-based scheme is proposed in this paper for streaming the omnidirectional video. The core concept of the SCP-based method is to manipulate the switching point pictures in a way that the frequent intra-coded pictures are no longer required for the viewport switching operations between different quality versions of the content. The experiments illustrated that the SCP-based method outperforms the MCTS-based method on average by 11% to 14% in terms of streaming bitrate reduction with only 4% extra decoding complexity.

Modeling the Perceptual Quality of Viewport Adaptive Omnidirectional Video Streaming

Instead of streaming the entire OmniDirectional Videos (ODVs) that are often sampled at ultra high definition and high frame rate, a viewport adaptive streaming is preferred in practice. We usually stream the High-Quality (HQ) content within current viewport, while Low-Quality (LQ) elsewhere to save the network bandwidth consumption. Such scheme would lead to a quality refinement after user adapts his/her focus to a new viewport. In this paper, we thus model the perceptual impact of the quality variations (through adjusting the Quantization Stepsize (QS or q) and Spatial Resolution (SR or s)) with respect to the Refinement Duration (RD or τ) when performing the refinement from an arbitrary LQ scale to an arbitrary HQ one. A number of quality variations are studied to cover sufficient use cases in practice, resulting in a unified analytical model, as a product of separable exponential functions that measure the QS and SR induced perceptual impacts in terms of the RD, and a perceptual index measuring the subjective quality of corresponding viewport video after refinement. This model is first validated in a managed lab environment via independent subjective assessments by constraining user's navigation to avoid unexpected noise, where both Pearson Correlation Coefficient (PCC) and Spearman's Rank Correlation Coefficient (SRCC) are around 0.97. We then extend the validations in a real-life viewport-dependent streaming system, still yielding PCC and SRCC about 0.96 when comparing collected subjective scores with model predictions.

Real-time Omnidirectional Video Streaming System

Real-time Omnidirectional Video Streaming System

An accurate and robust visual-compass algorithm for robot-mounted omnidirectional cameras

Due to their wide field of view, omnidirectional cameras are becoming ubiquitous in many mobile robotic applications. A challenging problem consists of using these sensors, mounted on mobile robotic platforms, as visual compasses (VCs) to provide an estimate of the rotational motion of the camera/robot from the omnidirectional video stream. Existing VC algorithms suffer from some practical limitations, since they require a precise knowledge either of the camera-calibration parameters, or the 3-D geometry of the observed scene. In this paper we present a novel multiple-view geometry constraint for paracatadioptric views of lines in 3-D, that we use to design a VC algorithm that does not require either the knowledge of the camera calibration parameters, or the 3-D scene geometry. In addition, our algorithm runs in real time since it relies on a closed-form estimate of the camera/robot rotation, and can address the image-feature correspondence problem. Extensive simulations and experiments with real robots have been performed to show the accuracy and robustness of the proposed method.

Telepresence by Real-Time View-Dependent Image Generation from Omnidirectional Video Streams

This paper describes a new approach to telepresence which realizes virtual tours into a visualized dynamic real world without significant time delay. We propose a novel concept ofinstantourwhich enables us to instantly look around a visualized space of a dynamic real world. Theinstantouris realized by the following two steps: (1) video-rate omnidirectional image acquisition and (2) real-time view-dependent perspective image generation from an omnidirectional video stream. The proposed technique is applicable to real-time telepresence in the situation where the real world to be seen is far from an observation site, because the time delay from the change of viewing direction to the change of displayed image does not depend on the actual distance between both sites. Moreover, multiple users can look around from a single viewpoint in a visualized dynamic real world in different directions at the same time. The proposed technique is also useful for another type of telepresence which uses recorded omnidirectional video streams. We have developed a prototype of virtualinstantoursystem and have successfully experimented with real dynamic scenes.