Adaptive Playback Research Articles

ALVS: Adaptive Live Video Streaming using deep reinforcement learning

Achieving a high Quality of Experience (QoE) in live event streaming is a challenging problem given a low-latency requirement and time-varying network conditions. Adaptive video bitrate and adaptive playback speed techniques are two separate control knobs to address this challenge. In this paper, we consider these two control parameters in a joint optimization problem and present a deep reinforcement learning (DRL) framework to maximize QoE for live streaming without any assumption about the environment or fixed rule-based heuristics. With the proposed DRL framework, our approach (ALVS) constructs the inference model to make a joint decision of adaptive playback speed and video quality level for the next video segment. Simulation results through real network traces show that ALVS outperforms both state-of-the-art DRL-based and rule-based algorithms in terms of QoE without sacrificing live latency and skipping any content.

Adaptive Playback Control: A Framework for Cinematic VR Creators to Embrace Viewer Interaction

Content creators have been trying to produce engaging and enjoyable Cinematic Virtual Reality (CVR) experiences using immersive media such as 360-degree videos. However, a complete and flexible framework, like the filmmaking grammar toolbox for film directors, is missing for creators working on CVR, especially those working on CVR storytelling with viewer interactions. Researchers and creators widely acknowledge that a viewer-centered story design and a viewer’s intention to interact are two intrinsic characteristics of CVR storytelling. In this paper, we stand on that common ground and propose Adaptive Playback Control (APC) as a set of guidelines to assist content creators in making design decisions about the story structure and viewer interaction implementation during production. Instead of looking at everything CVR covers, we set constraints to focus only at cultural heritage oriented content using a guided-tour style. We further choose two vital elements for interactive CVR: the narrative progression (director vs. viewer control) and visibility of viewer interaction (implicit vs. explicit) as the main topics at this stage. We conducted a user study to evaluate four variants by combining these two elements, and measured the levels of engagement, enjoyment, usability, and memory performance. One of our findings is that there were no differences in the objective results. Combining objective data with observations of the participants’ behavior we provide guidelines as a starting point for the application of the APC framework. Creators need to choose if the viewer will have control over narrative progression and the visibility of interaction based on whether the purpose of a piece is to invoke emotional resonance or promote efficient transfer of knowledge. Also, creators need to consider the viewer’s natural tendency to explore and provide extra incentives to invoke exploratory behaviors in viewers when adding interactive elements. We recommend more viewer control for projects aiming at viewer’s participation and agency, but more director control for projects focusing on education and training. Explicit (vs. implicit) control will also yield higher levels of engagement and enjoyment if the viewer’s uncertainty of interaction consequences can be relieved.

JustSpeak: Automated, User-Configurable, Interactive Agents for Speech Tutoring

Conversational agents are widely used in many situations, especially for speech tutoring. However, their contents and functions are often pre-defined and not customizable for people without technical backgrounds, thus significantly limiting their flexibility and usability. Besides, conventional agents often cannot provide feedback in the middle of training sessions because they lack technical approaches to evaluate users' speech dynamically. We propose JustSpeak: automated and interactive speech tutoring agents with various configurable feedback mechanisms, using any speech recordings with its transcription text as the template for speech training. In JustSpeak, we developed an automated procedure to generate customized tutoring agents from user-inputted templates. Moreover, we created a set of methods to dynamically synchronize speech recognizers' behavior with the agent's tutoring progress, making it possible to detect various speech mistakes dynamically such as being stuck, mispronunciation, and rhythm deviations. Furthermore, we identified the design primitives in JustSpeak to create different novel feedback mechanisms, such as adaptive playback, follow-on training, and passive adaptation. They can be combined to create customized tutoring agents, which we demonstrate with an example for language learning. We believe JustSpeak can create more personalized speech learning opportunities by enabling tutoring agents that are customizable, always available, and easy-to-use.

Viewer’s Role and Viewer Interaction in Cinematic Virtual Reality

Cinematic Virtual Reality (CVR) is a form of immersive storytelling widely used to create engaging and enjoyable experiences. However, issues related to the Narrative Paradox and Fear of Missing Out (FOMO) can negatively affect the user experience. In this paper, we review the literature about designing CVR content with the consideration of the viewer’s role in the story, the target scenario, and the level of viewer interaction, all aimed to resolve these issues. Based on our explorations, we propose a “Continuum of Interactivity” to explore appropriate spaces for creating CVR experiences to archive high levels of engagement and immersion. We also discuss two properties to consider when enabling interaction in CVR, the depth of impact and the visibility. We then propose the concept framework Adaptive Playback Control (APC), a machine-mediated narrative system with implicit user interaction and backstage authorial control. We focus on “swivel-chair” 360-degree video CVR with the aim of providing a framework of mediated CVR storytelling with interactivity. We target content creators who develop engaging CVR experiences for education, entertainment, and other applications without requiring professional knowledge in VR and immersive systems design.

Catching the Moment With LoL$^+$ in Twitch-Like Low-Latency Live Streaming Platforms

Our earlier Low-on-Latency (dubbed as LoL) solution offered an accurate bandwidth prediction and rate adaptation algorithm tailored for live streaming applications that targeted an end-to-end latency of up to two seconds. While LoL was a significant step forward in multi-bitrate low-latency live streaming, further experimentation and testing showed that there was room for improvement in three areas. First, LoL used hard-coded parameters computed from an offline training process in the rate adaptation algorithm and this was seen as a significant barrier in LoL’s wide deployment. Second, LoL’s objective was to maximize a collective QoE function. Yet, certain use cases have specific objectives besides the singular QoE and this had to be accommodated. Third, the adaptive playback speed control failed to produce satisfying results in some scenarios. Our goal in this paper is to address these areas and make LoL sufficiently robust to deploy. We refer to the enhanced solution as LoL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^+$</tex-math></inline-formula> , which has been integrated to the official dash.js player in v3.2.0.

Adaptive video streaming over HTTP through 4G wireless networks based on buffer analysis

One of the main challenges in video transmission is understanding and adapting to the varying network bandwidth. The traditional approach of bandwidth estimation is not accurate as there are many factors like congestion that can delay the arrival rate of the ping packet which may lead to a misconception that the bandwidth was low. Thus, the better approach to this problem will be to estimate the link conditions based on the buffer fullness. In this paper, a new system to support streaming of live and stored video through wireless network is proposed which is based on adaptive playback buffer management on the top of HTTP at the client. The buffer fullness is treated as a direct state variable that reflects the fluctuation of the network bandwidth. The buffer fullness estimation predicts the buffer status at a point in the future based on observations of the buffer over a stipulated period of time. The proposed algorithm uses non-linear exponential non-parametric regression for computing the decision parameter. A feedback message is then sent to the server in order to change the quality of the video stream for smoother video play at the client side. The synchronized update and feedback between the server and clients is achieved using HTTP protocol. During the experimentation with live video streaming, the proposed algorithm shows an improvement of 24.48% in average peak signal-to-noise ratio and 6.63% in average structural similarity index against the buffer underflow probability algorithm.

On Efficient Channel Modeling for Video Transmission over Cognitive Radio Networks

This paper investigates the problem of video transmission over cognitive radio networks with the objective of maintaining continuous video playback while gracefully degrading the quality of the reconstructed video sequences, if needed. We focus on modeling the channel availability to secondary users, which is a major limiting factor on the continuity of the streaming process. A Markov chain model for the channels availability in an M-channels system is developed. This model is used to estimate the likelihood of transmission interruptions a secondary user might experience due to the loss of a channel to a primary user. We also propose a joint adaptive mechanism where a simple source rate control scheme is integrated with an adaptive playback approach to reduce the impact of channels relocation/unavailability on the streaming process of active secondary users. Simulations and numerical investigations demonstrate the correctness of the proposed channel model. Simulation results also indicate that instants of playback buffer starvation at the secondary user ends could be avoided only when the hybrid approach is employed

APB: An Adaptive Playback Buffer Scheme for Wireless Streaming Media

The wireless streaming media communications are fragile to the delay jitter because the conditions and requirements vary frequently with the users' mobility. Buffering is a typical way to reduce the delay jitter of media packets before the playback, however, it will incur a longer end-to-end delay. Our motivation in this paper is to improve the balance between the elimination of delay jitter and the decrease of end-to-end delay. We propose a novel adaptive playback buffer (APB) based on the probing scheme. By utilizing the probing scheme, the instantaneous network situations are collected, and then the delay margin and the delay jitter margin are employed to calculate the step length (sl) which is used to adjust the playback buffer in each time. The adaptive adjustment to the playback buffer in APB enables the continuous and real-time representation of streaming media at the receiver. Unlike the previous studies, the novelty and contributions of the paper are: a) Accuracy: by employing the instantaneous network information, the adjustment to the playback buffer correctly reflects the current network situations and therefore achieves the improved balance between the elimination of delay jitter and the decrease of end-to-end delay; Hence. APB adjustment is accurate in terms of improving such balance; b) Efficiency: by utilizing the simple probing scheme, APB achieves the current network situations without the complex mathematic predictions, which enables the adjustment to be more timely and efficient. Performance data obtained through extensive simulations show that our APB is effective to reduce both delay jitter and playback buffer delay.