S-MDP: Streaming With Markov Decision Processes

Abstract

In recent years, finding adaptive players competing for network resources at a single bottleneck link has become common. This competition generally occurs at routers in household local area networks. Such competition severely reduces viewers’ quality of experience, such as fairness and stability. Researchers harness Markov decision process (MDP) models to optimize the adaptive video streaming process. Typically, players follow a policy based on numerous parameters, such as buffer occupancy or average bandwidth. In this study, we defied this traditional decentralized client-side MDP approach by allowing players to share a network state among themselves, which we called a streaming MDP (S-MDP). This state includes a discrete data rate measurement (DRM) value. The DRM value is a normalized value of a player's incoming bitrate and is an example of an interval measurement scale. Players use video bitrates to produce unique state transition matrices. The S-MDP reward matrix penalizes excessive switching along the DRM interval scale and thus encourages stability. At intervals during streaming, players create unique policies. The near-real-time update of policies enables players DRM values to converge. S-MDP shows relatively good performance in emulation experiments compared with four streaming methods, namely, k-chunk MDP, stochastic dynamic programming for adaptive streaming over hypertext transfer protocol (sdpDASH), MDP-based DASH, and RTRA_S. In Internet experiments, we compare the performance of streaming methods with a Roku, Amazon Fire TV, and Apple TV. We also compared it against users who play online games, download files, and/or chat online, and S-MDP outperforms the other methods in terms of both objective and subjective visual quality, except in the presence of transmission control protocol long-lived flows, such as Skype.