Inferring End-to-End Latency in Live Videos

Abstract

This paper provides an end-to-end latency model of real-time generated live videos composed of sub-models of capturing, encoding, network, decoding, rendering, and display refreshing. We build the model by deeply analyzing each sub part’s latency with parameters like video’s Spatial (frame scale <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$s$ </tex-math></inline-formula> ), Temporal (frame rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$t$ </tex-math></inline-formula> ), Amplitude (quantization step-size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula> ) Resolutions (STAR), Compressed Frame Bit Numbers ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n_{F}$ </tex-math></inline-formula> ), and other necessary parameters. The latency of 640 combinations of STAR for eight representative video sequences is measured to fit accurate models. We primarily focus on processing procedures, whose latency is becoming the principal affection, as network latency decreases due to the its development. We then introduce a baseline video sequence set to reduce the complexity when fitting the encoding and decoding sub-models by predicting the model parameters on new hardware from parameters of a known one, and improve models’ robustness crossing different scenarios. These sub-models correlate well with the realistic latency data, with the average Pearson Correlation Coefficient (PCC) of 0.99 and the average Spearman’s Rank Correlation Coefficient (SRCC) of 0.98, according to an independent validation test. Furthermore, we validate our end-to-end latency model in a simulated end-to-end live video transmission from generating to displaying, introducing the latency as a strict condition in adaptive bit-rate selection. The simulation shows that our model could significantly increase time during which the latency can satisfy the given tolerance at a low cost.