Video Streaming System Research Articles

COBIRAS: Offering a Continuous Bit Rate Slide to Maximize DASH Streaming Bandwidth Utilization

Reaching close-to-optimal bandwidth utilization in dynamic adaptive streaming over HTTP (DASH) systems can, in theory, be achieved with a small discrete set of bit rate representations. This includes typical bit rate ladders used in state-of-the-art DASH systems. In practice, however, we demonstrate that bandwidth utilization, and consequently the quality of experience (QoE), can be improved by offering a continuous set of bit rate representations, i.e., a continuous bit rate slide (COBIRAS). Moreover, we find that the buffer fill behavior of different standard adaptive bit rate (ABR) algorithms is sub-optimal in terms of bandwidth utilization. To overcome this issue, we leverage COBIRAS’ flexibility to request segments with any arbitrary bit rate and propose a novel ABR algorithm MinOff , which helps maximizing bandwidth utilization by minimizing download off-phases during streaming. To avoid extensive storage requirements with COBIRAS and to demonstrate the feasibility of our approach, we design and implement a proof-of-concept DASH system for video streaming that relies on just-in-time encoding ( JITE ), which reduces storage consumption on the DASH server. Finally, we conduct a performance evaluation on our testbed and compare a state-of-the-art DASH system with few bit rate representations and our JITE DASH system, which can offer a COBIRAS, in terms of bandwidth utilization and video QoE for different ABR algorithms.

Accelerated Bitrate Ladder Creation for Video Live Streaming Using Temporal Layer Injection

Video streaming systems aim to provide high-quality video adapted to clients’ device and network conditions. For this purpose, adaptive streaming architectures encode video content at a variety of quality levels, organized in a bitrate ladder. However, compressing a video into multiple streams is resource-intensive, which may become especially problematic in live streaming applications with real-time demands. Therefore, this paper proposes a novel solution for fast bitrate ladder creation, and provides the requirements for implementation in the H.266/VVC standard. More specifically, the proposed method creates new intermediate combined streams by injecting the lowest temporal layers of a higher-quality augmentation stream in a lower-quality base stream. Since the lowest layers are used as reference by the remaining layers, this procedure indirectly increases the quality of the frames in those untouched remaining layers as well. Although altering the reference frames may cause some drift-error artifacts, we demonstrate that, on average, the quality of the intermediate streams are higher than the base stream. Additionally, we demonstrate that injecting more layers brings both the quality and bitrate closer to that of the augmentation stream. The disadvantage of the combined streams is that their quality fluctuates more than the quality of the source streams, and that they are compressed less efficiently, comparable to going from a slower to fast or faster preset in the VVenC encoder. Most importantly, their main advantage is that they were generated at no significant additional computational complexity. In this way, the proposed method is of great benefit when generating a bitrate ladder of video streams under constrained computational resources. The code has been made open source and is available on https://github.com/IDLabMedia/NALUProcessing .

Optimizing live video streaming: Integrating 5G, IoT, and cloud computing with machine learning

AbstractIn this research, we optimize live video broadcast performance by incorporating advanced technologies such as 5G, the Internet of Things (IoT), and cloud computing. Our approach utilizes the Random Forest classifier to categorize data, achieving a 99% precision rate. A comparative study demonstrates that our proposed technique outperforms RCNN and Mask‐RCNN methods in optimizing video streaming efficacy. We show that our method efficiently enhances video streaming quality by integrating machine learning technologies. The combination of 5G, IoT, and cloud computing creates a robust environment for delivering optimized Live video streaming to users. This research underscores the importance of leveraging cutting‐edge technology to address optimization challenges in modern video streaming systems, focusing on the real‐time optimization of video streams in contemporary technological environments.

The clash between physical and digital realm: hybrid movement training during the pandemic

This article is based on ethnographic research conducted in one of the physical training institutions that offer the Laban/Bartenieff Movement Studies (LBMS) certification program in 2021. Because of the COVID-19 pandemic crisis, the training was held in a hybrid format in which some participants were in the studio while others were attending remotely via Zoom. Zoom-mediated movement training for long hours revealed how the intervention of telematic technologies challenged practitioners’ sensorial experience and sense-making process. Moreover, bringing co-located and remote participants’ experiences together in the hybrid setting disclosed different modes of interaction dynamics in the studio and online. Overall, participants described their hybrid experience as a clash. In the article, starting from unfolding those clashes from the enactive perspective, I discuss how remote intercorporeality through an audio–video streaming system, Zoom, challenges participants’ sensorial experience and how remote interaction affects the shared sense-making process in the hybrid format setting.

Adaptive Video Streaming in Multi-Tier Computing Networks: Joint Edge Transcoding and Client Enhancement

With the advancement of multimedia technology and wireless networks, there is a growing demand for high-quality video streaming. Delivering stable video streaming in extremely dynamic wireless networks, nevertheless, is still an open problem. Recent developments in client computing and mobile edge computing (MEC) technologies have both shown promise in enhancing the adaptive bitrate (ABR) streaming services. In this paper, we consider a video streaming system in multi-tier computing networks, enabled by joint edge-side video transcoding and client-side video enhancement. By “enhancement,” we mean that the client improves the video chunk quality via client-side image processing modules. In particular, we aim to design a joint bitrate adaptation, edge transcoding, and client image-processing algorithm, maximizing the quality of experience (QoE) of streaming services. The majority of the prior art has concentrated on super-resolution-enabled video streaming. Contrarily, we show that the video enhancement method outperforms the super-resolution approach in terms of signal-to-noise ratio and frames per second, implying a superior alternative for client-side processing in ABR streaming. We formulate the problem as an event-triggered Markov decision process (E-MDP), and propose a deep reinforcement learning (DRL)-based framework, named EDTEA. To deal with the delayed feedback induced by multi-tier computing, the entropy and the expected re-buffering terms are introduced to the objective and the reward, respectively. Extensive simulations based on real-world videos and bandwidth traces manifest that compared with state-of-the-art approaches, EDTEA provides <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$10.4\%\sim 78.4\%$</tex-math></inline-formula> extra QoE while reducing re-buffering time by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$85.5\%\sim 91.7\%$</tex-math></inline-formula> .

NERVE: Real-Time Neural Video Recovery and Enhancement on Mobile Devices

As mobile devices become increasingly popular for video streaming, it is crucial to optimize the streaming experience for these devices. Although deep learning-based video enhancement techniques are gaining attention, most of them cannot support real-time enhancement on mobile devices. Additionally, many of these techniques are focused solely on super-resolution and cannot handle partial or complete loss or corruption of video frames, which is common in the Internet and wireless networks. To overcome these challenges, we present NERVE, a novel approach in this paper. NERVE consists of (i) a novel video frame recovery scheme, (ii) a new super-resolution algorithm, and (iii) an enhancement-aware video bit rate adaptation algorithm. We implement NERVE on an iPhone 12, and it can support 30 frames per second (FPS). We evaluate NERVE in various networks such as 3G, 4G, 5G, and WiFi networks. Our evaluation shows that NERVE enables real-time video recovery and enhancement, and results in 24% - 83% increase in video Quality of Experience (QoE) in our video streaming system.

An automated ICU agitation monitoring system for video streaming using deep learning classification.

To address the challenge of assessing sedation status in critically ill patients in the intensive care unit (ICU), we aimed to develop a non-contact automatic classifier of agitation using artificial intelligence and deep learning. We collected the video recordings of ICU patients and cut them into 30-second (30-s) and 2-second (2-s) segments. All of the segments were annotated with the status of agitation as "Attention" and "Non-attention". After transforming the video segments into movement quantification, we constructed the models of agitation classifiers with Threshold, Random Forest, and LSTM and evaluated their performances. The video recording segmentation yielded 427 30-s and 6405 2-s segments from 61 patients for model construction. The LSTM model achieved remarkable accuracy (ACC 0.92, AUC 0.91), outperforming other methods. Our study proposes an advanced monitoring system combining LSTM and image processing to ensure mild patient sedation in ICU care. LSTM proves to be the optimal choice for accurate monitoring. Future efforts should prioritize expanding data collection and enhancing system integration for practical application.

MetaABR: A Meta-Learning Approach on Adaptative Bitrate Selection for Video Streaming

Video streaming is one of the most popular Internet applications that makes up a large amount of Internet traffic. A fundamental mechanism in video streaming is adaptive bitrate (ABR) selection which decides the proper compression level for each chunk of a video to optimize the users' quality of experience (QoE). The existing ABR algorithms require significant tuning and do not generalize to diverse network conditions and personalized QoE objectives. In this paper, we propose a novel framework for meta-learning based ABR design and discuss challenges of deploying learning based ABR mechanism in real-world video streaming systems. We utilize the proposed framework to design MetaABR, a novel adaptive bitrate selection algorithm based on meta-reinforcement learning to maximize users' QoE. By jointly training multiple learning tasks with a shared meta-critic, it can provide transferrable meta-knowledge to supervise bitrate selection across tasks, and can be applied to efficiently learn a new task in unseen environment with only a few trials. We implement MetaABR on an emulation platform which connects to the Linux network protocol stack through virtual network interfaces. Extensive experiments based on real-world traces and wireless testbed show that MetaABR achieves the best comprehensive QoE compared with the state-of-the-art ABR algorithms in a variety of network environments.

AMIS-MU: Edge Computing Based Adaptive Video Streaming for Multiple Mobile Users

The increasing demand for online high-quality video streaming has brought huge challenges to the traditional client-server video streaming systems due to the high feedback delay, rigorous bandwidth requirement, and the lack of a mechanism of centralized resource management between users. In this work, we propose AMIS-MU, an edge computing-based mobile video streaming system that optimizes the watching experience of users via playback adaptation and channel resource allocation. AMIS-MU fully explores the power of edge servers from three perspectives. First, by pre-caching videos from the cloud, AMIS-MU analyzes video contents at the edge, and achieves a nearly imperceptible content-based playback speed adaptation. Second, as the edge server controls the channel resources of users in a centralized fashion, AMIS-MU adaptively updates the channel configuration to optimize the overall watching experience. Last, the plenty of computational power available at the edge enables a more intelligent playback control by using deep reinforcement learning (DRL). We propose a novel usage of DRL which significantly reduces the complexity of the cross-layer joint optimization problem and solve the non-convex channel resource allocation problem by Lyapunov optimization. Experiments show that AMIS-MU outperforms other existing algorithms in terms of average QoE and fairness.

QoE Provisioning System for VoIP and Video Streaming Using Software-Defined Networking and IOTA Micropayment

QoE Provisioning System for VoIP and Video Streaming Using Software-Defined Networking and IOTA Micropayment

A Comparative Measurement Study of Point Cloud-Based Volumetric Video Codecs

Volumetric video is fully three-dimensional and immersive, which shows great potential in various application scenarios. However, the naive delivery of volumetric video over the Internet requires a huge amount of bandwidth and compute resources. A volumetric video should be compressed by a specific codec before transmission. In spite of the existence of a few pioneering efforts on prototype design, there is very limited work to investigate the efficiency and practicality of existing volumetric video codecs. In this paper, we conduct a comparative measurement study on five representative point cloud-based volumetric video codecs, including 3 conventional codecs (e.g., Draco, G-PCC, V-PCC) and 2 neural-based codecs (e.g., PU-GCN <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula> , MPU <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula> ), on six real volumetric video datasets. We investigate the applicability of the above codecs in different application scenarios, and examine how the features of point cloud (e.g., quality, texture, geometric and scene complexity) affect the quality of compressed volumetric video from multiple perspectives. In addition, we also study the impact of users’ viewing behaviors on the coding and delivery efficiency with Quality of Experience (QoE) metrics. Our results shed a number of important insights, which provide useful guidelines for optimizing the design of future volumetric video streaming systems.

A Framework for Multi-Task Learning in Dynamic Adaptive Streaming Over HTTP

This paper presents a framework with a taxonomy for multi-task learning in the context of Dynamic Adaptive Streaming over HTTP (DASH). DASH is a widely used technology for video streaming, and multi-task learning has emerged as a promising approach to enhance the performance and user experience of DASH systems by jointly optimizing multiple related tasks. The framework provides a structured approach to design, train, and evaluate multi-task learning models in DASH, while the taxonomy categorizes the key components and approaches within the framework. The taxonomy includes task types, multitask learning approaches, input features, and training strategies. Task types encompass video quality adaptation, buffer management, bandwidth estimation, content pre-fetching, and resource allocation, representing the specific tasks involved in DASH. Multi-task learning approaches encompass methodologies such as shared representation learning, task-specific layers, multi-head architectures, knowledge distillation, and reinforcement learning, offering flexibility in model design and optimization. Input features cover video characteristics, network conditions, device capabilities, and user preferences, providing the necessary information for informed decision-making across tasks. Training strategies include joint training, alternate training, hierarchical training, task weighting, and task balancing, determining how the multi-task learning model is trained and optimized. By following the presented framework and taxonomy, researchers and practitioners can systematically approach the design, training, and evaluation of multi-task learning models in DASH. The framework enables the development of efficient and adaptive video streaming systems by leveraging the interdependencies among tasks. The taxonomy helps organize the components and approaches within the framework, aiding in a better understanding of the various aspects of multi-task learning in the DASH context. Overall, this framework and taxonomy provide a valuable resource for advancing the field of multitask learning in the dynamic and complex domain of video streaming over HTTP.

A Taxonomy for Deep Learning in Dynamic Adaptive Video Streaming Over HTTP

Deep Learning (DL) has become a fundamental technology in the field of Dynamic Adaptive Video Streaming over HTTP (DASH), enabling significant advancements in video streaming systems. This taxonomy presents a novel framework for categorizing and organizing the diverse applications and methodologies of DL in DASH. The taxonomy encompasses various aspects of DL, including video representation, quality of experience (QoE) estimation, bitrate adaptation, buffer management, content- and context-aware adaptation, and network optimization. By providing a comprehensive overview of DL in DASH, this taxonomy serves as a valuable resource for researchers and practitioners, facilitating a better understanding of the different DL techniques and their applications in enhancing video streaming performance and user experience.

Research on the User Experience of Infinite Up-scrolling on Chinese Short Video Software Douyin

Douyin/TikTok is a popular short-form video application popular around the globe. The infinite up-scrolling video stream system that it is using is applying to more and more diversified applications. This research focused on the infinite up-scrolling video stream systems influence on the Chinese user group and the public reaction towards it. A survey was conducted, and 145 valid answers were received. The questionnaire focused on the publics daily interaction with the up-scrolling system. Through the statistics received, it is shown that most people like the feeling caused by up-swiping, and they easily use it over time. This phenomenon is explained from the perspective of psychology. The psychology analysis also explained why people often spend more time in this system than they planned or expected. Meanwhile, there is a large percentage of the public using the up-scrolling system in applications that are primarily for other usages. The system was adopted into these applications later on through program updates. With this system, applications are able to make their users more dependent on them. Therefore, this system is getting more and more popular in all kinds of applications.

JIRA: Joint Incentive Design and Resource Allocation for Edge-Based Real-Time Video Streaming Systems

Edge computing has been introduced as a promising technology for real-time video streaming systems. However, due to the lack of automatic incentives and the limitation of resources, traditional edge computing performs poorly in nowadays scenarios. To handle these two challenges, we propose a framework of Joint Incentive design and Resource Allocation (JIRA) for edge-based real-time video streaming systems. Technically, to ensure the trust and automatic distribution of incentives, we develop a novel smart contract based incentive mechanism and implement a prototype. Meanwhile, we propose an efficient online algorithm, i.e., JIRA, which dynamically adjusts compression ratio, offloading decision, and resource allocation to achieve performance optimization for video streaming under long-term latency and resource constraints. Specifically, JIRA is based on Lyapunov optimization, which decomposes the challenging long-term decision problem into a series of real-time optimization problems. Then we propose a multi-cut <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Generalized Benders Decomposition</i> based algorithm (MGA) to tackle the non-convexity of the decomposed problem. Through rigorous theoretical analysis, we prove the performance bound of JIRA. Extensive simulations demonstrate that the proposed schemes can achieve an efficient trade-off between accuracy performance and energy consumption.

Enhancing 360 Video Streaming through Salient Content in Head-Mounted Displays.

Predicting where users will look inside head-mounted displays (HMDs) and fetching only the relevant content is an effective approach for streaming bulky 360 videos over bandwidth-constrained networks. Despite previous efforts, anticipating users' fast and sudden head movements is still difficult because there is a lack of clear understanding of the unique visual attention in 360 videos that dictates the users' head movement in HMDs. This in turn reduces the effectiveness of streaming systems and degrades the users' Quality of Experience. To address this issue, we propose to extract salient cues unique in the 360 video content to capture the attentive behavior of HMD users. Empowered by the newly discovered saliency features, we devise a head-movement prediction algorithm to accurately predict users' head orientations in the near future. A 360 video streaming framework that takes full advantage of the head movement predictor is proposed to enhance the quality of delivered 360 videos. Practical trace-driven results show that the proposed saliency-based 360 video streaming system reduces the stall duration by 65% and the stall count by 46%, while saving 31% more bandwidth than state-of-the-art approaches.

Performability Evaluation and Sensitivity Analysis of a Video Streaming on Demand Architecture

In urban air mobility (UAM), video streaming platforms have gained significant attention from media companies due to their growing necessity for on-demand video streaming services-as-you-go in flights. Video streaming services can provide constant data transactions in a huge amount, especially in its operational digital twin (ODT). As a result, the ability to provide a satisfactory user experience through video streaming platforms is critical and complex. This requires continuously operating services while handling numerous user requests for near real-time video streaming. At the same time, high-quality video with high resolution and minimal interruptions is often expected. Therefore, the availability and performance (i.e., performability) of the Back-End video streaming infrastructure are crucial parameters for these platforms. However, evaluating novel video-on-demand architectures in real-world scenarios can be costly due to the numerous parameters involved. Analytical models, such as stochastic Petri nets (SPNs), can serve as an alternative in this complex scenario as they can be used to analyze systems during the design process. In this study, we developed a set of SPN models to assess the performance of a video-on-demand system. These models were designed to illustrate and to evaluate a video-on-demand architecture while considering performance. We had a base SPN model as well as three enhanced variants available. The extended models were generated using the Design of Experiments (DoE) technique and sensitivity analysis results. The DoE identified the factors with the greatest impacts on performance, and the most significant factor interactions. Redundancy strategies were applied to the extended models to increase the availability of the most important components. This redundancy increased the availability of “9 s” from three to five. It is worth noting that this study can help the designers of video streaming systems, to plan and to optimize their ideas based on the provided models.

Video coding deep learning-based modeling for long life video streaming over next network generation.

Availability is one of the primary goals of smart networks, especially, if the network is under heavy video streaming traffic. In this paper, we propose a deep learning based methodology to enhance availability of video streaming systems by developing a prediction model for video streaming quality, required power consumption, and required bandwidth based on video codec parameters. The H.264/AVC codec, which is one of the most popular codecs used in video steaming and conferencing communications, is chosen as a case study in this paper. We model the predicted consumed power, the predicted perceived video quality, and the predicted required bandwidth for the video codec based on video resolution and quantization parameters. We train, validate, and test the developed models through extensive experiments using several video contents. Results show that an accurate model can be built for the needed purpose and the video streaming quality, required power consumption, and required bandwidth can be predicted accurately which can be utilized to enhance network availability in a cooperative environment.

A Novel Hybrid Multikey Cryptography Technique for Video Communication

Online video streaming is becoming more widespread in people’s everyday entertainment routines. Protecting copyright and piracy has become a key concern in real-time video streaming systems. This research provides a revolutionary multi-key and hybrid cryptography approach to offer security. This work describes the software implementation of video encryption and decryption employing continuous systems based on the Elliptic Curve Cryptography approach as pseudorandom encryption key generators. This approach creates several keys to encrypt and decode small chunks of video files that are produced dynamically based on the video data. The suggested approach was implemented on the Android platform, where applications for sender and recipients had been created to enable streaming. The security and performance of the proposed system have been examined by implementing it on devices and streaming videos. The outcomes demonstrate superiority in terms of performance and security.

Multiple Description Coding for Best-Effort Delivery of Light Field Video Using GNN-Based Compression

In recent years, Light Field (LF) video has grabbed much attention as an emerging form of immersive media. LF collects, through a lens matrix, light information emanating in every direction, and obtains rich information about the scene, providing users with an immersive 6 Degrees of Freedom (DoF) experience. The visual content between different viewpoints is highly homogenized, suggesting the possibility of good compression and encoding. However, most fixed-structure LF coding schemes are difficult to adapt to the real-time requirements of different LF applications and best-effort network conditions causing packet loss. In this paper, we propose a dynamic adaptive LF video transmission scheme that can achieve high compression and yet provide near-distortion-free LF video when the network condition is stable. Additionally, for unstable network conditions a description scheduling algorithm is proposed, which can decode the LF video with the highest possible quality even if partial data cannot be received completely and/or timely. We achieve this by designing a Multiple Description Coding (MDC) based solution to transport the LF video compressed by a Graph Neural Network (GNN) model. Experimental results show that the scheduling algorithm can improve the quality of the decoding results by 3% to 15%. Compared with other similar schemes, our system greatly improves the reliability of the video streaming system against packet loss/error and supports heterogeneous receivers.