Video Streaming Research Articles

Mobility-aware proactive video caching based on asynchronous federated learning in mobile edge computing systems

With the rapid advance of the Internet of Things (IoT) and the surging user-centric smart devices, video services such as short videos, real-time video streaming, and Virtual Reality (VR) gaming are emerging. Traditional cloud-centric caching is no longer able to satisfy the low-latency needs of mobile user groups. Predicting the most popular content in advance and caching it on edge devices such as small base stations (SBS) is a promising solution to alleviate network congestion and supply better Quality of Service (QoS) for mobile users. However, frequently changing popular content, limited edge caching resources, privacy of user data, and user mobility pose significant challenges to edge video caching. To tackle these challenges, we propose an efficient asynchronous federated learning (AFL)-based mobility-aware proactive video caching scheme (AF-MPVC), which significantly improves the overall network performance while preventing the risk of user data leakage. First, we model the proactive video caching problem in mobile edge computing systems and consider the impact of user mobility on AFL, representing the problem as an autoencoder (AE) model loss minimization problem. Then, a proactive video caching algorithm is proposed based on a filtering model, which uses the hidden features obtained from the trained AE model to compute the user similarity to find the popular videos that are most likely to be accessed by users and combine them with the most frequently requested videos. Meanwhile, we take into account the short duration of high mobility users staying on the current SBS and cache some videos to neighboring edge servers to have more redundant space to store more less popular videos. Finally, experiments on three real-world datasets demonstrate that the presented caching scheme outperforms other baseline caching schemes with respect to cache hit rate and latency reduction.

Facial Feature-Based Attention Tracking System for Enhanced Online Learning Engagement

Recognizing and enhancing student engagement is crucial for improving learning outcomes, particularly in the context of online classes where monitoring can be challenging. Traditional methods of attendance tracking, such as calling out names, are impractical and susceptible to manipulation in the virtual environment. Students might appear 'online' without actively participating, and the absence of video feeds makes it difficult for teachers to verify attendance and attention. In order to realize a highly efficient and robust attendance management and engagement level prediction system for online learning, In the proposed System, the learner’s face is monitored by a video camera while attending a video lecture. Facial features were analyzed to predict reaction time (RT) to a task-irrelevant stimulus, which was assumed to be an index of the level of attention. Then apply a machine learning method, light Gradient Boosting Machine (LightGBM), to estimate RTs from facial features extracted as action units (AUs) corresponding to facial muscle movements by an open-source software (OpenFace). This project is to develops a user-friendly system integrated with private online learning and attendance recording system for teachers that can automatically record students ‘engagement state and attendance then generate attendance reports for online classrooms. It encompasses a novel design using the AI based FFCNN (Face Fiducial Convolution Neural Network) model to capture face biometric randomly from students’ video stream and record their attendance automatically. This integrated solution not only streamlines attendance management but also provides valuable insights into students' engagement levels through facial feature analysis.

The effect of differentiated video presentations of choral performances on aesthetic responses of undergraduate choral and instrumental ensemble musicians

Research examining affective response to media modalities including film, television, gaming, video streaming, and virtual reality is expanding. This study examined aesthetic responses to choral performances presented in two contrasting video formats (stationary and produced). Volunteer undergraduate students ( N = 94) enrolled in ensembles (choral, n = 45; instrumental, n = 49) indicated their aesthetic responses to choral performances of Ēriks Ešenvalds’ “Only in Sleep” by manipulating a Continuous Response Digital Interface (CRDI) dial. Participants viewed video performances of two ensembles, one of which was digitally altered to allow for use of the same audio content for both videos. CRDI data were used to create temporal line graphs depicting participant responses to each video and indicated similar shapes with more intense aesthetic response to the produced video. Statistical analysis of cumulative mean responses to each performance found a main effect of video format (stationary vs. produced) and a video format × presentation order interaction. Initial exposure to each video garnered a more intense response than that of the second video, and the produced video elicited greater magnitude than the stationary perspective video. Findings support further examination of video and other media, studying differentiated impact on aesthetic response to music performances.

Uses and Gratifications of Over-the-Top (OTT) Video Streaming Platforms in India: Predictors of User’s Affinity, Satisfaction and Cord-Cutting

Over-the-top (OTT) platforms have changed how people watch television due to technological affordance and interactivity. With the theoretical framework of the uses and gratifications theory ( N = 421), this study identified eight motivational factors for OTT viewing: convenient navigability, improved viewing experience, entertainment, relaxation, social interaction, companionship, voyeurism and information seeking. Some of the study’s prominent findings were: convenient navigability, improved viewing experience, entertainment and voyeurism were positive predictors of OTT affinity; Entertainment, voyeurism and information-seeking gratifications positively predicted satisfaction with the OTT, while convenient navigability negatively predicted satisfaction; Convenient navigability, improved viewing experience, entertainment and relaxation gratifications along with interactivity and viewing levels positively predicted the intention to cord-cut.

Intelligent Infrastructure for Traffic Monitoring Based on Deep Learning and Edge Computing

In the field of traffic management and control systems, we are witnessing a symbiotic evolution, where intelligent infrastructure is progressively collaborating with smart vehicles to produce benefits for traffic monitoring and security, by rapidly identifying hazardous behaviours. This exponential growth is due to the rapid development of deep learning in recent years, as well as the improvements in computer vision models. These technologies allow for monitoring tasks without the need to install numerous sensors or stop the traffic, using the extensive camera network of surveillance cameras already present in worldwide roads. This study proposes a computer vision-based solution that allows for real-time processing of video streams through edge computing devices, eliminating the need for Internet connectivity or dedicated sensors. The proposed system employs deep learning algorithms and vision techniques that perform vehicle detection, classification, tracking, speed estimation, and vehicle geolocation.

Sign Tone: Two Way Sign Language Recognition and Multilingual Interpreter System Using Deep Learning

In this paper, a comparative experimental assessment of computer vision-based methods for sign language reclog- nation is conducted. By implementing the most recent deep neural network methods in this field, a thorough evaluation on multiple publicly available datasets is performed. The aim of the present study is to provide insights on sign language recognition, focusing on mapping non-segmented video streams to glosses. For this task, two new sequence training criteria, known from the fields of speech and scene text recognition, are introduced. Furthermore, a plethora of pretraining schemes is thoroughly discussed. Finally, a new RGB+D dataset for the Greek sign language is created. To the best of our knowledge, this is the first sign language dataset where three annotation levels are provided (individual gloss, sentence and spoken language) for the same set of video captures

Instant interaction driven adaptive gaze control interface

Gaze estimation is long been recognised as having potential as the basis for human-computer interaction (HCI) systems, but usability and robustness of performance remain challenging . This work focuses on systems in which there is a live video stream showing enough of the subjects face to track eye movements and some means to infer gaze location from detected eye features. Currently, systems generally require some form of calibration or set-up procedure at the start of each user session. Here we explore some simple strategies for enabling gaze based HCI to operate immediately and robustly without any explicit set-up tasks. We explore different choices of coordinate origin for combining extracted features from multiple subjects and the replacement of subject specific calibration by system initiation based on prior models. Results show that referencing all extracted features to local coordinate origins determined by subject start position enables robust immediate operation. Combining this approach with an adaptive gaze estimation model using an interactive user interface enables continuous operation with the 75th percentile gaze errors of 0.7∘\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^\\circ $$\\end{document}, and maximum gaze errors of 1.7∘\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^\\circ $$\\end{document} during prospective testing. There constitute state-of-the-art results and have the potential to enable a new generation of reliable gaze based HCI systems.

A Deep Learning Framework for Monitoring Audience Engagement in Online Video Events

This paper introduces a deep learning methodology for analyzing audience engagement in online video events. The proposed deep learning framework consists of six layers and starts with keyframe extraction from the video stream and the participants’ face detection. Subsequently, the head pose and emotion per participant are estimated using the HopeNet and JAA-Net deep architectures. Complementary to video analysis, the audio signal is also processed using a neural network that follows the DenseNet-121 architecture. Its purpose is to detect events related to audience engagement, including speech, pauses, and applause. With the combined analysis of video and audio streams, the interest and attention of each participant are inferred more accurately. An experimental evaluation is performed on a newly generated dataset consisting of recordings from online video events, where the proposed framework achieves promising results. Concretely, the F1 scores were 79.21% for interest estimation according to pose, 65.38% for emotion estimation, and 80% for sound event detection. The proposed framework has applications in online educational events, where it can help tutors assess audience engagement and comprehension while hinting at points in their lectures that may require further clarification. It is effective for video streaming platforms that want to provide video recommendations to online users according to audience engagement.

A Closer Look into IPFS: Accessibility, Content, and Performance

The InterPlanetary File System (IPFS) has recently gained considerable attention. While prior research has focused on understanding its performance characterization and application support, it remains unclear: (1) what kind of files/content are stored in IPFS, (2) who are providing these files, (3) are these files always accessible, and (4) what affects the file access performance. To answer these questions, in this paper, we perform measurement and analysis on over 4 million files associated with CIDs (content IDs) that appeared in publicly available IPFS datasets. Our results reveal the following key findings: (1) Mixed file accessibility: while IPFS is not designed for a permanent storage, accessing a non-trivial portion of files, such as those of NFTs and video streams, often requires multiple retrieval attempts, potentially blocking NFT transactions and negatively affecting the user experience. (2) Dominance of NFT (non-fungible token) and video files: about 50% of stored files are NFT-related, followed by a large portion of video files, among which about half are pirated movies and adult content. (3) Centralization of content providers: a small number of peers (top-50), mostly cloud nodes hosted by tech companies, serve a large portion (95%) of files, deviating from IPFS's intended design goal. (4) High variation of downloading throughput and lookup time: large file retrievals experience lower average throughput due to more overhead for resolving file chunk CIDs, and looking up files hosted by non-cloud nodes takes longer. We hope that our findings can offer valuable insights for (1) IPFS application developers to take into consideration these characteristics when building applications on top of IPFS, and (2) IPFS system developers to improve IPFS and similar systems to be developed for Web3.

BONES: Near-Optimal Neural-Enhanced Video Streaming

Accessing high-quality video content can be challenging due to insufficient and unstable network bandwidth. Recent advances in neural enhancement have shown promising results in improving the quality of degraded videos through deep learning. Neural-Enhanced Streaming (NES) incorporates this new approach into video streaming, allowing users to download low-quality video segments and then enhance them to obtain high-quality content without violating the playback of the video stream. We introduce BONES, an NES control algorithm that jointly manages the network and computational resources to maximize the quality of experience (QoE) of the user. BONES formulates NES as a Lyapunov optimization problem and solves it in an online manner with near-optimal performance, making it the first NES algorithm to provide a theoretical performance guarantee. Comprehensive experimental results indicate that BONES increases QoE by 5% to 20% over state-of-the-art algorithms with minimal overhead. Our code is available at https://github.com/UMass-LIDS/bones.

Depression Detection System Using Facial Recognition Using Python

This paper presents a comprehensive mental health assessment toolkit implemented in Python, consisting of two distinct modules. The first module focuses on depression detection through a quiz, while the second module performs emotion detection using libraries such as DeepFace and OpenCV. The Depression Detection Module aims to provide a self-assessment tool for individuals who suspect they may be experiencing symptoms of depression. The emotion detection module uses advanced computer vision techniques and the DeepFace library to analyze facial expressions and recognize emotional states in real time via a webcam or recorded images. OpenCV is used to capture and process images or video streams, while DeepFace's deep learning models accurately classify emotions, including happiness, sadness, anger, fear, surprise, and neutrality. While the system shows promise in contributing to the field of mental health screening, it is essential to address ethical considerations related to user privacy and consent. Striking a balance between technological advancements and ethical guidelines ensures the responsible and effective deployment of such tools.

Violence Detection in Jails and Mental Asylums

This research investigates the use of Long- term Recurrent Convolutional Networks (LRCNs) for violence detection in video surveillance systems. LRCNs combine the strengths of Convolutional Neural Networks (CNNs) for capturing spatial information and Long Short-Term Memory (LSTM) networks for modeling temporal sequences. This combination allows the system to learn complex spatiotemporal patterns in video data, improving violence detection accuracy in environments like jails and mental health facilities. The project focuses on the integration of the LRCN model with a Telegram bot for real-time alerting and response. Upon detecting violent incidents in the video streams, the LRCN model triggers alerts through the Telegram bot, providing instant notifications to relevant authorities. The Telegram bot facilitates seamless communication and coordination among stakeholders, enabling swift action to mitigate potential risks and ensure the safety of occupants within these facilities. Through rigorous experimentation and evaluation, the effectiveness and reliability of the LRCN-based violence detection system integrated with the Telegram bot are demonstrated. The research contributes to advancing technology-driven solutions for proactive security measures in high-risk environments, fostering safer and more secure institutional settings

Detection of Human in Flame

This report explores the development of a real-time human-in-flames detection system using a YOLOv8 deep learning model. A carefully curated dataset of video frames depicting human-in-flames scenarios with corresponding bounding boxes was utilized to train the model. We explored various hyperparameter configurations and training techniques to optimize its performance. The model's effectiveness was evaluated on a separate validation dataset not used for training. Metrics like accuracy, precision, recall, and F1-score were employed to analyze the model's ability to generalize to unseen scenarios and identify people engulfed in flames within video streams. The findings of this project demonstrate the potential of YOLOv8 for human-in-flames detection, potentially contributing to improved fire safety systems and quicker response times in emergency situations.

Human Emotion Detection Using CNN and Transfer Learning

Facial emotion detection is a critical component in human-computer interaction, mental health assessment, and security systems. In this project, we propose a robust facial emotion detection system leveraging state-of-the-art deep learning techniques. Our system utilizes Convolutional Neural Networks (CNNs) to extract meaningful features from facial images and classify them into five distinct emotional categories: neutral, surprise, sad, happy, and angry. We conducted extensive experiments on a diverse dataset consisting of over 10,000 annotated facial images collected from various sources. Through data augmentation techniques such as rotation, translation, and flipping, we expanded the dataset to enhance model training. Additionally, we employed transfer learning by fine-tuning a pre-trained CNN model, ResNet50, on our dataset to leverage its learned features. This project presents a system for real-time emotion monitoring using computer vision techniques. The system utilizes the Haar Cascade Classifier for face detection in live webcam video streams. Furthermore, we evaluated the system's performance across different lighting conditions, poses, and facial occlusions to assess its robustness in real-world scenarios. Our results indicate that the system maintains consistent performance across diverse conditions, making it suitable for deployment in applications requiring real-time facial emotion recognition.

Multi-agent DRL-based Multipath Scheduling for Video Streaming with QUIC

The popularization of video streaming brings challenges in satisfying diverse Quality of Service (QoS) requirements. The multipath extension of the Quick UDP Internet Connection (QUIC) protocol, also called MPQUIC, has the potential to improve video streaming performance with multiple simultaneously transmitting paths. The multipath scheduler of MPQUIC determines how to distribute the packets onto different paths. However, while applying current multipath schedulers into MPQUIC, our experimental results show that they fail to adapt to various receive buffer sizes of different devices and comprehensive QoS requirements of video streaming. These problems are especially severe under heterogeneous and dynamic network environments. To tackle these problems, we propose MARS, a Multi-agent deep Reinforcement learning (MADRL)-based Multipath QUIC Scheduler, which is able to promptly adapt to dynamic network environments. It exploits the MADRL method to learn a neural network for each path and generate scheduling policy. Besides, it introduces a novel multi-objective reward function that takes out-of-order queue size and different QoS metrics into consideration to realize adaptive scheduling optimization. We implement MARS in an MPQUIC prototype and deploy in Dynamic Adaptive Streaming over HTTP system. Then, we compare it with the state-of-the-art multipath schedulers in both emulated and real-world networks. Experimental results show that MARS outperforms the other schedulers with better adaptive capability regarding the receive buffer sizes and QoS.

Laryngeal Cancer Screening During Flexible Video Laryngoscopy Using Large Computer Vision Models.

Develop an artificial intelligence assisted computer vision model to screen for laryngeal cancer during flexible laryngoscopy. Using laryngeal images and flexible laryngoscopy video recordings, we developed computer vision models to classify video frames for usability and cancer screening. A separate model segments any identified lesions on the frames. We used these computer vision models to construct a video stream annotation system. This system classifies findings from flexible laryngoscopy as "potentially malignant" or "probably benign" and segments any detected lesions. Additionally, the model provides a confidence level for each classification. The overall accuracy of the flexible laryngoscopy cancer screening model was 92%. For cancer screening, it achieved a sensitivity of 97.7% and a specificity of 76.9%. The segmentation model attained an average precision at a 0.50 intersection-over-union of 0.595. The confidence level for positive screening results can assist clinicians in counseling patients regarding the findings. Our model is highly sensitive and adequately specific for laryngeal cancer screening. Segmentation helps endoscopists identify and describe potential lesions. Further optimization is required to enable the model's deployment in clinical settings for real-time annotation during flexible laryngoscopy.

An Improved Fire Detection Approach Based On Yolo-v8 for Smart Cities

Systems for detecting fires are essential for preventing property damage and saving lives. defending people and property. Conventional techniques frequently depend on sensor-based strategies, which have limitations in intricate settings. In order to improve accuracy and efficiency, this study suggests an intelligent fire detection system that makes use of machine learning and computer vision techniques. The technology analyzes video streams in real time using deep learning algorithms to identify fire incidents based on visual patterns and attributes. Future research on fire detection systems will benefit from the information this study will provide for smoker and fire detection issues in both indoor and outdoor situations. The improved fire detection technique for smart cities that is based on the YOLOv8 algorithm is the smart fire detection system (SFDS), which uses deep learning to identify fire-specific properties in real-time. The SFDS strategy may be more cost-effective, reduce false alarms, and improve fire detection accuracy when compared to traditional methods. It can also be extended to find other intriguing aspects of smart cities, such as gas leakage or flooding. The proposed smart city framework consists of four primary levels: the application layer (i), cloud layer (iii), fog layer (ii), and internet of things layer (iv). The recommended technique uses fog, cloud computing, and the Internet of Things layer to collect and understand data in real time. This reduces the chance of damage to persons or property and enables faster reaction times. The SFDS demonstrated state- of-the-art performance in terms of precision and recall, with a high precision rate of 97.1% across all classes. Among the potential applications are intelligent security systems, forest fire monitoring, and public space fire safety management.

Study of the Correlation between Streaming Video Platform Content on Food Production Processes and the Behavioral Intentions of Generation Z.

With an estimated 2.4 million cases of foodborne illnesses recorded annually in the UK alone, food safety has become a paramount concern among stakeholders. Modern technology has positioned streaming platforms as pivotal conduits for disseminating information. Channels such as YouTube offer detailed recordings of the food production process, granting consumers extensive visibility of the food journey from farm to table. This increased transparency not only promotes vigilant monitoring of food safety practices but also solicits consumer feedback regarding the public exposure to food processing videos. Based on the Theory of Planned Behavior (TPB), this study augments its framework with constructs, such as perceived trust, perceived risk, community experience, and brand identity, to evaluate Taiwan's Generation Z consumer behavioral intentions. With 226 valid responses amassed, structural equation modeling facilitated elucidation of the relationships among the constructs. This analysis yielded three salient insights. First, Generation Z's engagement with food processing videos on streaming platforms is positively correlated with their subsequent purchasing behavior. Second, enriched community experience was correlated with strengthened brand identification. Third, both perceived trust and perceived risk had a constructive impact on behavioral intentions within Gen Z's demographic data. Based on these outcomes, food industry enterprises should proactively develop and bolster community experiential value, thereby encouraging streaming platform users to transform into brand consumers and advocates.

Adaptive Processing of Camera Video Stream with Limitations on the Network Data Transmission Bandwidth

Video surveillance systems, cameras, and video stream processing are actively used in many enterprises as a monitoring and control tool for regular and emergency situations, as well as staff activities. The application of intelligent algorithms allows tracking and minimizing operator errors, but these approaches are highly sensitive to the quality of the original video, presence of noise, and low resolution. On the other hand, such video surveillance systems may be limited by network bandwidth. Therefore, this work considers an adaptive video stream processing algorithm that ensures efficient operation of computer vision and object recognition methods while minimizing the amount of transmitted information within network bandwidth constraints. The proposed algorithm addresses the task of determining boundary conditions that ensure the functionality of object recognition algorithms with the least amount of video stream. Corresponding experimental studies were conducted to determine the minimum values of frame resolution and video bitrate. The algorithm was tested in organizing video surveillance at warehouse complexes. The obtained results can be used in developing decision support systems for enterprises in various industries requiring intelligent processing of large volumes of data.

Hadamard Error-Correcting Codes and Their Application in Digital Watermarking.

In communication technologies such as digital watermarking, wireless sensor networks (WSNs), and visual light communication (VLC), error-correcting codes are crucial. The Enhanced Hadamard Error-Correcting Code (EHC), which is based on 2D Hadamard Basis Images, is a novel error correction technique that is presented in this study. This technique is used to evaluate the effectiveness of the video watermarking scheme. Even with highly sophisticated embedding techniques, watermarks usually fail to resist such comprehensive attacks because of the extraordinarily high compression rate of approximately 1:200 that is frequently employed in video dissemination. It can only be used in conjunction with a sufficient error-correcting coding method. This study compares the efficacy of the well-known Reed-Solomon Code with this novel technique, the Enhanced Hadamard Error-Correcting Code (EHC), in maintaining watermarks in embedded videos. The main idea behind this newly created multidimensional Enhanced Hadamard Error-Correcting Code is to use a 1D Hadamard decoding approach on the 2D base pictures after they have been transformed into a collection of one-dimensional rows. Following that, the image is rebuilt, allowing for a more effective 2D decoding procedure. Using this technique, it is possible to exceed the theoretical error-correcting capacity threshold of ⌊dmin-12⌋ bits, where dmin is the Hamming distance. It may be possible to achieve better results by converting the 2D EHC into a 3D format. The new Enhanced Hadamard Code is used in a video watermarking coding scheme to show its viability and efficacy. The original video is broken down using a multi-level interframe wavelet transform during the video watermarking embedding process. Low-pass filtering is applied to the video stream in order to extract a certain frequency range. The watermark is subsequently incorporated using this filtered section. Either the Reed-Solomon Correcting Code or the Enhanced Hadamard Code is used to protect the watermarks. The experimental results show that EHC far outperforms the RS Code and is very resilient against severe MPEG compression.