Video Streaming Research Articles

ColorVideoVDP: A visual difference predictor for image, video and display distortions

ColorVideoVDP is a video and image quality metric that models spatial and temporal aspects of vision for both luminance and color. The metric is built on novel psychophysical models of chromatic spatiotemporal contrast sensitivity and cross-channel contrast masking. It accounts for the viewing conditions, geometric, and photometric characteristics of the display. It was trained to predict common video-streaming distortions (e.g., video compression, rescaling, and transmission errors) and also 8 new distortion types related to AR/VR displays (e.g., light source and waveguide non-uniformities). To address the latter application, we collected our novel XR-Display-Artifact-Video quality dataset (XR-DAVID), comprised of 336 distorted videos. Extensive testing on XR-DAVID, as well as several datasets from the literature, indicate a significant gain in prediction performance compared to existing metrics. ColorVideoVDP opens the doors to many novel applications that require the joint automated spatiotemporal assessment of luminance and color distortions, including video streaming, display specification, and design, visual comparison of results, and perceptually-guided quality optimization. The code for the metric can be found at https://github.com/gfxdisp/ColorVideoVDP.

Applying the extended UTAUT-2 to assess the factors contributing to consumers’ usage intention towards over-the-top video streaming platforms

The current study aims to examine consumers' over-the-top (OTT) video streaming platform usage intention and apply the extended unified theory of acceptance and use of technology version 2 (UTAUT-2) to develop a holistic evaluation model. Specifically, a decision-making trial and evaluation laboratory (DEMATEL) approach was utilised to discover the determinants of OTT video streaming service selection. This study comprehensively analyses data collected from 498 users of an OTT video streaming platform. The research rigorously examines and validates the proposed research questions by utilising multiple attribute causality modelling. The findings demonstrated that the primary factors for platform improvements were performance and effort expectancies, with facilitating conditions, personal innovativeness, hedonic motivations, and social influences significantly impacting consumers’ preferences. The findings provided insights into OTT platform providers to acquire more subscribers while improving respective competitiveness. The study discusses both theoretical and practical implications in depth.

Dynamic Bitrate Adjustment in Web-based Video Streaming Applications Using HTTP Live Streaming (HLS)

Dynamic Bitrate Adjustment in Web-based Video Streaming Applications Using HTTP Live Streaming (HLS)

An Interactive Multimedia Model for Developing QOE-Enhanced E-Learning Platforms

This paper provides an approach to providing QOE enhancement for a multifaceted learning platform that includes streamed media content. We demonstrate how the QOE of a multifaceted platform can be enhanced using the Lighthouse tool. The Lighthouse tool helped improve the page load speed by 75%. We also demonstrate how the playback quality of the video streaming content can be enhanced using CMAF-formatted DASH content. When compared to single-bit rate media, DASH-based media provided better video playback quality. The study also evaluates the learners` preferences for specific interactive features of a learning platform. The least preferred feature uniquely had static content. The results of this study provide empirical data that would help instructional designers make feature selection choices that`s guided by empirical data and improve the QOE of their instructional learning platforms.

Supporting Immersive Video Streaming via V2X Communication

With the rapid advancement of autonomous driving and network technologies, future vehicles will function as network nodes, facilitating information transmission. Concurrently, in-vehicle entertainment systems will undergo substantial enhancements. Beyond traditional broadcasting and video playback, future systems will integrate immersive applications featuring 360-degree views and six degrees of freedom (6DoF) capabilities. As autonomous driving technology matures, vehicle passengers will be able to engage in a broader range of entertainment activities while on the move. However, this evolution in video applications will significantly increase bandwidth demand for vehicular networks, potentially leading to bandwidth shortages in congested traffic areas. This paper presents a method for bandwidth allocation for vehicle video applications within the landscape of vehicle-to-everything (V2X) networks. By utilizing a millimeter-wave (mmWave), terahertz (THz) frequency band, and cell-free (CF) extremely large-scale multiple-input multiple-output (XL-MIMO) wireless communication technologies, we provide vehicle passengers with the necessary bandwidth resources. Additionally, we address bandwidth contention issues in congested road segments by incorporating communication methods tailored to the characteristics of vehicular environments. By classifying users and adjusting according to the unique requirements of various multimedia applications, we ensure that real-time applications receive adequate bandwidth. Simulation experiments validate the proposed method’s effectiveness in managing bandwidth allocation for in-vehicle video applications within V2X networks. It increases the available bandwidth during peak hours by 32%, demonstrating its ability to reduce network congestion and ensure smooth playback of various video application types.

Real Time Video Streaming Over Homogeneous Systems

Streaming videos from one device to another become more important due to the spread of mobile devices and social media applications. The process of sending video from one device to another have a set of challenges. The video must be delivered in time, otherwise the service quality will decrease. Packet drop and lip synchronization is another challenge that streaming video will encounter. Real Time Protocol (RTP) is a simple, lightweight protocol that used to perform video streaming with low burden. RTP uses User Datagram Protocol (UDP) in video transferring. This paper use streaming video through RTP over homogeneous systems and measure the Packet Loss Rate, Jitter, Out-of-Order Packets, Datagram Size, Packet Delivery Ratio, latency, and throughput. Four experiments were performed with (15, 30, 45, 60) seconds to check the effect on the previous measures. The results show that video jitter increases as streaming time increases, it also shows that latency and throughput doesn’t depend on the streaming period.

Hidden Semi-Markov Models for Semantic-Graph Language Modeling

Semantic communication is expected to play a critical role in reducing traffic load in future intelligent large-scale sensor networks. With advances in Machine Learning (ML) and Deep Learning (DL) techniques, design of semantically-aware systems has become feasible in recent years. This work focuses on improving the reliability of the semantic information represented in a graph-based language that has been recently proposed. Inaccuracies in the representation of the semantic information can arise due to multiple factors, such as algorithmic shortcomings or sensory errors, decreasing the performance of the semantic extractor. This study aims to model the temporal evolution of semantic information, represented using the graph language, to enhance its reliability. Each unique graph configuration is treated as a distinct state, leading to a Hidden Semi-Markov Model (HSMM) defined over the state space of the graph configurations. The HSMM formulation enables the integration of prior knowledge on the semantic signal into the graph sequences, enhancing the accuracy in identifying semantic innovations. Within the HSMM framework, algorithms designed for graph smoothing, semantic information fusion, and model learning are introduced. The efficacy of these algorithms in improving the reliability of the extracted semantic-graphs is demonstrated through simulations and video streams generated in the CARLA simulation environment.

Efficient Violence Recognition in Video Streams using ResDLCNN-GRU Attention Network

Detecting Violence in video streams is essential for public safety and security due to the rising frequency of violent incidents. Despite the extensive deployment of CCTV for surveillance, the available human monitoring resources still need to catch up with the need for vigilant supervision. This research presents a new lightweight model to address this gap by accurately identifying and categorizing violent behaviors in various scenarios, including CCTV footage. The proposed method leverages optical flow and RGB data to capture spatiotemporal features in the Violence data. Built on a Residual DLCNN architecture integrated with the Attention mechanism and GRU components, the model effectively handles high-dimensional video data, enhancing accuracy by prioritizing crucial frames containing violent and nonviolent instances. The proposed model's performance was validated on the Hockey Fights (HF), Movie Fights, and SCVD datasets, achieving impressive accuracies of 98.38%, 99.62%, and 90.57%, respectively. Here, we developed the Extended Automatic Violence Detection Dataset (EAVDD), featuring 1530 videos of violent scenes in movies, public spaces, social media, and sports. Testing the model with top fight scenes in rated movies yielded outstanding results. This research supports surveillance systems and advances short video analysis and understanding with applications in public safety, social media, sports, and law enforcement.

A Smart Visual Sensor for Smoke Detection Based on Deep Neural Networks.

The automatic detection of smoke by analyzing the video stream acquired by traditional surveillance cameras is becoming a more and more interesting problem for the scientific community thanks to the necessity to prevent fires at the very early stages. The adoption of a smart visual sensor, namely a computer vision algorithm running in real time, allows one to overcome the limitations of standard physical sensors. Nevertheless, this is a very challenging problem, due to the strong similarity of the smoke with other environmental elements like clouds, fog and dust. In addition to this challenge, data available for training deep neural networks is limited and not fully representative of real environments. Within this context, in this paper we propose a new method for smoke detection based on the combination of motion and appearance analysis with a modern convolutional neural network (CNN). Moreover, we propose a new dataset, called the MIVIA Smoke Detection Dataset (MIVIA-SDD), publicly available for research purposes; it consists of 129 videos covering about 28 h of recordings. The proposed hybrid method, trained and evaluated on the proposed dataset, demonstrated to be very effective by achieving a 94% smoke recognition rate and, at the same time, a substantially lower false positive rate if compared with fully deep learning-based approaches (14% vs. 100%). Therefore, the proposed combination of motion and appearance analysis with deep learning CNNs can be further investigated to improve the precision of fire detection approaches.

The influence of live video hosts' speech act on purchase behaviour

With the broad adoption of internet, China's e-commerce industry has grown vigorously. The influence mechanism between the host's speech act and the consumer's purchase behaviour deserves further exploration. Based on speech act theory, this study divided the speech acts into four categories: representative, expressive, commissive and directive. We established a model investing the effect of different speech acts on customer engagement and purchase behaviour. We recorded live streaming video and transferred the voice to the text. Our research shows that the representative act and the expressive act negatively affect purchase behaviour, whereas the commissive act has a positive impact on purchase behaviour. Furthermore, customer engagement mediates the negative impact of the directive act and the representative act on purchase behaviour. The conclusions can provide guidance for enhancing the speech skills of the hosts in live streaming.

Studying the Impact of Different TCP DoS Attacks on the Parameters of VoIP Streams

In today’s digital world, no one and nothing is safe from potential cyberattacks. There is also no 100% protection from such attacks. Therefore, it is advisable to carry out various studies related to the effects of the different cyberattacks on the performance of the specific devices under attack. In this work, a study was carried out to determine how individual TCP DoS attacks affect the parameters of VoIP (Voice over IP) voice and video streams. For the purpose of this work, a model of a simple IP network has been created using the GNS3 IP network-modeling platform. The VoIP platform used was Asterisk Free PBX. Tools from Kali Linux were used to implement the individual TCP DoS attacks; IP-network-monitoring tools and round-trip-delay-measurement tools were also used. The proposed study is applicable to multiple VoIP platforms wherein voice and video traffic are passed/processed by the VoIP server. From the obtained results, it was found that Asterisk Free PBX is very well secured against TCP DoS attacks, which do not affect the platform performance or the parameters of the voice and video streams. The values of the observed parameters, such as jitter, packet loss, round-trip delay, etc., are very far from the maximum allowable values. We also observed a low load on the CPU and RAM of the system during the whole study.

Energy and QoE Optimization for Mobile Video Streaming with Adaptive Brightness Scaling

Brightness scaling (BS) is an emerging and promising technique with outstanding energy efficiency on mobile video streaming. However, existing BS-based approaches totally neglect the inherent interaction effect between BS factor, video bitrate and environment context. Their combined impact on user’s visual perception in mobile scenario, leading to inharmonious between energy consumption and user’s quality of experience (QoE). In this paper, we propose PEO , a novel user- P erception-based video E xperience O ptimization for energy-constrained mobile video streaming, by jointly considering the inherent connection between a device’s state of motion, video quality and the resulting user-perceived quality. Specifically, by capturing the motion of the on-the-run device, PEO first infers the optimal bitrate and BS factor, therefore avoiding bitrate-inefficiency for energy saving while guaranteeing the user-perceived QoE. On that basis, we formulate the device motion-aware and user perception-aware video streaming as an optimization problem where we present an optimal algorithm to maximize the object function and adapt to user preference, and thus propose an online bitrate selection algorithm. Our evaluation (based on trace analysis and user study) shows that, compared with state-of-the-art techniques, PEO can raise the perceived quality by 23.8%-41.3% and save up to 25.2% energy consumption.

Face Recognition based Attendance System

In today’s era, facial recognition emerges as a pivotal tool, offering security, authentication, and identification benefits across various sectors. This paper introduces an automated attendance system that incorporates real-time face recognition technology. The system aims to overcome the limitations and inefficiencies associated with traditional manual attendance methods, particularly in educational institutions and workplace environments. Two approaches are outlined: one utilizing local servers and AWS cloud recognition API, and the other relying solely on AWS resources for processing. Both approaches entail constructing a database of individuals’ images and utilizing algorithms such as Haar-Cascade and Local Binary Pattern Histogram for face detection and recognition. By automating attendance through live video streams, the system mitigates issues like proxy attendance and inaccuracies. Technologies such as OpenCV, dlib, and MySQL are leveraged to streamline implementation. This integration of face recognition with cloud computing promises to revolutionize attendance tracking, offering scalability, efficiency and accountability.

Advancements in Real-Time Human Activity Recognition via Innovative Fusion of 3DCNN and ConvLSTM Models

Object detection (OD) is a computer vision procedure for locating objects in digital images. Our study examines the crucial need for robust OD algorithms in human activity recognition, a vital domain spanning human-computer interaction, sports analysis, and surveillance. Nowadays, three-dimensional convolutional neural networks (3DCNNs) are a standard method for recognizing human activity. Utilizing recent advances in Deep Learning (DL), we present a novel framework designed to create a fusion model that enhances conventional methods at integrates three-dimensional convolutional neural networks (3DCNNs) with Convolutional Long-Short-Term Memory (ConvLSTM) layers. Our proposed model focuses on utilizing the spatiotemporal features innately present in video streams. An important aspect often missed in existing OD methods. We assess the efficacy of our proposed architecture employing the UCF-50 dataset, which is well-known for its different range of human activities. In addition to designing a novel deep-learning architecture, we used data augmentation techniques that expand the dataset, improve model robustness, reduce overfitting, extend dataset size, and enhance performance on imbalanced data. The proposed model demonstrated outstanding performance through comprehensive experimentation, achieving an impressive accuracy of 98.11% in classifying human activity. Furthermore, when benchmarked against state-of-the-art methods, our system provides adequate accuracy and class average for 50 activity categories.

Selfredepth

Depth maps produced by consumer-grade sensors suffer from inaccurate measurements and missing data from either system or scene-specific sources. Data-driven denoising algorithms can mitigate such problems; however, they require vast amounts of ground truth depth data. Recent research has tackled this limitation using self-supervised learning techniques, but it requires multiple RGB-D sensors. Moreover, most existing approaches focus on denoising single isolated depth maps or specific subjects of interest highlighting a need for methods that can effectively denoise depth maps in real-time dynamic environments. This paper extends state-of-the-art approaches for depth-denoising commodity depth devices, proposing SelfReDepth, a self-supervised deep learning technique for depth restoration, via denoising and hole-filling by inpainting of full-depth maps captured with RGB-D sensors. The algorithm targets depth data in video streams, utilizing multiple sequential depth frames coupled with color data to achieve high-quality depth videos with temporal coherence. Finally, SelfReDepth is designed to be compatible with various RGB-D sensors and usable in real-time scenarios as a pre-processing step before applying other depth-dependent algorithms. Our results demonstrate our approach’s real-time performance on real-world datasets shows that it outperforms state-of-the-art methods in denoising and restoration performance at over 30 fps on Commercial Depth Cameras, with potential benefits for augmented and mixed-reality applications.

Semantic guidance incremental network for efficiency video super-resolution

AbstractIn video streaming, bandwidth constraints significantly affect client-side video quality. Addressing this, deep neural networks offer a promising avenue for implementing video super-resolution (VSR) at the user end, leveraging advancements in modern hardware, including mobile devices. The principal challenge in VSR is the computational intensity involved in processing temporal/spatial video data. Conventional methods, uniformly processing entire scenes, often result in inefficient resource allocation. This is evident in the over-processing of simpler regions and insufficient attention to complex regions, leading to edge artifacts in merged regions. Our innovative approach employs semantic segmentation and spatial frequency-based categorization to divide each video frame into regions of varying complexity: simple, medium, and complex. These are then processed through an efficient incremental model, optimizing computational resources. A key innovation is the sparse temporal/spatial feature transformation layer, which mitigates edge artifacts and ensures seamless integration of regional features, enhancing the naturalness of the super-resolution outcome. Experimental results demonstrate that our method significantly boosts VSR efficiency while maintaining effectiveness. This marks a notable advancement in streaming video technology, optimizing video quality with reduced computational demands. This approach, featuring semantic segmentation, spatial frequency analysis, and an incremental network structure, represents a substantial improvement over traditional VSR methodologies, addressing the core challenges of efficiency and quality in high-resolution video streaming.

Building a searchable online corpus of Australian and New Zealand aligned speech

ABSTRACT Advances in automatic speech recognition technology, increases in bandwidth availability, and the widespread use of video streaming and sharing platforms have opened new horizons for corpus phonetics. CoANZSE Audio, a searchable online version of the Corpus of Australian and New Zealand Spoken English, provides access to over 195 million words of transcribed speech from transcripts of videos uploaded to YouTube by councils and other local government entities in Australia and New Zealand. Audio and forced alignment files are also available, making the resource suitable for the investigation of a range of research questions pertaining to morphosyntax, phonetics, and discourse. The resource, which is freely available via login through CLARIN, Europe’s main language resources infrastructure network, was created through the use of open-source tools and software: yt-dlp, a Python library for collecting data from video and streaming websites; the Montreal Forced Aligner, a recent neural network alignment suite; and Parselmouth-Praat, Python bindings for the Praat acoustic analysis software. The website is powered by BlackLab, which combines a powerful search engine based on Apache Lucene with an intuitive web frontend. CoANZSE Audio may be useful for the investigation of regional differentiation of language features, and with additional annotation, differences in feature use according to social or demographic groups. Recent applications have included studies of double modals, a rare syntactic feature, and apology sequences. The nature of the audio and alignment data may make the resource especially suitable for the study of regional phonetic variation. Furthermore, the methods used to create the resource may be of interest to researchers seeking to adopt a pipeline approach for the creation of specialized corpora from publicly available online content.

The rise and fall of political comedy on late night TV

ABSTRACT The emergence and demise of political comedy – a form of political humor distinct from political satire – on late night television, can be understood as a commercial response to the media landscape of the time. During the “golden era” of broadcast television, three networks dominated the television landscape and crafted their content to appeal to a mass national audience. This meant avoiding overtly political programming and messaging that favored either side of the political spectrum. But by the early 2000s, the widespread penetration of cable and satellite television, and later, the advent of the Internet and television streaming services meant that content providers could no longer hope to appeal to mass audiences. Thus, they began to target their programming toward smaller, more specific audiences. In this environment, political satire could thrive. In addition, the inclinations of late night talk show hosts and particular political circumstances helped push political comedy off the late night airwaves. Political satire now rules the late night television political humor landscape and will likely do so in the future.

Learned fractional downsampling network for adaptive video streaming

Given increasing demand for very large format contents and displays, spatial resolution changes have become an important part of video streaming. In particular, video downscaling is a key ingredient that streaming providers implement in their encoding pipeline as part of video quality optimization workflows. Here, we propose a downsampling network architecture that progressively reconstructs residuals at different scales. Since the layers of convolutional neural networks (CNNs) can only be used to alter the resolutions of their inputs by integer scale factors, we seek new ways to achieve fractional scaling, which is crucial in many video processing applications. More concretely, we utilize an alternative building block, formulated as a conventional convolutional layer followed by a differentiable resizer. To validate the efficacy of our proposed downsampling network, we integrated it into a modern video encoding system for adaptive streaming. We extensively evaluated our method using a variety of different video codecs and upsampling algorithms to show its generality. The experimental results show that improvements in coding efficiency over the conventional Lanczos algorithm and state-of-the-art methods are attained, in terms of PSNR, SSIM, and VMAF, when tested on high-resolution test videos. In addition to quantitative experiments, we also carried out a subjective quality study, validating that the proposed downsampling model yields favorable results.

Joint Optimization for Quality Selection and Resource Allocation of Live Video Streaming in Internet of Vehicles

Joint Optimization for Quality Selection and Resource Allocation of Live Video Streaming in Internet of Vehicles