Video Streaming Research Articles

Predictive and Near-Optimal Sampling for View Materialization in Video Databases

Scalable video query optimization has re-emerged as an attractive research topic in recent years. The OTIF system, a video database with cutting-edge efficiency, has introduced a new paradigm of utilizing view materialization to facilitate online query processing. Specifically, it stores the results of multi-object tracking queries to answer common video queries with sub-second latency. However, the cost associated with view materialization in OTIF is prohibitively high for supporting large-scale video streams. In this paper, we study efficient MOT-based view materialization in video databases. We first conduct a theoretical analysis and establish two types of optimality measures that serve as lower bounds for video frame sampling. In order to minimize the number of processed video frames, we propose a novel predictive sampling framework, namely LEAP, exhibits near-optimal sampling performance. Its efficacy relies on a data-driven motion manager that enables accurate trajectory prediction, a compact object detection model via knowledge distillation, and a robust cross-frame associator to connect moving objects in two frames with a large time gap. Extensive experiments are conducted in 7 real datasets, with 7 baselines and a comprehensive query set, including selection, aggregation and top-k queries. The results show that with comparable query accuracy to OTIF, our LEAP can reduce the number of processed video frames by up to 9× and achieve 5× speedup in query processing time. Moreover, LEAP demonstrates impressive throughput when handling large-scale video streams, as it leverages a single NVIDIA RTX 3090ti GPU to support real-time MOT-based view materialization from 160 video streams simultaneously.

An Optimized Link State Routing Protocol with a Blockchain Framework for Efficient Video-Packet Transmission and Security over Mobile Ad-Hoc Networks

A mobile ad-hoc network (MANET) necessitates appropriate routing techniques to enable optimal data transfer. The selection of appropriate routing protocols while utilizing the default settings is required to solve the existing problems. To enable effective video streaming in MANETs, this study proposes a novel optimized link state routing (OLSR) protocol that incorporates a deep-learning model. Initially, the input videos are collected from the Kaggle dataset. Then, the black-hole node is detected using a novel twin-attention-based dense convolutional bidirectional gated network (SA_ DCBiGNet) model. Next, the neighboring nodes are analyzed using trust values, and routing is performed using the extended osprey-aided optimized link state routing protocol (EO_OLSRP) technique. Similarly, the extended osprey optimization algorithm (EOOA) selects the optimal feature based on parameters such as node stability and link stability. Finally, blockchain storage is included to improve the security of MANET data using interplanetary file system (IPFS) technology. Additionally, the proposed blockchain system is validated utilizing a consensus technique based on delegated proof-of-stake (DPoS). The proposed method utilizes Python and it is evaluated using data acquired from various mobile simulator models accompanied by the NS3 simulator. The proposed model performs better with a packet-delivery ratio (PDR) of 91.6%, average end delay (AED) of 23.6 s, and throughput of 2110 bytes when compared with the existing methods which have a PDR of 89.1%, AED of 22 s, and throughput of 1780 bytes, respectively.

Cloud media video encoding: review and challenges

AbstractIn recent years, Internet traffic patterns have been changing. Most of the traffic demand by end users is multimedia, in particular, video streaming accounts for over 53%. This demand has led to improved network infrastructures and computing architectures to meet the challenges of delivering these multimedia services while maintaining an adequate quality of experience. Focusing on the preparation and adequacy of multimedia content for broadcasting, Cloud and Edge Computing infrastructures have been and will be crucial to offer high and ultra-high definition multimedia content in live, real-time, or video-on-demand scenarios. For these reasons, this review paper presents a detailed study of research papers related to encoding and transcoding techniques in cloud computing environments. It begins by discussing the evolution of streaming and the importance of the encoding process, with a focus on the latest streaming methods and codecs. Then, it examines the role of cloud systems in multimedia environments and provides details on the cloud infrastructure for media scenarios. After doing a systematic literature review, we have been able to find 49 valid papers that meet the requirements specified in the research questions. Each paper has been analyzed and classified according to several criteria, besides to inspect their relevance. To conclude this review, we have identified and elaborated on several challenges and open research issues associated with the development of video codecs optimized for diverse factors within both cloud and edge architectures. Additionally, we have discussed emerging challenges in designing new cloud/edge architectures aimed at more efficient delivery of media traffic. This involves investigating ways to improve the overall performance, reliability, and resource utilization of architectures that support the transmission of multimedia content over both cloud and edge computing environments ensuring a good quality of experience for the final user.

A video compression-cum-classification network for classification from compressed video streams

A video compression-cum-classification network for classification from compressed video streams

A Real-Time Deep Learning-Based Facial Mask Detection System for Preventing the Transmission of Respiratory Viruses

In order to prevent the spread of respiratory viruses like COVID-19 and influenza, an effective protection method is to wear facial masks in densely populated areas. This has led to a growing need for smart services that automatically detect facial masks and replace manual reminding. To address this challenging task and to contribute towards health safety, this paper introduces MedNetV2 system, which is an efficient deep learning-based facial mask detector with a low computational cost. In comparison with existing systems, the main specificities of the proposed system are: (1) the adoption of an effective deep learning-based framework to deal with both the large scale diversity and position variations of masked faces involved in natural scenes, (2) the interaction between face localization and facial mask detection modules to achieve the overall system goal, (3) the lightweight design and the real-time response well suited for real-world scenarios. Extensive experiments on public dataset and real-world video streaming are carried out to validate quantitatively and visually the effectiveness of the proposed system. Promising results, in terms of detection accuracy as well as time response, are achieved when compared it with other state-of-the-art systems.

Streamlined Workforce Management System Using AI

In today's digital landscape, face recognition technology has emerged as a versatile and non-invasive tool used across sectors, despite its lower accuracy compared to iris or fingerprint recognition. Its applications range from security and authentication to optimizing attendance systems in educational institutions and workplaces. This paper introduces an innovative class attendance system employing face recognition technology, replacing manual processes prone to inaccuracies like proxy attendance. The system involves database creation from student images, utilizing Haar-Cascade classifiers for face detection, and Local Binary Pattern Histogram algorithms for live recognition from classroom video streams. It automates attendance data distribution to faculty members, enhancing efficiency and precision. Simultaneously, the security industry necessitates alert and vigilant security personnel to prevent breaches. A sophisticated hybrid system, leveraging machine learning techniques like OpenCV and EAR/MAR algorithms alongside UHD cameras, detects guard drowsiness and alerts them promptly using sound cues and preventive measures, ensuring continual vigilance. This innovative hybrid system fortifies security guard surveillance, mitigating potential vulnerabilities due to lapses in attentiveness. Key Word: Face Recognition Technology, Biometrics, Attendance Management, Security, Authentication, Surveillance, Machine Learning, Open CV, Security Guard Monitoring, Vigilance, Drowsiness Detection.

Visual Communication Design Method in Folk Art Based on Multimedia Data Transmission Technology

With the rapid development of science and technology and China's economy, the internet, big data, computers, and multimedia technology are widely used in all walks of life to promote the application and improvement of visual communication design concepts, the gradual implementation of multimedia technology, and the continuous improvement of innovation. In this paper, from the multimedia transmission technology based on network coding, some key technologies in the network communication of data and video streams are studied in depth and, based on the transmission-quality assessment model, the working effect of the jitter-buffering algorithm and WebRTC jitter-buffering algorithm is compared in different network environments. The experimental results show that the jitter-buffering algorithm proposed in this paper has better working effect. This research is of great significance for the realization of multimedia transmission technology in next-generation networks.

GreenABR+: Generalized Energy-Aware Adaptive Bitrate Streaming

Adaptive bitrate (ABR) algorithms play a critical role in video streaming by making optimal bitrate decisions in dynamically changing network conditions to provide a high quality of experience (QoE) for users. However, most existing ABRs suffer from limitations such as predefined rules and incorrect assumptions about streaming parameters. They often prioritize higher bitrates and ignore the corresponding energy footprint, resulting in increased energy consumption, especially for mobile device users. Additionally, most ABR algorithms do not consider perceived quality, leading to suboptimal user experience. This paper proposes a novel ABR scheme called GreenABR+, which utilizes deep reinforcement learning to optimize energy consumption during video streaming while maintaining high user QoE. Unlike existing rule-based ABR algorithms, GreenABR+ makes no assumptions about video settings or the streaming environment. GreenABR+ model works on different video representation sets and can adapt to dynamically changing conditions in a wide range of network scenarios. Our experiments demonstrate that GreenABR+ outperforms state-of-the-art ABR algorithms by saving up to 57% in streaming energy consumption and 57% in data consumption while providing up to 25% more perceptual QoE due to up to 87% less rebuffering time and near-zero capacity violations. The generalization and dynamic adaptability make GreenABR+ a flexible solution for energy-efficient ABR optimization.

Face Identification And Blurring the Face Using Deep Learning Based Approaches In Videos

In an era where privacy concerns and ethical considerations dominate technological advancements, our study presents a pioneering solution at the intersection of facial recognition and privacy preservation in video streams. By amalgamating sophisticated facial recognition algorithms from the face recognition library [1] with Gaussian blur techniques, our system redefines the landscape of real-time face recognition. Key to our approach is the judicious application of blur effects, selectively safeguarding the identities of individuals while maintaining the integrity of facial recognition processes. Through meticulous encoding and storage of known faces [2], our system seamlessly identifies familiar individuals within video data. Leveraging facial recognition capabilities [3], it swiftly discerns between known and unknown faces, ensuring that only unidentified individuals are subject to the privacy-enhancing blur treatment.

AP-assisted adaptive video streaming in wireless networks with high-density clients

Adaptive video streaming over wireless networks has experienced tremendous growth in past few years. In order to guarantee users’ quality of experience (QoE), adaptive bitrate (ABR) algorithms have been extensively studied. With the recent emergency of high-density Wi-Fi networks, these solutions no longer perform well. On the one hand, bitrate decisions respond slowly to high network fluctuations, and on the other hand, wireless channel resources are insufficient especially under the cases that multiple clients compete for the limited bandwidth. Hence, users’ personalized QoE metrics need to be taken into account to cope with the QoE degradation. To this end, we design an access point (AP) assisted wireless dynamic adaptive video streaming over Hypertext Transfer Protocol (HTTP) solution called Wi-DASH, which aims to improve users’ QoE while considering channel utilization. In Wi-DASH, the video server aggregates real-time network status with the assistance of AP, and estimates clients’ player status through statistical analysis of chunk request logs. On this basis, a deep reinforcement learning (DRL) based ABR algorithm is designed, where the DRL model can deal with the complicated global status information including network status, player status, and QoE preferences. Finally, we implement Wi-DASH system, and conduct experiments with 4K resolution videos. Experimental results reveal that the Wi-DASH can more fully utilize wireless channel resources, and significantly improve users’ QoE.

AN EFFCIENT SYSTEM FOR DETECTING TRAFFIC VIOLATIONS SUCH AS OVER SPEED, DISREGARDING SIGNALS, AND INSTANCES OF TRIPLE RIDING

In recent time surveys, the deaths and injuries due to traffic violations have increased chiefly in Indian roads. So, this needed the assistance of an automated computer vision-based object detection model, as manually identifying the vehicles violating traffic is hectic. The principle of this paper is to detect multiple violations using single video frames. The input video stream obtained from the surveillance camera is processed and annotated to carry out multiple processes. The dataset used for red-light jumping is COCO and the dataset for over boarding is created by annotating the images obtained from google. The model is trained, and the output is visualized using tensor board. The accuracy for red light skipping is 93% and the mAP value for over boarding is 0.5:0.95. This system utilizes the video stream at its maximum to detect various violations.

Player Experience of Minimalist Video Game Design: Case Study of Indie Horror Iron Lung

Player experiences of minimalist video games are not well documented. This article addresses this gap by exploring players’ experiences of the minimalist horror video game Iron Lung. This game is identified as employing unique methods of gameplay toward affecting players. A case study design was framed using Caroux et al.’s player–video game interactions and Nealen et al.’s minimalist video game design characteristics and observed through affect theory to consider how bodies (video games) affect other bodies (players). A selection of player reviews was thematically analyzed, and three video streams were observed and analyzed. Results demonstrate that most players were affected as the developer intended using obfuscation, deliberately clumsy mechanics, and unique gameplay pacing. This article contributes to the field of minimalist design by filling the gap of player experiences of minimalist game design. This is useful to video game designers, minimalist designers, interactive experience designers, scenographers, or environmental design specialists.

The effects of COVID‐19 lockdown on the body weight and lifestyle behaviors of U.S. adults

AbstractThe COVID‐19 pandemic and associated policies may have impacted body weight and related lifestyle behaviors. Using 2017–2020 Behavioral Risk Factor Surveillance System data, we estimate the effects of COVID‐19 lockdown on Body Mass Index (BMI) and the probability of obesity among adults in the United States. Our triple‐differences approach, using recent pre‐pandemic years as counterfactuals, identifies statistically significant lockdown‐induced increases in body weight. Event study results highlight a lagged effect of lockdown on body weight, with BMI increases manifesting 2 months post‐lockdown. These effects are particularly pronounced among non‐white adults, low‐income individuals, females, and metropolitan residents. The lockdown's impact on mental health, a potential mechanism driving BMI increases, is concentrated among non‐white, low‐income adults. We observe potential post‐lockdown changes in diet and sedentary behaviors, evidenced by increased Google searches for high‐calorie foods, food delivery apps, and video streaming. Google searches related to healthy habits, such as exercise, yield mixed results.

MetaABR: A Meta-Learning Approach on Adaptative Bitrate Selection for Video Streaming

Video streaming is one of the most popular Internet applications that makes up a large amount of Internet traffic. A fundamental mechanism in video streaming is adaptive bitrate (ABR) selection which decides the proper compression level for each chunk of a video to optimize the users' quality of experience (QoE). The existing ABR algorithms require significant tuning and do not generalize to diverse network conditions and personalized QoE objectives. In this paper, we propose a novel framework for meta-learning based ABR design and discuss challenges of deploying learning based ABR mechanism in real-world video streaming systems. We utilize the proposed framework to design MetaABR, a novel adaptive bitrate selection algorithm based on meta-reinforcement learning to maximize users' QoE. By jointly training multiple learning tasks with a shared meta-critic, it can provide transferrable meta-knowledge to supervise bitrate selection across tasks, and can be applied to efficiently learn a new task in unseen environment with only a few trials. We implement MetaABR on an emulation platform which connects to the Linux network protocol stack through virtual network interfaces. Extensive experiments based on real-world traces and wireless testbed show that MetaABR achieves the best comprehensive QoE compared with the state-of-the-art ABR algorithms in a variety of network environments.

Video Anomaly Detection in Crime Analysis using Deep learning Architecture- A survey

The importance of developing automated video surveillance systems for public safety and security, particularly in crime analysis, has witnessed significant growth in recent years. This survey delves into the current landscape of automated video surveillance systems, emphasizing advancements in crime analysis and exploring existing methodologies and technologies. The study underscores the significance of employing deep learning models in video analysis. Furthermore, the study suggests a deep learning architecture to address the challenges of the existing methods. The goal of the suggested approach is to help security and law enforcement organizations quickly react to any dangers by precisely identifying unusual occurrences or actions in video sequences. The DenseNet-121 architecture is used for efficient spatial and temporal data acquisition from the video frames. This architecture is characterized by a dense connection structure in which all levels get feature mappings from all layers before them. The characteristics of DenseNet-121 can help in the accurate identification of anomalies in video streams and differentiate between normal and abnormal actions. In addition, the study also delves into the topic of using a cell structure with varied sizes to effectively split video sequences. This allows for flexible analysis and can accommodate different sorts of abnormalities. Anomaly detection accuracy can be further improved by adding size, motion, and location information to prediction and measurement models. This study serves as a foundation for the future research that aims to develop a more robust and efficient automated video surveillance solutions.

Automating the Human Action of First-Trimester Biometry Measurement from Real-World Freehand Ultrasound

Objective: Automated medical image analysis solutions should closely mimic complete human actions to be useful in clinical practice. However, more often an automated image analysis solution only represents part of a human task which restricts its practical utility. In the case of ultrasound-based fetal biometry, an automated solution should ideally recognize key fetal structures in free-hand video guidance, select a standard plane from a video stream and perform biometry. A complete automated solution should automate all three sub-actions.Methods: This paper considers how to automate the complete human action of first-trimester biometry measurement from real-world freehand ultrasound. In the proposed hybrid Convolutional Neural Network (CNN) architecture design, a classification-regression-based guidance model detects and tracks fetal anatomical structures (using visual cues) in the ultrasound video. Several high-quality standard planes which contain the mid-sagittal view of the fetus are sampled at multiple timestamps (using a custom-designed confident-frame detector) based on the estimated probability values associated with predicted anatomical structures that define the biometry plane. Automated semantic segmentation is performed on the selected frames to extract fetal anatomical landmarks. A crown-rump length (CRL) estimate is calculated as the mean CRL from these multiple frames.Results: Our fully automated method shows a high correlation with clinical expert CRL measurement (Pearson ρ=0.92, R-Squared (R2)=0.84) and a low mean absolute error of 0.834 (weeks) for fetal age estimation on a test data set of 42 videos.Conclusion: A novel algorithm for standard plane detection employs a quality detection mechanism defined by clinical standards and ensuring precise biometric measurements.

Shots segmentation-based optimized dual-stream framework for robust human activity recognition in surveillance video

Nowadays, for controlling crime, surveillance cameras are typically installed in all public places to ensure urban safety and security. However, automating Human Activity Recognition (HAR) using computer vision techniques faces several challenges such as lowlighting, complex spatiotemporal features, clutter backgrounds, and inefficient utilization of surveillance system resources. Existing attempts in HAR designed straightforward networks by analyzing either spatial or motion patterns resulting in limited performance while the dual streams methods are entirely based on Convolutional Neural Networks (CNN) that are inadequate to learning the long-range temporal information for HAR. To overcome the above-mentioned challenges, this paper proposes an optimized dual stream framework for HAR which mainly consists of three steps. First, a shots segmentation module is introduced in the proposed framework to efficiently utilize the surveillance system resources by enhancing the lowlight video stream and then it detects salient video frames that consist of human. This module is trained on our own challenging Lowlight Human Surveillance Dataset (LHSD) which consists of both normal and different levels of lowlighting data to recognize humans in complex uncertain environments. Next, to learn HAR from both contextual and motion information, a dual stream approach is used in the feature extraction. In the first stream, it freezes the learned weights of the backbone Vision Transformer (ViT) B-16 model to select the discriminative contextual information. In the second stream, ViT features are then fused with the intermediate encoder layers of FlowNet2 model for optical flow to extract a robust motion feature vector. Finally, a two stream Parallel Bidirectional Long Short-Term Memory (PBiLSTM) is proposed for sequence learning to capture the global semantics of activities, followed by Dual Stream Multi-Head Attention (DSMHA) with a late fusion strategy to optimize the huge features vector for accurate HAR. To assess the strength of the proposed framework, extensive empirical results are conducted on real-world surveillance scenarios and various benchmark HAR datasets that achieve 78.6285%, 96.0151%, and 98.875% accuracies on HMDB51, UCF101, and YouTube Action, respectively. Our results show that the proposed strategy outperforms State-of-the-Art (SOTA) methods. The proposed framework gives superior performance in HAR, providing accurate and reliable recognition of human activities in surveillance systems.

Technical and functional design considerations for a real-world interpretable AI solution for NIR perfusion analysis (including cancer)

Near infrared (NIR) analysis of tissue perfusion via indocyanine green fluorescence assessment is performed clinically during surgery for a range of indications. Its usefulness can potentially be further enhanced through the application of interpretable artificial intelligence (AI) methods to improve dynamic interpretation accuracy in these and also open new applications. While its main use currently is for perfusion assessment as a tissue health check prior to performing an anastomosis, there is increasing interest in using fluorophores for cancer detection during surgical interventions with most research being based on the paradigm of static imaging for fluorophore uptake hours after preoperative dosing. Although some image boosting and relative estimation of fluorescence signals is already inbuilt into commercial NIR systems, fuller implementation of AI methods can enable actionable predictions especially when applied during the dynamic, early inflow-outflow phase that occurs seconds to minutes after ICG (or indeed other fluorophore) administration. Already research has shown that such methods can accurately differentiate cancer from benign tissue in the operating theatre in real time in principle based on their differential signalling and could be useful for tissue perfusion classification more generally. This can be achieved through the generation of fluorescence intensity curves from an intra-operative NIR video stream. These curves are processed to adjust for image disturbances and curve features known to be influential in tissue characterisation are extracted. Existing machine learning based classifiers can then use these features to classify the tissue in question according to prior training sets. The use of this interpretable methodology enables accurate classification algorithms to be built with modest training sets in comparison to those required for deep learning modelling in addition to achieving compliance with medical device regulations. Integration of the multiple algorithms required to achieve this classification into a desktop application or medical device could make the use of this method accessible and useful to (as well as useable by) surgeons without prior training in computer technology. This document details some technical and functional design considerations underlying such a novel recommender system to advance the foundational concept and methodology as software as medical device for in situ cancer characterisation with relevance more broadly also to other tissue perfusion applications.

Ground-based response robotics for the safe operation of TV journalists in emergency zone

The article considers an approach to ensuring safe working conditions for TV journalists conducting TV reports from the emergency zone by means of ground-based response robotics. It is proposed to use a remote controlled (teleoperated) ground-based mobile service robots equipped with vision systems and professional television film-ing equipment. Remote-controlled robots developed at the Russian State Scientific Center for Robotics and Tech-nical Cybernetics (St. Petersburg) can be used as such robotic systems. Equipping robots with a stereoscopic vi-sion systems and a special TV camera to organize teleoperation of a mobile robot’s movement requires a broad-band communication channel, since it is necessary to transmit several video streams simultaneously. To solve this problem, it is proposed to use an image compression methods and the experience of using a network technolo-gies obtained during the development of the multiple-camera vision systems for installation on the mobile robotic systems BROKK-110D and BROKK-330 (Sweden) at the instructions of the Center for Special Risk Rescue Opera-tions «Leader» (Russia). The technical solution is presented to provide TV journalists with special remote-controlled ground-based robotics, which allows them to conduct television reports directly from the scene of hostili-ties, while being at a safe distance from the scene of events and the location of radio antennas.

Digital Apparel Try-Ons: A Technological Odyssey into the World of Virtual Dressing Rooms and Consumer Engagement

This individual can now enjoy internet shopping whenever they want and from anywhere in the world thanks to the invention of smartphones and tablets. Online shopping has unquestionably taken the place of conventional methods for purchasing apparel and daily necessities. People benefit from a reputation as they pick internet buying. Almost all online retailers offer payment on delivery, free shipment, and special rates these days. These internet merchants have eliminated the hassles of parking, stuck in traffic, and standing in line to make purchases. They have also helped folks who consistently lament their lack of time. This is the primary justification for why most consumers now shop online. They may easily access a wide choice of affordable prices here, and they also get quick customer service and free home delivery. There is no denying that these are a few of the alluring aspects that draw customers in. Even though there is a minor problem that can make people lose interest in online buying, they might not be able to try on garments in such situations. By establishing a virtual changing room setting, the aim is to improve accessibility and time efficiency for trying on clothing. Our proposed methodology consists of three main components: skin color recognition, model alignment, and user image extraction from the video stream. Through user extraction, we may create an augmented reality environment by taking the user area out of the video stream and superimposing it on top of a simulated world within the user interface. To match the 2D cloth models with the use of scaling, rotation, and positioning using the 3D locations of the joints.