Analysis Of Video Streams Research Articles

A Deep Learning Framework for Monitoring Audience Engagement in Online Video Events

This paper introduces a deep learning methodology for analyzing audience engagement in online video events. The proposed deep learning framework consists of six layers and starts with keyframe extraction from the video stream and the participants’ face detection. Subsequently, the head pose and emotion per participant are estimated using the HopeNet and JAA-Net deep architectures. Complementary to video analysis, the audio signal is also processed using a neural network that follows the DenseNet-121 architecture. Its purpose is to detect events related to audience engagement, including speech, pauses, and applause. With the combined analysis of video and audio streams, the interest and attention of each participant are inferred more accurately. An experimental evaluation is performed on a newly generated dataset consisting of recordings from online video events, where the proposed framework achieves promising results. Concretely, the F1 scores were 79.21% for interest estimation according to pose, 65.38% for emotion estimation, and 80% for sound event detection. The proposed framework has applications in online educational events, where it can help tutors assess audience engagement and comprehension while hinting at points in their lectures that may require further clarification. It is effective for video streaming platforms that want to provide video recommendations to online users according to audience engagement.

A real-time feeding decision method based on density estimation of farmed fish

With the global population growth and increasing demand for high-quality protein, aquaculture has experienced rapid development. Fish culture management and feed supply are crucial components of aquaculture. Traditional baiting management relies on experiential judgment and regular observation, which often leads to inefficient baiting practices and wastage. To address these issues, intelligent bait casting decisions have emerged. Leveraging advanced artificial intelligence algorithms, intelligent bait casting decisions can overcome most drawbacks of traditional bait management and enhance breeding efficiency. However, most of the current intelligent baiting decisions are focused on using methods such as image processing and target detection to identify different feeding actions and patterns. These methods do not discuss based on video streams and do not consider the changes in fish behavior during the baiting process. Therefore, we proposed a real-time analysis method based on the density distribution of fish feeding behavior (FishFeed). Firstly, this method upgrades the input mechanism, not only handling static images but also capable of real-time video stream analysis. Secondly, by evaluating the fish school density distribution through a new intelligent baiting strategy, this method can monitor the feeding behavior of fish school during the baiting process in real time. Finally, we constructed a dataset for fish school density analysis (DlouFishDensity) that includes a wealth of video and image frames, providing a valuable resource for research. Experimental results indicate that our algorithm outperforms MCNN, improving MAE by 1.63 and 1.35, MSE by 1.92 and 1.58, and reducing prediction time by 2.56 seconds on the same dataset. By implementing real-time analysis of fish feeding behavior density distribution, our method offers a more efficient and effective approach to baiting management in aquaculture, contributing to improved breeding efficiency and resource utilization.

A Liquidity Analysis System for Large-scale Video Streams in the Oilfield

This article introduces LinkStream, a liquidity analysis system based on multiple video streams designed and implemented for oilfield. LinkStream combines a variety of technologies to solve several problems in computing power and network latency. First, the system adopts an edge-central architecture and tailoring based on spatio-temporal correlation, which greatly reduces computing power requirements and network costs, and enables real-time analysis of large-scale video stream on limited edge devices. Second, it designed a set of liquidity information to describe the liquidity status in the oilfield. Finally, it uses object tracking technology to design a counting algorithm for the unique tubing object in the oilfield. We have deployed LinkStream in an oilfield in Iraq. LinkStream can perform real-time inference on over 200 video streams with acceptable resource overhead.

Vehicle Detection in Traffic Monitoring

Traffic Monitoring is a challenging task on crowded roads. Traffic Monitoring procedures are manual, expensive, time consuming and involve human operators. Large-scale storage and analysis of video streams were not possible due to limited availability. However, it is now possible to implement object detection and tracking, behavioral analysis of traffic patterns, number plate recognition and surveillance on video streams produced by traffic monitoring. In Big data, video datasets are so large that typical database systems are not able to store and analyses the datasets. Storing and Processing big volumes of data requires Scalability, Fault Tolerance and Availability. Thus, Big Data and Cloud computing are two compatible concepts as cloud enables big data for traffic monitoring. In this paper, we proposed vehicle detection for traffic monitoring. We have applied Support Vector Machine (SVM) machine learning algorithm for detecting vehicles.

Climbing Technique Evaluation by Means of Skeleton Video Stream Analysis.

Due to the growing interest in climbing, increasing importance has been given to research in the field of non-invasive, camera-based motion analysis. While existing work uses invasive technologies such as wearables or modified walls and holds, or focuses on competitive sports, we for the first time present a system that uses video analysis to automatically recognize six movement errors that are typical for novices with limited climbing experience. Climbing a complete route consists of three repetitive climbing phases. Therefore, a characteristic joint arrangement may be detected as an error in a specific climbing phase, while this exact arrangement may not considered to be an error in another climbing phase. That is why we introduced a finite state machine to determine the current phase and to check for errors that commonly occur in the current phase. The transition between the phases depends on which joints are being used. To capture joint movements, we use a fourth-generation iPad Pro with LiDAR to record climbing sequences in which we convert the climber's 2-D skeleton provided by the Vision framework from Apple into 3-D joints using the LiDAR depth information. Thereupon, we introduced a method that derives whether a joint moves or not, determining the current phase. Finally, the 3-D joints are analyzed with respect to defined characteristic joint arrangements to identify possible motion errors. To present the feedback to the climber, we imitate a virtual mentor by realizing an application on the iPad that creates an analysis immediately after the climber has finished the route by pointing out the detected errors and by giving suggestions for improvement. Quantitative tests with three experienced climbers that were able to climb reference routes without any errors and intentionally with errors resulted in precision-recall curves evaluating the error detection performance. The results demonstrate that while the number of false positives is still in an acceptable range, the number of detected errors is sufficient to provide climbing novices with adequate suggestions for improvement. Moreover, our study reveals limitations that mainly originate from incorrect joint localizations caused by the LiDAR sensor range. With human pose estimation becoming increasingly reliable and with the advance of sensor capabilities, these limitations will have a decreasing impact on our system performance.

A Comprehensive Analysis of Real-Time Car Safety Belt Detection Using the YOLOv7 Algorithm

Using a safety belt is crucial for preventing severe injuries and fatalities during vehicle accidents. In this paper, we propose a real-time vehicle occupant safety belt detection system based on the YOLOv7 (You Only Look Once version seven) object detection algorithm. The proposed approach aims to automatically detect whether the occupants of a vehicle have buckled their safety belts or not as soon as they are detected within the vehicle. A dataset for this purpose was collected and annotated for validation and testing. By leveraging the efficiency and accuracy of YOLOv7, we achieve near-instantaneous analysis of video streams, making our system suitable for deployment in various surveillance and automotive safety applications. This paper outlines a comprehensive methodology for training the YOLOv7 model using the labelImg tool to annotate the dataset with images showing vehicle occupants. It also discusses the challenges of detecting seat belts and evaluates the system’s performance on a real-world dataset. The evaluation focuses on distinguishing the status of a safety belt between two classes: “buckled” and “unbuckled”. The results demonstrate a high level of accuracy, with a mean average precision (mAP) of 99.6% and an F1 score of 98%, indicating the system’s effectiveness in identifying the safety belt status.

A novel multi-modal neural network approach for dynamic and generic sports video summarization

Video Summarization is a video compression/compaction technique to create a shorter yet informative version of original video. Video summarization has offered solutions to plenty of media, user and engineering applications. Though sports video summarization has been an active research topic for some time; there still exists a void for multi-modal, dynamic, generic and domain knowledge based approach for Cricket Sport video summarization. This paper presents a multi-modal video summarization approach to summarize Cricket sport videos. This work captures the domain knowledge acquired from multi-modal (audio-visual) cues. A dual neural network architecture pipeline is proposed to dynamically segment and dynamically summarize Cricket videos for generic target audience. The former Neural Network is grounded on Cricket bowling activity (visual feature) for dynamic video segmentation of Cricket videos. The segments are then forwarded to the latter Neural Network for identification of key segments. The key segment detection module relies on Audio analysis of Cricket video stream to identify exciting, content representative and informative segments as per Cricket domain. Experimental analysis on two novel proposed benchmark datasets, i.e. DPCS (Delivery Play Cricket Sport) image dataset and EXINP (Excited Interval Normal Play) Cricket Dataset (audio based) shows promising results. The results indicate that the proposed multi-modal approach generates exciting, content representative, informative, generic and dynamic summary incorporating domain knowledge of the sport.

Continuous Human Action Recognition for Human-machine Interaction: A Review

With advances in data-driven machine learning research, a wide variety of prediction models have been proposed to capture spatio-temporal features for the analysis of video streams. Recognising actions and detecting action transitions within an input video are challenging but necessary tasks for applications that require real-time human-machine interaction. By reviewing a large body of recent related work in the literature, we thoroughly analyse, explain, and compare action segmentation methods and provide details on the feature extraction and learning strategies that are used on most state-of-the-art methods. We cover the impact of the performance of object detection and tracking techniques on human action segmentation methodologies. We investigate the application of such models to real-world scenarios and discuss several limitations and key research directions towards improving interpretability, generalisation, optimisation, and deployment.

Modeling of neurocognitive rehabilitation using intelligent analysis of video streaming and neurophysiological information (EEG, ERPs)

Modeling of neurocognitive rehabilitation using intelligent analysis of video streaming and neurophysiological information (EEG, ERPs)

DEEP REINFORCEMENT LEARNING-BASED OPTIMIZATION AND ENHANCEMENT OF MULTIMEDIA DATA: AN INNOVATIVE APPROACH

The rapid growth of multimedia data in various domains has necessitated the development of efficient techniques to enhance and optimize its quality. Traditional approaches often struggle to address the complexity and diversity of multimedia data, leading to suboptimal results. This paper presents a novel approach to tackle this challenge by leveraging the power of deep reinforcement learning (DRL). The proposed method utilizes DRL to learn and optimize multimedia data in an improvised manner. By employing a combination of convolutional neural networks and deep Q-networks, the model can effectively extract high-level features and make informed decisions to enhance the quality of multimedia data. The reinforcement learning framework enables the system to learn from its actions, continuously improving its performance through an iterative process. To evaluate the effectiveness of the proposed method, extensive experiments were conducted using a diverse set of multimedia datasets. The results demonstrate significant improvements in various quality metrics, including image resolution, video frame rate, and audio clarity. Additionally, the proposed approach exhibits robustness across different types of multimedia data, ensuring consistent enhancement performance across various domains. Furthermore, the computational efficiency of the proposed method is also highlighted, as it demonstrates faster convergence and lower computational overhead compared to traditional optimization methods. This makes the approach practical for real-time applications where multimedia data needs to be processed efficiently. Overall, this paper introduces an innovative framework that combines deep reinforcement learning with multimedia data optimization. The results indicate its potential for enhancing multimedia data quality, offering a promising solution to the challenges associated with traditional approaches. The proposed method not only improves the visual and auditory aspects of multimedia content but also provides a scalable and efficient solution for real-world applications in domains such as image processing, video streaming, and audio analysis.

Obstacle Detection Algorithm for Railroad-Road Crossings Based on Video Stream Analysis

The paper presents a computationally efficient method developed to automatically detect obstacles that appear at railroad-road crossings. The proposed algorithm works maintenance-free and automatically detects obstacles that appear and stay or are left at the crossing. These obstacles can pose a threat for the trains causing even a trail derailment. When a detected obstacle is a person it is crucial to generate a warning in order to react and safely remove the person before the train arrives. The proposed algorithm is based on live video stream analysis. The standard image processing methods were used in order to detect obstacles in the acquired images. However, several techniques were proposed to obtain the desired results dealing with image vibrations, heavy rain and snow. Such robustness was obtained by building and maintaining a fuzzy background frame and appropriate filtering. The obstacle detection system was implemented and tested. The results confirmed its correct operation and usefulness in real-world conditions.

American Cultural Insularity and Global Online Video: Are Netflix, Amazon Prime and Other Digital Streaming Platforms Broadening Americans’ Foreign Film Consumption Horizons?

The United States has long been characterized by American Cultural Insularity (ACI). According to a theory of ACI that I have developed in previous work ( Author 2019 , 2020 ), compared to most people in most other countries, Americans tend to consume much more of their own cultural media products and much fewer cultural media products produced in other countries than people in other countries consume. This paper compares long-running and deeply-entrenched American resistance to foreign and non-English language film in movie theaters to the (lack of) popularity of foreign, non-American and feature film-length content originally produced in a language other than English on major digital online video streaming platforms such as Netflix, Amazon Prime and Google Video. It does so primarily via a discussion of, and analysis of, digital online video streaming platform popularity charts compiled by flixpatrol.com (Flixpatrol), a Netherlands-based online video streaming data collection and analysis web site/company. An analysis of Flixpatrol's Top Streaming in the United States popularity chart for 10 major digital online video streaming platforms from February 2020 to September 2021 shows little evidence of a movement among American-based consumers toward more consumption of foreign, non-English-language feature length films.

Algorithmic fairness in computational medicine.

SummaryMachine learning models are increasingly adopted for facilitating clinical decision-making. However, recent research has shown that machine learning techniques may result in potential biases when making decisions for people in different subgroups, which can lead to detrimental effects on the health and well-being of specific demographic groups such as vulnerable ethnic minorities. This problem, termed algorithmic bias, has been extensively studied in theoretical machine learning recently. However, the impact of algorithmic bias on medicine and methods to mitigate this bias remain topics of active discussion. This paper presents a comprehensive review of algorithmic fairness in the context of computational medicine, which aims at improving medicine with computational approaches. Specifically, we overview the different types of algorithmic bias, fairness quantification metrics, and bias mitigation methods, and summarize popular software libraries and tools for bias evaluation and mitigation, with the goal of providing reference and insights to researchers and practitioners in computational medicine.

Person Fall Detection System Based on Video Stream Analysis

The purpose of this paper is to develop an information system that would detect a fall based on video stream analysis. This work contains research and a description of the subject environment and the structure of the system. An analysis of current scientific works in this area was conducted, which allowed defining the purpose and primary objectives of the thesis. In addition, this analysis showed that this system would be in demand in the market of software that uses computer vision. The practical significance of the results can be determined by the fact that the proposed application performs on the level of other research works in this field and can be used by a broad market of customers. We applied pre-trained TensorFlow Lite CNN real-time model PoseNet to detect keypoints of a subject. Keypoints sequence is analyzed by LSTM as it works best when processing long sequences of data. The system was trained using a combination of the two biggest fall detection datasets. The system extracts the locations of 17 joint points of the human body and recognizes human movement by detecting the joint point location changes. By using joint points location as input to LSTM, the training and inference time was improved drastically. It also helped minimize the adverse effects of light sources and shadows

A Smart System for Personal Protective Equipment Detection in Industrial Environments Based on Deep Learning at the Edge

Real-time object detection is currently used to automate various tasks in industrial environments. One of the most important tasks is to improve the safety of workers by monitoring the correct use of Personal Protective Equipment (PPE) in dangerous areas. In this context, usually, a monitoring system analyzes the stream of videos from surveillance cameras to assess PPE usage in real time. When a worker not wearing the appropriate PPE is detected, an acoustic or visual alarm is triggered automatically to raise attention and awareness. The solutions proposed so far are mostly cloud-based systems: images from the site are continuously offloaded to the cloud for analysis. This centralized architecture requires significant network bandwidth to transmit the video feeds through an internet connection that must be reliable, as a network outage would disrupt the service. In this work, we propose a system for real-time PPE detection based on video streaming analysis and Deep Neural Network (DNN). We adopt the edge computing model in which the application for image analysis and classification is deployed on an embedded system installed in proximity of the camera and directly connected to it. The system does not require continuous image transmission towards a cloud system, thus ensuring bandwidth efficiency, reliability, and workers’ privacy. A prototype of the proposed system is developed exploiting a low-cost commercial embedded system, i.e. a Raspberry PI, equipped with an Intel Neural Compute Stick 2. We tested the system with five different pre-trained convolutional neural networks (CNNs), fine-tuned to detect different PPEs, namely helmets, vests, and gloves. In our experimental evaluation, we first compared the five CNNs in terms of classification performance and inference latency. Then, we deployed each CNN on the real system and evaluated the system’s throughput regarding the number of video frames analyzed each second.

Edge-Based Detection of Varroosis in Beehives with IoT Devices with Embedded and TPU-Accelerated Machine Learning

One of the causes of mortality in bees is varroosis, a bee disease caused by the Varroa destructor mite. Varroa destructor mites may occur suddenly in beehives, spread across them, and impair bee colonies, which finally die. Edge IoT (Internet of Things) devices capable of processing video streams in real-time, such as the one we propose, may allow for the monitoring of beehives for the presence of Varroa destructor. Additionally, centralization of monitoring in the Cloud data center enables the prevention of the spread of this disease and reduces bee mortality through monitoring entire apiaries. Although there are various IoT or non-IoT systems for bee-related issues, such comprehensive and technically advanced solutions for beekeeping and Varroa detection barely exist or perform mite detection after sending the data to the data center. The latter, in turn, increases communication and storage needs, which we try to limit in our approach. In the paper, we show an innovative Edge-based IoT solution for Varroa destructor detection. The solution relies on Tensor Processing Unit (TPU) acceleration for machine learning-based models pre-trained in the hybrid Cloud environment for bee identification and Varroa destructor infection detection. Our experiments were performed in order to investigate the effectiveness and the time performance of both steps, and the study of the impact of the image resolution on the quality of detection and classification processes prove that we can effectively detect the presence of varroosis in beehives in real-time with the use of Edge artificial intelligence invoked for the analysis of video streams.

Deep-Framework: A Distributed, Scalable, and Edge-Oriented Framework for Real-Time Analysis of Video Streams.

Edge computing is the best approach for meeting the exponential demand and the real-time requirements of many video analytics applications. Since most of the recent advances regarding the extraction of information from images and video rely on computation heavy deep learning algorithms, there is a growing need for solutions that allow the deployment and use of new models on scalable and flexible edge architectures. In this work, we present Deep-Framework, a novel open source framework for developing edge-oriented real-time video analytics applications based on deep learning. Deep-Framework has a scalable multi-stream architecture based on Docker and abstracts away from the user the complexity of cluster configuration, orchestration of services, and GPU resources allocation. It provides Python interfaces for integrating deep learning models developed with the most popular frameworks and also provides high-level APIs based on standard HTTP and WebRTC interfaces for consuming the extracted video data on clients running on browsers or any other web-based platform.

Quality of service analysis of video streaming in fourth generation-Long Term Evolution for telemedicine applications: An experimental learning with game theory approach

Wireless technologies have become the model for Entertainment, Communication and Education all over the world. Electrical and Electronics Engineering Departments of various Universities across the globe offer courses on Wireless Communication and Networks for Undergraduate and Postgraduate students by focusing on the latest teaching pedagogy. Recent advancements in health care services demand reliable transmission of medical images and videos for better diagnosis and treatment. The effective utilization of available spectrum is one of the requirements for stable transmission of medical information. A Long Term Evolution (LTE) system model for a real-time multimedia application must provide high Quality of Service (QoS). Scheduling and allocating resources for real-time downlink flows in LTE networks is a challenging issue, as the network excellence and assured output have to be confirmed together with decreasing the Head of Line (HOL) suspension. The ever increasing demand for higher data rates in uplink and downlink transmission services constraints on delay and bandwidth requirements. This leads the existing fourth generation LTE wireless networks to concentrate more on the resource scheduling techniques with acceptable levels of Quality of Service (QoS). In this paper, a Game theory based scheduling (GTBS) and resource allocation Algorithm multimedia traffic in LTE network is designed and studied. The Algorithm is simulated in LTE-sim for real time (RT) and Non-real time (NRT) data traffic. The work identified a new teaching pedagogy, the work is explained to the undergraduate Engineering students so that they can apply the knowledge to develop new innovative projects.

Dissociation of tic generation from tic expression during the sleep-wake cycle

SummaryMotor tics, the hallmark of Tourette syndrome (TS), are modulated by different behavioral and environmental factors. A major modulating factor is the sleep-wake cycle in which tics are attenuated to a large extent during sleep. This study demonstrates a similar reduction in tic expression during sleep in an animal model of chronic tic disorders and investigates the underlying neural mechanism. We recorded the neuronal activity during spontaneous sleep-wake cycles throughout continuous GABAA antagonist infusion into the striatum. Analysis of video streams and concurrent kinematic assessments indicated tic reduction during sleep in both frequency and intensity. Extracellular recordings in the striatum revealed a state-dependent dissociation between motor tic expression and their macro-level neural correlates (“LFP spikes”) during the sleep-wake cycle. Local field potential (LFP) spikes, which are highly correlated with tic expression during wakefulness, persisted during tic-free sleep and did not change their properties despite the reduced behavioral expression. Local, micro-level, activity near the infusion site was time-locked to the LFP spikes during wakefulness, but this locking decreased significantly during sleep. These results suggest that whereas LFP spikes encode motor tic generation and feasibility, the behavioral expression of tics requires local striatal neural activity entrained to the LFP spikes, leading to the propagation of the activity to downstream targets and consequently their motor expression. These findings point to a possible mechanism for the modulation of tic expression in patients with TS during sleep and potentially during other behavioral states.

DeepROD: a deep learning approach for real-time and online detection of a panic behavior in human crowds

Detecting a panic behavior within a human crowd is of a high importance as it allows to prevent disasters. Online analysis of video streams, real-time processing and accurate detection are required to ensure effective surveillance of the crowded places. However, these requirements are not simultaneously fulfilled by the existing techniques. Rapid advances in artificial intelligence are investing the power for automatic public surveillance and timely detection of a possible abnormal behavior. Thus, the aim of the present work is to propose an online, real-time and effective technique for panic behavior detection. It relies on a handcrafted feature that accounts for the characteristics of the crowd to understand people behaviors and a long short-term memory neural network to predict future feature values. Experiments are performed on well-known datasets of panic situations to evaluate the performance and accuracy of the proposed algorithm. Results show the system yields excellent performances in terms of accuracy and processing time with respect to the state of the art techniques.