Video Analytics System Research Articles

SketchQL: Video Moment Querying with a Visual Query Interface

Localizing video moments based on the movement patterns of objects is an important task in video analytics. Existing video analytics systems offer two types of querying interfaces based on natural language and SQL, respectively. However, both types of interfaces have major limitations. SQL-based systems require high query specification time, whereas natural language-based systems require large training datasets to achieve satisfactory retrieval accuracy. To address these limitations, we present SketchQL, a video database management system (VDBMS) for offline, exploratory video moment retrieval that is both easy to use and generalizes well across multiple video moment datasets. To improve ease-of-use, SketchQL features a visual query interface that enables users to sketch complex visual queries through intuitive drag-and-drop actions. To improve generalizability, SketchQL operates on object-tracking primitives that are reliably extracted across various datasets using pre-trained models. We present a learned similarity search algorithm for retrieving video moments closely matching the user's visual query based on object trajectories. SketchQL trains the model on a diverse dataset generated with a novel simulator, that enhances its accuracy across a wide array of datasets and queries. We evaluate SketchQL on four real-world datasets with nine queries, demonstrating its superior usability and retrieval accuracy over state-of-the-art VDBMSs.

Spatialyze: A Geospatial Video Analytics System with Spatial-Aware Optimizations

Videos that are shot using commodity hardware such as phones and surveillance cameras record various metadata such as time and location. We encounter such geospatial videos on a daily basis and such videos have been growing in volume significantly. Yet, we do not have data management systems that allow users to interact with such data effectively. In this paper, we describe Spatialyze, a new framework for end-to-end querying of geospatial videos. Spatialyze comes with a domain-specific language where users can construct geospatial video analytic workflows using a 3-step, declarative, build-filter-observe paradigm. Internally, Spatialyze leverages the declarative nature of such workflows, the temporal-spatial metadata stored with videos, and physical behavior of real-world objects to optimize the execution of workflows. Our results using real-world videos and workflows show that Spatialyze can reduce execution time by up to 5.3×, while maintaining up to 97.1% accuracy compared to unoptimized execution.

Video Analytics in Moodle Using xAPI

AbstractThis article presents the first iteration of the video analytics system VA4ME that enables the logging of video interactions in Moodle without the need to provide the videos by a separate plugin or website. Instead, the logging plugin injects, if allowed within a course, JavaScript code that logs video interactions and transforms these logs into xAPI statements. Using this data format creates the foundation for combining video interactions with other logs for more in-depth analyses. The analyses are conducted periodically, every 24 h. Advantages and challenges resulting from this approach are presented. It is described how the dashboard, visualizing the analysis results, was designed and how the analysis results are stored such that they can be retrieved as fast as possible. Overall, the setup of the video analytics system is quite extensive, as two Moodle plugins and Excalibur LA need to be set up. The advantages however, especially in the long term, including analyzing video interactions with other logs or easy extensibility with new analyses, outweigh the initial effort. The system was pilot tested in a blended learning course where videos were an addition to the lectures. The analysis results show little consumption of the videos, which limits the insights into the video usage. Still, the dashboard’s usability was rated good and the reports were considered promising in terms of providing more insights when more data is given. To further evaluate this potential, the system was integrated in a flipped classroom and a self-learning course where video usage was more central than within the pilot test course.

ПОКРАЩЕННЯ ВІДЕОПОСЛІДОВНОСТІ В СИСТЕМАХ ВІДЕОАНАЛІТИКИ

A video enhancement method based on applying gamma correction is proposed. Its realization implies processing all video frames using a single algorithm. The proposed algorithm, due to the preservation of inter-frame coherence throughout the entire video, significantly reduces the time spent searching for the optimal value of the gamma parameter, which ensures the highest quality of the frame reaching the maximum possible level in the automatic mode in the sense of both visual observation and the determination of key points and the selection of contours of objects in images. The method is characterized by high adaptability to sudden changes in the lighting of the scene, preservation of interframe coherence, and the absence of any side negative artifacts in the enhanced video. A toolkit for automatically determining the optimal value of the gamma parameter for video frames is developed. It significantly increases the efficiency of video analytics systems, image and video segmentation and processing processes due to reducing the negative impact of the scene lighting mode on image quality. Keywords: gamma-correction, video analytics system, video sequence enhancement, histogram, cumulative histogram, video processing, interframe coherence.

ОПТИМИЗАЦИЯ ТЕХНОЛОГИЧЕСКОЙ КАРТЫ ДОПУСТИМЫХ СИСТЕМОТЕХНИЧЕСКИХ РЕШЕНИЙ ЗАДАЧИ ВИДЕОАНАЛИТИКИ АКВАКУЛЬТУРЫ

The paper considers the problem of video analytics of aquatic organisms in fish farming industrial complexes. It proposes a conceptual model of the video analytics problem, and a corresponding mathematical formulation of the problem with the search for a subset of acceptable options for the technological map, which boils down to the search for options that satisfy functional criteria. The final decision on the structure and functions of video analytics hardware and software takes into account the cost of the entire life cycle of the equipment. The study emphasizes the importance of automation and intellectualization of technological processes in agriculture, the most pressing modern problems, which are the main focus of the research of advanced teams. The paper discusses primary data for the development and implementation of a video analytics system when solving the problem of counting fish and their mass during transplantation, shipment, and reception into the processing shop. The data were obtained in the framework of cooperation with the Ostrov company, specializing in trout cultivation and paying serious attention to the implementation of modern means of automatization and artificial intelligence in technological processes of industrial aquaculture. Full-scale experiments were carried out and images of fishes were collected in a transparent narrow pipe directed by a stream of water, and in air while moving on a conveyor belt. Further research will be aimed at developing model-algorithmic and software necessary for testing the proposed mathematical models and optimizing options for the technological map of video analytics system.

An Edge-Side Real-Time Video Analytics System With Dual Computing Resource Control

An Edge-Side Real-Time Video Analytics System With Dual Computing Resource Control

Internet-of-Things Edge Computing Systems for Streaming Video Analytics: Trails Behind and the Paths Ahead

The falling cost of IoT cameras, the advancement of AI-based computer vision algorithms, and powerful hardware accelerators for deep learning have enabled the widespread deployment of surveillance cameras with the ability to automatically analyze streaming video feeds to detect events of interest. While streaming video analytics is currently largely performed in the cloud, edge computing has emerged as a pivotal component due to its advantages of low latency, reduced bandwidth, and enhanced privacy. However, a distinct gap persists between state-of-the-art computer vision algorithms and the successful practical implementation of edge-based streaming video analytics systems. This paper presents a comprehensive review of more than 30 research papers published over the last 6 years on IoT edge streaming video analytics (IE-SVA) systems. The papers are analyzed across 17 distinct dimensions. Unlike prior reviews, we examine each system holistically, identifying their strengths and weaknesses in diverse implementations. Our findings suggest that certain critical topics necessary for the practical realization of IE-SVA systems are not sufficiently addressed in current research. Based on these observations, we propose research trajectories across short-, medium-, and long-term horizons. Additionally, we explore trending topics in other computing areas that can significantly impact the evolution of IE-SVA systems.

Edge Video Analytics With Adaptive Information Gathering: A Deep Reinforcement Learning Approach

With growing popularity of enormous public safety and transportation infrastructure cameras, there are increasing demands for automatic mobile video analytics. The emerging multi-access edge computing (MEC) technology has been recently applied to improve the accuracy-latency tradeoff of mobile video analytics. In this paper, we study an MEC-enabled multi-device video analytics system and formulate the problem as a Markov decision process (MDP) to meet two practical challenges: i) the absence of ground truth in real-time and ii) the content-varying degradation-accuracy relation. In particular, we aim to design an online joint frame degradation and bandwidth allocation algorithm with the time-varying function and limited feedback from each device. Thanks to the MDP formulation and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> -step return technique, the long-term goal offers adaptive information gathering and thus improves the average accuracy and latency. For sample efficiency, we decompose the MDP problem into discrete degradation adaptation subproblems and continuous bandwidth allocation subproblems. Based on the decomposition, we propose a deep reinforcement learning (DRL) based framework, referred to as DBAG, to solve the decomposed subproblems. DBAG integrates model-based optimization and model-free DRL to solve the MDP problem with a discrete-continuous hybrid action space. Under various network setups and public datasets, DBAG greatly improves the accuracy-latency tradeoff.

Система событийного мониторинга для автоматизированного обнаружения инцидентов

Currently, computer vision technologies used in event monitoring systems to solve security problems in the field of transport, data protection, medicine are becoming an increasingly promising direction. Video surveillance sys-tems generate petabytes of data every day, and only a small part is used in processing. The use of video analytics will eliminate the need for storing and processing unnecessary data, their manual viewing, which will directly affect the cost, complexity and speed of solving operational production tasks of responding to incidents. The data from video cameras, information collected from different sources and used together for analysis would make it possible to more effectively and quickly identify and prevent various undesirable events. It is possible to automate the analysis of complex structured data, reducing the influence of the human factor, eliminating errors and abuses, using artificial intelligence methods, neural networks. But modern intelligent video analytics systems have drawbacks. Many systems are focused on the recognition of a certain type of images, can work in limited subject areas and under certain environmental conditions. Recognition algorithms are associated with a large number of false positives, especially in conditions of a rapidly increasing data volume, the degree of uncertainty of input information, therefore, it is proposed to supplement event monitoring systems. The systems contain a large number of settings and rules, which complicates the understanding of the system. There have been described the difficulties of using biometric data in recognition systems due to the legal restrictions, the main stages of designing an event monitoring system, its model, which combines elements of fuzzy logic and pattern recognition methods.

Evaluating digital video transcoding for forensic derivative results

Video data received for analysis often come in a variety of file formats and compression schemes. These data are often transcoded to a consistent file format for forensic examination and/or ingesting into a video analytic system. The file format often requested is an MP4 file format. The MP4 file format is a very common and a universally accepted file format. The practical application of this transcoding process, across the analytical community, has generated differences in video quality. This study sought to explore possible origins of the differences and assist the practitioner by defining minimum recommendations to ensure that quality of the video data is maintained through the transcoding process. This study sought to generate real world data by asking participants to transcode provided video files to an MP4 file format using programs they would typically utilize to perform this task. The transcoded results were evaluated based on measurable metrics of quality. As the results were analyzed, determining why these differences might have occurred became less about a particular software application and more about the settings employed by the practitioner or of the capabilities of the program. This study supports the need for any video examiner who is transcoding video data to be cognizant of the settings utilized by the programs employed for transcoding video data, as loss of video quality can affect analytics as well as further analysis.

Video analytic system for activity profiling, fall detection, and unstable motion detection

Real time detection of falls and unstable movement by elderly people is vital to their quality of life and safety. We present an edge processing device integrated with a cloud computation framework that can be used for activity profiling as well as trigger alerts for falls and unstable motion by elderly people at home. The proposed system uses fixed cameras to track and analyze each visible person in the scene, classifying their actions into nine ordinary activities, a fall, or unstable movement. An alert notification is sent to caregivers whenever a fall or unstable movement is detected. The major components of the system include an embedded device (NVIDIA JETSON TX2) and cloud-based storage and analysis infrastructure. The system is composed of modules for detecting, tracking and recognizing humans, a cascaded hierarchical classifier for nine ordinary activities and falls, and a long short-term memory (LSTM) module to predict unstable movement in video. The system is designed for accuracy, usability, and cost. A prototype system has been subjected to individual module tests along with a field test within a volunteer’s household. It achieved an accuracy of 91.6% for ordinary actions and falls with a recall of 97.02% for unstable motion. Future phases will expand deployment to multiple homes.

Enabling Edge-Cloud Video Analytics for Robotics Applications

Emerging deep learning-based video analytics tasks demand computation-intensive neural networks and powerful computing resources on the cloud to achieve high accuracy. Due to the latency requirement and limited network bandwidth, edge-cloud systems adaptively compress the data to strike a balance between overall analytics accuracy and bandwidth consumption. However, the degraded data leads to another issue of poor tail accuracy, which means the extremely low accuracy of a few semantic classes and video frames. Autonomous robotics applications especially value the tail accuracy performance but suffer using the prior edge-cloud systems. We present Runespoor, an edge-cloud video analytics system to manage the tail accuracy and enable emerging robotics applications. We train and deploy a super-resolution model tailored for the tail accuracy of analytics tasks on the server to significantly improves the performance on hard-to-detect classes and sophisticated frames. During online operation, we use an adaptive data rate controller to further improve the tail performance by instantly adjusting the data rate policy according to the video content. Our evaluation shows that Runespoor improves class-wise tail accuracy by up to 300%, frame-wise 90%/99% tail accuracy by up to 22%/54%, and greatly improves the overall accuracy and bandwidth trade-off.

The method of the real-time human detection and tracking

Today, data collected from video surveillance systems require processing. Video content analysis (VCA) or video analytics (VA) has found applications in security systems, retail, the automotive industry, smart home technologies, etc. The tasks of video analytics include the detection and tracking of objects. Usually, video analytics systems are specialized software for analyzing video data collected from webcams and intelligently assessing the situation. There are also separate video surveillance cameras with built-in video analytics functions. Software approaches to human detection and tracking are quite diverse, special applications and web applications or separate software modules are created. This work presents an approach to solving these problems using convolutional neural networks. The aim of the study is to increase the efficiency of human detection and tracking in video sequences. For this purpose, an overview of existing methods of detecting objects in images was conducted, in particular: the Viola-Jones algorithm, the histogram of oriented gradients. The choice of convolutional neural networks for solving the subtask of object detection is substantiated. The working principles, advantages and disadvantages of Faster R-CNN, YOLO, SSD and RetinaNet networks are considered. Their comparative analysis was carried out according to the indicators of speed and accuracy of recognition on the HABBOF dataset. A hybrid neural network for human detection and tracking has been developed: a convolutional neural network of the YOLO type has been improved. The created network achieved an accuracy of 39.2% at 43 frames per second. Experiments were carried out with the created network in order to evaluate the operation of the network in various conditions. It has been established that it works well in poor lighting conditions, but the issue of detecting small objects remains open.

Implementation and evaluation of the effectiveness of dynamic object detection tools for multithreaded and distributed processing of graphical data

Modern monitoring and operational decision-making systems based on image series have become widely used for a wide variety of applications. At the stage of identification of graphic objects, it is established that the found dynamic object belongs to the class of objects of interest based on a comparative analysis of its contours with a given template. The article is the final in a series of articles devoted to theoretical, algorithmic and software components of the video analytics system. The description of the structural components of the video analytics system, the features of the implementation of the dynamic object detection module on video sequences, and the evaluation of the efficiency of the system is presented.

Інформаційні технології та цифрова трансформація в системі адаптивного розвитку підприємств роздрібної торгівлі

The modern information space of adaptive development of retail enterprises is characterized and the distinctive properties of information technologies are defined. Digital technologies are systematized and their functional purpose is defined. The general advantages of data analytics are formulated and the advantages of modern video analytics systems, chatbots, Internet of Things (IoT), RFID tags in the organization of retail trade enterprises are characterized. The need for a situational approach to digital changes is argued. A model of the formation of a comprehensive mechanism for the readiness of a retail trade enterprise for digital transformation has been developed. Digital competencies of personnel are considered as key criteria for establishing the level of readiness for digital transformation of business processes at various levels in the conditions of a highly dynamic business environment. The list of DigComp 2.0 digital competencies was considered. Two groups of digital competencies have been formed: basic and special for solving specific tasks of adaptive development of a retail enterprise.

End-to-end evaluation of practical video analytics systems for face detection and recognition

Practical video analytics systems that are deployed in bandwidth constrained environments like autonomous vehicles perform computer vision tasks such as face detection and recognition. In an end-to-end face analytics system, inputs are first compressed using popular video codecs like HEVC and then passed onto modules that perform face detection, alignment, and recognition sequentially. Previously, the modules of these systems have been evaluated independently using task-specific imbalanced datasets that can misconstrue performance estimates. In this paper, we perform a thorough end-to-end evaluation of a face analytics system using a driving-specific dataset, which enables meaningful interpretations. We demonstrate how independent task evaluations and dataset imbalances can overestimate system performance. We propose strategies to balance the evaluation dataset and to make its annotations consistent across multiple analytics tasks and scenarios. We then evaluate the end-to-end system performance sequentially to account for task interdependencies. Our experiments show that our approach provides a true estimate of the end-to-end performance for critical real-world systems.

Cognitive Cameras on the Edge for Crowd Physical Distancing Monitoring in the Covid-19 Era

With the rise of the Internet of Things (IoT) architectures and protocols, new video analytics systems and surveillance applications have been developed. In conventional systems, all the streams produced by cameras are sent to a centralized node where they can be seen by human operators whose task is to identify uncommon on abnormal situations. However, this way, much bandwidth is necessary for the system to work, and the number of necessary resources is proportional to the number of cameras and streams involved. In this paper, we propose an interesting approach to this problem: transforming any IP camera into a cognitive object. A cognitive camera (CC) can be considered a classic connected camera with onboard computational power for intelligent video processing. A CC can understand and interact with the surroundings, intelligently analyze complex scenes, and interact with the users. The IoT Edge Computing approach decreases latency in the decision-making process and consumes a tiny portion of bandwidth concerning the stream of a video, even in low resolution. CCs can help to address COVID-19. As a preventive measure, proper crowd monitoring and management systems must be installed in public places to limit sudden outbreaks and improve healthcare. The number of new infections can be significantly reduced by adopting physical distancing measures earlier. Motivated by this notion, a real-time crowd monitoring and management system for physical distance classification using CCs is proposed in this research paper. The experiment on Movidius board, an AI accelerator device, provides promising results of our proposed method in which the accuracies can achieve more than 85% from different datasets.

VaBUS: Edge-Cloud Real-Time Video Analytics via Background Understanding and Subtraction

Edge-cloud collaborative video analytics is transforming the way data is being handled, processed, and transmitted from the ever-growing number of surveillance cameras around the world. To avoid wasting limited bandwidth on unrelated content transmission, existing video analytics solutions usually perform temporal or spatial filtering to realize aggressive compression of irrelevant pixels. However, most of them work in a context-agnostic way while being oblivious to the circumstances where the video content is happening and the context-dependent characteristics under the hood. In this work, we propose VaBUS, a real-time video analytics system that leverages the rich contextual information of surveillance cameras to reduce bandwidth consumption for semantic compression. As a task-oriented communication system, VaBUS dynamically maintains the background image of the video on the edge with minimal system overhead and sends only highly confident Region of Interests (RoIs) to the cloud through adaptive weighting and encoding. With a lightweight experience-driven learning module, VaBUS is able to achieve high offline inference accuracy even when network congestion occurs. Experimental results show that VaBUS reduces bandwidth consumption by 25.0%-76.9% while achieving 90.7% accuracy for both the object detection and human keypoint detection tasks.

MLink: Linking Black-Box Models from Multiple Domains for Collaborative Inference.

The cost efficiency of model inference is critical to real-world machine learning (ML) applications, especially for delay-sensitive tasks and resource-limited devices. A typical dilemma is: in order to provide complex intelligent services (e.g. smart city), we need inference results of multiple ML models, but the cost budget (e.g. GPU memory) is not enough to run all of them. In this work, we study underlying relationships among black-box ML models and propose a novel learning task: model linking, which aims to bridge the knowledge of different black-box models by learning mappings (dubbed model links) between their output spaces. We propose the design of model links which supports linking heterogeneous black-box ML models. Also, in order to address the distribution discrepancy challenge, we present adaptation and aggregation methods of model links. Based on our proposed model links, we developed a scheduling algorithm, named MLink. Through collaborative multi-model inference enabled by model links, MLink can improve the accuracy of obtained inference results under the cost budget. We evaluated MLink on a multi-modal dataset with seven different ML models and two real-world video analytics systems with six ML models and 3,264 hours of video. Experimental results show that our proposed model links can be effectively built among various black-box models. Under the budget of GPU memory, MLink can save 66.7% inference computations while preserving 94% inference accuracy, which outperforms multi-task learning, deep reinforcement learning-based scheduler and frame filtering baselines.

Mathematical model of the passage of audio signals in network-based audio analytics systems

Relevance. Nowadays, more and more audio and video analytics systems work together. But in many cases, it is necessary to quickly transfer the audio stream. Therefore, the task of reducing delays in transmission of audio streams in network-based audio analytics systems becomes relevant. The object of research is the process of audio signal transmission. The subject of research is mathematical models of audio and video streams in network systems. The purpose of this paper is to develop a mathematical model for the passage of audio signals in network-based audio analytics systems. Research results. The statement of the problem of optimizing the network of connecting lines according to the criterion of minimum capital costs is proposed. Optimization equations for a unidirectional three-node basic structure are compiled based on the problem of minimizing capital costs. Optimization equations are obtained for a bidirectional three-node basic structure. Obtained optimization equations are generalized to any structure of a network of connecting lines. The direction of further research is the development of an approximate method for quickly solving optimization equations for a network of connecting lines and assessing the adequacy of the developed model.