Video Analytics Applications Research Articles

Live Migration of Video Analytics Applications in Edge Computing

In order to schedule resources efficiently or maintain applications' continuity for mobile customers, edge platforms often need to adaptively migrate the applications on them. However, our measurement shows that existing migration solutions cannot solve the issue of migrating video analytics applications in edge computing because the memory states of video analytics applications have different characteristics from other applications. We conduct a breakdown analysis of the memory states of video analytics applications, and propose to treat three types of states separately with three different techniques, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , warm-up, sync, and replay, to minimize the negative influence of migrations on application performance. Based on this idea, we implement a prototype system in which two new components, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">state store</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sidecar</i> , are designed to achieve near-transparent live migration with minimal application code modifications. Evaluation experiments demonstrate that the time of application interruption caused by migrating a video analytics application with our solution is less than 405ms, and our solution does not consume much resources.

SDN-LB: A novel server workload balancing algorithm for IoT video analytics

The wide deployment of Internet of Things (IoT) cameras combined with powerful deep learning models for object detection and recognition will usher in a new generation of IoT video analytics applications. However, IoT video analytics require intensive computing resources, which often are not present in IoT devices and are provided by edge and/or cloud servers. This paper proposes a hierarchical edge/cloud-based solution for processing IoT video streaming flows. We devised a queuing model that considers the characteristics of the IoT video flows (i.e., frame rate and frame resolution), the network backbone (i.e., communication latency of the routing path from IoT devices to cloud/edge servers), and the characteristics of the servers (i.e., processing resources capabilities) to estimate the expected end-to-end latency experienced by the frames when processed at the edge and cloud servers. We then propose a software-defined networking (SDN)-based architecture to balance the workload (i.e., video frames to be processed) at the edge and cloud servers aimed at reducing the average latency when processing video frames. We design the SDN-LB algorithm to periodically collect data from programmable switches, determine the expected latency in each server, and re-assign IoT video streaming flows to edge and cloud servers aimed at reducing the end-to-end latency for the processing of video streaming. Extensive numerical evaluation results show that the proposed balancing solution can effectively balance the IoT video streaming flows across the edge and cloud servers, and reduce the average latency incurred in the processing of IoT video frames.

Resource Management in Mobile Edge Computing: A Comprehensive Survey

With the evolution of 5G and Internet of Things technologies, Mobile Edge Computing (MEC) has emerged as a major computing paradigm. Compared to cloud computing, MEC integrates network control, computing, and storage to customizable, fast, reliable, and secure distributed services that are closer to the user and data site. Although a popular research topic, MEC resource management comes in many forms due to its emerging nature and there exists little consensus in the community. In this survey, we present a comprehensive review of existing research problems and relevant solutions within MEC resource management. We first describe the major problems in MEC resource allocation when the user applications have diverse performance requirements. We discuss the unique challenges caused by the dynamic nature of the environments and use cases where MEC is adopted. We also explore and categorize existing solutions that address such challenges. We particularly explore traditional optimization-based methods and deep learning-based approaches. In addition, we take a deeper dive into the most popular applications and use cases that adopt MEC paradigm and how MEC provides customized solutions for each use cases, in particular, video analytics applications. Finally, we outline the open research challenges and future directions. 1

Solar-powered Parking Analytics System Using Deep Reinforcement Learning

Advances in deep vision techniques and the ubiquity of smart cameras will drive the next generation of video analytics. However, video analytics applications consume vast amounts of energy as deep learning techniques are power-hungry. In this article, we focus on a parking video analytics platform and propose RL-CamSleep, a deep reinforcement learning-based technique, to actuate the cameras to reduce the energy footprint while retaining the system’s utility. Our key insight is that many video-analytics applications do not need to be always operational, and we can design policies to activate video analytics only when necessary. We design two modes of operation for the reinforcement learning (RL) controller: (i) cloud-based mode and (ii) grid-isolated solar-powered mode. In the cloud-based mode, the controller runs on the cloud to control the cameras, whereas, in the solar-powered mode, the RL controller is constrained by the energy produced by solar. We evaluate our approach on a city-scale parking dataset having 76 streets spread across a city. Our analysis shows RL-CamSleep can learn an adaptive policy that reduces the average energy consumption by 76% and achieves an average accuracy of 98%. For the grid-isolated mode, RL-CamSleep outperforms other baseline techniques demonstrating the need for adaptive policy in energy-constrained environments.

A Video Target Tracking and Correction Model with Blockchain and Robust Feature Location

In this paper, a cutting-edge video target tracking system is proposed, combining feature location and blockchain technology. The location method makes full use of feature registration and received trajectory correction signals to achieve high accuracy in tracking targets. The system leverages the power of blockchain technology to address the challenge of insufficient accuracy in tracking occluded targets, by organizing the video target tracking tasks in a secure and decentralized manner. To further enhance the accuracy of small target tracking, the system uses adaptive clustering to guide the target location process across different nodes. In addition, the paper also presents an unmentioned trajectory optimization post-processing approach, which is based on result stabilization, effectively reducing inter-frame jitter. This post-processing step plays a crucial role in maintaining a smooth and stable track of the target, even in challenging scenarios such as fast movements or significant occlusions. Experimental results on CarChase2 (TLP) and basketball stand advertisements (BSA) datasets show that the proposed feature location method is better than the existing methods, achieving a recall of 51% (27.96+) and a precision of 66.5% (40.04+) in the CarChase2 dataset and recall of 85.52 (11.75+)% and precision of 47.48 (39.2+)% in the BSA dataset. Moreover, the proposed video target tracking and correction model performs better than the existing tracking model, showing a recall of 97.1% and a precision of 92.6% in the CarChase2 dataset and an average recall of 75.9% and mAP of 82.87% in the BSA dataset, respectively. The proposed system presents a comprehensive solution for video target tracking, offering high accuracy, robustness, and stability. The combination of robust feature location, blockchain technology, and trajectory optimization post-processing makes it a promising approach for a wide range of video analytics applications, such as surveillance, autonomous driving, and sports analysis.

ECCVideo: A Scalable Edge Cloud Collaborative Video Analysis System

Video analysis drives a wide range of applications in the fields of public safety, autonomous vehicles, etc., with the great potential to impact society. Traditional cloud-based approaches are not applicable because of prohibitive bandwidth consumption and high response latency, while simply edge-based video analysis suffers from large computation delay, considering the restricted computing capacity of edge servers. Therefore, in this article, we focus on low-latency edge-cloud collaborative video analytic applications (ECCVApps) by making full use of resources at both the edge and cloud. Particularly, we present an edge-cloud collaborative video analysis system called ECCVideo, to support the unified management of heterogeneous servers and facilitate the development and deployment of large-scale ECCVApps. Under ECCVideo, we design the application architecture of ECCVApps, including presentation paradigm, transparent communication services, and full lifecycle management. To validate the proposed system, a real-time object detection application is deployed on the ECCVideo prototype.

ALGORITHMS FOR TRACKING OBJECTS ON A VIDEO STREAM

Algorithms for monitoring moving objects are of great interest in the field of computer vision. The paper presents the results of the development and research of image analysis algorithms for monitoring vehicles in a video stream. Currently, the monitoring of vehicles, especially in production areas, is performed manually by operators. Developing practical applications of video analytics will reduce operator workload, increase speed and accuracy of decision making, and automate the process of collecting and generating reporting statistics on plant transportation. Research was conducted and a video analysis system was developed that includes detection algorithms, classifications by vehicle type, color, and model, and vehicle tracking in the video stream. It should be noted that the architecture of existing solutions, as a rule, is monolithic and does not allow to refine and embed new unique modules to solve emerging problems. In modern conditions, in order to cover new requirements in the field of object recognition and detection in large vehicle flows, it is necessary to develop and implement new solutions. The results of the work of the algorithms on real images of existing datasets and the developed own dataset are presented. In this case, the objects are under different lighting conditions, in different angles, are rebuilt in the stream, there is an overlap of objects. The developed system has a microservice architecture, which allows to adapt the solutions to specific tasks.

Towards a resource efficient and privacy-preserving framework for campus-wide video analytics-based applications

Video surveillance and analytics solutions based on Artificial Intelligence (AI) are increasingly being deployed across industries, including academia. There are a number of use-cases for campus-wide video analytics applications. Detecting events of interest in real-time and generating alerts is a core requirement for such applications, making them both network and compute intensive. Thus, the underlying framework needs to be resource optimized in terms of latency, compute and storage requirements for a multitude of video applications. Increasingly privacy concerns have been voiced against the pervasive deployment of video analytics-based applications. Thus, protecting the privacy of students and staff in a campus setting shall be a major design consideration for such systems going forward. This paper presents a resource optimized and privacy preserving framework for campus-wide video analytics applications. Several use-cases are presented and early results from the deployment of the proposed framework establish its feasibility and effectiveness.

Serverless Empowered Video Analytics for Ubiquitous Networked Cameras

Ubiquitous deployment of networked cameras has boosted the prevalence of video analytics. Leveraging machines to automatically analyze captured videos has become the driving force behind a variety of contemporary applications. However, it is known to be resource-hungry with highly dynamic demands, which mismatches the existing monolithic cloud service deployment with coarse-grained resource allocation. Recent advances in serverless computing, which offers ultra-fast and fine-grained autoscaling, would become a game-changer. This article closely examines the potential and challenges of serverless computing in building modern video analytics applications. We accordingly present an integrated framework with geo-distributed resources, and identify the critical design issues toward its implementation. We further discuss a series of promising research directions in this field.

Tracking more than 100 arbitrary objects at 25 FPS through deep learning

Most video analytics applications rely on object detectors to localize objects in frames. However, when real-time is a requirement, running the detector at all the frames is usually not possible. This is somewhat circumvented by instantiating visual object trackers between detector calls, but this does not scale with the number of objects. To tackle this problem, we present SiamMT, a new deep learning multiple visual object tracking solution that applies single-object tracking principles to multiple arbitrary objects in real-time. To achieve this, SiamMT reuses feature computations, implements a novel crop-and-resize operator, and defines a new and efficient pairwise similarity operator. SiamMT naturally scales up to several dozens of targets, reaching 25 fps with 122 simultaneous objects for VGA videos, or up to 100 simultaneous objects in HD720 video. SiamMT has been validated on five large real-time benchmarks, achieving leading performance against current state-of-the-art trackers.

Deep-Framework: A Distributed, Scalable, and Edge-Oriented Framework for Real-Time Analysis of Video Streams.

Edge computing is the best approach for meeting the exponential demand and the real-time requirements of many video analytics applications. Since most of the recent advances regarding the extraction of information from images and video rely on computation heavy deep learning algorithms, there is a growing need for solutions that allow the deployment and use of new models on scalable and flexible edge architectures. In this work, we present Deep-Framework, a novel open source framework for developing edge-oriented real-time video analytics applications based on deep learning. Deep-Framework has a scalable multi-stream architecture based on Docker and abstracts away from the user the complexity of cluster configuration, orchestration of services, and GPU resources allocation. It provides Python interfaces for integrating deep learning models developed with the most popular frameworks and also provides high-level APIs based on standard HTTP and WebRTC interfaces for consuming the extracted video data on clients running on browsers or any other web-based platform.

Performance characterization of video analytics workloads in heterogeneous edge infrastructures

SummaryPowered by deep learning, video analytic applications process millions of camera feeds in real‐time to extract meaningful information from their surroundings. And this number grows by the minute. To avoid saturating the backhaul network and provide lower latencies, a distributed and heterogeneous edge cloud is postulated as a key enabler for widespread video analytics. This article provides a complete characterization of end‐to‐end video analytics across a set of hardware platforms and different neural network architectures. Each platform is selected to fill a different gap in a distributed, shared, and heterogeneous infrastructure. Moreover, we analyze how performance scales on each of these platforms with respect to the amount of resources dedicated to video analytics. Finally, we extract the key conclusions of the characterization to build an experimental model to estimate performance and cost of end‐to‐end video analytics in different edge scenarios. Our experiments show that managing video analytics workloads efficiently requires awareness of both, the platforms in which these are executed, and the full end‐to‐end pipeline. To the best of our knowledge, this is the first work that provides a complete characterization of end‐to‐end video analytics in heterogeneous edge platforms.

Real-Time Multiple Object Visual Tracking for Embedded GPU Systems

Real-time visual object tracking provides every object of interest with a unique identity and a trajectory across video frames. This is a fundamental task of many video analytics applications, such as traffic monitoring or video surveillance in general. The development of real-time multiple object tracking systems on low-power edge devices as IoT nodes, without compromising accuracy, is a challenge due to the limited computing capacity of said devices. This might rule out the best in-class computer vision solutions, which, nowadays, are based on deep learning, and thus, they are very hardware demanding. This article meets this challenge with a multiple object detection and tracking system that employs cutting-edge deep learning architectures on an embedded GPU while operating in real time. For this purpose, a system has been designed that extends a joint architecture of tracking and detection by adding a module comprised of appearance-based and movement-based trackers that allow to maintain the identity of the objects of interest for longer periods of time while alleviating the burden of the detector. Our system is mapped onto an embedded GPU platform, cutting down power consumption significantly with respect to a server GPU. Tracking performance metrics show a 51.1% in multiple object tracking accuracy (MOTA) on the MOT16 data set. This, in conjunction with a real-time processing speed of 25.2 FPS for up to 45 simultaneous objects and low-power consumption of 15 W, make our system an ideal solution for a wide range of video analytics applications.

Edge-based Video Analytic for Smart Cities

Video analytic is the important tool for smart city development. The video analytic application requires more memories and high processing devices. The problems of cloud-based approach for video analytic are high latency and more network bandwidth to transfer data into the cloud. To overcome these problems, we propose a model based on dividing the jobs into smaller sub-tasks with less processing requirements in a typical video analytics application for the development of smart city. The object detection, tracking and pattern recognition method to reduce the size of videos based on edge network will be proposed. We will design a video analytic model, and simulation is performed using iFogSim simulator. We will also propose Convolutional Neural Network (CNN) based object tracking model. The experimental verification shows that our tracking model is more than 96% accurate, and the proposed edge and cloud-based model is more than 80% effective than only cloud-based approach for video analytic applications.

Video Based Person Re Identification Methods, Datasets, and Deep Learning

Video Analytics applications like security and surveillance face a critical problem of person re-identification abbreviated as re-ID. The last decade witnessed the emergence of large-scale datasets and deep learning methods to use these huge data volumes. Most current re-ID methods are classified into either image-based or video-based re-ID. Matching persons across multiple camera views have attracted lots of recent research attention. Feature representation and metric learning are major issues for person re-identification. The focus of re-ID work is now shifting towards developing end-to-end re-Id and tracking systems for practical use with dynamic datasets. Most previous works contributed to the significant progress of person re-identification on still images using image retrieval models. This survey considers the more informative and challenging video-based person re-ID problem, pedestrian re-ID in particular. Publicly available datasets and codes are listed as a part of this work. Current trends which include open re-identification systems, use of discriminative features and deep learning is marching towards new applications in security and surveillance, typically for tracking.

A Novel Edge-Cloud Interworking Framework in the Video Analytics of the Internet of Things

This letter proposes a novel edge-cloud interworking framework in the video analytics of the Internet of Things (IoT) that consists of cost-effective job load balancing and scheduling schemes for computation-intensive video analytics applications. The proposed framework aims to minimize the cost of cloud resource usage while guaranteeing deadlines when conducting concurrent operations. A formulation of a two-stage mixed-integer problem and its heuristic greedy algorithms is presented, which captures all intertwined goals. From the numerical analysis, we reveal that the proposed framework outperforms the existing schemes in terms of monetary cost and service latency with a practical complexity bound.

Analysis of Customer Emotion from Video based Feedback of a Product

Due to the high levels of competition in a global market, companies have put more effort on building strong customer relationships and increasing customer satisfaction levels. Now-a-days due to technological improvements in information and communication technologies gives a highly anticipated key contributor to improve the customer experience and satisfaction in service episodes is through the application of video analytics, such as to evaluate the customer’s emotions over the complete service cycle. Currently, emotion recognition from video could be a difficult analysis space. One of the foremost effective solutions to deal with this challenge is to utilize each audio and visual part as two sources contained within the video knowledge to form an overall assessment of the emotion. The combined use of audio and visual knowledge sources presents further challenges, such as determining the optimal data fusion technique prior to classification. In this paper, we propose an audio–visual emotion recognition system to detect the universal six emotions (happy, angry, sad, disgust, surprise, and fear) from video data. The detected customer emotions are then mapped and translated to provide client satisfaction scores. The projected client satisfaction video analytics system will operate over video conferencing or video chat. The effectiveness of our proposal is verified through numerical results.

A Comprehensive Survey of Video Datasets for Background Subtraction

Background subtraction is an effective method of choice when it comes to detection of moving objects in videos and has been recognized as a breakthrough for the wide range of applications of intelligent video analytics (IVA). In recent years, a number of video datasets intended for background subtraction have been created to address the problem of large realistic datasets with accurate ground truth. The use of these datasets enables qualitative as well as quantitative comparisons and allows benchmarking of different algorithms. Finding the appropriate dataset is generally a cumbersome task for an exhaustive evaluation of algorithms. Therefore, we systematically survey standard video datasets and list their applicability for different applications. This paper presents a comprehensive account of public video datasets for background subtraction and attempts to cover the lack of a detailed description of each dataset. The video datasets are presented in chronological order of their appearance. Current trends of deep learning in background subtraction along with top-ranked background subtraction methods are also discussed in this paper. The survey introduced in this paper will assist researchers of the computer vision community in the selection of appropriate video dataset to evaluate their algorithms on the basis of challenging scenarios that exist in both indoor and outdoor environments.

Video Analytics for Customer Emotion and Satisfaction at Contact Centers

Due to the high levels of competition in a global market, companies have put more emphasis on building strong customer relationships and increasing customer satisfaction levels. With technological improvements in information and communication technologies, a highly anticipated key contributor to improve the customer experience and satisfaction in service episodes is through the application of video analytics, such as to evaluate the customer's emotions over the full service cycle. Currently, emotion recognition from video is a challenging research area. One of the most effective solutions to address this challenge is to utilize both the audio and visual components as two sources contained in the video data to make an overall assessment of the emotion. The combined use of audio and visual data sources presents additional challenges, such as determining the optimal data fusion technique prior to classification. In this paper, we propose an audio-visual emotion recognition system to detect the universal six emotions (happy, angry, sad, disgust, surprise, and fear) from video data. The detected customer emotions are then mapped and translated to give customer satisfaction scores. The proposed customer satisfaction video analytics system can operate over video conferencing or video chat. The effectiveness of our proposal is verified through numerical results.

A Systematic Evaluation and Benchmark for Person Re-Identification: Features, Metrics, and Datasets.

Person re-identification (re-id) is a critical problem in video analytics applications such as security and surveillance. The public release of several datasets and code for vision algorithms has facilitated rapid progress in this area over the last few years. However, directly comparing re-id algorithms reported in the literature has become difficult since a wide variety of features, experimental protocols, and evaluation metrics are employed. In order to address this need, we present an extensive review and performance evaluation of single- and multi-shot re-id algorithms. The experimental protocol incorporates the most recent advances in both feature extraction and metric learning. To ensure a fair comparison, all of the approaches were implemented using a unified code library that includes 11 feature extraction algorithms and 22 metric learning and ranking techniques. All approaches were evaluated using a new large-scale dataset that closely mimics a real-world problem setting, in addition to 16 other publicly available datasets: VIPeR, GRID, CAVIAR, DukeMTMC4ReID, 3DPeS, PRID, V47, WARD, SAIVT-SoftBio, CUHK01, CHUK02, CUHK03, RAiD, iLIDSVID, HDA+, and Market1501. The evaluation codebase and results will be made publicly available for community use.