Real-world Video Surveillance Research Articles

Background separation network for video anomaly detection

Video anomaly detection refers to the automatic identification of abnormal behaviors, objects, or events in videos. However, current methods for anomaly detection based on original frames lack a comprehensive understanding of the importance of foreground information, making it challenging to efficiently address video anomaly detection in the presence of complex background interference. In this paper, we propose a video anomaly detection algorithm based on Background Separation Network (BSN) to address this issue. Firstly, we utilize a video stabilization algorithm to reduce video jitter and enhance the quality of input video frames. Secondly, BSN shifts the focus from the entire frame to the foreground region with higher anomaly detection value. BSN utilizes the motion pixel distribution of the video as the basis for foreground extraction, enabling pixel-level background separation to obtain more accurate and complete foreground targets. Lastly, a certain proportion of foreground targets in the foreground image are masked as background, reducing the interference caused by redundant targets on the detection results. The proposed method achieves an accuracy of 96.2% on the UCSD ped2 dataset, demonstrating its effectiveness. This method contributes to accurately detecting abnormal behaviors in real-world surveillance videos to protect the safety of public lives and assets.

Anomaly Detection in Heterogeneous Data Using Graph Convolutional Networks

Closed-circuit television (CCTV) surveillance systems are ubiquitous tools for ensuring public and private safety. However, manual monitoring of surveillance footage is arduous and time-consuming, often resulting in delayed identification and response to potential threats. In this paper, we present a novel real-time threat detection system tailored for CCTV surveillance, employing cutting-edge deep learning methodologies, particularly graph convolutional networks (GCNs), to detect and classify high-movement levels within video frames. By treating videos as segments and delineating between anomalous (potentially threatening) and normal (safe) segments, our system facilitates continuous real-time monitoring of surveillance footage, enabling the timely identification of potential threats such as abuse, burglaries, explosions, shootings, fighting, shoplifting, road accidents, arson, robbery, stealing, assault, and vandalism. To assess the efficacy of our system, we conducted extensive experiments on a large-scale dataset comprising CCTV footage, yielding promising results. Additionally, we introduce a novel, comprehensive dataset comprising 128 hours of real-world surveillance videos, encompassing various anomalies and normal activities. This dataset serves dual purposes: general anomaly detection, and the recognition of specific anomalous activities. Our experimental findings demonstrate the superior performance of our method in anomaly detection compared to state-of-the-art approaches. Furthermore, we provide an analysis of several recent deep learning baselines for anomalous activity recognition, revealing the challenging nature of our dataset and the potential it holds for future research endeavors. Through the integration of advanced deep learning techniques and comprehensive dataset creation, our proposed system stands to significantly enhance the efficiency and effectiveness of CCTV surveillance, facilitating faster response times and bolstering security for individuals in diverse environments.

CARgram: CNN-based accident recognition from road sounds through intensity-projected spectrogram analysis

Road surveillance systems play an important role in traffic monitoring and detecting hazardous events. In recent years, several artificial intelligence-based approaches have been proposed for this purpose, typically based on the analysis of the acquired video streams. However, occlusions, poor lighting conditions, and heterogeneity of the events may often reduce their effectiveness and reliability. To overcome the limitations mentioned, scientific and industrial research has therefore focused on integrating such solutions with audio recognition methods. By automatically identifying anomalous traffic sounds, e.g., car crashes and skids, they help reduce false positives and missed alarms. Following this trend, in this work, we propose an innovative pipeline for the analysis of intensity-projected audio spectrograms from streams of traffic sounds, which exploits both (i) a visual approach based on a custom, special-purpose Convolutional Neural Network for the identification of anomalous events on the sound signal; and, (ii) a novel multi-representational encoding of the input, which proved to significantly improve the recognition accuracy of the neural models. The validation results of the proposed pipeline on the public MIVIA dataset, with a 0.96% of false positive rate, showed to be the best performance against the state-of-the-art competitors. Notably, following such findings, a prototype implementation has been deployed on a real-world video surveillance infrastructure.

Collaborative Multi-Agent Video Fast-Forwarding

Multi-agent applications have recently gained significant popularity. In many computer vision tasks, a network of agents, such as a team of robots with cameras, could work collaboratively to perceive the environment for efficient and accurate situation awareness. However, these agents often have limited computation, communication, and storage resources. Thus, reducing resource consumption while still providing an accurate perception of the environment becomes an important goal when deploying multi-agent systems. To achieve this goal, we identify and leverage the overlap among different camera views in multi-agent systems for reducing the processing, transmission and storage of redundant/unimportant video frames. Specifically, we have developed two collaborative multi-agent video fast-forwarding frameworks in distributed and centralized settings, respectively. In these frameworks, each individual agent can selectively process or skip video frames at adjustable paces based on multiple strategies via reinforcement learning. Multiple agents then collaboratively sense the environment via either 1) a consensus-based distributed framework called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DMVF</b> that periodically updates the fast-forwarding strategies of agents by establishing communication and consensus among connected neighbors, or 2) a centralized framework called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MFFNet</b> that utilizes a central controller to decide the fast-forwarding strategies for agents based on collected data. We demonstrate the efficacy and efficiency of our proposed frameworks on a real-world surveillance video dataset VideoWeb and a new simulated driving dataset CarlaSim, through extensive simulations and deployment on an embedded platform with TCP communication. We show that compared with other approaches in the literature, our frameworks achieve better coverage of important frames, while significantly reducing the number of frames processed at each agent.

Introducing shape priors in Siamese networks for image classification

The efficiency of deep neural networks is increasing, and so is the amount of annotated data required for training them. We propose a solution improving the learning process of a classification network with less labeled data. Our approach is to inform the classifier of the elements it should focus on to make its decision by supplying it with some shape priors. These shape priors are expressed as binary masks, giving a rough idea of the shape of the relevant elements for a given class. We resort to Siamese architecture and feed it with image/mask pairs. By inserting shape priors, only the relevant features are retained. This provides the network with significant generalization power without requiring a specific domain adaptation step. This solution is tested on some standard cross-domain digit classification tasks and on a real-world video surveillance application. Extensive tests show that our approach outperforms the classical classifier by generating a good latent space with less training data. Code is available at https://github.com/halqasir/MG-Siamese.

A new framework for deep learning video based Human Action Recognition on the edge

Nowadays, video surveillance systems are commonly found in most public and private spaces. These systems typically consist of a network of cameras that feed into a central node. However, the processing aspect is evolving towards distributed approaches, leveraging edge-computing. These distributed systems are capable of effectively addressing the detection of people or events at each individual node. Most of these systems, rely on the use of deep-learning and segmentation algorithms which enable them to achieve high performance, but usually with a significant computational cost, hindering real-time execution. This paper presents an approach for people detection and action recognition in the wild, optimized for running on the edge, and that is able to work in real-time, in an embedded platform. Human Action Recognition (HAR) is performed by using a Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM). The input to the LSTM is an ad-hoc, lightweight feature vector obtained from the bounding box of each detected person in the video surveillance image. The resulting system is highly portable and easily scalable, providing a powerful tool for real-world video surveillance applications (in the wild and real-time action recognition). The proposal has been exhaustively evaluated and compared against other state-of-the-art (SOTA) proposals in five datasets, including four widely used (KTH, WEIZMAN, WVU, IXMAX) and a novel one (GBA) recorded in the wild, that includes several people performing different actions simultaneously. The obtained results validate the proposal, since it achieves SOTA accuracy within a much more complicated video surveillance real scenario, and using a lightweight embedded hardware.

Hybrid Attention and Motion Constraint for Anomaly Detection in Crowded Scenes

Crowds often appear in surveillance videos in public places, from which anomaly detection is of great importance to public safety. Since the abnormal cases are rare, variable and unpredictable, autoencoders with encoder and decoder structures using only normal samples have become a hot topic among various approaches for anomaly detection. However, since autoencoders have excessive generalization ability, they can sometimes still reconstruct abnormal cases very well. Recently, some researchers construct memory modules under normal conditions and use these normal memory items to reconstruct test samples during inference to increase the reconstruction errors for anomalies. However, in practice, the errors of reconstructing normal samples with the memory items often increase as well, which makes it still difficult to distinguish between normal and abnormal cases. In addition, the memory-based autoencoder is usually available only in the specific scene where the memory module is constructed and almost loses the prospect of cross-scene applications. We mitigate the overgeneralization of autoencoders from a different perspective, namely, by reducing the prediction errors for normal cases rather than increasing the prediction errors for abnormal cases. To this end, we propose an autoencoder based on hybrid attention and motion constraint for anomaly detection. The hybrid attention includes the channel attention used in the encoding process and spatial attention added to the skip connection between the encoder and decoder. The hybrid attention is introduced to reduce the weight of the feature channels and regions representing the background in the feature matrix, which makes the autoencoder features more focused on optimizing the representation of the normal targets during training. Furthermore, we introduce motion constraint to improve the autoencoder’s ability to predict normal activities in crowded scenes. We conduct experiments on real-world surveillance videos, UCSD, CUHK Avenue, and ShanghaiTech datasets. The experimental results indicate that the prediction errors of the proposed method for frequent normal crowd activities are smaller than those of other approaches, which increases the gap between the prediction errors for normal frames and the prediction errors for abnormal frames. In addition, the proposed method does not depend on a specific scene. Therefore, it balances good anomaly detection performance and strong cross-scene capability.

Video-Based Vehicle Counting and Analysis using YOLOv5 and DeepSORT with Deployment on Jetson Nano

In recent years, the advancements in deep learning and high-performance edge-computing systems have increased tremendously and have become the center of attention when it comes to the analysis of video-based systems on edge by making use of computer vision techniques. Intelligent Transportation Systems (ITS) is one area where deep learning can be used for several tasks including highway-based vehicle counting systems whereby making use of computer vision techniques, an edge computing device and cameras installed in specific locations on the road, we are able to obtain very accurate vehicle counting results and replace the use of traditional and laborious hardware devices with modern low-cost solutions. This paper proposes and implements a modern, compact, and reliable vehicle counting system based on the most recent and popular object detection algorithm as of writing this paper, known as the YOLOv5, combined with a state-of-the-art object tracking algorithm known as DeepSORT. The YOLOv5 will be used in the following system to detect and classify four different classes of vehicles, whereas DeepSORT will be used to track those vehicles across different frames in the video sequence. Finally, a unique and efficient vehicle counting method will be implemented and used to count tracked vehicles across the highway scenes. A new highway vehicle dataset consisting of four vehicle classes, namely: car, motorcycle, bus, and truck, was collected, cleaned, and annotated with a total of 11,982 images published in the following study and used for the training of our robust vehicle detection model. From the results obtained over real-world highway surveillance videos, the following system was able to obtain an average vehicle detection mAP score of 96.1% and a vehicle counting accuracy of 95.39%, all while being able to be deployed on a compact Nvidia Jetson Nano edge-computing device with an average speed of 15 FPS which outperforms other previously proposed tools in terms of both accuracy and speed.

Deep learning based detection and localization of road accidents from traffic surveillance videos

Real-world traffic surveillance videos need continuous supervision to monitor and take appropriate actions in case of fatal accidents. However, continuously monitoring them with human supervision is error prone and tedious. Therefore, a deep learning approach for automatic detection and localization of road accidents has been proposed by formulating the problem as anomaly detection. The method follows one-class classification approach and applies spatio-temporal autoencoder and sequence-to-sequence long short-term memory autoencoder for modeling spatial and temporal representations in the video. The model is executed on a real-world video traffic surveillance datasets and significant results have been achieved both qualitatively and quantitatively.

An Efficient Person Search Method Using Spatio-Temporal Features for Surveillance Videos

Existing person search methods mainly focus on searching for the target person using database images. However, this is different from real-world surveillance videos which involve a temporal relationship between video frames. To solve this problem, we propose an efficient person search method that employs spatio-temporal features in surveillance videos. This method not only considers the spatial features of persons in each frame, but also utilizes the temporal relationship of the same person between adjacent frames. For this purpose, the spatial features are extracted by combining Yolo network with Resnet-50 model, and the temporal relationship is processed by gated recurrent unit. The spatio-temporal features are generated by the following average pooling layer and used to represent persons in the videos. To ensure search efficiency, locality sensitive hashing is used to organize massive spatio-temporal features and calculate the similarity. A surveillance video database is also constructed to evaluate the proposed method, and the experimental results demonstrate that our method improves search accuracy while ensuring search efficiency.

Deep Reinforcement Learning-based anomaly detection for Video Surveillance

The anomaly detection in automated video surveillance is considered as one of the most critical tasks to be solved, in which we aim to detect a variety of real-world abnormalities. This paper introduces a novel approach for anomaly detection based on deep reinforcement learning. In recent years, deep reinforcement learning has been achieving a significant success in various applications with data of a high degree of complexity such as robotics and games, by mimicking the way humans learn from experiences. Generally, the state-of-the-art methods classify a video as normal or abnormal without pinpointing the exact location of the anomaly in the input video due to the unlabeled clip-level data in training videos. We focus on adapting the prioritized Dueling deep Q-networks to the anomaly detection problem. This model learns to evaluate the anomaly in video clips by exploiting the video-level label to obtain a better detection accuracy. Extensive experiments on 13 cases class of real-word anomaly show that our DRL agent achieved a near optimal performance with a high accuracy in the real world video surveillance system compared to the state-of-the-art approaches.

Eye of Devil: Face Recognition in Real World Surveillance Video with Feature Extraction and Pattern Matching

Abstract: People lost, people missing etc., these are the words we come across whenever there is any mass gathering events going on or in crowded areas. To solve this issue some traditional approaches like announcements are in use. One idea is to identify the person using face recognition and pattern matching techniques. There are several techniques to implement face recognition like extraction of facial features by using the position of eyes, nose, jawbone or skin texture analysis etc., By using these techniques a unique dataset can be created for each human. Here the photograph of the missing person can be used to extract these facial features. After getting the dataset of that individual, by using pattern matching techniques, there is a scope to find the person with same facial features in the crowd images or videos. Keywords: Face-Recognition, Image-Processing, Feature extraction, Video-Processing, Pattern-Matching.

Toward Intelligent Surveillance as an Edge Network Service (iSENSE) Using Lightweight Detection and Tracking Algorithms

Edge computing extends the realm of information technology beyond the boundaries defined by cloud computing. Performing computation near the sensors, edge computing is promising to address the challenges in many bandwidth-and delay-sensitive applications. Although recently many smart video surveillance approaches based on Machine Learning (ML) algorithms become available, it is still challenging to efficiently migrate those smart algorithms to edge. In this paper, we propose an intelligent Surveillance as an Edge Network Service (iSENSE), which explores the feasibility of moving ML to the edge by testing two popular human-object detection schemes. Besides, a lightweight Convolutional Neural Network (L-CNN) is introduced to improve computational execution by leveraging the depth-wise separable convolution. To enhance performance on edge, we propose a hybrid lightweight tracking algorithm, Kerman (Kernelized Kalman filter), which is a decision tree based hybrid Kernelized Correlation Filter algorithm designed for human-object tracking. We have implemented both Kerman and L-CNN algorithms on edge by using different types of single board computers. The proposed iSENSE system was validated using both real-world campus surveillance video and open image sets. The experimental results present that the proposed algorithms can track the human objects in real-time with a good accuracy with limited resource in edge devices.

Analyzing drivers’ hazard recognition: Precursors to single-vehicle collisions

Extensive driver behavior and performance information provided by real-world video surveillance and sensor data in the SHRP2 Naturalistic Driving Study has enabled the examination of new layers and pathways leading to crash outcomes. We note that the prominence of hazards and the importance of recognizing them vary systematically across single vs. multi-vehicle crashes, and address a fundamental question about safety: why do around three-quarters of drivers involved in single-vehicle crashes not recognize, perceive, or react to the precipitating event (PE)? Using a path-analytic framework through marginal effects, this study investigates factors correlated to recognition of the PE in single-vehicle events, and how these correlations may act as crash precursors. Logit models, accounting for heterogeneity among events and drivers by estimating both fixed and random parameters, quantified correlations among key variables, given a crash or near-crash event (N = 543). The type of PE, roadway environment factors, and driving maneuvers heavily influenced recognition chances. Drivers had a harder time recognizing less conspicuous hazards (e.g. departing the travel way, decreased recognition chances by 48.29%), but seemed better at recognizing prominent hazards (e.g. vehicle losing control, increased recognition chances by 46.71%). In addition, drivers are less likely to recognize PEs when executing less involved driving maneuvers in more relaxed environments, such as daylight (decreased recognition chances by 16.00%), but are more adept in environments that already demand more attention. Recognition reduced the chances of a crash by 12.23%, so we found similar correlations with crash outcome. Future intelligent transportation systems may focus on increasing driver recognition of potential hazards by bringing attention to less conspicuous hazards and less involved driving environments and actions.

Human face super-resolution on poor quality surveillance video footage

Most super-resolution (SR) methods proposed to date do not use real ground-truth high-resolution (HR) and low-resolution (LR) image pairs; instead, the vast majority of methods use synthetic LR images generated from the HR images. This approach yields excellent performance on synthetic datasets, but on real-world poor quality surveillance video footage, they suffer from performance degradation. A promising alternative is to apply recent advances in style transfer for unpaired datasets, but state-of-the-art work along these lines has used LR images and HR images from completely different datasets, introducing more variation between the HR and LR domains than necessary. In this paper, we propose methods that overcome both of these shortcomings, applying unpaired style transfer learning methods to face SR but using HR and LR datasets that share important properties. The key is to acquire roughly paired training data from a high-quality main stream and a lower-quality sub-stream of the same IP camera. Based on this principle, we have constructed four datasets comprising more than 400 people, with 1–15 weakly aligned real HR–LR pairs for each subject. We adopt a cycle generative adversarial networks (Cycle GANs) approach that produces impressive super-resolved images for low-quality test images never seen during training. Experiments prove the efficacy of the method. The approach to face SR advocated for in this paper makes possible many real-world applications requiring the extraction of high-quality face images from low-resolution video streams such as those produced by security cameras. Developers of diverse applications such as face recognition, 3D face reconstruction, face alignment, face parsing, human–computer interaction, remote sensing, and access control will benefit from the methods introduced in this work.

A study on attention-based LSTM for abnormal behavior recognition with variable pooling

Behavior recognition is a well-known computer vision mobile technology. It has been used in many applications such as video surveillance, motion detection on devices, human-computer interaction and sports video, etc. However, most of the existing works ignored the depth and spatio-temporal information so that they resulted in over-fitting and inferior performance. Consequently, a novel framework for behavior recognition is proposed in this paper. In this framework, we propose a target depth estimation algorithm to calculate the 3D spatial position information of the target, and take this information as the input of the behavior recognition model. Simultaneously, in order to obtain more Spatio-temporal information and better handle long-term video, combining with the idea of attention mechanism, we propose a skeleton behavior recognition model which is based on spatio-temporal convolution and attention-based LSTM (ST-CNN & ATT-LSTM). The deep spatial information is merged into each segment, and the model focuses on the key information extraction, which is essential for improving behavior recognition performance. Meanwhile, we use a feature compression method based on variable pooling to solve the problem of inconsistent input sizes caused by multi-person behavior recognition, so that the network can flexibly recognize multi-person skeleton sequences. Finally, the proposed framework is evaluated with real-world surveillance video data, and the results indicate that our framework is superior to existing methods.

Scene Adaptive Online Surveillance Video Synopsis via Dynamic Tube Rearrangement Using Octree.

Visual surveillance produces a significant amount of raw video data that can be time consuming to browse and analyze. In this work, we present a video synopsis methodology called "scene adaptive online video synopsis via dynamic tube rearrangement using octree (SSOcT)" that can effectively condense input surveillance videos. Our method entailed summarizing the input video by analyzing scene characteristics and determining an effective spatio-temporal 3D structure for video synopsis. For this purpose, we first analyzed the attributes of each extracted tube with respect to scene geometry and complexity. Then, we adaptively grouped the tubes using an online grouping algorithm that exploits these scene characteristics. Finally, the tube groups were dynamically rearranged using the proposed octree-based algorithm that efficiently inserted and refined tubes containing high spatio-temporal movements in real time. Extensive video synopsis experimental results are provided, demonstrating the effectiveness and efficiency of our method in summarizing real-world surveillance videos with diverse scene characteristics.

Model-Based Transfer Learning and Sparse Coding for Partial Face Recognition

With the growing needs of practical applications such as security monitoring, partial face recognition is a challenging but important issue, because the captured faces in real-world surveillance videos may be occluded or with variations. Though current face recognition methods perform well in relatively constrained scenes, they may suffer from degradation for partial faces. In this paper, we propose a framework of model-based transfer learning and sparse coding (MTLSC) for partial face recognition. First, due to less information in partial face image, we exploit the mirrored image of an original probe sample as sample augment to provide further information. Considering the inadequacy of training face samples, we obtain face features based on model-based transfer learning VGGNet that is pre-trained on VGGFace dataset. Then we reconstruct face features by sliding window in view of different sizes of partial face hard to extract the same feature dimension. Finally we carry out sparse coding with rectification and calculate the minimum score of the probe and mirrored samples among all classes to get the results. Thus, by model-based transfer learning, sliding window for feature reconstruction and sparse coding with rectification, the proposed framework improves partial face recognition performance. Experimental results on three face databases (LFW, AR and NIR), and two person re-identification databases (iLIDS-VID and PKU-Reid) demonstrate our method is effective for partial face recognition.

Towards the sustainable development of smart cities through mass video surveillance: A response to the COVID-19 pandemic

Sustainable smart city initiatives around the world have recently had great impact on the lives of citizens and brought significant changes to society. More precisely, data-driven smart applications that efficiently manage sparse resources are offering a futuristic vision of smart, efficient, and secure city operations. However, the ongoing COVID-19 pandemic has revealed the limitations of existing smart city deployment; hence; the development of systems and architectures capable of providing fast and effective mechanisms to limit further spread of the virus has become paramount. An active surveillance system capable of monitoring and enforcing social distancing between people can effectively slow the spread of this deadly virus. In this paper, we propose a data-driven deep learning-based framework for the sustainable development of a smart city, offering a timely response to combat the COVID-19 pandemic through mass video surveillance. To implementing social distancing monitoring, we used three deep learning-based real-time object detection models for the detection of people in videos captured with a monocular camera. We validated the performance of our system using a real-world video surveillance dataset for effective deployment.

Deep anomaly detection through visual attention in surveillance videos

This paper describes a method for learning anomaly behavior in the video by finding an attention region from spatiotemporal information, in contrast to the full-frame learning. In our proposed method, a robust background subtraction (BG) for extracting motion, indicating the location of attention regions is employed. The resulting regions are finally fed into a three-dimensional Convolutional Neural Network (3D CNN). Specifically, by taking advantage of C3D (Convolution 3-dimensional), to completely exploit spatiotemporal relation, a deep convolution network is developed to distinguish normal and anomalous events. Our system is trained and tested against a large-scale UCF-Crime anomaly dataset for validating its effectiveness. This dataset contains 1900 long and untrimmed real-world surveillance videos and splits into 950 anomaly events and 950 normal events, respectively. In total, there are approximately ~ 13 million frames are learned during the training and testing phase. As shown in the experiments section, in terms of accuracy, the proposed visual attention model can obtain 99.25 accuracies. From the industrial application point of view, the extraction of this attention region can assist the security officer on focusing on the corresponding anomaly region, instead of a wider, full-framed inspection.