Video Anomaly Research Articles

A New Unsupervised Video Anomaly Detection Using Multi-Scale Feature Memorization and Multipath Temporal Information Prediction

Anomaly detection in video is an advanced computer vision challenge that recognizes video segments containing out-of-the-ordinary motions or objects. Most recent techniques in video anomaly detection have focused on reconstruction and prediction methods; however, in practice, frame reconstruction methods deliver suboptimal results due to the outstanding generalization abilities of convolutional neural networks when reconstructing abnormal frames. Meanwhile, frame prediction methods have drawn much attention and are a powerful way of simulating the dynamics of natural scenes. This paper provides a new unsupervised frame prediction-based algorithm for anomaly detection that improves overall performance. Our suggested strategy follows a U-Net-like architecture that employs a Time-distributed 2D CNN-based encoder and 2D CNN-based decoder. A memory module is used in the design to retrieve and store the most relevant prototypical pattern of the normal scenario in the memory slots during training giving our model the capacity to produce poor predictions in the case of unusual input. For the memory module to fully retain normal semantic patterns on multiple scales, we propose an upstream multi-branch structure composed of dilated convolutions to extract contextual information. We also provide a multi-path structure that, as a great substitute for the optical flow loss function, directly includes temporal information into the network design. Experiments on the UCSD Ped1, UCSD Ped2, and CUHK Avenue benchmark datasets revealed that our design outperforms most competing models.

Contrastive-Regularized U-Net for Video Anomaly Detection

Video anomaly detection aims to identify anomalous segments in a video. It is typically trained with weakly supervised video-level labels. This paper focuses on two crucial factors affecting the performance of video anomaly detection models. First, we explore how to capture the local and global temporal dependencies more effectively. Previous architectures are effective at capturing either local and global information, but not both. We propose to employ a U-Net like structure to model both types of dependencies in a unified structure where the encoder learns global dependencies hierarchically on top of local ones; then the decoder propagates this global information back to the segment level for classification. Second, overfitting is a non-trivial issue for video anomaly detection due to limited training data. We propose weakly supervised contrastive regularization which adopts a feature-based approach to regularize the network. Contrastive regularization learns more generalizable features by enforcing inter-class separability and intra-class compactness. Extensive experiments on the UCF-Crime dataset shows that our approach outperforms several state-of-the-art methods.

OSIN: Object-Centric Scene Inference Network for Unsupervised Video Anomaly Detection

Video Anomaly Detection (VAD) is an essential yet challenging task in the signal processing community, which aims to understand the spatial and temporal contextual interactions between objects and surrounding scenes to detect unexpected events in surveillance videos. However, existing unsupervised methods either use a single network to learn global prototype patterns without making a unique distinction between foreground objects and background scenes or try to strip objects from frames, ignoring that the essence of anomalies lies in unusual object-scene interactions. To this end, this letter proposes an Object-centric Scene Inference Network (OSIN) that uses a well-designed three-stream structure to learn both global scene normality and local object-specific normal patterns as well as explore the object-scene interactions using scene memory networks. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed OSIN model, which achieves frame-level AUCs of 91.7%, 79.6%, and 98.3% on the CUHK Avenue, ShanghaiTech, and UCSD Ped2 datasets, respectively.

Learning Graph Enhanced Spatial-Temporal Coherence for Video Anomaly Detection

Video Anomaly Detection (VAD) is a critical yet challenging task in the signal processing community. Since part abnormal events cannot be detected by analyzing spatial or temporal information alone, learning spatial-temporal coherence has been proven the key to effective VAD. To this end, we propose a Graph Enhanced Spatial-Temporal Attention (GESTA) to address unsupervised VAD by learning the spatial-temporal coherence of normal events. Firstly, we propose a Dynamic Graph Recurrent Neural Network (DGRNN) to extract the motion patterns. Then, we propose a Spatial-Temporal Attention Module (STAM) to better model spatial-temporal coherence by integrating the prototypical spatial and temporal information. Finally, the fused spatial-temporal features are fed into the decoder to predict future frames. In testing phase, the anomaly with irregular information will result in poor prediction results. Experiments on three benchmarks demonstrate that our GESTA performs comparably to the state-of-the-art methods, and extensive analysis proves the effectiveness of DGRNN and STAM.

Weakly Supervised Video Anomaly Detection Combining Deep Features with Shallow Neural Networks

Deep features have outgrown hand-sketched features in many applications. The availability of pre-trained deep feature extractors helps to overcome one of the deep learning main drawbacks, which is the need for large volumes of data for training. Multiple Instance Learning (MIL) has become an attractive solution for video surveillance literature once it allows working with weakly labeled bases. This work evaluates a video anomaly detection approach based on the MIL paradigm combining deep features with shallow Neural Networks. For computational efficiency, we apply Principal Component Analysis (PCA) for dimensionality reduction before classification. We performed the experiments from a set of I3D (Inflated 3D) features, which corresponds to the ShanghaiTech benchmark dataset, and the MLP and SVM shallow classifiers achieved competitive results.

Bidirectional Spatio-Temporal Feature Learning With Multiscale Evaluation for Video Anomaly Detection

Video anomaly detection aims to detect the segments containing abnormal events from video sequence, which is a current research hotspot due to the importance in maintaining social security. Recent detection methods tend to build frame reconstruction or frame prediction model based on deep learning to learn features of events. The reconstruction-based methods reproduce the input frame one-to-one, inevitably losing some temporal features. The prediction-based methods predict frames according to the natural time order, but ignore the reverse time information, causing the deviation in information learning. Besides, anomaly evaluation methods based on patch-level error neglect the diversity of object sizes in complex scenes, and it is difficult to determine the optimal size of the error patch accurately. For these issues, we propose a bidirectional spatio-temporal feature learning framework with multi-scale anomaly evaluation strategy. A video sequence is input to a double-encoder double-decoder network, and bidirectional spatio-temporal features are obtained for bidirectional prediction by fusing forward and backward features extracted from the two encoders. The multi-scale anomaly evaluation method is implemented based on error pyramid and mean pooling, which effectively detects target objects with different sizes. Experiments on several publicly video datasets show that our method outperforms most of existing methods.

An efficient deep neural model for detecting crowd anomalies in videos

An efficient deep neural model for detecting crowd anomalies in videos

Future Frame Prediction Network for Video Anomaly Detection.

Video Anomaly detection in videos refers to the identification of events that do not conform to expected behavior. However, almost all existing methods cast this problem as the minimization of reconstruction errors of training data including only normal events, which may lead to self-reconstruction and cannot guarantee a larger reconstruction error for an abnormal event. In this paper, we propose to formulate the video anomaly detection problem within a regime of video prediction. We advocate that not all video prediction networks are suitable for video anomaly detection. Then, we introduce two principles for the design of a video prediction network for video anomaly detection. Based on them, we elaborately design a video prediction network with appearance and motion constraints for video anomaly detection. Further, to promote the generalization of the prediction-based video anomaly detection for novel scenes, we carefully investigate the usage of a meta learning within our framework, where our model can be fast adapted to a new testing scene with only a few starting frames. Extensive experiments on both a toy dataset and three real datasets validate the effectiveness of our method in terms of robustness to the uncertainty in normal events and the sensitivity to abnormal events.

Weakly-Supervised and Unsupervised Video Anomaly Detection

As surveillance technology is continuously improving, an ever-increasing number of cameras are being deployed everywhere. Relying on manual detection of anomalies through cameras may be unreliable and untimely. Therefore, the application of deep learning in video anomaly detection is being extensively studied. Anomaly Detection (AD) refers to identifying events that deviate from the desired actions. This article discusses representative unsupervised and weakly-supervised learning methods applied to various data types. In these machine learning methods, Generative Adversarial Network, Auto Encoder, Recurrent Neural Network, etc. are broadly adopted for AD. Some renowned and new datasets are reviewed. Furthermore, we also proposed several future directions of research in video anomaly detection.

Influence-Aware Attention Networks for Anomaly Detection in Surveillance Videos

Detecting anomalies in videos is a fundamental issue in public security. The majority of existing deep learning methods often perform anomaly detection based on the behavior or the trajectory of a single target. However, due to the overlaps of the crowd and the low-resolution of monitoring images, the segmentation of population is hard to implement and the features cannot be learned thoroughly, which make the methods be easily disturbed by visual elements and thus may lead to false detection sometimes. To tackle these problems, we propose the influence-aware attention to learn the representative attributes of the whole crowd. Walking pedestrians can be divided into numbers of flows, and in this paper, we aim to measure the consistency of movement patterns in the same stream and the interactions between different streams. Meanwhile, great importance is given to the relation between pedestrians and the circumstance for certain anomalies occur as a result of environmental issues. Specifically, the influence-aware attention module is composed of the motion attention and the location attention, which is designed to quantify the relations in the scene from spatial and temporal aspects. For the lack of abnormal samples, we utilize a dual generator-based framework to learn interactions among normal scenes. Experimental results on six benchmarks verify the effectiveness and robustness of our proposed method.

A Dynamic Convolutional Generative Adversarial Network for Video Anomaly Detection

A Dynamic Convolutional Generative Adversarial Network for Video Anomaly Detection

Comprehensive Regularization in a Bi-directional Predictive Network for Video Anomaly Detection

Video anomaly detection aims to automatically identify unusual objects or behaviours by learning from normal videos. Previous methods tend to use simplistic reconstruction or prediction constraints, which leads to the insufficiency of learned representations for normal data. As such, we propose a novel bi-directional architecture with three consistency constraints to comprehensively regularize the prediction task from pixel-wise, cross-modal, and temporal-sequence levels. First, predictive consistency is proposed to consider the symmetry property of motion and appearance in forwards and backwards time, which ensures the highly realistic appearance and motion predictions at the pixel-wise level. Second, association consistency considers the relevance between different modalities and uses one modality to regularize the prediction of another one. Finally, temporal consistency utilizes the relationship of the video sequence and ensures that the predictive network generates temporally consistent frames. During inference, the pattern of abnormal frames is unpredictable and will therefore cause higher prediction errors. Experiments show that our method outperforms advanced anomaly detectors and achieves state-of-the-art results on UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets.

Self-Training Multi-Sequence Learning with Transformer for Weakly Supervised Video Anomaly Detection

Weakly supervised Video Anomaly Detection (VAD) using Multi-Instance Learning (MIL) is usually based on the fact that the anomaly score of an abnormal snippet is higher than that of a normal snippet. In the beginning of training, due to the limited accuracy of the model, it is easy to select the wrong abnormal snippet. In order to reduce the probability of selection errors, we first propose a Multi-Sequence Learning (MSL) method and a hinge-based MSL ranking loss that uses a sequence composed of multiple snippets as an optimization unit. We then design a Transformer-based MSL network to learn both video-level anomaly probability and snippet-level anomaly scores. In the inference stage, we propose to use the video-level anomaly probability to suppress the fluctuation of snippet-level anomaly scores. Finally, since VAD needs to predict the snippet-level anomaly scores, by gradually reducing the length of selected sequence, we propose a self-training strategy to gradually refine the anomaly scores. Experimental results show that our method achieves significant improvements on ShanghaiTech, UCF-Crime, and XD-Violence.

A Causal Inference Look at Unsupervised Video Anomaly Detection

Unsupervised video anomaly detection, a task that requires no labeled normal/abnormal training data in any form, is challenging yet of great importance to both industrial applications and academic research. Existing methods typically follow an iterative pseudo label generation process. However, they lack a principled analysis of the impact of such pseudo label generation on training. Furthermore, the long-range temporal dependencies also has been overlooked, which is unreasonable since the definition of an abnormal event depends on the long-range temporal context. To this end, first, we propose a causal graph to analyze the confounding effect of the pseudo label generation process. Then, we introduce a simple yet effective causal inference based framework to disentangle the noisy pseudo label's impact. Finally, we perform counterfactual based model ensemble that blends long-range temporal context with local image context in inference to make final anomaly detection. Extensive experiments on six standard benchmark datasets show that our proposed method significantly outperforms previous state-of-the-art methods, demonstrating our framework's effectiveness.

Detection of Appearance and Behavior Anomalies in Stationary Camera Videos Using Convolutional Neural Networks

The automatic detection and tracking of appearance and behavior anomalies in video surveillance systems is one of the promising areas for the development and implementation of artificial intelligence. In this paper, we present a formalization of these problems. Based on the proposed generalization, a detection and tracking algorithm that uses the tracking-by-detection paradigm and convolutional neural networks (CNNs) is developed. At the first stage, people are detected using the YOLOv5 CNN and are marked with bounding boxes. Then, their faces in the selected regions are detected and the presence or absence of face masks is determined. Our approach to face-mask detection also uses YOLOv5 as a detector and classifier. For this problem, we generate a training dataset by combining the Kaggle dataset and a modified Wider Face dataset, in which face masks were superimposed on half of the images. To ensure a high accuracy of tracking and trajectory construction, the CNN features of the images are included in a composite descriptor, which also contains geometric and color features, to describe each person detected in the current frame and compare this person with all people detected in the next frame. The results of the experiments are presented, including some examples of frames from processed video sequences with visualized trajectories for loitering and falls.

Robust Unsupervised Video Anomaly Detection by Multipath Frame Prediction.

Video anomaly detection is commonly used in many applications, such as security surveillance, and is very challenging. A majority of recent video anomaly detection approaches utilize deep reconstruction models, but their performance is often suboptimal because of insufficient reconstruction error differences between normal and abnormal video frames in practice. Meanwhile, frame prediction-based anomaly detection methods have shown promising performance. In this article, we propose a novel and robust unsupervised video anomaly detection method by frame prediction with a proper design which is more in line with the characteristics of surveillance videos. The proposed method is equipped with a multipath ConvGRU-based frame prediction network that can better handle semantically informative objects and areas of different scales and capture spatial-temporal dependencies in normal videos. A noise tolerance loss is introduced during training to mitigate the interference caused by background noise. Extensive experiments have been conducted on the CUHK Avenue, ShanghaiTech Campus, and UCSD Pedestrian datasets, and the results show that our proposed method outperforms existing state-of-the-art approaches. Remarkably, our proposed method obtains the frame-level AUROC score of 88.3% on the CUHK Avenue dataset.

Collaborative Normality Learning Framework for Weakly Supervised Video Anomaly Detection

Video anomaly detection (VAD) under weak supervision aims to temporally locate abnormal clips using the easy-to-obtain video-level labels. In this brief, we introduce the underlying thought of unsupervised VAD to the weakly supervised VAD and propose a collaborative normality learning framework to obtain more discriminative deep representations. Specifically, a deep auto-encoder is first trained in an unsupervised manner to learn the prototypical spatial-temporal patterns of normal videos. Then, both the normal and abnormal videos are used to train a regression module, where the objective is to make the average score of the abnormal videos higher than the maximum score of the normal videos. Finally, the clips in abnormal videos with an anomaly score lower than the average are regarded as normal and used to fine-tune the trained auto-encoder. The unsupervised auto-encoder collaborates with the weakly supervised regression model to extract prototypical features of normal clips, making the learned features of normal and abnormal events more distinguishable. Experimental results on three benchmark datasets show that the proposed framework achieves comparable performance to the state-of-the-art methods. Additionally, the results of ablation studies demonstrate the validity of collaborative normality learning.

A multi-stream deep neural network with late fuzzy fusion for real-world anomaly detection

Abnormal event detection in video is alternatively known as outlier detection, where machine learning can be highly effective. While testing an unknown video, the objective of such methods is to verify the video’s category, e.g. normal or abnormal. This paper exploits visual information from normal as well as abnormal videos to train a deep multiple instance learning classifier for classification of videos. Existing multiple instance learning classifiers presume that the training videos contain only short-duration anomalous events. This assumption may not be valid for all real-world anomalies. Also, multiple occurrences of anomalies in training videos cannot be ruled out. This paper shows that by injecting temporal information in feature extraction, anomaly detection performance can be improved. To accomplish this, two spatio-temporal deep feature extractors have been applied in parallel on the training videos. These streams are then used to train a modified multiple instance learning-based classifier. Finally, a fuzzy aggregation is employed to fuse the anomaly scores. Additionally, two lightweight deep-learning classifiers have been used to substantiate the model’s efficacy for classifying fire and accident events. To understand the reliability and performance of the proposed method, extensive experiments have been carried out using UCF-Crime video dataset containing 13 anomaly categories. The dataset has been restructured into five broad categories based on the severity of actions to study the robustness of the proposed method. The paper provides sufficient empirical evidence which proves that by incorporating temporal features in the pipeline, anomaly detection accuracy can be significantly improved. Moreover, the model helps to detect long-duration anomalies in videos, which was not possible using existing methods. The proposed end-to-end multi-stream architecture performs abnormal event detection with accuracy as high as 84.48%, which is better than the performance of existing video anomaly detection methods. Moreover, the class-wise detection accuracy has improved by 6%–14% across various broad categories.

Real-World Video Anomaly Detection by Extracting Salient Features in Videos

We propose a lightweight and accurate method for detecting anomalies in videos. Existing methods used multiple-instance learning (MIL) to determine the normal/abnormal status of each segment of the video. Recent successful researches argue that it is important to learn the temporal relationships among segments to achieve high accuracy, instead of focusing on only a single segment. Therefore we analyzed the existing methods that have been successful in recent years, and found that while it is indeed important to learn all segments together, the temporal orders among them are irrelevant to achieving high accuracy. Based on this finding, we do not use the MIL framework, but instead propose a lightweight model with a self-attention mechanism to automatically extract features that are important for determining normal/abnormal from all input segments. As a result, our neural network model has 1.3% of the number of parameters of the existing method. We evaluated the frame-level detection accuracy of our method on three benchmark datasets (UCF-Crime, ShanghaiTech, and XD-Violence) and demonstrate that our method can achieve the comparable or better accuracy than state-of-the-art methods.

Multi Chunk Learning Based Auto Encoder for Video Anomaly Detection

Video anomaly detection is essential to distinguish abnormal events in large volumes of surveillance video and can benefit many fields such as traffic management, public security and failure detection. However, traditional video anomaly detection methods are unable to accurately detect and locate abnormal events in real scenarios, while existing deep learning methods are likely to omit important information when extracting features. In order to avoid omitting important features and improve the accuracy of abnormal event detection and localization, this paper proposes a novel method called Multi Chunk Learning based Skip Connected Convolutional Auto Encoder (MCSCAE). The proposed method improves the accuracy of video anomaly detection by obtaining more vital information. In the data sorting phase, non-uniform chunking is proposed to divide the video frame into several chunks of different sizes to avoid obtaining unnecessary information and omitting crucial information. In order to well reflect the abnormal motion of objects in the video, a new feature, the inter frame flow feature, which is obtained by merging inter frame difference and optical flow features, is proposed to extract motion feature. Moreover, in this paper, skip connection in the auto encoder is utilized during the training phase to reduce the reconstruction error between the original frames and the reconstruction frames, so that the reconstruction error can be used to detect abnormal events during testing. Experience on three public datasets verifies the effectiveness and accuracy of our proposed method. Experimental results show that the proposed method can detect and locate abnormal events outperforms other recent methods significantly.