Video Anomaly Research Articles

RETRACTED ARTICLE: Temporal and non-temporal contextual saliency analysis for generalized wide-area (GWA) search within UAV video

Unmanned aerial vehicles (UAVs) can be used to great effect for wide-area searches such as search and rescue operations. UAV enables search and rescue teams to cover large areas more efficiently and in less time. However, using UAV for this purpose involves the creation of large amounts of data, typically in video format, which must be analyzed before any potential findings can be uncovered and actions taken. This is a slow and expensive process, which can result in significant delays to the response time after a target is seen by the UAV. To solve this problem, a deep model using a visual saliency approach to automatically analyze and detect anomalies in UAV video is proposed. Temporal contextual saliency model is based on the state of the art in visual saliency detection using deep convolutional neural networks and considers local and scene context, with novel additions in utilizing temporal information through a convolutional LSTM layer and modifications to the base model. Additionally, the impact of temporal versus non-temporal reasoning for this task is evaluated. This model achieves improved results on a benchmark dataset with the addition of temporal reasoning showing significantly improved results compared to the state of the art in saliency detection.

An Approach to Detect Anomaly in Video Using Deep Generative Network

Anomaly detection in the video has recently gained attention due to its importance in the intelligent surveillance system. Even though the performance of the state-of-art methods has been competitive in the benchmark dataset, the trade-off between the computational resource and the accuracy of the anomaly detection should be considered. In this paper, we present a framework to detect anomalies in video. We proposed a “multi-scale U-Net” network architecture, the unsupervised learning for anomaly detection in video based on generative adversarial network (GAN) structure. Shortcut Inception Modules (SIMs) and residual skip connection are employed to the generator network to increase the ability of the training and testing of the neural network. An asymmetric convolution has been applied instead of traditional convolution layers to decrease the number of training parameters without performance penalty in terms of detection accuracy. In the training phase, the generator network was trained to generate the normal events and attempt to make the generated image and the ground truth to be similar. A multi-scale U-Net kept useful features of an image that were lost during training caused by the convolution operator. The generator network is trained by minimizing the reconstruction error on the normal data and then using the reconstruction error as an indicator of anomalies in the testing phase. Our proposed framework has been evaluated on three benchmark datasets, including UCSD pedestrian, CHUK Avenue, and ShanghaiTech. As a result, the proposed framework surpasses the state-of-the-art learning-based methods on all these datasets, which achieved 95.7%, 86.9%, and 73.0% in terms of AUC. Moreover, the numbers of training and testing parameters in our framework are reduced compared to the baseline network architecture, while the detection accuracy is still improved.

Usefulness of Background Removal for Video Anomaly Detection Using Generative Adversarial Networks

Usefulness of Background Removal for Video Anomaly Detection Using Generative Adversarial Networks

Main-Auxiliary Aggregation Strategy for Video Anomaly Detection

The aim of surveillance video anomaly detection is to detect events that rarely or never happened in a certain scene. Generally, different detectors can detect different anomalies. This paper proposes an efficient strategy to aggregate multiple detectors. First, the aggregation strategy chooses one detector as master detector by experience, and sets the remaining detectors as auxiliary detectors. Then, the aggregation strategy extracts credible information from auxiliary detectors, including credible abnormal (Cred-a) frames and credible normal (Cred-n) frames. After that, the frequencies that each video frame being judged as Cred-a and Cred-n are counted. Applying the events' time continuity property, more Cred-a and Cred-n frames can be inferred. Finally, the aggregation strategy utilizes the Cred-a and Cred-n frequencies to vote to calculate soft weights, and uses the soft weights to assist the master detector. Experiments are carried out on multiple datasets. Comparing with existing aggregation strategies, the proposed strategy achieves state-of-the-art performance.

A Comparative Study of Transfer Learning Approaches for Video Anomaly Detection

Recent research has shown that features obtained from pretrained Convolutional Neural Network (CNN) models can be promptly applied to a variety of problems they were not originally designed to solve. This concept, often referred to as Transfer Learning (TL), is a common practice when labeled data is limited. In some fields, such as video anomaly detection, TL is still an underexplored subject in the sense that it is not clear whether the architecture of the pretrained CNN model impacts on the video anomaly detection performance. In order to clarify this issue, we perform an extensive benchmark using 12 different pretrained CNN models on ImageNet as feature extractors and apply the features obtained to seven video anomaly detection benchmark datasets. This work presents some interesting findings about video anomaly detection using TL. The highlights of our findings were revealed by our experiments, which have shown that a simple classification process using One-Class Support Vector Machines yields similar results to state-of-the-art models. Moreover, a statistical analysis suggests that architectural differences are negligible when choosing a pretrained model for video anomaly detection, since all models presented similar performance. At last, we present an in-depth visual analysis of the Avenue dataset, and reveal several aspects that may be limiting the performance of state-of-the-art video anomaly detection methods.

Automatic detection of subsea entities and anomalies in inspection videos

Automatic detection of subsea entities and anomalies in inspection videos

Normality Learning in Multispace for Video Anomaly Detection

Video anomaly detection is a challenging task owing to the rare and diverse nature of abnormal events. However, most of the existing methods only learn the normality in a single space, focusing on low-level detailed features, which is easily affected by unimportant pixels. To address this issue, in this study, we propose a semi-supervised method based on the generative adversarial network and frame prediction, wherein the normality is learned in both the original image space and latent space, and the events deviating from the normality are detected as anomalies. In particular, given a video clip, we first predict a future frame and minimize the prediction errors between the generated frame and its ground truth. Thereafter, we encode the predicted frames and their ground truths in the latent space and minimize their differences. In the testing phase, we calculate the normal scores of each frame in both the image and latent spaces to obtain a comprehensive evaluation. Utilizing the multispace can capture more normality distribution information of the data, which can benefit anomaly detection. The results of experiments on three benchmark datasets demonstrate the effectiveness of the proposed method.

Real-time and accurate abnormal behavior detection in videos

Abnormal crowd behavior detection is a hot research topic in the field of computer vision. In order to solve the problems of high computational cost and the imbalance between positive and negative samples, we propose an efficient algorithm that can detect and locate anomalies in videos. In order to solve the problem of less negative samples, the algorithm uses the spatiotemporal autoencoder to identify and extract the negative samples (contain abnormal behaviors) in the dataset in an unsupervised learning method. On this basis, a spatiotemporal convolutional neural network (CNN) is constructed with simple structure and low computational complexity. The supervised training method is used to train the spatiotemporal CNN with positive and negative samples to generate the detection model. Experiments are conducted on the UCSD and UMN datasets. The experiment results show that the proposed algorithm can detect and locate abnormal behaviors in real time (using only CPU), and the accuracy of the algorithm exceeds those of the existing algorithms at both the pixel level and frame level.

REMOVED: Detecting anomalies within Unmanned Aerial Vehicle (UAV) video based on contextual saliency

Abstract Unmanned Aerial Vehicles (UAV) can be used to great effect for the purposes of surveillance or search and rescue operations. UAV enable search and rescue teams to cover large areas more efficiently and in less time. However, using UAV for this purpose involves the creation of large amounts of data (typically video) which must be analyzed before any potential findings can be uncovered and actions taken. This is a slow and expensive process which can result in significant delays to the response time after a target is seen by the UAV. To solve this problem, it is proposed a deep model using a visual saliency approach to automatically analyze and detect anomalies in UAV video. Contextual Saliency for Anomaly Detection in UAV Video (CSADUV) model is based on the state-of-the-art in visual saliency detection using deep convolutional neural networks and considers local and scene context, with novel additions in utilizing temporal information through a convolutional LSTM layer and modifications to the base model. This model achieves promising results with the addition of the temporal implementation producing significantly improved results compared to the state-of-the-art in saliency detection. However, due to limitations in the dataset used the model fails to generalize well to other data, failing to beat the state-of-the-art in anomaly detection in UAV footage. The approach taken shows promise with the modifications made yielding significant performance improvements and is worthy of future investigation. The lack of a publicly available dataset for anomaly detection in UAV video poses a significant roadblock to any deep learning approach to this task, however despite this paper shows that leveraging temporal information for this task, which the state-of-the-art does not currently do, can lead to improved performance.

프레임 레벨 비디오 이상탐지를 위한 단일 클래스 적대학습 방법

이상탐지분야에서는 이상치의 수가 매우 적어 지도학습 방법을 이용하면 데이터 불균형 및 새로운 이상치 탐지에 문제가 발생한다. 따라서 비지도 방식의 단일 클래스 분류기는 효율적으로 사용될 수 있다. 본 논문에서는 단일 클래스 적대방법을 통해 정상데이터만의 분포를 따르는 잠재공간을 학습하여 이상치를 탐지할 수 있는 모델을 제안한다. 이것은 인코더, 디코더 및 분류기의 3가지 모듈로 이루어진다. 인코더와 디코더는 입력데이터를 압축하여 잠재공간에 전사하고 전사된 데이터를 복원하는 오토인코더 구조를 가진다. 분류기는 전사된 데이터를 입력받아 왜곡된 정상 데이터인지 기존 정상 데이터인지 판단하도록 학습된다. 인코더는 왜곡된 정상 데이터가 전사된 값도 분류기를 통해 기존으로 분류되도록 적대학습하며 정상 데이터에 대한 분포를 정교하게 학습한다. 우리의 모델은 이상탐지분야에서 많이 사용되는 Modified National Institute of Standards and Technology (MNIST)와 University of California San Diego (UCSD) Pedestrian (Ped) 2 데이터를 사용하여 각각 Area Under the Curve (AUC) 0.91 및 0.72의 탐지 성능을 보여준다.

Malicious attacks detection in crowded areas using deep learning-based approach

With the increasing need to ensure people's safety in crowded areas, the development of a systematic anomaly detection mechanism is becoming indispensable. Here are a few examples of recent malicious attacks targeting crowded areas in big cities: in 2016, a truck driver attacked and killed 84 persons walking in the promenade in Nice, France; and on 19 December, 2016, a truck was deliberately driven into the Christmas market, in Berlin, Germany, killing 12 people and injuring 56 others. These attacks demonstrate the need for efficient monitoring systems to avoid such devastating attacks. To do so, early detection and prevention abilities are vital. Detecting and localizing abnormal events in crowded scenes is important and has significant implications in video surveillance applications. Video surveillance can be challenging, as abnormal events can be unpredictable and changing, based on the context. Accurately detecting and localizing anomalies in videos is a powerful tool that can help to improve security and understand the behavior of anomalies. In this paper, we present an automated visionbased monitoring scheme specifically designed for atypical event-detection and localization in crowded areas.

Automatic Detection of Violent Incidents from Video Footage of CCTV Cameras

In the current era, the majority of public places such as supermarket, public garden, malls, university campus, etc. are under video surveillance. There is a need to provide essential security and monitor unusual anomaly activities at such places. The major drawback in the traditional approach, that there is a need to perform manual operation for 24 ? 7 and also there are possibilities of human errors. This paper focuses on anomaly detection and activity recognition of humans in the videos. Computer vision has evolved in the last decade as a key technology for numerous applications replacing human supervision. We present an e?cient method for detecting anomalies in videos. Recent applications of convolutional neural networks have shown promises of convolutional layers for object detection and recognition, especially in images. Experimental results on challenging datasets show the superiority of the proposed method compared to the state of the art in both frame-level and pixel-level in anomaly detection task.

Anomaly Detection Based on Locality Sensitive Hashing with Genetic Algorithm

With the increase of public safety awareness, video anomaly detection has attracted researchers’ attention. In the paper, a novel approach is proposed to detect anomalies in the video. It is based on Locality Sensitive Hashing (LSH), which maps similar data to the same bucket with high probabilities, and non-similar data is mapped to the same bucket with a low probability to detect abnormal videos that are not similar to normal videos. In order to improve the probability of similar data mapping into the same bucket, the Genetic Algorithm (GA) is used to optimize the entire hash function group while maintaining the diversity of the hash function group. The algorithm gets AUC 0.78 on the dataset UCSD ped1 and AUC 0.94 on the dataset UCSD ped2, which confirmed the effectiveness of the algorithm.

IBaggedFCNet: An Ensemble Framework for Anomaly Detection in Surveillance Videos

The prevalent use of surveillance cameras in public places and advancements in computer vision warrant most sought-after research in the domain of anomalous activity detection. Anomaly detection has shown promising applications for suspicious activity detection. In this paper, we propose a bagging framework IBaggedFCNet that leverages the power of ensembles for robust classification to detect anomalies in videos. Our approach, which investigates state-of-the-art Inception-v3 image classification network, requires no video segmentation prior to feature extraction that can produce unstable segmentation results and cause a high memory footprint. We show improvement empirically on multiple benchmark datasets, most prominently on the UCF-Crime dataset. Moreover, we experiment with different ensemble fusion methods, including static and dynamic techniques, and also prove our single model’s predictive accuracy in localizing anomaly in surveillance videos.

Dual Discriminator Generative Adversarial Network for Video Anomaly Detection

Video anomaly detection is an essential task because of its numerous applications in various areas. Because of the rarity of abnormal events and the complicated characteristic of videos, video anomaly detection is challenging and has been studied for a long time. In this paper, we propose a semi-supervised approach with a dual discriminator-based generative adversarial network structure. Our method considers more motion information in video clips compared with previous approaches. Specifically, in the training phase, we predict future frames for normal events via a generator and attempt to force the predicted frames to be similar to their ground truths. In addition, we utilize both a frame discriminator and motion discriminator to adverse the generator to generate more realistic and consecutive frames. The frame discriminator attempts to determine whether the input frames are generated or original frames sampled from the normal video. The motion discriminator attempts to determine whether the given optical flows are real or fake. Fake optical flows are estimated from generated frames and adjacent frames, and real optical flows are estimated from the real frames sampled from original videos. Then, in the testing phase, we evaluate the quality of predicted frames to obtain the regular score, and we consider those frames with lower prediction qualities as abnormal frames. The results of experiments on three publicly available datasets demonstrate the effectiveness of our proposed method.

Learning Normal Patterns via Adversarial Attention-Based Autoencoder for Abnormal Event Detection in Videos

Automatically detecting anomalies in videos is a challenging problem due to non-deterministic definitions of abnormal events and lack of sufficient training data. To address these issues, we propose an autoencoder coupled with attention model to discover normal patterns in videos via adversarial learning. Abnormal events are detected by diverging them from the normal patterns with the reconstruction error produced by the autoencoder. To this end, we build an end-to-end trainable adversarial attention-based autoencoder network, called Ada-Net, to make the reconstructed frames indistinguishable from original frames. The Ada-Net combines an autoencoder network and a GAN model that is used to benefit enhancing the reconstruction ability of the autoencoder. To further improve the reconstruction performance, we integrate an attention model into the decoder to dynamically select informative parts of encoding features for decoding. The attenion mechanism is helpful to preserving important information for learning intrinsic normal patterns. Evaluations on four challenging datasets, including the Subway, the UCSD Pedestrian, the CUHK Avenue, and the ShanghaiTech datasets, demonstrate the effectiveness of the proposed method.

Wasserstein Clustering based Video Anomaly Detection for Traffic Surveillance

Anomaly Detection is very important in present scenario with huge availability of data and enormous difficulty in extraction of meaningful information out of it. In this paper we present an approach for video anomaly detection based on trajectory features and spatio – temporal features. Clustering of spatio – temporal features and trajectory features are performed in Wasserstein metric space and cluster distance and span in Wasserstein metric space is exploited to perform anomaly detection. The Performance of the Anomaly detection with Wasserstein distance based K – means and Wasserstein distance based DBSCAN clustering of the 3D wavelet features and trajectory features was studied. The method is robust and suffers from fewer false alarms.

Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks.

This paper presents an anomaly detection method that is based on a sparse coding inspired Deep Neural Networks (DNN). Specifically, in light of the success of sparse coding based anomaly detection, we propose a Temporally-coherent Sparse Coding (TSC), where a temporally-coherent term is used to preserve the similarity between two similar frames. The optimization of sparse coefficients in TSC with the Sequential Iterative Soft-Thresholding Algorithm (SIATA) is equivalent to a special stacked Recurrent Neural Networks (sRNN) architecture. Further, to reduce the computational cost in alternatively updating the dictionary and sparse coefficients in TSC optimization and to alleviate hyperparameters selection in TSC, we stack one more layer on top of the TSC-inspired sRNN to reconstruct the inputs, and arrive at an sRNN-AE. We further improve sRNN-AE in the following aspects: i) rather than using a predefined similarity measurement between two frames, we propose to learn a data-dependent similarity measurement between neighboring frames in sRNN-AE to make it more suitable for anomaly detection; ii) to reduce computational costs in the inference stage, we reduce the depth of the sRNN in sRNN-AE and, consequently, our framework achieves real-time anomaly detection; iii) to improve computational efficiency, we conduct temporal pooling over the appearance features of several consecutive frames for summarizing information temporally, then we feed appearance features and temporally summarized features into a separate sRNN-AE for more robust anomaly detection. To facilitate anomaly detection evaluation, we also build a large-scale anomaly detection dataset which is even larger than the summation of all existing datasets for anomaly detection in terms of both the volume of data and the diversity of scenes. Extensive experiments on both a toy dataset under controlled settings and real datasets demonstrate that our method significantly outperforms existing methods, which validates the effectiveness of our sRNN-AE method for anomaly detection. Codes and data have been released at https://github.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection.

Video Anomaly Detection and Localization Based on an Adaptive Intra-Frame Classification Network

Video anomaly detection and localization is still a challenging task in the computer vision field. Previous methods took this task as an outlier detection problem, which computed the deviation between the test samples and the normal patterns. In this paper, an adaptive intra-frame classification network (AICN) is proposed to transform this task to a multi-class classification problem. The contributions of our method are as follows. AICN is an end-to-end network for anomaly detection and localization. By using the motion convolutional layers and the shape convolutional layers, spatial-temporal features are extracted without resizing or splitting the frames before forward propagation. AICN enhances the adaptiveness of model. By using the adaptive region pooling layer and the intra-frame classifier, AICN is adaptive to frames with different resolutions and is easier to be applied on other scenes. AICN evaluates the abnormality of frames based on the intra-frame classification results. The intra-frame classification strategy reserves more connection information of sub-regions and makes the model outperform previous methods. The proposed method is examined on four public datasets with different background complexities and resolutions: UCSD Ped1 dataset, UCSD Ped2 dataset, Avenue dataset and Subway dataset. The results are further compared with previous approaches to confirm the effectiveness and the advantage of our method.

Robust Anomaly Detection in Videos Using Multilevel Representations

Detecting anomalies in surveillance videos has long been an important but unsolved problem. In particular, many existing solutions are overly sensitive to (often ephemeral) visual artifacts in the raw video data, resulting in false positives and fragmented detection regions. To overcome such sensitivity and to capture true anomalies with semantic significance, one natural idea is to seek validation from abstract representations of the videos. This paper introduces a framework of robust anomaly detection using multilevel representations of both intensity and motion data. The framework consists of three main components: 1) representation learning using Denoising Autoencoders, 2) level-wise representation generation using Conditional Generative Adversarial Networks, and 3) consolidating anomalous regions detected at each representation level. Our proposed multilevel detector shows a significant improvement in pixel-level Equal Error Rate, namely 11.35%, 12.32% and 4.31% improvement in UCSD Ped 1, UCSD Ped 2 and Avenue datasets respectively. In addition, the model allowed us to detect mislabeled anomalies in the UCDS Ped 1.