Video Anomaly Research Articles

Enhancing Video Anomaly Detection with Improved UNET and Cascade Sliding Window Technique

Computer vision video anomaly detection still needs to be improved, especially when identifying images with unusual motions or objects. Current approaches mainly concentrate on reconstruction and prediction methods, and unsupervised video anomaly detection faces difficulties because there are not enough tagged abnormalities, which reduces accuracy. This paper presents a novel framework called the Improved UNET (I-UNET), designed to counteract overfitting by addressing the need for complex models that can extract subtle information from video anomalies. Video frame noise can be eliminated by preprocessing the frames with a Weiner filter. Moreover, the system uses Convolution Long Short-Term Memory (ConvLSTM) layers to smoothly integrate temporal and spatial data into its encoder and decoder portions, improving the accuracy of anomaly identification. The Cascade Sliding Window Technique (CSWT) is used post-processing to identify anomalous frames and generate anomaly scores. Compared to baseline approaches, experimental results on the UCF, UCSDped1, and UCSDped2 datasets demonstrate notable performance gains, with 99% accuracy, 90.8% Area Under Curve (AUC), and 10.9% Equal Error Rate (EER). This study provides a robust and accurate framework for video anomaly detection with the highest accuracy rate.

Video Anomaly Detection via self-supervised and spatio-temporal proxy tasks learning

Video Anomaly Detection (VAD) aims to identify events in videos that deviate from typical patterns. Given the scarcity of anomalous samples, previous research has primarily focused on learning regular patterns from datasets exclusively containing normal behaviors, and treating deviations from these patterns as anomalies. However, most of these methods are constrained by coarse-grained modeling approaches that renders them incapable of learning highly-discriminative features, which are necessary to effectively distinguish between the subtle differences between normal and abnormal behaviors. To better capture these features, we propose an innovative method. Initially, pseudo-anomalous samples for appearance and motion are generated through geometric transformations (2D rotations) and the scrambling of video sequences. Subsequently, a dual-branch network featuring spatio-temporal decoupling is proposed, in which the spatial and temporal branches each handle a specific proxy task. These tasks are designed to distinguish between normal and pseudo-anomalous samples, involving operations such as predicting patch-based 2D rotation angles and classifying video frame triplets as total-anomaly, left-anomaly, right-anomaly, and non-anomaly. Our approach employs an end-to-end training methodology, without relying on pre-trained models (except for the object detector). Evaluations on the UCSD Ped2, Avenue, and ShanghaiTech datasets show that our method achieved AUC scores of 99.1%, 91.9%, and 81.1%, respectively, demonstrating its effectiveness. The code is publicly accessible at the following link: https://spatio-temporal-tasks.

Delving into CLIP latent space for Video Anomaly Recognition

We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, the first to combine Vision and Language Models (VLMs), such as CLIP, with multiple instance learning for joint video anomaly detection and classification. Our approach specifically involves manipulating the latent CLIP feature space to identify the normal event subspace, which in turn allows us to effectively learn text-driven directions for abnormal events. When anomalous frames are projected onto these directions, they exhibit a large feature magnitude if they belong to a particular class. We also leverage a computationally efficient Transformer architecture to model short- and long-term temporal dependencies between frames, ultimately producing the final anomaly score and class prediction probabilities. We compare AnomalyCLIP against state-of-the-art methods considering three major anomaly detection benchmarks, i.e. ShanghaiTech, UCF-Crime, and XD-Violence, and empirically show that it outperforms baselines in recognising video anomalies. Project website and code are available at https://lucazanella.github.io/AnomalyCLIP/.

Triplet-set feature proximity learning for video anomaly detection

The identification of anomalies in videos is a particularly complex visual challenge, given the wide variety of potential real-world events. To address this issue, our paper introduces a unique approach for detecting divergent behavior in surveillance videos, utilizing triplet-loss for video anomaly detection. Our method involves selecting a triplet set of video segments from normal (n) and abnormal (a) data points for deep feature learning. We begin by creating a database of triplet sets of two types: a-a-n and n-n-a. By computing a triplet loss, we model the proximity between n-n chunks and the distance between ‘a’ chunks from the n-n ones. Additionally, we train the deep network to model the closeness of a-a chunks and the divergent behavior of ‘n’ from the a-a chunks.The model acquired in the initial stage can be viewed as a prior, which is subsequently employed for modeling normality. As a result, our method can leverage the advantages of both straightforward classification and normality modeling-based techniques. We also present a data selection mechanism for the efficient generation of triplet sets. Furthermore, we introduce a novel video anomaly dataset, AnoVIL, designed for human-centric anomaly detection. Our proposed method is assessed using the UCF-Crime dataset encompassing all 13 categories, the IIT-H accident dataset, and AnoVIL. The experimental findings demonstrate that our method surpasses the current state-of-the-art approaches. We conduct further evaluations of the performance, considering various configurations such as cross-dataset evaluation, loss functions, siamese structure, and embedding size. Additionally, an ablation study is carried out across different settings to provide insights into our proposed method.

Video anomaly localization using modified faster RCNN with soft NMS algorithm

Video anomaly localization using modified faster RCNN with soft NMS algorithm

Enhancing Video Anomaly Detection Using a Transformer Spatiotemporal Attention Unsupervised Framework for Large Datasets

This work introduces an unsupervised framework for video anomaly detection, leveraging a hybrid deep learning model that combines a vision transformer (ViT) with a convolutional spatiotemporal relationship (STR) attention block. The proposed model addresses the challenges of anomaly detection in video surveillance by capturing both local and global relationships within video frames, a task that traditional convolutional neural networks (CNNs) often struggle with due to their localized field of view. We have utilized a pre-trained ViT as an encoder for feature extraction, which is then processed by the STR attention block to enhance the detection of spatiotemporal relationships among objects in videos. The novelty of this work is utilizing the ViT with the STR attention to detect video anomalies effectively in large and heterogeneous datasets, an important thing given the diverse environments and scenarios encountered in real-world surveillance. The framework was evaluated on three benchmark datasets, i.e., the UCSD-Ped2, CHUCK Avenue, and ShanghaiTech. This demonstrates the model’s superior performance in detecting anomalies compared to state-of-the-art methods, showcasing its potential to significantly enhance automated video surveillance systems by achieving area under the receiver operating characteristic curve (AUC ROC) values of 95.6, 86.8, and 82.1. To show the effectiveness of the proposed framework in detecting anomalies in extra-large datasets, we trained the model on a subset of the huge contemporary CHAD dataset that contains over 1 million frames, achieving AUC ROC values of 71.8 and 64.2 for CHAD-Cam 1 and CHAD-Cam 2, respectively, which outperforms the state-of-the-art techniques.

Weakly-Supervised Video Anomaly Detection With Snippet Anomalous Attention

Weakly-Supervised Video Anomaly Detection With Snippet Anomalous Attention

AEMNet: Unsupervised Video Anomaly Detection Method Based on Attention-Enhanced Memory Networks

Video anomaly detection has always been a challenging task in computer vision due to data imbalance and susceptibility to scene variations such as lighting and occlusions. In response to this challenge, this paper proposes an unsupervised video anomaly detection method based on an attention-enhanced memory network. The method utilizes a dual-stream network structure of autoencoders, enhancing the model’s learning ability for important features in appearance and motion by introducing coordinate attention mechanisms and variance attention mechanisms, emphasizing significant characteristics of static objects and rapidly moving regions. By adding memory modules to both the appearance and motion branches, the network structure’s memory information is reinforced, enabling it to capture long-term spatiotemporal dependencies in videos and thereby improving the accuracy of anomaly detection. Furthermore, by optimizing the network structure’s activation functions to handle negative inputs, it enhances its nonlinear modeling capabilities, enabling better adaptation to complex environments, including variations in lighting and occlusions, further improving the effectiveness of anomaly detection. The paper conducts comparative experiments and ablation studies using three public available datasets and various models. The results demonstrate that compared to baseline models, the AUC performance is improved by 3.9%, 4.7%, and 1.7% on UCSD Ped2, CHUK Avenue, and ShanghaiTech datasets, respectively. When compared with the other models, the average AUC performance is improved by 4.3%, 5.4%, and 6.2%, with an average improvement of 8.75% in the ERR metric, validating the effectiveness and adaptability of the proposed method. The code can be obtained at the following URL: https://github.com/AcademicWhite/AEMNet .

A Survey on Self-Supervised Learning-Based Video Anomaly Detection

Video anomaly detection (VAD) exhibits promising applications across diverse domains, bolstering intelligence, security, and operational efficiency, thereby catalyzing industry growth. This paper begins by examining the research background and significance of VAD, providing an in-depth analysis of its relevance across various sectors. Subsequently, from a machine learning standpoint, recent advancements in self-supervised learning (SSL)-based VAD models are systematically categorized and summarized, elucidating their underlying principles and deployment scenarios. Additionally, commonly utilized datasets in VAD are introduced to facilitate readers' understanding of model assessment and comparative analysis. Lastly, discussions on future trajectories and extant challenges in VAD are undertaken to foster deeper exploration and propel the advancement of this domain.

Optimal object stitching and video condensation-aided improved YOLOv5-oriented anomaly object detection framework

Recent advances in intelligent surveillance to boost public security have increased interest in anomaly location and identification among computer vision researchers in congested situations. Unrestricted video monitoring is required due to its extensive use in private and public spheres. Particularly, research on finding anomalies in surveillance videos has drawn much interest. Video condensation is an efficient method for quickly scanning and retrieving lengthy surveillance films. Reordering the video events in the spatial or temporal domain seeks to condense large video sequences into their equivalent compact video representation. Additionally, if a delay occurs in anomaly detection, it leads to time consumption for rectifying it. Therefore, the core aim of this task is to reduce the time consumption of detecting the instance of video anomaly detection faster by introducing the optimal object stitching through the deep learning-based video condensation concept. The steps include video condensation and deep learning-based anomaly detection. Initially, the necessary video for anomaly detection is gathered from the datasets. The gathered video is converted into frames and is used for object detection in each frame using Improved YOLOv5 (I-YOLOv5). Then, the optimal frame selection is performed using the hybrid Fitness of Wild Geese and Path Finder (FWGPF) for stitching the detected objects into the selected optimal frames, thus compressing the relevant frames. In the second phase, the condensed (compressed) frames are utilized for determining the anomaly frames with the support of Parameter Tuning in Dilated MobileNet (PTDM), and further, the anomaly objects are detected on the classified condensed frames based on the I-YOLOv5. Here, the tuning of constraint is done in the anomaly detection phase by the same hybrid FWGPF to reduce the time consumption. The experimental analysis of the developed model is performed with baseline algorithms and detection techniques to confirm the developed model’s efficiency.

Progressive prediction: Video anomaly detection via multi‐grained prediction

AbstractVideo Anomaly Detection (VAD) has been an active research field for several decades. However, most existing approaches merely extract a single type of feature from videos and define a single paradigm to indicate the extent of abnormalities. A coarse‐to‐fine three‐level prediction is built by integrating different levels of spatio‐temporal representations, better highlighting the difference between normal and abnormal behaviors. First, an object‐level trajectory prediction is proposed to model human historical position using a graph transformer network. Subsequently, skeleton‐level prediction is achieved by incorporating the positional information from the trajectory prediction. More importantly, based on the predicted skeleton, a skeleton‐guided pixel‐level region prediction is performed. A novel Skeleton Conditioned Generative Adversarial Network (SCGAN) is designed to explore the correlation between skeleton‐level and pixel‐level motion prediction. Benefiting from SCGAN, the prediction of human regions is contributed by both coarse‐grained and fine‐grained motion features. This three‐level prediction, namely Progressive Prediction Video Anomaly Detection (P3VAD), enlarges the prediction error on irregular motion patterns. Besides, a pixel‐level analysis method is proposed to achieve Background‐bias Elimination (BE) and denoise the predicted region. Experimental results validate the effectiveness of P3VAD on the four benchmark datasets (ShanghaiTech, CUHK Avenue, IITB‐Corridor, and ADOC).

VPE-WSVAD: Visual prompt exemplars for weakly-supervised video anomaly detection

Weakly Supervised Video Anomaly Detection (WSVAD) plays a crucial role in visual surveillance by effectively distinguishing anomalies from normality with only video-level annotations; nevertheless, due to inherent limitations, including imbalanced data, intra-bag similarity, and snippet entanglement, existing methodologies prone to a bias towards normality. This paper proposes a novel framework, Visual Prompt Exemplars-based WSVAD termed VPE-WSVAD, to enhance discriminative representations of anomalies against normality by incorporating scenario-awareness visual prompt exemplars. The proposed VPE-WSVAD framework comprises three key components. Firstly, the prompt retrieval network generates potential abnormal proposals consistent with visual prompt exemplars at the frame level. Secondly, the proposal filtering mechanism aims to select the proposal with the highest confidence score to represent potential anomalies in the snippet. Finally, prompt likelihood learning is designed to capture the correlations between proposals (w.r.t prompt exemplars) and snippets, generating discriminative representations for each snippet. By incorporating visual prompt exemplars, our method provides more detailed reporting of abnormal events, including when, where, and what. To validate the efficacy of our method, we conducted comprehensive evaluations on three publicly available datasets. The experimental results conclusively demonstrate the superiority of our approach, achieving an impressive frame-level area under the curve (AUC) of 96.88% on the ShanghaiTech dataset and 99.86% on the UCSD Ped2 dataset. Our code will be released in the future.

Leveraging YOLOv5s with optimization‐based effective anomaly detection in pedestrian walkways

AbstractCurrently, video surveillance is generally used to safeguard safety in public places like railway stations, traffic signals, malls, and so on. Video anomaly recognition and localization are the main components of the intelligent video surveillance method. Video anomaly recognition refers to the procedure of spatiotemporal localization of the abnormal design existing in the video. A main task in video surveillance is the classification of anomalies that occur in it like thefts, crimes, and so forth. Also, anomaly recognition in pedestrian walkways has enlarged major attention among the computer vision (CV) groups to improve pedestrian protection. The current developments in Deep Learning (DL) methods have great attention to dissimilar procedures like image classification, object recognition, and so forth. This study designs an Optimal Deep Learning for Effective Anomaly Detection in Pedestrian Walkways (ODL‐EADPW) model. The ODL‐EADPW technique employs a fine‐tuned DL model for the identification of pedestrians and anomalies in the walkways. In the ODL‐EADPW technique, the image pre‐processing is primarily involved in two stages median filtering (MF) based noise removal and adaptive histogram equalization (AHE)‐based contrast enhancement. For anomaly detection in pedestrian walkways, the ODL‐EADPW technique uses the YOLOv5s model with EfficientRep as a backbone network. To enhance the detection results of the ODL‐EADPW technique, a stochastic gradient descent (SGD) optimizer was employed to perfect the hyperparameters of the EfficientRep model. The performance evaluation of the ODL‐EADPW methodology is implemented on the UCSD Anomaly detection dataset. An extensive comparison study stated that the ODL‐EADPW technique gains effectual detection results over other DL models in terms of different measures.

Anomaly Detection for Video Surveillance

Anomaly detection in video surveillance is critical for identifying abnormal behaviours or events that deviate from typical patterns. This abstract explores various techniques used to detect anomalies in surveillance videos, including traditional methods like motion detection and background subtraction, as well as advanced approaches such as deep learning and anomaly scoring algorithms. By leveraging these techniques, surveillance systems can automatically detect suspicious activities such as trespassing, theft, or violence, enabling timely intervention and improved security measures. Highlighting the challenges in anomaly detection, such as dealing with complex scenes, occlusions and discusses ongoing research efforts to enhance the accuracy and efficiency of anomaly detection systems in real-world surveillance scenarios. Key Words: anomaly, suspicious, pattern, motion, detection, security measure

Normality learning reinforcement for anomaly detection in surveillance videos

Video Anomaly Detection (VAD) is a key technology that enables automatic anomaly detection in surveillance video systems. Due to the considerable dimensions and fine-grained spatial details of video sequences, VAD tasks necessitate the collaboration of learning both spatial and temporal contextualized information. Due to the restricted understanding of spatial details, previous unsupervised VAD works have yet to fully tap into the potential of prediction-based methods to fully capture the spatial–temporal correlations. Inspired by the lateral geniculate nucleus in visual processing, we propose a Normality Learning Reinforcement framework of VAD (NLR-VAD) that enables learning more sophisticated and efficient spatial–temporal interactions. Specifically, NLR-VAD introduces a Normality Learning Reinforcement Unit (NLRU) rooted in diffusion models. NLRU refines the irrelevant information and amplifies the knowledge retrieval of spatial normality, particularly in high-resolution details. NLRU collaborates with Memory-Augmented Auto-Encoder (MAAE) to enhance spatial–temporal normality through fine-grained extractions. Moreover, a cross-NLRU fusion unit is proposed for information integration as a bridge between NLRU and MAAE. Quantitative results on three benchmarks show that NLR-VAD performs competitively compared with the previous methods. Extended analysis and visualization fully demonstrate the effectiveness of the proposed units.

Deep learning based anomaly detection in real-time video

AbstractMany security cameras have been put up in places like airports, roads, and banks for the safety of these public places. These cameras make a lot of video data, and most security camera recordings are only ever seen when something strange happens. This means that monitoring has to be done by people, which is time-consuming and often wrong, so automatic ways of monitoring have to be used. In this paper, we propose a system that automatically detects irregular events in videos based on the integration of Inflated 3D Convolution Network (I3D-ResNet50) and deep Multiple Instance Learning (MIL). This system considers both regular and unusual videos as negative and positive packets, respectively. Each video snippet is a case of that packet. An anomaly score is generated for each video snippet using a fully connected Neural Network (NN). After processing videos, we used an I3D-ResNet50 to extract features after applying 10-crop augmentations to the UCF-101 dataset that contains 130 GB of videos with 13 abnormal events such as fighting, stealing, abuse, etc., as well as normal events. Our experimental results show that the AUC is 82.85% with only 10,000 iterations compared with other approaches. This means that our model is better at spotting anomalies in real-time videos.

Weakly Supervised Video Anomaly Detection via Self-Guided Temporal Discriminative Transformer.

Weakly supervised video anomaly detection is generally formulated as a multiple instance learning (MIL) problem, where an anomaly detector learns to generate frame-level anomaly scores under the supervision of MIL-based video-level classification. However, most previous works suffer from two drawbacks: 1) they lack ability to model temporal relationships between video segments and 2) they cannot extract sufficient discriminative features to separate normal and anomalous snippets. In this article, we develop a weakly supervised temporal discriminative (WSTD) paradigm, that aims to leverage both temporal relation and feature discrimination to mitigate the above drawbacks. To this end, we propose a transformer-styled temporal feature aggregator (TTFA) and a self-guided discriminative feature encoder (SDFE). Specifically, TTFA captures multiple types of temporal relationships between video snippets from different feature subspaces, while SDFE enhances the discriminative powers of features by clustering normal snippets and maximizing the separability between anomalous snippets and normal centers in embedding space. Experimental results on three public benchmarks indicate that WSTD outperforms state-of-the-art unsupervised and weakly supervised methods, which verifies the superiority of the proposed method.

Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models

Video Anomaly Detection (VAD) serves as a pivotal technology in the intelligent surveillance systems, enabling the temporal or spatial identification of anomalous events within videos. While existing reviews predominantly concentrate on conventional unsupervised methods, they often overlook the emergence of weakly-supervised and fully-unsupervised approaches. To address this gap, this survey extends the conventional scope of VAD beyond unsupervised methods, encompassing a broader spectrum termed Generalized Video Anomaly Event Detection (GVAED). By skillfully incorporating recent advancements rooted in diverse assumptions and learning frameworks, this survey introduces an intuitive taxonomy that seamlessly navigates through unsupervised, weakly-supervised, supervised and fully-unsupervised VAD methodologies, elucidating the distinctions and interconnections within these research trajectories. In addition, this survey facilitates prospective researchers by assembling a compilation of research resources, including public datasets, available codebases, programming tools, and pertinent literature. Furthermore, this survey quantitatively assesses model performance, delves into research challenges and directions, and outlines potential avenues for future exploration.

Video anomaly detection based on a multi-layer reconstruction autoencoder with a variance attention strategy

In this paper, we propose a comprehensive framework for detecting anomalies in videos based on autoencoder (AE). Traditional AE models solely rely on input and final reconstruction, potentially limiting their capacity to fully utilize the intermediate neural network layers. To mitigate this limitation, we introduce a novel approach that concurrently trains the model using corresponding intermediate layers from both the encoder and decoder. This allows the model to capture more intricate features, thus enhancing its anomaly detection capabilities. Furthermore, we introduce a motion loss function that exclusively relies on original video frames rather than optical flow, rendering it more efficient and capable of extracting motion features. Additionally, we have devised a variance attention strategy that is parameter-free and can automatically directs our model's focus towards moving objects, further boosting the performance of our approach. Our experiments on three public datasets demonstrate the effectiveness and efficiency of our method in identifying abnormal events in complex scenarios. The code is publicly available at https://github.com/lsf2008/multRecLossAEPub.

TransGANomaly: Transformer based Generative Adversarial Network for Video Anomaly Detection

Video anomaly detection aims to identify a set of abnormal events in videos. Deep reconstruction and prediction-based models have been employed to detect anomalies. Deep reconstruction models sometimes recreate the abnormal events along with the normal ones. However, the prediction-based approaches have demonstrated encouraging results. This paper presents a video vision transformer (ViViT) based generative adversarial network (GAN), TransGANomaly, a novel approach for detecting anomalies. The proposed framework is a video frame predictor and trained only on normal video data adversarially. The generator of the GAN is a ViViT network that receives 3D input tokens from the video snippets. The generator aims to predict the future frame based on past sequences. After that, the predicted and original frames are given to the model’s discriminator for binary classification. Extensive experiments have been performed on UCSD Pedestrian, CUHK Avenue, and ShanghiaTech datasets to validate the efficacy of the proposed method.