UCSD Dataset Research Articles

Variational Abnormal Behavior Detection With Motion Consistency.

Abnormal crowd behavior detection has recently attracted increasing attention due to its wide applications in computer vision research areas. However, it is still an extremely challenging task due to the great variability of abnormal behavior coupled with huge ambiguity and uncertainty of video contents. To tackle these challenges, we propose a new probabilistic framework named variational abnormal behavior detection (VABD), which can detect abnormal crowd behavior in video sequences. We make three major contributions: (1) We develop a new probabilistic latent variable model that combines the strengths of the U-Net and conditional variational auto-encoder, which also are the backbone of our model; (2) We propose a motion loss based on an optical flow network to impose the motion consistency of generated video frames and input video frames; (3) We embed a Wasserstein generative adversarial network at the end of the backbone network to enhance the framework performance. VABD can accurately discriminate abnormal video frames from video sequences. Experimental results on UCSD, CUHK Avenue, IITB-Corridor, and ShanghaiTech datasets show that VABD outperforms the state-of-the-art algorithms on abnormal crowd behavior detection. Without data augmentation, our VABD achieves 72.24% in terms of AUC on IITB-Corridor, which surpasses the state-of-the-art methods by nearly 5%.

Abnormal Event Detection in Videos Based on Deep Neural Networks

Abnormal event detection has attracted widespread attention due to its importance in video surveillance scenarios. The lack of abnormally labeled samples makes this problem more difficult to solve. A partially supervised learning method only using normal samples to train the detection model for video abnormal event detection and location is proposed. Assuming that the distribution of all normal samples complies to the Gaussian distribution, the abnormal sample will appear with a lower probability in this Gaussian distribution. The method is developed based on the variational autoencoder (VAE), through end-to-end deep learning technology, which constrains the hidden layer representation of the normal sample to a Gaussian distribution. Given the test sample, its hidden layer representation is obtained through the variational autoencoder, which represents the probability of belonging to the Gaussian distribution. It is judged abnormal or not according to the detection threshold. Based on two publicly available datasets, i.e., UCSD dataset and Avenue dataset, the experimental are conducted. The results show that the proposed method achieves 92.3% and 82.1% frame-level AUC at a speed of 571 frames per second on average, which demonstrate the effectiveness and efficiency of our framework compared with other state-of-the-art approaches.

Human Anomalous Activity Detection: Shape and Motion Approach in Crowded Scenes

Detecting anomalous activities in crowded scenes is a very challenging task in computer vision. An enhanced video anomaly detection framework is proposed for frame-wise anomalous activity detection in crowded scenes that is based on both shape and motion based features. The Histogram of Oriented Gradients (HOG) is used to represent the shape based features of the video frames and for representing the motion, Histogram of Oriented Optical Flow (HOOF) is used. These features are modeled using two-class Support Vector Machines (SVM) to detect abnormal events in every frame. The proposed method is modeled with both normal and abnormal behaviors which are learnt from the training data and it is capable of detecting abnormal activities in a live surveillance video. To evaluate the performance of the proposed work, experiments are conducted on the standard benchmark UCSD data set and the results are compared with the HOOF feature bin values.

Video Abnormal Event Detection Based on Optical Flow and 3DCNN

With the advancement of technology, the detection of abnormal events in videos has extensive theoretical and practical significance. Most events in videos are normal events, but a small number of abnormal events contain a lot of information. How to identify individual abnormal events from massive videos is the focus of concern. The paper made full use of the spatio-temporal continuity of video images. Firstly, a space-time cube was constructed by integrating optical flow information; secondly, a multi-layer three-dimensional convolutional neural network model was constructed; finally, the entire process of video abnormal event detection was realized based on this model. The experiment on the UCSD dataset showed that the method in this paper can effectively detect abnormal events in videos.

Detection and Localization of Abnormalities in Surveillance Video Using Timerider-Based Neural Network

Abstract Automatic anomaly detection in surveillance videos is a trending research domain, which assures the detection of the anomalies effectively, relieves the time-consumed by the manual interpretation methods without the requirement of the domain knowledge about the anomalous object. Accordingly, this research work proposes an effective anomaly detection approach, named, TimeRide Neural network (TimeRideNN), by modifying the standard RideNN using the Taylor series such that an extra group of rider, named as timerider, is included in the standard rider optimization algorithm. Initially, the face in the videos is subjected to face detection using the Viola Jones algorithm. Then, the object tracking is performed using the knocker and holoentropy-based Bhattacharya distance, which is a modification of the Bhattacharya distance using the knocker and holoentropy. After that, the features, such as object-level features and speed-level features of the objects, are extracted and the features are employed to the proposed TimeRideNN classifier, which declares the anomalous objects in the video. The experimentation of the proposed anomaly detection method is done using the UCSD dataset (Ped1), subway dataset and QMUL junction dataset, and the analysis is performed based on accuracy, sensitivity and specificity. The proposed TimeRideNN classifier obtains the accuracy, sensitivity and specificity of 0.9724, 0.9894 and 0.9691, respectively.

Two-stream deep spatial-temporal auto-encoder for surveillance video abnormal event detection

With the improvement of public security awareness, video anomaly detection has become an indispensable demand in surveillance videos. To improve the accuracy of video anomaly detection, this paper proposes a novel two-stream spatial-temporal architecture called Two-Stream Deep Spatial-Temporal Auto-Encoder (Two-Stream DSTAE), which is composed of a spatial stream DSTAE and a temporal stream DSTAE. Firstly, the spatial stream extracts appearance characteristics whereas the temporal stream extracts the motion patterns, respectively. Then, based on the novel policy joint reconstruction error, this model fuses the spatial stream and the temporal stream to extract spatial-temporal characteristics to detect anomalies. Furthermore, since the optical flow is invariant to appearances such as color or light, we introduce optical flow to enhance the capability of extracting continuity between adjacent frames and inter-frame motion information. We demonstrate the accuracy of the proposed method on the publicly available standard datasets: UCSD, Avenue and UMN datasets. Our experiments demonstrate high accuracy, which is superior to the state-of-the-art methods.

Video Anomaly Detection Using the Optimization-Enabled Deep Convolutional Neural Network

Abstract In video surveillance, automatic detection of the anomalies is the active research area in computer technology. Even though various video anomaly detection methods are introduced, detecting anomalous events, such as illegal actions and crimes, is a major challenging issue in video surveillance. Thus, an effective automatic video anomaly detection strategy based on the deep convolutional neural network (deep CNN) is developed in this research. Initially, the input video surveillance is passed into the spatiotemporal feature descriptor, named Histograms of Optical Flow Orientation and Magnitude. The features obtained from the descriptor provide the optical flow details with the aspect of normal patterns from the scene. These patterns are further subjected to the deep CNN, which is trained using the proposed dragonfly-rider optimization algorithm (DragROA) to assure the classification either as an anomalous activity or normal. The proposed DragROA is the combination of the standard dragonfly optimization algorithm and the standard rider optimization algorithm. The implementation of the proposed DragROA-based deep CNN is carried out using two datasets, namely anomaly detection dataset and UMN dataset; the performance is analyzed using the metrics, namely accuracy, sensitivity and specificity. From the analysis, it is depicted that the proposed method obtains the maximum accuracy, sensitivity and specificity of 0.9922, 0.9809 and 1, respectively, for the UCSD dataset.

ViT-P: Classification of Genitourinary Syndrome of Menopause From OCT Images Based on Vision Transformer Models

Genitourinary syndrome of menopause (GSM) is a disease caused by a physiological decline in estrogen levels, and it can negatively affect a woman’s overall health and quality of life in terms of sexual function. Real-time optical biopsy images can now be obtained with optical coherence tomography (OCT) systems. In this study, we introduce vision transformer (ViT) to the field of medical OCT images for the first time and propose a deep learning-based approach for GSM lesion screening. Specifically, we first build a GSM dataset to train and evaluate the experimental model performance. The study aims to propose a method that combines null convolution with a deep convolutional adversarial generative network classifier to generate the samples needed for training to alleviate the hindrance of such problems, in response to certain practical problems, such as category imbalance that occur during data collection. Next, the experiments present ViT PLUS (ViT-P) for the vaginal OCT image classification task used, which effectively improves the shortcomings of ViT in extracting Patch Embedding using a multibranch convolutional neural network combined with a channel attention mechanism. The clinical images acquired by the OCT device are then automatically classified on the basis of the OCT device to reduce the medical workload of gynecologists. Experimental results show that the ViT-P model outperforms the CNN model and ViT for case screening in the GSM and UCSD datasets, and the accuracy can reach 99.9% and 99.69%, respectively.

Anomaly Detection With Particle Filtering for Online Video Surveillance

With growing security threats, many online and offine frameworks have been proposed for anomaly detection in video sequences. However, existing online anomaly detection techniques are either computationally very expensive or lack desirable accuracy. This research work proposes a novel particle filtering based framework for online anomaly detection which detects video frames with anomalous activities based upon the posterior probability of activities in a video sequence. The proposed method also detects anomalous regions in anomalous video frames. We propose novel prediction and measurement models to accurately detect anomalous video frames and anomalous regions in video frames. Novel prediction model for particle prediction and likelihood model for assigning weights to these particles are proposed. These models efficiently utilise variable sized cell structure which creates variable sized sub-regions of scenes in video frames. Furthermore, they efficiently extract and utilise information from the video frame in the form of size, motion and location features. The proposed framework is tested on UCSD and LIVE datasets and compared with the existing state-of-the-art algorithms in the literature. The proposed anomaly detection algorithm outperforms the state-of-the art algorithms in terms of reduced Equal Error Rate (EER) with comparatively lesser processing time.

A New Semantic and Statistical Distance‐Based Anomaly Detection in Crowd Video Surveillance

Recently, attention toward autonomous surveillance has been intensified and anomaly detection in crowded scenes is one of those significant surveillance tasks. Traditional approaches include the extraction of handcrafted features that need the subsequent task of model learning. They are mostly used to extract low‐level spatiotemporal features of videos, neglecting the effect of semantic information. Recently, deep learning (DL) methods have been emerged in various domains, especially CNN for visual problems, with the ability to extract high‐level information at higher layers of their architectures. On the other side, topic modeling‐based approaches like NMF can extract more semantic representations. Here, we investigate a new hybrid visual embedding method based on deep features and a topic model for anomaly detection. Features per frame are computed hierarchically through a pretrained deep model, and in parallel, topic distributions are learned through multilayer nonnegative matrix factorization entangling information from extracted deep features. Training is accomplished through normal samples. Thereafter, K‐means is applied to find typical normal clusters. At test time, after achieving feature representation through deep model and topic distribution for test frames, a statistical earth mover distance (EMD) metric is evaluated to measure the difference between normal cluster centroids and test topic distributions. High difference versus a threshold is detected as an anomaly. Experimental results on the benchmark Ped1 and Ped2 UCSD datasets demonstrate the effectiveness of our proposed method in anomaly detection.

A New Microscopic Approach to Traffic Flow Classification Using a Convolutional Neural Network Object Detector and a Multi-Tracker Algorithm

Traffic congestion is a significant issue in urban areas and can cause adverse effects. In this paper, our proposal categorizes the traffic activity from video through three steps: vehicle monitoring, feature extraction, and classification. The vehicle monitoring step comprises an object detector based on a convolutional neural network and multi-object tracker. The feature extraction step uses information related to each detected vehicle, in various points of the road, to represent the traffic condition through three features: density, flow, and velocity. We tested on the UCSD dataset and achieved the best performance with 98.82% of accuracy, which outperformed the state-of-the-art methods.

Predicting relative topological stability of mobile users in a P2P mobile cloud

In a P2P mobile cloud (PMC), an ad-hoc cloud is formed using the closely adjacent smart mobile devices (SMD). Here, a resource-deficient SMD user seeks its neighbouring resource-rich SMDs’ help to carry out a resource-wanting job. To maintain reliability and QoS of the PMC, it is necessary to be assured of the availability of the SMDs before submitting a job request. If the job is distributed to multiple SMDs, instead of assessing the availability of a single SMD, the availability of a group of SMDs is required for a certain period of time. The objective of the paper is to find a stable group of SMD users to increase the QoS of PMC. We aim to propose a novel mobility prediction method to assess the probability of a group of SMD users to be relatively static with respect to each other. We propose an algorithm to estimate the relative stability of an SMD with respect to its neighbourhood over a period of time, irrespective of its geographical location. We assess the short-term relative stability between a group of SMD users as well as the long-term mobility pattern between them. For mobility calculation, we have used real user traces from the UCSD dataset. The estimated results are analyzed and validated using different metrics.

Real-time and accurate abnormal behavior detection in videos

Abnormal crowd behavior detection is a hot research topic in the field of computer vision. In order to solve the problems of high computational cost and the imbalance between positive and negative samples, we propose an efficient algorithm that can detect and locate anomalies in videos. In order to solve the problem of less negative samples, the algorithm uses the spatiotemporal autoencoder to identify and extract the negative samples (contain abnormal behaviors) in the dataset in an unsupervised learning method. On this basis, a spatiotemporal convolutional neural network (CNN) is constructed with simple structure and low computational complexity. The supervised training method is used to train the spatiotemporal CNN with positive and negative samples to generate the detection model. Experiments are conducted on the UCSD and UMN datasets. The experiment results show that the proposed algorithm can detect and locate abnormal behaviors in real time (using only CPU), and the accuracy of the algorithm exceeds those of the existing algorithms at both the pixel level and frame level.

Inter- and Intra-Subject Transfer Reduces Calibration Effort for High-Speed SSVEP-Based BCIs.

Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) that can deliver a high information transfer rate (ITR) usually require subject's calibration data to learn the class- and subject-specific model parameters (e.g. the spatial filters and SSVEP templates). Normally, the amount of the calibration data for learning is proportional to the number of classes (or visual stimuli), which could be huge and consequently lead to a time-consuming calibration. This study presents a transfer learning scheme to substantially reduce the calibration effort. Inspired by the parameter-based and instance-based transfer learning techniques, we propose a subject transfer based canonical correlation analysis (stCCA) method which utilizes the knowledge within subject and between subjects, thus requiring few calibration data from a new subject. The evaluation study on two SSVEP datasets (from Tsinghua and UCSD) shows that the stCCA method performs well with only a small amount of calibration data, providing an ITR at 198.18±59.12 (bits/min) with 9 calibration trials in the Tsinghua dataset and 111.04±57.24 (bits/min) with 3 trials in the UCSD dataset. Such performances are comparable to those from using the multi-stimulus CCA (msCCA) and the ensemble task-related component analysis (eTRCA) methods with the minimally required calibration data (i.e., at least 40 trials in the Tsinghua dataset and at least 12 trials in the UCSD dataset), respectively. Inter- and intra-subject transfer helps the recognition method achieve high ITR with extremely little calibration effort. The proposed approach saves much calibration effort without sacrificing the ITR, which would be significant for practical SSVEP-based BCIs.

Abnormal activity detection using shear transformed spatio-temporal regions at the surveillance network edge

This paper presents a method of detecting abnormal activity in crowd videos while considering the direction of the dominant crowd motion. One main goal of our approach is to be able to run at the edge of the surveillance network close to the surveillance cameras so as to reduce network congestion and decision latency. To capture motion features while considering the direction of dominant crowd direction we propose a generalised shear transform based spatio-temporal region. To detect abnormal activity, an autoencoder based method is adopted considering the requirement for running the method at the network edge. During training, the autoencoder learns motion features for each spatio-temporal region from video frames containing normal activity. While testing, those motion features from each spatio-temporal region that cannot be reconstructed satisfactorily by the autoencoder indicate abnormal activity. This approach allows coarse localisation as well as detection of abnormal activity. The approach demonstrated $\mathcal {O}(n)$ behaviour with ability to work at higher frame rates by trading off accuracy. The approach has been verified against recent works on standard abnormal activity datasets: UCSD dataset and Subway dataset.

An adaptive training-less framework for anomaly detection in crowd scenes

Anomaly detection in crowd videos has become a popular area of research for the computer vision community. Several existing methods have determined anomaly as a deviation from scene normalcy learned via separate training with/without labeled information. However, owing to rare and sparse nature of anomalous events, any such learning can be misleading as there exist no hardcore segregation between anomalous and non-anomalous events. To address such challenge, we propose an adaptive training-less system capable of detecting anomaly on-the-fly. Our solution pipeline consists of three major components, namely, adaptive 3D-DCT model for multi-object detection-based association, local motion descriptor generation through an improved saliency guided optical flow, and anomaly detection based on Earth mover’s distance (EMD). The proposed model, despite being training-free, is found to achieve comparable performance with several state-of-the-art methods on publicly available UCSD, UMN, CUHK-Avenue and ShanghaiTech datasets.

Incremental learning of an abnormal behavior detection algorithm based on principal components

In this paper, we propose an incremental learning scheme for the abnormal behavior detection algorithm based on principal component. The results obtained on a UCSD dataset and our experimental videos at a different number of training samples show that error rates are similar to conventional learning. Moreover, the proposed scheme allows the incremental learning time to be significantly reduced in comparison with a method based on matrix eigendecomposition.

A Novel Framework for Anomaly Detection in Video Surveillance using Convolutional LSTM

Today, due to public safety requirements, surveillance systems have gained increased attention. Video data processing technologies such as the identification of activity [1], object tracking [2], crowd counting [3], and the detection of anomalies [ 4] have therefore been rapidly developing. In this study, we establish an unattended method for the detection of anomaly events in videos based on a ConvLSTM encoder-decoder to learn about the evolution of spatial characteristics. Our model only covers typical video events during preparation, whereas in testing the videos are both usual and abnormal. Experiments on the UCSD datasets confirm the validity of the suggested approach to abnormal event detection.

Arbitrary perspective crowd counting via local to global algorithm

Crowd counting is getting more and more attention. More and more collective activities, such as the Olympics Games and the World Expo, are also important to control the crowd number. In this paper, we address the problem of crowd counting in the crowded scene. Our model accurately estimated the count of people in the crowded scene. Firstly, we proposed a novel and simple convolutional neural network, called Global Counting CNN (GCCNN). The GCCNN can learn a mapping, transforms the appearance of image patches to estimated density maps. Secondly, the Local to Global counting CNN (LGCCNN), calculating the density map from local to global. Stiching the local patches constrains the final density map of the larger area, which makes up for the difference values in the perspective map. In general, it makes the final density map more accurate. The dataset we used is a set of public dataset, which are WorldExpo’10 dataset, Shanghaitech dataset, the UCF_CC_50 dataset and the UCSD dataset. The experiments have proved our method achieves the state-of-the-art result over other algorithms.

Video Frame Prediction by Deep Multi-Branch Mask Network

Future frame prediction in video is one of the most important problem in computer vision, and useful for a range of practical applications, such as intention prediction or video anomaly detection. However, this task is challenging because of the complex and dynamic evolution of scene. The difficulty of video frame prediction is to model the inherent spatio-temporal correlation between frames and pose an adaptive and flexible framework for large motion change or appearance variation. In this paper, we construct a deep multi-branch mask network (DMMNet) which adaptively fuses the advantages of optical flow warping and RGB pixel synthesizing methods, i.e., the common two kinds of approaches in this task. In the procedure of DMMNet, we add mask layer in each branch to adaptively adjust the magnitude range of estimated optical flow and the weight of predicted frames by optical flow warping and RGB pixel synthesizing, respectively. In other words, we provide a more flexible masking network for motion and appearance fusion on video frame prediction. Exhaustive experiments on Caltech pedestrian and UCF101 datasets show that the proposed model can obtain favorable video frame prediction performance compared with the state-of-the-art methods. In addition, we also put our model into the video anomaly detection problem, and the superiority is verified by the experiments on UCSD dataset.