Abnormal Crowd Behavior Research Articles

A Review of Abnormal Crowd Behavior Recognition Technology Based on Computer Vision

Abnormal crowd behavior recognition is one of the research hotspots in computer vision. Its goal is to use computer vision technology and abnormal behavior detection models to accurately perceive, predict, and intervene in potential abnormal behaviors of the crowd and monitor the status of the crowd system in public places in real time, to effectively prevent and deal with public security risks and ensure public life safety and social order. To this end, focusing on the abnormal crowd behavior recognition technology in the computer vision system, a systematic review study of its theory and cutting-edge technology is conducted. First, the crowd level and abnormal behaviors in public places are defined, and the challenges faced by abnormal crowd behavior recognition are expounded. Then, from the dimensions based on traditional methods and based on deep learning, the mainstream technologies of abnormal behavior recognition are discussed, and the design ideas, advantages, and limitations of various methods are analyzed. Next, the mainstream software tools are introduced to provide a comprehensive reference for the technical framework. Secondly, typical abnormal behavior datasets at home and abroad are sorted out, and the characteristics of these datasets are compared in detail from multiple perspectives such as scale, characteristics, and uses, and the performance indicators of different algorithms on the datasets are compared and analyzed. Finally, the full text is summarized and the future development direction of abnormal crowd behavior recognition technology is prospected.

Hybrid Classifiers for Spatio-Temporal Abnormal Behavior Detection, Tracking, and Recognition in Massive Hajj Crowds

Individual abnormal behaviors vary depending on crowd sizes, contexts, and scenes. Challenges such as partial occlusions, blurring, a large number of abnormal behaviors, and camera viewing occur in large-scale crowds when detecting, tracking, and recognizing individuals with abnormalities. In this paper, our contribution is two-fold. First, we introduce an annotated and labeled large-scale crowd abnormal behavior Hajj dataset, HAJJv2. Second, we propose two methods of hybrid convolutional neural networks (CNNs) and random forests (RFs) to detect and recognize spatio-temporal abnormal behaviors in small and large-scale crowd videos. In small-scale crowd videos, a ResNet-50 pre-trained CNN model is fine-tuned to verify whether every frame is normal or abnormal in the spatial domain. If anomalous behaviors are observed, a motion-based individual detection method based on the magnitudes and orientations of Horn–Schunck optical flow is proposed to locate and track individuals with abnormal behaviors. A Kalman filter is employed in large-scale crowd videos to predict and track the detected individuals in the subsequent frames. Then, means and variances as statistical features are computed and fed to the RF classifier to classify individuals with abnormal behaviors in the temporal domain. In large-scale crowds, we fine-tune the ResNet-50 model using a YOLOv2 object detection technique to detect individuals with abnormal behaviors in the spatial domain. The proposed method achieves 99.76% and 93.71% of average area under the curves (AUCs) on two public benchmark small-scale crowd datasets, UMN and UCSD, respectively, while the large-scale crowd method achieves 76.08% average AUC using the HAJJv2 dataset. Our method outperforms state-of-the-art methods using the small-scale crowd datasets with a margin of 1.66%, 6.06%, and 2.85% on UMN, UCSD Ped1, and UCSD Ped2, respectively. It also produces an acceptable result in large-scale crowds.

Scale-aware CNN for crowd density estimation and crowd behavior analysis

Visual surveillance is a highly demanded system in various real-time applications for civil and military sectors. For huge gatherings such as rallies and stadiums, crowd monitoring becomes a tedious task. Moreover, crowd density estimation also plays an important role to improve the performance of crowd monitoring system. Several techniques have been presented for crowd counting or density estimation and crowd behavior analysis. The crowd scenes contain noise, occlusion, cluttered environment which creates complexity to analyze the crowd behavior. To overcome the challenges of crowd monitoring, we present a combined solution to perform the crowd density estimation and crowd behavior analysis. The crowd density estimation is carried out by developing a new CNN based architecture which considers scale information and overcome the issues of scale variations. In order to deal with this issue of scale variation, we introduce a scale-aware attention module where multiple branches of self-attention module are concatenated to improve the scale variation realization. Further, we present the crowd behavior analysis model which uses motion map and energy level distribution based features to detect the abnormal crowd behavior. The Proposed approach achieves average accuracy of abnormal event detection as 98.95, and crowd counting accuracy as 97.60. Furthermore, the average MAE and MSE is achieved as 58.60, 98.55 for ShanghaiTech Part A dataset and 7.55, 8.50 for Part B dataset, respectively. Similarly, the MAE and MSE for UCF 50 dataset is obtained as 210.20, and 260.80, respectively which shows a significant improvement over other techniques discussed in the experimental study.

Abnormal Crowd Behavior Detection Using Optimized Pyramidal Lucas-Kanade Technique

Abnormal behavior detection is challenging and one of the growing research areas in computer vision. The main aim of this research work is to focus on panic and escape behavior detections that occur during unexpected/uncertain events. In this work, Pyramidal Lucas Kanade algorithm is optimized using EMEHOs to achieve the objective. First stage, OPLKT-EMEHOs algorithm is used to generate the optical flow from MIIs. Second stage, the MIIs optical flow is applied as input to 3 layer CNN for detect the abnormal crowd behavior. University of Minnesota (UMN) dataset is used to evaluate the proposed system. The experimental result shows that the proposed method provides better classification accuracy by comparing with the existing methods. Proposed method provides 95.78% of precision, 90.67% of recall, 93.09% of f-measure and accuracy with 91.67%.

Taxonomy of Anomaly Detection Techniques in Crowd Scenes.

With the widespread use of closed-circuit television (CCTV) surveillance systems in public areas, crowd anomaly detection has become an increasingly critical aspect of the intelligent video surveillance system. It requires workforce and continuous attention to decide on the captured event, which is hard to perform by individuals. The available literature on human action detection includes various approaches to detect abnormal crowd behavior, which is articulated as an outlier detection problem. This paper presents a detailed review of the recent development of anomaly detection methods from the perspectives of computer vision on different available datasets. A new taxonomic organization of existing works in crowd analysis and anomaly detection has been introduced. A summarization of existing reviews and datasets related to anomaly detection has been listed. It covers an overview of different crowd concepts, including mass gathering events analysis and challenges, types of anomalies, and surveillance systems. Additionally, research trends and future work prospects have been analyzed.

Crowd Abnormality Detection Using Optical Flow and GLCM-Based Texture Features

Detection of abnormal crowd behavior is one of the important tasks in real-time video surveillance systems for public safety in public places such as subway, shopping malls, sport complexes and various other public gatherings. Due to high density crowded scenes, the detection of crowd behavior becomes a tedious task. Hence, crowd behavior analysis becomes a hot topic of research and requires an approach with higher rate of detection. In this work, the focus is on the crowd management and present an end-to-end model for crowd behavior analysis. A feature extraction-based model using contrast, entropy, homogeneity, and uniformity features to determine the threshold on normal and abnormal activity has been proposed in this paper. The crowd behavior analysis is measured in terms of receiver operating characteristic curve (ROC) & area under curve (AUC) for UMN dataset for the proposed model and compared with other crowd analysis methods in literature to prove its worthiness. YouTube video sequences also used for anomaly detection.

Simulation of crowd dynamics in pedestrian evacuation concerning panic contagion: A cellular automaton approach

The study of the panic evacuation process is of great significance to emergency management. Panic not only causes negative emotions such as irritability and anxiety, but also affects the pedestrians decision-making process, thereby inducing the abnormal crowd behavior. Prompted by the epidemiological SIR model, an extended floor field cellular automaton model was proposed to investigate the pedestrian dynamics under the threat of hazard resulting from the panic contagion. In the model, the conception of panic transmission status (PTS) was put forward to describe pedestrians’ behavior who could transmit panic emotions to others. The model also indicated the pedestrian movement was governed by the static and hazard threat floor field. Then rules that panic could influence decision-making process were set up based on the floor field theory. The simulation results show that the stronger the pedestrian panic, the more sensitive pedestrians are to hazards, and the less able to rationally find safe exits. However, when the crowd density is high, the panic contagion has a less impact on the evacuation process of pedestrians. It is also found that when the hazard position is closer to the exit, the panic will propagate for a longer time and have a greater impact on the evacuation. The results also suggest that as the extent of pedestrian’s familiarity with the environment increases, pedestrians spend less time to escape from the room and are less sensitive to the hazard. In addition, it is essential to point out that, compared with the impact of panic contagion, the pedestrian’s familiarity with environment has a more significant influence on the evacuation.

Variational Abnormal Behavior Detection With Motion Consistency.

Abnormal crowd behavior detection has recently attracted increasing attention due to its wide applications in computer vision research areas. However, it is still an extremely challenging task due to the great variability of abnormal behavior coupled with huge ambiguity and uncertainty of video contents. To tackle these challenges, we propose a new probabilistic framework named variational abnormal behavior detection (VABD), which can detect abnormal crowd behavior in video sequences. We make three major contributions: (1) We develop a new probabilistic latent variable model that combines the strengths of the U-Net and conditional variational auto-encoder, which also are the backbone of our model; (2) We propose a motion loss based on an optical flow network to impose the motion consistency of generated video frames and input video frames; (3) We embed a Wasserstein generative adversarial network at the end of the backbone network to enhance the framework performance. VABD can accurately discriminate abnormal video frames from video sequences. Experimental results on UCSD, CUHK Avenue, IITB-Corridor, and ShanghaiTech datasets show that VABD outperforms the state-of-the-art algorithms on abnormal crowd behavior detection. Without data augmentation, our VABD achieves 72.24% in terms of AUC on IITB-Corridor, which surpasses the state-of-the-art methods by nearly 5%.

Motion-shape-based deep learning approach for divergence behavior detection in high-density crowd

We propose a novel method of abnormal crowd behavior detection in surveillance videos. Mainly, our work focuses on detecting crowd divergence behavior that can lead to serious disasters like a stampede. We introduce a notion of physically capturing motion in the form of images and classify crowd behavior using a convolution neural network (CNN) trained on motion-shape images (MSIs). First, the optical flow (OPF) is computed, and finite-time Lyapunov exponent (FTLE) field is obtained by integrating OPF. Lagrangian coherent structure (LCS) in the FTLE field represents crowd-dominant motion. A ridge extraction scheme is proposed for the conversion of LCS-to-grayscale MSIs. Lastly, a supervised training approach is utilized with CNN to predict normal or divergence behavior for any unknown image. We test our method on six real-world low- as well as high-density crowd datasets and compare performance with state-of-the-art methods. Experimental results show that our method is not only robust for any type of scene but also outperform existing state-of-the-art methods in terms of accuracy. We also propose a divergence localization method that not only identifies divergence starting (source) points but also comes with a new feature of generating a ‘localization mask’ around the diverging crowd showing the size of divergence. Finally, we also introduce two new datasets containing videos of crowd normal and divergence behaviors at the high density.

Efficient Anomaly Detection in Crowd Videos Using Pre-Trained 2D Convolutional Neural Networks

Surveillance of crowded places can benefit from improved techniques of anomaly detection in crowd videos. Several existing methods have detected various types of crowd abnormal behaviors by using spatial and temporal information got from videos. So far as real-time detection of anomalies is concerned, special attention must be given to reducing the model complexity that leads to computational and memory loads. This paper proposes a low computational cost approach to detect crowd anomalies. The proposed approach avoids the expensive optical flow calculations by adopting a pre-trained 2D convolutional neural network (CNN) for motion information and implements a lighter form of the 2D CNN to achieve high recognition accuracy at low computational cost. Experiments on public datasets show that the proposed model outperforms the existing approaches in terms of detection accuracy alongside providing better performance in generating input frames.

Crowd Counting and Abnormal Behavior Detection via Multiscale GAN Network Combined with Deep Optical Flow

Aiming at the problem of low performance of crowd abnormal behavior detection caused by complex backgrounds and occlusions, this paper proposes a single-image crowd counting and abnormal behavior detection via multiscale GAN network. The proposed method firstly designed an embedded GAN module with a multibranch generator and a regional discriminator to initially generate crowd-density maps; and then our proposed multiscale GAN module is added to further strengthen the generalization ability of the model, which can effectively improve the accuracy and robustness of the prediction detection and counting. On the basis of single-image crowd counting, synthetic optical-flow feature descriptor is adopted to obtain the crowd motion trajectory, and the classification of abnormal behavior is finally implemented. The simulation results show that the proposed algorithm can significantly improve the accuracy and robustness of crowd counting and abnormal behavior detection in real complex scenarios compared with the existing mainstream algorithms, which is suitable for engineering applications.

Crowd density estimation method based on floor area

Crowd density estimation is a hot issue in the analysis of crowd abnormal behavior. At present, the crowd density is calculated by the ratio of population and area. However, these methods ignore the influence of monitoring perspective and local crowd distribution in the selection and processing of area. In this paper, we propose a crowd density estimation method based on the floor area, which can accurately estimate the density of low and medium density crowd. Based on crowd density estimation, we propose a new division method of crowd density rank, which can effectively judge the crowd density rank at each moment. The proposed method is executed by the data sets of Mall, Smarty City and USCD. The experimental results show that proposed method can effectively estimate the crowd density value in the low and medium density crowd.

Real-time adversarial GAN-based abnormal crowd behavior detection

Detecting abnormal events in the crowd is a challenging problem. Insufficient samples make those traditional model-based methods cannot cope with sophisticated anomaly monitoring. Therefore, we design a real-time generative adversarial network plus an add-on encoder to deal with the continually changing environment. After the generator reconstructs the compressed pattern to generate the design to the latent vector, a discriminator is used to construct better videos by minimizing the adversarial loss function. We calculated the abnormal score by the distance between the two underlying patterns encoded by the first and the second encoders. The unusual event is detected when the anomaly score is above the threshold. To accelerate the processing efficiency, we introduced the grouped pointwise convolution method to decrease the computing complexity. The frame-level and video-level experiments on the benchmark dataset show the accuracy and reliance of our approach. The acceleration approach can increase the efficiency of the network with only limited accuracy loss.

A Multitask Cascading CNN with MultiScale Infrared Optical Flow Feature Fusion-Based Abnormal Crowd Behavior Monitoring UAV.

Visual-based object detection and understanding is an important problem in computer vision and signal processing. Due to their advantages of high mobility and easy deployment, unmanned aerial vehicles (UAV) have become a flexible monitoring platform in recent years. However, visible-light-based methods are often greatly influenced by the environment. As a result, a single type of feature derived from aerial monitoring videos is often insufficient to characterize variations among different abnormal crowd behaviors. To address this, we propose combining two types of features to better represent behavior, namely, multitask cascading CNN (MC-CNN) and multiscale infrared optical flow (MIR-OF), capturing both crowd density and average speed and the appearances of the crowd behaviors, respectively. First, an infrared (IR) camera and Nvidia Jetson TX1 were chosen as an infrared vision system. Since there are no published infrared-based aerial abnormal-behavior datasets, we provide a new infrared aerial dataset named the IR-flying dataset, which includes sample pictures and videos in different scenes of public areas. Second, MC-CNN was used to estimate the crowd density. Third, MIR-OF was designed to characterize the average speed of crowd. Finally, considering two typical abnormal crowd behaviors of crowd aggregating and crowd escaping, the experimental results show that the monitoring UAV system can detect abnormal crowd behaviors in public areas effectively.

Real-time and accurate abnormal behavior detection in videos

Abnormal crowd behavior detection is a hot research topic in the field of computer vision. In order to solve the problems of high computational cost and the imbalance between positive and negative samples, we propose an efficient algorithm that can detect and locate anomalies in videos. In order to solve the problem of less negative samples, the algorithm uses the spatiotemporal autoencoder to identify and extract the negative samples (contain abnormal behaviors) in the dataset in an unsupervised learning method. On this basis, a spatiotemporal convolutional neural network (CNN) is constructed with simple structure and low computational complexity. The supervised training method is used to train the spatiotemporal CNN with positive and negative samples to generate the detection model. Experiments are conducted on the UCSD and UMN datasets. The experiment results show that the proposed algorithm can detect and locate abnormal behaviors in real time (using only CPU), and the accuracy of the algorithm exceeds those of the existing algorithms at both the pixel level and frame level.

Using Deep Learning Models Combined with Crowd Emotion Models to Identify Abnormal Behaviors in Crowds

Aiming at the problem that the definition of crowd abnormal behavior detection is ambiguous and difficult to combine with context semantics, an algorithm using OCC human emotion model combined with crowd entropy is proposed. First calculate the crowd entropy for the crowd, and determine whether the entropy value is abnormal, if it is abnormal, further extract the optical flow OF and HOG. Then project it into two-dimensional vector data, send it to CNN for local feature extraction and combine with OCC model to achieve the description of crowd emotions. Finally, predict whether the abnormality occurs according to the judgment factor. Verified on the data set, this method shows a high accuracy.

A novel framework and concept-based semantic search Interface for abnormal crowd behaviour analysis in surveillance videos

Monitoring continuously captured surveillance videos is a challenging and time consuming task. To assist this issue, a new framework is introduced that applies anomaly detection, semantic annotation and provides a concept-based search interface. In particular, novel optical flow based features are used for abnormal crowd behaviour detection. Then, processed surveillance videos are annotated using a new semantic metadata model based on multimedia standards using Semantic Web technologies. In this way, globally inter-operable metadata about abnormal crowd behaviours are generated. Finally, for the first time, based on crowd behaviours, a novel concept-based semantic search interface is proposed. In the proposed interface, along with search results (video segments), statistical data about crowd behaviours are also presented. With extensive user studies, it is demonstrated that the proposed concept-based semantic search interface enables efficient search and analysis of abnormal crowd behaviours. Although there are existing works to achieve (a) crowd anomaly detection, (b) semantic annotation and (c) semantic search interface, none of the existing works combine these three system components in a novel framework like the one proposed in this paper. In each system component, we introduce contributions to the field as well as use the Semantic Web technologies to combine and standardize output of different system components; output of the anomaly detection is automatically annotated with metadata and stored to a semantic database. When continuous surveillance videos are processed, only the semantic database is updated. Finally, the user interface queries the updated database for searching/analyzing surveillance videos without changing any coding. Thus, the framework supports re-usability. This paper explains and evaluates different components of the framework.

Crowd Analysis System for Images of CCTV Camera

It is important to monitor and analyze the crowd for peaceful event organization and minimum number of persons killed or injured in a stampede, war or accident in public or religious places. Ultimately, crowd management strategies should be adopted for crowd safety. Detecting the abnormal crowd behaviour is the interesting research area. Traditionally to monitor the crowd closed circuit television is used. Video analyses are used in many application areas. This work is a paper of traditional approach and is a study of different attributes of crowd which include crowd counting and estimation of density, detection of crowd motion. It also presents the study about behaviour understanding of crowd and crowd tracking. We have also presents crowd analysis using different deep learning methods. This paper showed results of crowd analysis using convolutional neural network and different recurrent neural network training model.

Abnormal Crowd Behavior Detection Using Motion Information Images and Convolutional Neural Networks

We introduce a novel method for abnormal crowd event detection in surveillance videos. Particularly, our work focuses on panic and escape behavior detection that may appear because of violent events and natural disasters. First, optical flow vectors are computed to generate a motion information image (MII) for each frame, and then MIIs are used to train a convolutional neural network (CNN) for abnormal crowd event detection. The proposed MII is a new formulation that provides a visual appearance of crowd motion. The proposed MIIs make the discrimination between normal and abnormal behaviors easier. The MII is mainly based on the optical flow magnitude, and angle difference computed between the optical flow vectors in consecutive frames. A CNN is employed to learn normal and abnormal crowd behaviors using MIIs. The MII generation, and the combination with a CNN is a new approach in the context of abnormal crowd behavior detection. Experiments are performed on commonly used datasets such as UMN and PETS2009. Evaluation indicates that our method achieves the best results.

ABNORMAL CROWD BEHAVIOUR DETECTION BASED ON DEEP LEARNING AND SPARSE REPRESENTATION

This article proposes a new method combining improved principal component analysis network (PCANet) and sparse representation for abnormal crowd behaviour detection on the consideration that “abnormal events are not “normal events.