Crowd Analysis Research Articles

Network as a Sensor for Smart Crowd Analysis and Service Improvement

Network as a Sensor for Smart Crowd Analysis and Service Improvement

Density-based clustering with fully-convolutional networks for crowd flow detection from drones

Crowd analysis from drones has attracted increasing attention in recent times due to the ease of use and affordable cost of these devices. However, how this technology can provide a solution to crowd flow detection is still an unexplored research question. To this end, we propose a crowd flow detection method for video sequences shot by a drone. The method is based on a fully-convolutional network that learns to perform crowd clustering in order to detect the centroids of crowd-dense areas and track their movement in consecutive frames. The proposed method proved effective and efficient when tested on the Crowd Counting datasets of the VisDrone challenge, characterized by video sequences rather than still images. The encouraging results show that the proposed method could open up new ways of analyzing high-level crowd behavior from drones.

Audio-visual multi-modality driven hybrid feature learning model for crowd analysis and classification.

The high pace emergence in advanced software systems, low-cost hardware and decentralized cloud computing technologies have broadened the horizon for vision-based surveillance, monitoring and control. However, complex and inferior feature learning over visual artefacts or video streams, especially under extreme conditions confine majority of the at-hand vision-based crowd analysis and classification systems. Retrieving event-sensitive or crowd-type sensitive spatio-temporal features for the different crowd types under extreme conditions is a highly complex task. Consequently, it results in lower accuracy and hence low reliability that confines existing methods for real-time crowd analysis. Despite numerous efforts in vision-based approaches, the lack of acoustic cues often creates ambiguity in crowd classification. On the other hand, the strategic amalgamation of audio-visual features can enable accurate and reliable crowd analysis and classification. Considering it as motivation, in this research a novel audio-visual multi-modality driven hybrid feature learning model is developed for crowd analysis and classification. In this work, a hybrid feature extraction model was applied to extract deep spatio-temporal features by using Gray-Level Co-occurrence Metrics (GLCM) and AlexNet transferrable learning model. Once extracting the different GLCM features and AlexNet deep features, horizontal concatenation was done to fuse the different feature sets. Similarly, for acoustic feature extraction, the audio samples (from the input video) were processed for static (fixed size) sampling, pre-emphasis, block framing and Hann windowing, followed by acoustic feature extraction like GTCC, GTCC-Delta, GTCC-Delta-Delta, MFCC, Spectral Entropy, Spectral Flux, Spectral Slope and Harmonics to Noise Ratio (HNR). Finally, the extracted audio-visual features were fused to yield a composite multi-modal feature set, which is processed for classification using the random forest ensemble classifier. The multi-class classification yields a crowd-classification accurac12529y of (98.26%), precision (98.89%), sensitivity (94.82%), specificity (95.57%), and F-Measure of 98.84%. The robustness of the proposed multi-modality-based crowd analysis model confirms its suitability towards real-world crowd detection and classification tasks.

Abnormal Event Detection and Localization Based on Crowd Analysis in Video Surveillance

Abnormal Event Detection and Localization Based on Crowd Analysis in Video Surveillance

Multiple Pedestrian Detection and Tracking in Night Vision Surveillance Systems

Pedestrian detection and tracking are vital elements of today’s surveillance systems, which make daily life safe for humans. Thus, human detection and visualization have become essential inventions in the field of computer vision. Hence, developing a surveillance system with multiple object recognition and tracking, especially in low light and night-time, is still challenging. Therefore, we propose a novel system based on machine learning and image processing to provide an efficient surveillance system for pedestrian detection and tracking at night. In particular, we propose a system that tackles a two-fold problem by detecting multiple pedestrians in infrared (IR) images using machine learning and tracking them using particle filters. Moreover, a random forest classifier is adopted for image segmentation to identify pedestrians in an image. The result of detection is investigated by particle filter to solve pedestrian tracking. Through the extensive experiment, our system shows 93% segmentation accuracy using a random forest algorithm that demonstrates high accuracy for background and roof classes. Moreover, the system achieved a detection accuracy of 90% using multiple template matching techniques and 81% accuracy for pedestrian tracking. Furthermore, our system can identify that the detected object is a human. Hence, our system provided the best results compared to the state-of-art systems, which proves the effectiveness of the techniques used for image segmentation, classification, and tracking. The presented method is applicable for human detection/tracking, crowd analysis, and monitoring pedestrians in IR video surveillance.

Focal Inverse Distance Transform Maps for Crowd Localization

In this paper, we focus on the crowd localization task, a crucial topic of crowd analysis. Most regression-based methods utilize convolution neural networks (CNN) to regress a density map, which can not accurately locate the instance in the extremely dense scene, attributed to two crucial reasons: 1) the density map consists of a series of blurry Gaussian blobs, 2) severe overlaps exist in the dense region of the density map. To tackle this issue, we propose a novel Focal Inverse Distance Transform (FIDT) map for the crowd localization task. Compared with the density maps, the FIDT maps accurately describe the persons' locations without overlapping in dense regions. Based on the FIDT maps, a Local-Maxima-Detection-Strategy (LMDS) is derived to effectively extract the center point for each individual. Furthermore, we introduce an Independent SSIM (I-SSIM) loss to make the model tend to learn the local structural information, better recognizing local maxima. Extensive experiments demonstrate that the proposed method reports state-of-the-art localization performance on six crowd datasets and one vehicle dataset. Additionally, we find that the proposed method shows superior robustness on the negative and extremely dense scenes, which further verifies the effectiveness of the FIDT maps. The code and model will be available at https://github.com/dk-liang/FIDTM.

Sensing Technologies for Crowd Management, Adaptation, and Information Dissemination in Public Transportation Systems: A Review

Management of crowd information in public transportation (PT) systems is crucial, both to foster sustainable mobility, by increasing the user's comfort and satisfaction during normal operation, as well as to cope with emergency situations, such as pandemic crises, as recently experienced with COVID-19 limitations. This paper presents a taxonomy and review of sensing technologies based on Internet of Things (IoT) for real-time crowd analysis, which can be adopted in the different segments of the PT system (buses/trams/trains, railway/metro stations, and bus/tram stops). To discuss such technologies in a clear systematic perspective, we introduce a reference architecture for crowd management, which employs modern information and communication technologies (ICT) in order to: (i) monitor and predict crowding events; (ii) implement crowd-aware policies for real-time and adaptive operation control in intelligent transportation systems (ITSs); (iii) inform in real-time the users of the crowding status of the PT system, by means of electronic displays installed inside vehicles or at bus/tram stops/stations, and/or by mobile transport applications. It is envisioned that the innovative crowd management functionalities enabled by ICT/IoT sensing technologies can be incrementally implemented as an add-on to state-of-the-art ITS platforms, which are already in use by major PT companies operating in urban areas. Moreover, it is argued that, in this new framework, additional services can be delivered to the passengers, such as, e.g., on-line ticketing, vehicle access control and reservation in severely crowded situations, and evolved crowd-aware route planning.

Deformable Density Estimation via Adaptive Representation.

Crowd counting is the basic task of crowd analysis and it is of great significance in the field of public safety. Therefore, it receives more and more attention recently. The common idea is to combine the crowd counting task with convolutional neural networks to predict the corresponding density map, which is generated by filtering the dot labels with specific Gaussian kernels. Although the counting performance is promoted by the newly proposed networks, they all suffer one conjunct problem, which is due to the perspective effect, there is significant scale contrast among targets in different positions within one scene, but the existing density maps can not represent this scale change well. To address the prediction difficulties caused by target scale variation, we propose a scale-sensitive crowd density map estimation framework, which focuses on dealing with target scale change from density map generation, network design, and model training stage. It consists of the Adaptive Density Map (ADM), Deformable Density Map Decoder (DDMD), and Auxiliary Branch. To be specific, the Gaussian kernel size variates adaptively based on target size to generate ADM that contains scale information for each specific target. DDMD introduces the deformable convolution to fit the Gaussian kernel variation and boosts the model's scale sensitivity. The Auxiliary Branch guides the learning of deformable convolution offsets during the training phase. Finally, we construct experiments on different large-scale datasets. The results show the effectiveness of the proposed ADM and DDMD. Furthermore, the visualization demonstrates that deformable convolution learns the target scale variation.

Co-Communication Graph Convolutional Network for Multi-View Crowd Counting

We study and address the multi-view crowd counting (MVCC) problem which poses more realistic challenges than single-view crowd counting for better facilitating crowd management / public safety systems. Its major challenge lies in how to fully distill and aggregate useful, complementary information among multiple camera views to create powerful ground-plane representations for wide-area crowd analysis. In this paper, we present a graph-based, multi-view learning model called Co-Communication Graph Convolutional Network (CoCo-GCN) to jointly investigate intra-view contextual dependencies and inter-view complementary relations. More specifically, CoCo-GCN builds a view-agnostic graph interaction space for each camera view to conduct efficient contextual reasoning, and extends the intra-view reasoning by using a novel Graph Communication Layer (GCL) to also take between-graph (cross-view), complementary information into account. Moreover, CoCo-GCN uses a new Co-Memory Layer (CoML) to jointly coarsen the graphs and close the ‘representational gap’ among them for further exploiting the compositional nature of graphs and learning more consistent representations. Finally, these jointly learned features of multiple views can be easily fused to create ground-plane representations for wide-area crowd counting. Experiments show that the proposed CoCo-GCN achieves state-of-the-art results on three MVCC datasets, i.e., PETS2009, DukeMTMC, and City Street, significantly improving the scene-level accuracy over previous models.

The Development of an Improvised Crowd Analysis Model

Abstract: The goal of this project is to create a deep learning-based visual crowd counting system. The objective of this project is to build a functioning system that can analyze pictures and determine the number of individuals present within these images. It will also demonstrate its approximate density map, a graph comparing the expected count to the actual count, and information on its accuracy in terms of Mean Absolute Error (MAE). To routinely supervise audiences, researchers recently moved to computer vision. This research analysis proposed the implementation of a Deep learning algorithm CNN for which the aim was to detect the crowd and estimate increased influx of people which has been successfully achieved by employing deep learning technique. We have used the shanghai tech dataset part B for our research purpose. CNN model detected the crowd and estimated the density with an absolute error of 21. In 312 we obtained a validation mean absolute error of 21.3, which means on average, the model will estimate 21 persons in excess or deficit.

A Research Perceptive on Deep Learning Framework for Pedestrian Detection in a Crowd

In populated cities, we often find crowded events like political meetings, religious festivals, music concerts, and events in shopping malls, which have more safety issues. Smart surveillance systems are used in big cities to keep crowds safe and make crowd security less complicated and more accurate. However, the surveillance systems proposed for a crowd are monitored by human agents, which are inefficient, error-prone, and overwhelming. Even with deep learning-based feature engineering in crowds, many variants of crowd analysis still lack attention and are technically unaddressed. Considering this scenario, the smart system requires the most advanced techniques to monitor the security of the crowd. Crowd analysis is commonly divided into crowd statics and behavior analysis. This paper explores more about crowd behaviour analysis, pedestrian and group detection which describes the movements that are noticed in the crowd image. Subsequently, the issues of the current methodology of pedestrian detection, datasets, and evaluation criteria are analyzed. Keyword : Crowd Analysis, Pedestrian and group detection, deep learning, Crowd IoT analysis, Human Activity Recognition.

A point and density map hybrid network for crowd counting and localization based on unmanned aerial vehicles

ABSTRACT Crowd counting and localisation are essential tasks in crowd analysis and are vital to ensure public safety. However, these tasks via UAV bring new obstacles compared with video surveillance (e.g. viewpoint and scale variations, background clutter, and small scales). To overcome the difficulties, this research presents a novel network named PDNet. It employs the multi-task learning approach to combine the point regression and density map regression. PDNet includes a backbone to extract multi-scale features, a Dilated Feature Fusion module (DFF), a Density Map Attention module (DMA), a density map branch and a point branch. Aims of DFF is to address the difficulties of small targets and scale variations by establishing relationships between targets and their surroundings. DMA is created to address the challenges of complicated backgrounds, allowing the PDNet to focus on the target's location. In addition, the density map branch and point branch are designed for density maps regression and point regression, respectively. Experiments on the DroneCrowd dataset demonstrate that our proposed network outperforms state-of-the-art approaches in terms of localisation, L-mAP (53.85%), L-AP@10 (59.14%), L-AP@15 (63.64%), and L-AP@20 (66.21%), and we improved counting performance and significantly reduced inference time. In addition, ablation experiments are conducted to prove the modules' effectiveness.

A Unified Multiple-Motion-Mode Framework for Socially Compliant Navigation in Dense Crowds

Mobile robots are expected to move safely and efficiently in socially-compliant ways in dense crowds. In this paper, a unified multiple-motion-mode framework is proposed to achieve socially-compliant navigation in dense crowds. The proposed framework consists of three successive phases, the identification of pedestrian groups and prediction of their trajectories, the generation of candidate local trajectories of the robot, and the determination of an optimal local trajectory. The pedestrians are grouped according to distances among pedestrians and their velocities and a predictor predicts the groups’ trajectories. Three robot motion modes, moving-solo, pedestrian-following, and courteous-stopping, are proposed based on a systematic analysis and classification of dense crowds and they are utilized to generate candidate local trajectories. A composite metric is proposed to determine the best local trajectory by simultaneously considering the robot motion safety, efficiency, and stability of the trajectory. The proposed robot navigation framework is evaluated both in simulations and real-world experiments with various scenarios. The results show that the proposed framework surpasses the state-of-the-art methods in terms of efficiency and social compliance. Note to Practitioners—This paper is motivated by the socially-compliant navigation problem of mobile robots in densely crowded environments such as hospitals and airports. Robots navigation in these scenes need to follow collective social conventions to ensure ordered and efficient public pedestrian traffics, and individual social conventions for acceptance by human society. Existing approaches generally consider only one type of social conventions cannot accomplish a truly socially-compliant navigation and cannot be naturally accepted in the human pedestrian environment. This paper proposes to imitate the pedestrian’s walking behavior for robot navigation by naturally choosing one of the three walking modes, moving-solo, following, and courteous-stopping. We construct the candidate trajectory sampling strategies for different motion modes based on the global path and the two types of social conventions. Experimental results show that the proposed framework can be applied in large-scale, densely crowded, unstructured, and human-robot coexisting environments for socially-compliant navigation. However, the proposed navigation framework has limited improvement in navigation efficiency in a highly crowded environment due to the limited feasible space. Even pedestrians are difficult to walk in these highly crowded environments. In future work, we will study how to auto-adjust the weights to steer the robot to efficient navigation in highly crowded environments.

A human behavior analysis model to track object behavior in surveillance videos

In potential disasters, real world scenarios and public events, understanding of crowd psychology is a challenging process and detection of crowd behaviors in those events is quite complicated. Therefore, in this article, an effective crowd analysis mechanism is introduced using human behavior analysis model based on motion heat flow enabled optical flow method. Here, OCP descriptor evaluates required object points from motion heat map to detect objects and generate feature weights. Here, three type of histogram gradients are evaluated using proposed HBA model such as inconsistencies between neighboring points and OCP descriptor, gradients obtained considering spatial angle and gradients obtained considering temporal angle. Features obtained considering spatial as well as temporal domain are encoded using feature encoding scheme. The performance of proposed Human behavior Analysis (HBA) model is tested on Web and Violent-Flow Video Dataset. The performance of the proposed HBA model is compared against several traditional behavior analysis methods considering performance matrices like Accuracy, Precision and Recall to detect behaviors in public events. Performance of proposed HBA model is superior to multiple state-of-art-crowd behavior analysis techniques.

A review on crowd analysis of evacuation and abnormality detection based on machine learning systems

A review on crowd analysis of evacuation and abnormality detection based on machine learning systems

Taxonomy of Anomaly Detection Techniques in Crowd Scenes.

With the widespread use of closed-circuit television (CCTV) surveillance systems in public areas, crowd anomaly detection has become an increasingly critical aspect of the intelligent video surveillance system. It requires workforce and continuous attention to decide on the captured event, which is hard to perform by individuals. The available literature on human action detection includes various approaches to detect abnormal crowd behavior, which is articulated as an outlier detection problem. This paper presents a detailed review of the recent development of anomaly detection methods from the perspectives of computer vision on different available datasets. A new taxonomic organization of existing works in crowd analysis and anomaly detection has been introduced. A summarization of existing reviews and datasets related to anomaly detection has been listed. It covers an overview of different crowd concepts, including mass gathering events analysis and challenges, types of anomalies, and surveillance systems. Additionally, research trends and future work prospects have been analyzed.

Deep Dilated Convolutional Neural Network for Crowd Density Image Classification with Dataset Augmentation for Hajj Pilgrimage.

Almost two million Muslim pilgrims from all around the globe visit Mecca each year to conduct Hajj. Each year, the number of pilgrims grows, creating worries about how to handle such large crowds and avoid unpleasant accidents or crowd congestion catastrophes. In this paper, we introduced deep Hajj crowd dilated convolutional neural network (DHCDCNNet) for crowd density analysis. This research also presents augmentation technique to create additional dataset based on the hajj pilgrimage scenario. We utilized a single framework to extract both high-level and low-level features. For creating additional dataset we divide the process of images augmentation into two routes. In the first route, we utilized magnitude extraction followed by the polar magnitude. In the second route, we performed morphological operation followed by transforming the image into skeleton. This paper presented a solution to the challenge of measuring crowd density using a surveillance camera pointed at a distance. An FCNN-based technique for crowd analysis is included in the proposed methodology, particularly for classifying crowd density. There are several obstacles in video analysis when there are a large number of pilgrims moving around the tawaf area, with densities of between 7 and 8 per square meter. The proposed DHCDCNNet method has achieved accuracy of 97%, 89% and 100% for the JHU-CROWD dataset, the UCSD dataset and the proposed Hajj-Crowd dataset, respectively. The proposed Hajj-Crowd dataset, the UCSD dataset, and the JHU-CROW dataset all had accuracy of 98%, 97% and 97%, respectively, using the VGGNet approach. Using the ResNet50 approach, the proposed Hajj-Crowd dataset, the UCSD dataset, and the JHU-CROW dataset all had an accuracy of 99%, 91% and 97%, respectively.

Frequency Feature Pyramid Network With Global-Local Consistency Loss for Crowd-and-Vehicle Counting in Congested Scenes

Context prediction plays a crucial role in implementing autonomous driving applications. As one of important context-prediction tasks, crowd-and-vehicle counting is critical for achieving real-time traffic and crowd analysis, consequently facilitating decision-making processes for autonomous vehicles. However, the completion of crowd-and-vehicle counting also faces challenges, such as large-scale variations, imbalanced data distribution, and insufficient local patterns. To tackle these challenges, we put forth a novel frequency feature pyramid network (FFPNet) in this paper. Our proposed FFPNet extracts the multi-scale information by frequency feature pyramid module, which can tackle the issue of large-scale variations. Meanwhile, the frequency feature pyramid module uses different frequency branches to obtain different scale information. We also adopt the attention mechanism to strength the extraction of different scale information. Moreover, we devise a novel loss function, namely global-local consistency loss, to address the existing problems of imbalanced data distribution and insufficient local patterns. Furthermore, we conduct extensive experiments on six datasets to evaluate our proposed FFPNet. It is worth mentioning that we also construct a novel crowd-and-vehicle dataset (CROVEH), which is the only dataset that contains both crowd-and-vehicle annotations. The experimental results show that FFPNet achieves the best performance on different backbones, e.g., 52.69 mean absolute error (MAE) on P2PNet with FFP module. The codes are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/MUST-AI-Lab/FFPNet</uri> .

Comparative Study of Various Crowd Detection and Classification Methods for Safety Control System

A crowd is a distinct collection of people or anything that is involved in community or society. The phenomenon of a crowd is fairly well known in a wide range of academic fields, including sociology, civil engineering, and physics, amongst others. At this point in time, it has developed into the most active-oriented research and fashionable issue in the field of computer vision. Pre-processing, object detection, and event or behavior identification are the three stages of processing that are traditionally included in crowd analysis. These stages are pre-processing, object detection, and event recognition. Pre-processing, object detection, and event or behaviour identification are the three stages of processing that are traditionally included in crowd analysis. These stages are pre-processing, object detection, and event recognition. This study gives a model of crowd analysis as well as a taxonomy of the most prevalent method to crowd analysis. It may be helpful to researchers and would serve as a good introduction connected to the area of work that has been conducted.

Extrinsic Behavior Prediction of Pedestrians via Maximum Entropy Markov Model and Graph-Based Features Mining

With the change of technology and innovation of the current era, retrieving data and data processing becomes a more challenging task for researchers. In particular, several types of sensors and cameras are used to collect multimedia data from various resources and domains, which have been used in different domains and platforms to analyze things such as educational and communicational setups, emergency services, and surveillance systems. In this paper, we propose a robust method to predict human behavior from indoor and outdoor crowd environments. While taking the crowd-based data as input, some preprocessing steps for noise reduction are performed. Then, human silhouettes are extracted that eventually help in the identification of human beings. After that, crowd analysis and crowd clustering are applied for more accurate and clear predictions. This step is followed by features extraction in which the deep flow, force interaction matrix and force flow features are extracted. Moreover, we applied the graph mining technique for data optimization, while the maximum entropy Markov model is applied for classification and predictions. The evaluation of the proposed system showed 87% of mean accuracy and 13% of error rate for the avenue dataset, while 89.50% of mean accuracy rate and 10.50% of error rate for the University of Minnesota (UMN) dataset. In addition, it showed a 90.50 mean accuracy rate and 9.50% of error rate for the A Day on Campus (ADOC) dataset. Therefore, these results showed a better accuracy rate and low error rate compared to state-of-the-art methods.