Crowded Scenarios Research Articles

Cascade Transformer Decoder Based Occluded Pedestrian Detection With Dynamic Deformable Convolution and Gaussian Projection Channel Attention Mechanism

Occluded pedestrian detection is very challenging in computer vision, because the pedestrians are frequently occluded by various obstacles or persons, especially in crowded scenarios. In this paper, an occluded pedestrian detection method is proposed under a basic DEtection TRansformer (DETR) framework. Firstly, Dynamic Deformable Convolution (DyDC) and Gaussian Projection Channel Attention (GPCA) mechanism are proposed and embedded into the low layer and high layer of ResNet50 respectively, to improve the representation capability of features. Secondly, Cascade Transformer Decoder (CTD) is proposed, which aims to generate high-score queries, avoiding the influence of low-score queries in the decoder stage, further improving the detection accuracy. The proposed method is verified on three challenging datasets, namely CrowdHuman, WiderPerson, and TJU-DHD-pedestrian. The experimental results show that, compared with the state-of-the-art methods, it can obtain a superior detection performance.

Crowd Localization from Gaussian Mixture Scoped Knowledge and Scoped Teacher.

Crowd localization is to predict each instance head position in crowd scenarios. Since the distance of pedestrians being to the camera are variant, there exists tremendous gaps among scales of instances within an image, which is called the intrinsic scale shift. The core reason of intrinsic scale shift being one of the most essential issues in crowd localization is that it is ubiquitous in crowd scenes and makes scale distribution chaotic. To this end, the paper concentrates on access to tackle the chaos of the scale distribution incurred by intrinsic scale shift.We propose Gaussian Mixture Scope (GMS) to regularize the chaotic scale distribution. Concretely, the GMS utilizes a Gaussian mixture distribution to adapt to scale distribution and decouples the mixture model into sub-normal distributions to regularize the chaos within the sub-distributions. Then, an alignment is introduced to regularize the chaos among sub-distributions. However, despite that GMS is effective in regularizing the data distribution, it amounts to dislodging the hard samples in training set, which incurs overfitting. We assert that it is blamed on the block of transferring the latent knowledge exploited by GMS from data to model. Therefore, a Scoped Teacher playing a role of bridge in knowledge transform is proposed. What' s more, the consistency regularization is also introduced to implement knowledge transform. To that effect, the further constraints are deployed on Scoped Teacher to derive feature consistence between teacher and student end. With proposed GMS and Scoped Teacher implemented on four mainstream datasets of crowd localization, the extensive experiments demonstrate the superiority of our work. Moreover, comparing with existing crowd locators, our work achieves state-of-the-art via F1-measure comprehensively on four datasets.

Dataset of Large Gathering Images for Person Identification and Tracking

This paper presents a large gathering dataset of images extracted from publicly filmed videos by 24 cameras installed on the premises of Masjid Al-Nabvi, Madinah, Saudi Arabia. This dataset consists of raw and processed images reflecting a highly challenging and unconstraint environment. The methodology for building the dataset consists of four core phases; that include acquisition of videos, extraction of frames, localization of face regions, and cropping and resizing of detected face regions. The raw images in the dataset consist of a total of 4613 frames obtained from video sequences. The processed images in the dataset consist of the face regions of 250 persons extracted from raw data images to ensure the authenticity of the presented data. The dataset further consists of 8 images corresponding to each of the 250 subjects (persons) for a total of 2000 images. It portrays a highly unconstrained and challenging environment with human faces of varying sizes and pixel quality (resolution). Since the face regions in video sequences are severely degraded due to various unavoidable factors, it can be used as a benchmark to test and evaluate face detection and recognition algorithms for research purposes. We have also gathered and displayed records of the presence of subjects who appear in presented frames; in a temporal context. This can also be used as a temporal benchmark for tracking, finding persons, activity monitoring, and crowd counting in large crowd scenarios.

Use of agent-based modeling to model intermediate force capabilities in (counter)mobility crowd scenarios

In this paper, we use an agent-based model (ABM) to run (counter)mobility scenarios to explore which characteristics of intermediate force capabilities (IFC) are relevant to these, and how they can affect outcomes in gray zone conflicts. Using an ABM called Map-Aware Non-Uniform Automata (MANA), developed by the New Zealand Defense Technology Agency, we implemented two scenarios where the friendly forces’ mobility was limited by large groups of civilians. Then, we employed data farming and analytics methods to analyze the data and identify key parameters influencing the outcomes. The main parameters appeared to be the IFC Range, Power (a measure of the duration of the effect), and Crowd Density. Future research could include a wide range of mobility scenarios and possibly a more detailed IFC representation.

Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-Based Approaches

In crowd scenarios, predicting trajectories of pedestrians is a complex and challenging task depending on many external factors. The topology of the scene and the interactions between the pedestrians are just some of them. Due to advancements in data-science and data collection technologies deep learning methods have recently become a research hotspot in numerous domains. Therefore, it is not surprising that more and more researchers apply these methods to predict trajectories of pedestrians. This paper compares these relatively new deep learning algorithms with classical knowledge-based models that are widely used to simulate pedestrian dynamics. It provides a comprehensive literature review of both approaches, explores technical and application oriented differences, and addresses open questions as well as future development directions. Our investigations point out that the pertinence of knowledge-based models to predict local trajectories is nowadays questionable because of the high accuracy of the deep learning algorithms. Nevertheless, the ability of deep-learning algorithms for large-scale simulation and the description of collective dynamics remains to be demonstrated. Furthermore, the comparison shows that the combination of both approaches (the hybrid approach) seems to be promising to overcome disadvantages like the missing explainability of the deep learning approach.

An efficient anchor‐free method for pig detection

Given the rapid growth of commercial pig farms, the need to automatically monitor pig behaviour becomes more important in order to assist farmers. Recent advances in convolutional neural networks may pave the way for new solutions. However, the primary task of individual pig detection under real-world conditions is still a challenging task. Previous studies used anchor-based frameworks that are unsuitable for such crowded scenarios with extreme overlapping. Furthermore, most applications focus on specific levels of brightness, farm facilities, or pig species without considering generalization. To tackle these problems, an anchor-free pig detection method based on pig centre localization is first proposed. Then, a novel negative training data augmentation technique is introduced using examples from outside the training distribution. Furthermore, using the test time augmentation technique is proposed to improve the model performance. Experiments are conducted on two online pig detection datasets; the network surpasses state-of-the-art results for both datasets. It is also found that the proposed method outperforms the latest anchor-free techniques commonly used in crowded scenarios. The method can detect pigs individually, even if their bounding boxes overlap strongly or occlude each other. Moreover, the real-time system achieves an improvement of 10% in F measure $F_{\text{measure}}$ when testing in unconstrained real-world conditions.

Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking

An effective understanding of the environment and accurate trajectory prediction of surrounding dynamic obstacles are indispensable for intelligent mobile systems (e.g., autonomous vehicles, social robots) to achieve safe and high-quality planning when they navigate in highly interactive and crowded scenarios. Due to the existence of frequent interactions and uncertainty in the scene evolution, it is desired for the prediction system to enable relational reasoning on different entities and provide a distribution of future trajectories for each agent. In this paper, we propose a generic generative neural system (called STG-DAT) for multi-agent trajectory prediction involving heterogeneous agents. The system takes a step forward to explicit interaction modeling by incorporating relational inductive biases with a dynamic graph representation and leverages both trajectory and scene context information. We also employ an efficient kinematic constraint layer applied to vehicle trajectory prediction. The constraint not only ensures physical feasibility but also enhances model performance. Moreover, the proposed prediction model can be easily adopted by multi-target tracking frameworks. The tracking accuracy proves to be improved by empirical results. The proposed system is evaluated on three public benchmark datasets for trajectory prediction, where the agents cover pedestrians, cyclists and on-road vehicles. The experimental results demonstrate that our model achieves better performance than various baseline approaches in terms of prediction and tracking accuracy.

Crowd flow forecasting via agent-based simulations with sequential latent parameter estimation from aggregate observation

Unlike conventional crowd simulations for what-if analysis, agent-based crowd simulations for real-time applications are an emerging research topic and an important tool for better crowd managements in smart cities. Recent studies have attempted to incorporate the real-time crowd observations into crowd simulations for real-time crowd forecasting and management; however, crowd flow forecasting considering individual-level microscopic interactions, especially for large crowds, is still challenging. Here, we present a method that incorporates crowd observation data to forecast a large crowd flow, including thousands of individuals, using a microscopic agent-based model. By sequentially estimating both the crowd state and the latent parameter behind the crowd flows from the aggregate crowd density observation with the particle filter algorithm, the present method estimates and forecasts the large crowd flow using agent-based simulations that incorporate observation data. Numerical experiments, including a realistic evacuation scenario with 5000 individuals, demonstrated that the present method could successfully provide reasonable crowd flow forecasting for different crowd scenarios, even with limited information on crowd movements. These results support the feasibility of real-time crowd flow forecasting and subsequent crowd management, even for large but microscopic crowd problems.

Cluster-Aided Collision Resolution Random Access in Distributed Massive MIMO Systems

Massive machine-type communication (MTC) is expected to be one of the key concerns in future B5G and 6G. However, the explosive growth of MTC devices (MTDs) in Internet of Things (IoT) poses great challenges for the initial access procedure. To address this issue, we propose a cluster-aided collision resolution random access (CACRRA) protocol for grant-based random access (RA) in the distributed massive multiple input multiple output (mMIMO) systems, which takes full advantage of the spatial multiplexing provided by distributed mMIMO. By exploiting the difference of large-scale fading coefficients between different access points (APs) and MTDs, we further propose an exclusive clustering algorithm to cluster APs for each pilot. Moreover, the pilot collision resolution is performed distributedly in each cluster by leveraging the channel hardening and the favorable propagation of mMIMO channels, which achieves a large reuse factor of pilots. Simulation results show that the proposed CACRRA can efficiently resolve the pilot collision and significantly improve the access performance. Specifically, the proposed protocol can improve the RA throughput more than nine times compared to the conventional protocol in incredibly crowded scenarios.

Selection of object detections using overlap map predictions

Advances in deep neural networks have led to significant improvement of object detection accuracy. However, object detection in crowded scenarios is a challenging task for neural networks since extremely overlapped objects provide fewer visible cues for a model to learn from. Further complicating the detection of overlapping objects is the fact that most object detectors produce multiple redundant detections for single objects, which are indistinguishable from detections of separate overlapped objects. Most existing works use some variant of non-maximum suppression to prune duplicate candidate bounding boxes based on their confidence scores and the amount of overlap between predicted bounding boxes. These methods are unaware of how much overlap there actually is between the objects in the image, and are therefore inclined to merge detections for highly overlapped objects. In this paper, we propose an overlap aware box selection solution that uses a predicted overlap map to help it decide which highly overlapping bounding boxes are associated with actual overlapping objects and should not be pruned. We show our solution outperforms the state-of-the-art set-NMS bounding box selection algorithm for both the crowdHuman dataset and a sports dataset.

Spatial-Frequency Attention Network for Crowd Counting.

Counting the number of people in crowded scenarios is a crucial task in video surveillance and urban security system. Widely deployed surveillance cameras provide big data for training, a compelling deep learning-based counting network. However, large-scale variations in dense crowds are still not entirely solved. To address this problem, we propose a spatial-frequency attention network (SFANet) for crowd counting in this article. A bottleneck spatial attention module is built to emphasize features in various spatial locations and select a region containing individuals adaptively in the spatial domain. As a complementary, in the frequency domain, a multispectral channel attention module is adopted to obtain a more complete set of frequency components for representing each channel. The two attention modules are combined to focus on the discriminative region and suppress the misleading information by their mutual promotion. Experimental results on five benchmark crowd data sets demonstrate that the SFANet can achieve the state-of-the-art performance in terms of accuracy and robustness.

COMBINING LIDAR SLAM AND DEEP LEARNING-BASED PEOPLE DETECTION FOR AUTONOMOUS INDOOR MAPPING IN A CROWDED ENVIRONMENT

Abstract. In this paper, we present a mapping system based on an autonomous mobile robot equipped with a LiDAR device and a camera, that can deal with the presence of people. Thanks to a deep learning approach, the position of humans is identified and a new surveying path is planned that brings the robot to scan occluded areas, so as to obtain a complete point cloud of the environment. Experimental results are performed with a wheeled mobile robot in different crowded scenarios, showing the applicability of the proposed approach to perform an autonomous survey avoiding occlusions and automatically removing from the map noisy and spurious objects caused by people presence.

SIMCD: SIMulated crowd data for anomaly detection and prediction

Smart Crowd management (SCM) solutions can mitigate overcrowding disasters by implementing efficient crowd learning models that can anticipate critical crowd conditions and potential catastrophes. Developing an SCM solution involves monitoring crowds and modelling their dynamics. Crowd monitoring produces vast amounts of data, with features such as densities and speeds, which are essential for training and evaluating crowd learning models. By and large, crowd datasets can be classified as real (e.g., real monitoring of crowds) or synthetic (e.g., simulation of crowds). Using real crowd datasets can produce effective and reliable crowd learning models. However, acquiring real crowd data faces several challenges, including the expensive installation of a sensory infrastructure, the data pre-processing costs and the lack of real datasets that cover particular crowd scenarios. Consequently, crowd management literature has adopted simulation tools for generating synthetic datasets to overcome the challenges associated with their real counterparts. The majority of existing datasets, whether real or synthetic, can be used for crowd counting applications or analysing the activities of individuals rather than collective crowd behaviour. Accordingly, this paper demonstrates the process of generating bespoke synthetic crowd datasets that can be used for crowd anomaly detection and prediction, using the MassMotion crowd simulator. The developed datasets present two types of crowd anomalies; namely, high densities and contra-flow walking direction. These datasets are: SIMulated Crowd Data (SIMCD)-Single Anomaly and SIMCD-Multiple Anomalies for anomaly detection tasks, besides two SIMCD-Prediction datasets for crowd prediction tasks. Furthermore, the paper demonstrates the data preparation (pre-processing) process by aggregating the data and proposing new essential features, such as the level of crowdedness and the crowd severity level, that are useful for developing crowd prediction and anomaly detection models.

Crowd behavior anomaly detection using correlation of optical flow magnitude

Nowadays, crowd monitoring is a contentious issue. Because of the increasing population and diversity of human activities, crowd scenarios in the real world are becoming more common, demanding the need for an automotive anomaly detection system. Crowd behavior is influenced by the thoughts and attitudes of others around them. An unexpected event can turn a peaceful crowd into a riot. A mechanism based on optical flow must be implemented to compensate for all of these factors. The amount of motion present in two successive frames is estimated using optical flow. It includes information on velocity in the x & y plane, along with magnitude and line of action. By means of “anomalous event” in this paper is quick and sudden dispersal of the crowd. For detecting an event the magnitude of two successive frames should be taken into account followed by estimating a correlation. We expect a high correlation, slight motion, and low rate of change in velocities at non-anomalous events, but as soon as an anomalous event occurs, the correlation begins to decrease with a significant change in velocity and large motion vectors. The methodology was tested on a dataset from the University of Minnesota that included 11 movies from three different circumstances. Almost all anomalous occurrences in videos were successfully detected using this method.

The One-Man-Crowd: Single User Generation of Crowd Motions Using Virtual Reality.

Crowd motion data is fundamental for understanding and simulating realistic crowd behaviours. Such data is usually collected through controlled experiments to ensure that both desired individual interactions and collective behaviours can be observed. It is however scarce, due to ethical concerns and logistical difficulties involved in its gathering, and only covers a few typical crowd scenarios. In this work, we propose and evaluate a novel Virtual Reality based approach lifting the limitations of real-world experiments for the acquisition of crowd motion data. Our approach immerses a single user in virtual scenarios where he/she successively acts each crowd member. By recording the past trajectories and body movements of the user, and displaying them on virtual characters, the user progressively builds the overall crowd behaviour by him/herself. We validate the feasibility of our approach by replicating three real experiments, and compare both the resulting emergent phenomena and the individual interactions to existing real datasets. Our results suggest that realistic collective behaviours can naturally emerge from virtual crowd data generated using our approach, even though the variety in behaviours is lower than in real situations. These results provide valuable insights to the building of virtual crowd experiences, and reveal key directions for further improvements.

Correlating the Local and Global Dynamics of an Enzyme in the Crowded Milieu.

Enzymes are dynamic biological macromolecules, with their catalytic function(s) being largely influenced by the changes in local fluctuations of amino acid side chains as well as global structural modulations that the enzyme undergoes. Such local and global motions can be highly affected inside the crowded physiological interior of the cell. Here, we have addressed the role of dynamic structural flexibility in affecting the activation energy barrier of a flexible multidomain enzyme adenylate kinase (AK3L1, UniProtKB: Q9UIJ7). Activation energy profiles of both local (at three different sites along the polypeptide backbone) and global dynamics of the enzyme have been monitored using solvation studies on the subnanosecond time scale and tryptophan quenching studies over the temperature range of 278-323 K, respectively, under crowded conditions (Ficoll 70, Dextran 40, Dextran 70, and PEG 8). This study not only provides the site-specific mapping of dynamics but reveals that the activation energies associated with these local motions undergo a significant decrease in the presence of macromolecular crowders, providing new insights into how crowding affects internal protein dynamics. The crowded scenario also aids in enhancing the coupling between the local and global motions of the enzyme. Moreover, select portions/regions of the enzyme when taken together can well mirror the overall dynamics of the biomolecule, showing possible energy hotspots along the polypeptide backbone.

Step Attention: Sequential Pedestrian Trajectory Prediction

In this paper we propose a deep learning model, which we call step attention, for pedestrian trajectory prediction. The proposed model has a special architecture which consists of recurrent neural networks, convolutional neural networks, and an augmented attention mechanism. Rather than developing architectures to model factors that may affect the walking behavior, the proposed model learns trajectory patterns directly from input sequences. We evaluate the performance of the step attention model using TrajNet–a publicly available benchmark dataset collected from diverse real-world crowded scenarios. We compare the performance of step attention with three existing state-of-the-art algorithms, including social LSTM, social GAN, and occupancy LSTM on the TrajNet benchmark dataset. Our experiments show that the average displacement error (ADE) of step attention for a 4.8-seconds-long prediction horizon is about 0.53 m. The final displacement error (FDE) is 1.72 m. Both average and final displacement errors are favorable compared to the benchmark methods. We conduct a second set of experiments using data collected from a four-way intersection through roadside camera sensor platforms to study the effectiveness of the proposed model in uncrowded environments. On this dataset, the proposed model has an ADE of 0.74 m and a FDE of about 1.40 m for a 6-seconds-long prediction horizon. A complementary set of experiments is conducted to further investigate model performance in a real-world intersection. In these experiments, the model gains an ADE/FDE of 0.76/1.70 m. The proposed model also produces accurate prediction results on different scenarios composed of different walking patterns (e.g., straight and curvy) and different environments (e.g., sidewalk and street). The average displacement errors on all investigated datasets are within the length of a single step of an adult. The experiments also indicate that the displacement error grows almost linearly with the prediction horizon.

Semantic-driven multi-camera pedestrian detection

In the current worldwide situation, pedestrian detection has reemerged as a pivotal tool for intelligent video-based systems aiming to solve tasks such as pedestrian tracking, social distancing monitoring or pedestrian mass counting. Pedestrian detection methods, even the top performing ones, are highly sensitive to occlusions among pedestrians, which dramatically degrades their performance in crowded scenarios. The generalization of multi-camera setups permits to better confront occlusions by combining information from different viewpoints. In this paper, we present a multi-camera approach to globally combine pedestrian detections leveraging automatically extracted scene context. Contrarily to the majority of the methods of the state-of-the-art, the proposed approach is scene-agnostic, not requiring a tailored adaptation to the target scenario–e.g., via fine-tuning. This noteworthy attribute does not require ad hoc training with labeled data, expediting the deployment of the proposed method in real-world situations. Context information, obtained via semantic segmentation, is used (1) to automatically generate a common area of interest for the scene and all the cameras, avoiding the usual need of manually defining it, and (2) to obtain detections for each camera by solving a global optimization problem that maximizes coherence of detections both in each 2D image and in the 3D scene. This process yields tightly fitted bounding boxes that circumvent occlusions or miss detections. The experimental results on five publicly available datasets show that the proposed approach outperforms state-of-the-art multi-camera pedestrian detectors, even some specifically trained on the target scenario, signifying the versatility and robustness of the proposed method without requiring ad hoc annotations nor human-guided configuration.

Video analytics using deep learning for crowd analysis: a review

Gathering a large number of people in a shared physical area is very common in urban culture. Although there are limitless examples of mega crowds, the Islamic religious ritual, the Hajj, is considered as one of the greatest crowd scenarios in the world. The Hajj is carried out once in a year with a congregation of millions of people when the Muslims visit the holy city of Makkah at a given time and date. Such a big crowd is always prone to public safety issues, and therefore requires proper measures to ensure safe and comfortable arrangement. Through the advances in computer vision based scene understanding, automatic analysis of crowd scenes is gaining popularity. However, existing crowd analysis algorithms might not be able to correctly interpret the video content in the context of the Hajj. This is because the Hajj is a unique congregation of millions of people crowded in a small area, which can overwhelm the use of existing video and computer vision based sophisticated algorithms. Through our studies on crowd analysis, crowd counting, density estimation, and the Hajj crowd behavior, we faced the need of a review work to get a research direction for abnormal behavior analysis of Hajj pilgrims. Therefore, this review aims to summarize the research works relevant to the broader field of video analytics using deep learning with a special focus on the visual surveillance in the Hajj. The review identifies the challenges and leading-edge techniques of visual surveillance in general, which may gracefully be adaptable to the applications of Hajj and Umrah. The paper presents detailed reviews on existing techniques and approaches employed for crowd analysis from crowd videos, specifically the techniques that use deep learning in detecting abnormal behavior. These observations give us the impetus to undertake a painstaking yet exhilarating journey on crowd analysis, classification and detection of any abnormal movement of the Hajj pilgrims. Furthermore, because the Hajj pilgrimage is the most crowded domain for video-related extensive research activities, this study motivates us to critically analyze the crowd on a large scale.

Multi-scale features fused network with multi-level supervised path for crowd counting

Many CNN-based methods which utilize the density map to regress the count number of crowd are introduced to solve the crowd counting problem lately. Due to the head scale variations caused by the perspective change and background noise, these methods cannot address these two problems well in highly crowded scenario. In order to solve these two problems, we introduce a multi-scale features fused network with multi-level supervised path to produce the high-quality density map in this paper. Our model utilizes the first 13 layers of VGG16 model as the backbone, the multi-level supervised path in our model employs the multi-level dilated convolution module (MLD) to supervise the whole network at multi-level, and generate the attention map for the density map, which is used to handle the scale variations. The other path is used to fuse multi-scale features to generate the density map with soft spatial-channel attention module (SSCA) which aims to produce a saliency weight map of same size. In the end, the final density map is captured by the feature map multiply the attention map. In addition, a new objective function is proposed to train our network. A large number of experimental results show that compared with other networks, our method achieves better experimental results on four challenging datasets (UCF_CC_50, ShanghaiTech, UCF-QRNF and WorldExpo’10 dataset).