Pedestrian Dataset Research Articles

Video Anomaly Detection Based on Global–Local Convolutional Autoencoder

Video anomaly detection (VAD) plays a crucial role in fields such as security, production, and transportation. To address the issue of overgeneralization in anomaly behavior prediction by deep neural networks, we propose a network called AMFCFBMem-Net (appearance and motion feature cross-fusion block memory network), which combines appearance and motion feature cross-fusion blocks. Firstly, dual encoders for appearance and motion are employed to separately extract these features, which are then integrated into the skip connection layer to mitigate the model’s tendency to predict abnormal behavior, ultimately enhancing the prediction accuracy for abnormal samples. Secondly, a motion foreground extraction module is integrated into the network to generate a foreground mask map based on speed differences, thereby widening the prediction error margin between normal and abnormal behaviors. To capture the latent features of various models for normal samples, a memory module is introduced at the bottleneck of the encoder and decoder structures. This further enhances the model’s anomaly detection capabilities and diminishes its predictive generalization towards abnormal samples. The experimental results on the UCSD Pedestrian dataset 2 (UCSD Ped2) and CUHK Avenue anomaly detection dataset (CUHK Avenue) demonstrate that, compared to current cutting-edge video anomaly detection algorithms, our proposed method achieves frame-level AUCs of 97.5% and 88.8%, respectively, effectively enhancing anomaly detection capabilities.

YOLO-ESL: An Enhanced Pedestrian Recognition Network Based on YOLO

Pedestrian detection is a critical task in computer vision; however, mainstream algorithms often struggle to achieve high detection accuracy in complex scenarios, particularly due to target occlusion and the presence of small objects. This paper introduces a novel pedestrian detection algorithm, YOLO-ESL, based on the YOLOv7 framework. YOLO-ESL integrates the ELAN-SA module, designed to enhance feature extraction, with the LGA module, which improves feature fusion. The ELAN-SA module optimizes the flexibility and efficiency of small object feature extraction, while the LGA module effectively integrates multi-scale features through local and global attention mechanisms. Additionally, the CIOUNMS algorithm addresses the issue of target loss in cases of high overlap, improving boundary box filtering. Evaluated on the VOC2012 pedestrian dataset, YOLO-ESL achieved an accuracy of 93.7%, surpassing the baseline model by 3.0%. Compared to existing methods, this model not only demonstrates strong performance in handling occluded and small object detection but also remarkable robustness and efficiency.

Ranking-based adaptive query generation for DETRs in crowded pedestrian detection

Variants of DEtection TRansformer (DETRs) have shown promising performance in crowded pedestrian detection. However, we observe that DETRs are sensitive to the hyper-parameter (the number of queries). Adjusting this hyper-parameter is crucial for achieving competitive performance across different crowded pedestrian datasets. Existing query generation methods are limited to generate a fixed number of queries based on this hyper-parameter, which often leads to missed detections and incorrect detections due to the varied number and density of pedestrians in crowded scenes. To address this challenge, we propose an adaptive query generation method called Ranking-based Adaptive Query Generation (RAQG). RAQG comprises three components: a ranking prediction head, a query supplementer, and Soft Gradient L1 Loss (SGL1). Specifically, we leverage the ranking of the lowest confidence score positive training sample to generate queries adaptively. The ranking prediction head predicts this ranking, which guides our query generation. Additionally, to refine the query generation process, we introduce a query supplementer that adjusts the number of queries based on the predicted ranking. Furthermore, we introduce SGL1, a novel loss function for training the ranking prediction head over a wide regression range. Our method is designed to be lightweight and universal, suitable for integration into any DETRs framework for crowded pedestrian detection. Experimental results on Crowdhuman and Citypersons datasets demonstrate that our RAQG method can generate queries adaptively and achieves competitive results. Notably, our approach achieves a state-of-the-art 39.4% MR on Crowdhuman.

Towards faster yet accurate video prediction for resource-constrained platforms

Video prediction has emerged as a critical task in the field of computer vision. While conventional deep learning-based video prediction models excel in accuracy, they often impose significant computational overhead. Moreover, they tend to disregard crucial factors like inference latency and memory consumption, which may render them unsuitable for real-time video prediction applications. This issue is further exacerbated by the necessity to deploy many of these applications on resource-limited embedded devices. In this study, we introduce an innovative faster yet accurate video prediction (FAVP) model designed to mitigate inference latency and memory footprint, all the while maintaining predictive accuracy. Our model employs an encoder-convertor-decoder architecture with a novel multi-layer convertor. Each layer within the convertor uses consistent kernel sizes to precisely capture dimensional features, while varying kernel sizes between layers effectively capture both local and global features. This combination of uniformity within layers and diversity between layers enhances the model’s ability to grasp dynamic changes. Furthermore, we demonstrate that replacing conventional large-kernel convolutions with involutions significantly trims the model’s parameter count and inference latency, without compromising prediction accuracy. Through comprehensive experiments conducted on resource-adequate x86 platforms, utilizing KTH, TrafficBJ, Human3.6M, Moving MNIST, and KTH-Enhanced datasets, we showcase our model’s exceptional prediction performance, particularly in terms of inference latency. Additionally, we evaluate the model on resource-constrained NVIDIA Jetson Nano platform using KITTI & CalTech Pedestrian dataset. The results underscore our model’s superior inference speed and its competence in meeting real-time video prediction requirements, even under stringent resource limitations.

Data compensation and feature fusion for sketch based person retrieval

Sketch re-identification (Re-ID) aims to retrieve pedestrian photo in the gallery dataset by a query sketch drawn by professionals. The sketch Re-ID task has not been adequately studied because collecting such sketches is difficult and expensive. In addition, the significant modality difference between sketches and images makes extracting the discriminative feature information difficult. To address above issues, we introduce a novel sketch-style pedestrian dataset named Pseudo-Sketch dataset. Our proposed dataset maximizes the utilization of the existing person dataset resources and is freely available, thus effectively reducing the expenses associated with the training and deployment phases. Furthermore, to mitigate the modality gap between sketches and visible images, a cross-modal feature fusion network is proposed that incorporates information from each modality. Experiment results show that the proposed Pseudo-Sketch dataset can effectively complement the real sketch dataset, and the proposed network obtains competitive results than SOTA methods. The dataset will be released later.

Pseudo-Multispectral Pedestrian Detection with Deep Thermal Feature Guidance

With complementary multi-modal information (i.e. visible and thermal), multispectral pedestrian detection is essential for around-the-clock applications, such as autonomous driving, video surveillance, and vicinagearth security. Despite its broad applications, the requirements for expensive thermal device and multi-sensor alignment limit the utilization in real-world applications. In this paper, we propose a pseudo-multispectral pedestrian detection (called PseudoMPD) method, which employs the gray image converted from the RGB image to replace the real thermal image, and learns the pseudo-thermal feature through deep thermal feature guidance (TFG). To achieve this goal, we first introduce an image base-detail decomposition (IBD) module to decompose image information into base and detail parts. Afterwards, we design a base-detail hierarchical feature fusion (BHFF) module to deeply exploit the information between these two parts, and employ a TFG module to guide pseudo-thermal base and detail feature learning. As a result, our proposed method does not require the real thermal image during inference. The comprehensive experiments are performed on two public multispectral pedestrian datasets. The experimental results demonstrate the effectiveness of our proposed method.

An Improved Pedestrian Detection Model Based on YOLOv8 for Dense Scenes

In dense scenes, pedestrians often exhibit a variety of symmetrical features, such as symmetry in body contour, posture, clothing, and appearance. However, pedestrian detection poses challenges due to the mutual occlusion of pedestrians and the small scale of distant pedestrians in the image. To address these challenges, we propose a pedestrian detection algorithm tailored for dense scenarios called YOLO-RAD. In this algorithm, we integrate the concept of receiving field attention (RFA) into the Conv and C2f modules to enhance the feature extraction capability of the network. A self-designed four-layer adaptive spatial feature fusion (ASFF) module is introduced, and shallow pedestrian feature information is added to enhance the multi-scale feature fusion capability. Finally, we introduce a small-target dynamic head structure (DyHead-S) to enhance the capability of detecting small-scale pedestrians. Experimental results on WiderPerson and CrowdHuman, two challenging dense pedestrian datasets, show that compared with YOLOv8n, our YOLO-RAD algorithm has achieved significant improvement in detection performance, and the detection performance of mAP@0.5 has increased by 2.5% and 6%, respectively. The detection performance of mAP@0.5:0.95 was improved by 2.7% and 6.8%, respectively. Therefore, the algorithm can effectively improve the performance of pedestrian detection in dense scenes.

GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction

Predicting pedestrian behavior is the key to ensure safety and reliability of autonomous vehicles. While deep learning methods have been promising by learning from annotated video frame sequences, they often fail to fully grasp the dynamic interactions between pedestrians and traffic, crucial for accurate predictions. These models also lack nuanced common sense reasoning. Moreover, the manual annotation of datasets for these models is expensive and challenging to adapt to new situations. The advent of Vision Language Models (VLMs) introduces promising alternatives to these issues, thanks to their advanced visual and causal reasoning skills. To our knowledge, this research is the first to conduct both quantitative and qualitative evaluations of VLMs in the context of pedestrian behavior prediction for autonomous driving. We evaluate GPT-4V(ision) on publicly available pedestrian datasets: JAAD and WiDEVIEW. Our quantitative analysis focuses on GPT-4V's ability to predict pedestrian behavior in current and future frames. The model achieves a 57% accuracy in a zero-shot manner, which, while impressive, is still behind the state-of-the-art domain-specific models (70%) in predicting pedestrian crossing actions. Qualitatively, GPT-4V shows an impressive ability to process and interpret complex traffic scenarios, differentiate between various pedestrian behaviors, and detect and analyze groups. However, it faces challenges, such as difficulty in detecting smaller pedestrians and assessing the relative motion between pedestrians and the ego vehicle.

PEDESTRIAN ALERT SYSTEM USING DEEP LEARNING

Pedestrian detection is an essential and significant task in any intelligent video surveillance system, as it provides the fundamental information for semantic understanding of the video footages. It has an obvious extension to automotive applications due to the potential for improving safety systems. Many car manufacturers (e.g., Volvo, Ford, GM, Nissan) offer this as an ADAS option in 2017. It is used in many cars and vehicles for automatic driver assistance systems. But when the ADAS system detects pedestrians but if fails to apply brakes on time accidents might happened. To overcome this situation our project provides an audio alert to the human passengers or driver in the vehicle so that they can apply manual brakes. In this project we will require Python libraries such as Pytorch and OpenCV. Pytorch is a widely used Machine Learning library. It is popular for the YOLO (You Only Look Once) algorithm which is built for Object Detection, we are using the YOLO algorithm and customize it to detect objects on a pedestrian dataset. Python has a library pyttsx3, that is capable to convert text-to-speech offline. Itextracts the label from the detected pedestrians in the video and converts the text label into speech. The algorithm detects the pedestrians when they are in a close approach to the vehicles and sends the identifies labels to the pyttsx3 speaker engine. An audio alert is generated by the engine and alerts the passengers or drivers. Keywords - YOLO, Pytorch, Pyttsx3, Object Detection, Python.

Enhanced YOLOX with United Attention Head for Road Detetion When Driving

Object detection plays a crucial role in autonomous driving assistance systems. It requires high accuracy for prediction, a small size for deployment on mobile devices, and real-time inference speed to ensure safety. In this paper, we present a compact and efficient algorithm called YOLOX with United Attention Head (UAH-YOLOX) for detection in autonomous driving scenarios. By replacing the backbone network with GhostNet for feature extraction, the model reduces the number of parameters and computational complexity. By adding a united attention head before the YOLO head, the model effectively detects the scale, position, and contour features of targets. In particular, an attention module called Spatial Self-Attention is designed to extract spatial location information, demonstrating great potential in detection. In our network, the IOU Loss (Intersection of Union) has been replaced with CIOU Loss (Complete Intersection of Union). Further experiments demonstrate the effectiveness of our proposed methods on the BDD100k dataset and the Caltech Pedestrian dataset. UAH-YOLOX achieves state-of-the-art results by improving the detection accuracy of the BDD100k dataset by 1.70% and increasing processing speed by 3.37 frames per second (FPS). Visualization provides specific examples in various scenarios.

Pedestrian trajectory prediction method based on automatic driving

Pedestrian trajectory prediction plays a crucial role in autonomous driving, as its accuracy directly affects the autonomous driving system’s comprehension of the environment and subsequent decision-making processes. Current trajectory prediction methods tend to oversimplify pedestrians to mere point coordinates, utilizing positional information to infer interactions among individuals while overlooking the temporal correlations between them, thereby excessively simplifying pedestrian characteristics. To address the aforementioned issues, this paper proposes a trajectory prediction model for autonomous driving applications, that takes into account both pedestrian motion characteristics and scene interaction. The model optimizes the LSTM unit structure twice, serving to learn correlations among long trajectories of pedestrians and to integrate multiple forms of information into the neighborhood interaction module. Furthermore, our model introduces dual attention mechanisms for individuals and scenes, focusing on the key motion points of individual pedestrians and their interactive behavior with others in busy scenarios. The efficacy of the model was validated on the MOT16 pedestrian dataset and the Daimler pedestrian path prediction dataset, outperforming mainstream methods with 8% and 10% reductions in Average Displacement Error and Final Displacement Error respectively.

Automated Region of Interest-Based Data Augmentation for Fallen Person Detection in Off-Road Autonomous Agricultural Vehicles.

Due to the global population increase and the recovery of agricultural demand after the COVID-19 pandemic, the importance of agricultural automation and autonomous agricultural vehicles is growing. Fallen person detection is critical to preventing fatal accidents during autonomous agricultural vehicle operations. However, there is a challenge due to the relatively limited dataset for fallen persons in off-road environments compared to on-road pedestrian datasets. To enhance the generalization performance of fallen person detection off-road using object detection technology, data augmentation is necessary. This paper proposes a data augmentation technique called Automated Region of Interest Copy-Paste (ARCP) to address the issue of data scarcity. The technique involves copying real fallen person objects obtained from public source datasets and then pasting the objects onto a background off-road dataset. Segmentation annotations for these objects are generated using YOLOv8x-seg and Grounded-Segment-Anything, respectively. The proposed algorithm is then applied to automatically produce augmented data based on the generated segmentation annotations. The technique encompasses segmentation annotation generation, Intersection over Union-based segment setting, and Region of Interest configuration. When the ARCP technique is applied, significant improvements in detection accuracy are observed for two state-of-the-art object detectors: anchor-based YOLOv7x and anchor-free YOLOv8x, showing an increase of 17.8% (from 77.8% to 95.6%) and 12.4% (from 83.8% to 96.2%), respectively. This suggests high applicability for addressing the challenges of limited datasets in off-road environments and is expected to have a significant impact on the advancement of object detection technology in the agricultural industry.

A spatiotemporal motion prediction network based on multi-level feature disentanglement

The prediction task is significantly challenged by the intricate scene information and motion variations present in spatiotemporal data. Existing prediction methods struggle to accurately forecast long-term outcomes, particularly for transient motions characterized by notable trends, such as hand lifts, jumps, or vehicle turns. To address these challenges, we introduce a Spatiotemporal Motion Prediction Network based on Multi-level Feature Disentanglement (FDPNet). The model delineates spatiotemporal prediction into two distinct stages: feature disentanglement and motion prediction. We first devise a Multi-level Feature Disentanglement (MFD) model to disentangle the multilayer features of motion within the temporal sequence, encompassing period, trend, and residual components. This disentanglement is based on disentangling spatiotemporal coupling, enabling the network to comprehensively grasp the genuine laws governing motion in the spatiotemporal evolution process. Second, to enhance the prediction accuracy of the network over extended periods, we introduce the Motion Differential Self-Attention LSTM unit (MDSA-LSTM). This unit employs differential operations to extract inter-frame motion trends, elevating the network's proficiency in capturing spatiotemporal correlations over long distances through an enhanced self-attention mechanism. FDPNet attains state-of-the-art performance on the Moving MNIST, UCF101, KITTI, and Caltech pedestrian datasets. These outcomes substantiate the substantial potential of this research within the realm of spatiotemporal prediction.

Mm-CasGAN: A cascaded adversarial neural framework for mmWave radar point cloud enhancement

Handling and interpreting sparse 3D point clouds, especially from mmWave radar, presents unique challenges due to the inherent data sparsity and the vast domain difference compared to denser point clouds like those from LiDAR. In this paper, we introduce a novel cascaded generative adversarial network (GAN) approach to bridge this domain gap. The core principle is to progressively refine the radar-based point cloud through a series of GANs, each targeting a higher resolution. By leveraging multi-level features and a hybrid loss function that combines adversarial, geometric, and consistency components, our method ensures a smooth transition from the sparse radar representation to a high-resolution LiDAR-like point cloud. Our cascaded approach operates at a patch level, and the integrated loss function ensures that the generated points not only resemble the target domain but also maintain geometric and structural fidelity. Real-life dataset consisting mostly of moving pedestrians were collected using a system made of Radar, LiDAR, and RGB Camera. Through an extensive experiment on the collected real-world pedestrian dataset, we validate the efficacy of our approach. Inference from the network indicates that our method can upsample mmWave radar point clouds with enhanced density, uniformity, and closer alignment to the ground truth LiDAR point clouds, which is the first of its kind network to do so.

CrossFormer: Cross-guided attention for multi-modal object detection

Object detection is one of the essential tasks in a variety of real-world applications such as autonomous driving and robotics. In a real-world scenario, unfortunately, there are numerous challenges such as illumination changes, adverse weather conditions, and geographical changes, to name a few. To tackle the problem, we propose a novel multi-modal object detection model that is built upon a hierarchical transformer and cross-guidance between different modalities. The proposed hierarchical transformer consists of domain-specific feature extraction networks where intermediate features are connected by the proposed Cross-Guided Attention Module (CGAM) to enrich their representational power. Specifically, in the CGAM, one domain is regarded as a guide and the other is assigned to a base. After that, the cross-modal attention from the guide to the base is applied to the base feature. The CGAM works bidirectionally in parallel by exchanging roles between modalities to refine multi-modal features simultaneously. Experimental results on FLIR-aligned, LLVIP, and KAIST multispectral pedestrian datasets demonstrate that the proposed method is superior to previous multi-modal detection algorithms quantitatively and qualitatively.

Improving RGB-infrared object detection with cascade alignment-guided transformer

The integration of multispectral data in object detection, especially visible and infrared images, has been the subject of considerable attention recently. Complementary information from visible (RGB) and infrared (IR) images can ameliorate the challenges posed by variable lighting conditions, rendering them an invaluable resource in many fields, including RGB-IR object detection, RGB-IR semantic segmentation, and RGB-IR crowd counting. However, existing methods still suffer from weak misalignment and fusion imprecision problems. These two problems present significant challenges for accurate object detection. In this paper, our primary focus is to solve the above problems in RGB-IR object detection tasks. Specifically, we first propose a Translation-Scale-Rotation Alignment (TSRA) module to align two modality features from region proposals. Based on the aligned region features, we introduce a Complementary Fusion Transformer (CFT) module to capture the complementary features. These two modules can be coupled in an unified Region of Interest (RoI) detection head called Cascade Alignment-Guided Transformer (CAGT) to obtain the robust fused features. Finally, based on CAGT, a region feature alignment and fusion detector called CAGTDet is constructed for RGB-IR object detection. Through comprehensive experiments on the aerial DroneVehicle dataset, our method effectively mitigates the impact of these two issues, resulting in robust detection results. Moreover, to evaluate the generalization of our method, we also perform experiments on the nature images sampled from the KAIST multispectral pedestrian dataset. The results show that our method surpasses other state-of-the-art methods.

Pedestrian detection algorithm integrating large kernel attention and YOLOV5 lightweight model.

In the context of intelligent driving, pedestrian detection faces challenges related to low accuracy in target recognition and positioning. To address this issue, a pedestrian detection algorithm is proposed that integrates a large kernel attention mechanism with the YOLOV5 lightweight model. The algorithm aims to enhance long-term attention and dependence during image processing by fusing the large kernel attention module with the C3 module. Furthermore, it addresses the lack of long-distance relationship information in channel and spatial feature extraction and representation by introducing the Coordinate Attention mechanism. This mechanism effectively extracts local information and focused location details, thereby improving detection accuracy. To improve the positioning accuracy of obscured targets, the alpha CIOU bounding box regression loss function is employed. It helps mitigate the impact of occlusions and enhances the algorithm's ability to precisely localize pedestrians. To evaluate the effectiveness of trained model, experiments are conducted on the BDD100K pedestrian dataset as well as the Pascal VOC dataset. Experimental results demonstrate that the improved attention fusion YOLOV5 lightweight model achieves an average accuracy of 60.3%. Specifically, the detection accuracy improves by 1.1% compared to the original YOLOV5 algorithm, and the accuracy performance index reaches 73.0%. These findings strongly indicate the proposed algorithm in significantly enhancing the accuracy of pedestrian detection in road scenes.

A comprehensive review of pedestrian re-identification based on deep learning

Pedestrian re-identification (re-ID) has gained considerable attention as a challenging research area in smart cities. Its applications span diverse domains, including intelligent transportation, public security, new retail, and the integration of face re-ID technology. The rapid progress in deep learning techniques, coupled with the availability of large-scale pedestrian datasets, has led to remarkable advancements in pedestrian re-ID. In this paper, we begin the study by summarising the key datasets and standard evaluation methodologies for pedestrian re-ID. Second, we look into pedestrian re-ID methods that are based on object re-ID, loss functions, research directions, weakly supervised classification, and various application scenarios. Moreover, we assess and display different re-ID approaches from deep learning perspectives. Finally, several challenges and future directions for pedestrian re-ID development are discussed. By providing a holistic perspective on this topic, this research serves as a valuable resource for researchers and practitioners, enabling further advancements in pedestrian re-ID within smart city environments.

Contour Information-Guided Multi-Scale Feature Detection Method for Visible-Infrared Pedestrian Detection.

Infrared pedestrian target detection is affected by factors such as the low resolution and contrast of infrared pedestrian images, as well as the complexity of the background and the presence of multiple targets occluding each other, resulting in indistinct target features. To address these issues, this paper proposes a method to enhance the accuracy of pedestrian target detection by employing contour information to guide multi-scale feature detection. This involves analyzing the shapes and edges of the targets in infrared images at different scales to more accurately identify and differentiate them from the background and other targets. First, we propose a preprocessing method to suppress background interference and extract color information from visible images. Second, we propose an information fusion residual block combining a U-shaped structure and residual connection to form a feature extraction network. Then, we propose an attention mechanism based on a contour information-guided approach to guide the network to extract the depth features of pedestrian targets. Finally, we use the clustering method of mIoU to generate anchor frame sizes applicable to the KAIST pedestrian dataset and propose a hybrid loss function to enhance the network's adaptability to pedestrian targets. The extensive experimental results show that the method proposed in this paper outperforms other comparative algorithms in pedestrian detection, proving its superiority.

NIRPed: A Novel Benchmark for Nighttime Pedestrian and Its Distance Joint Detection

In nighttime driving scenes, due to insufficient and uneven lighting, and the scarcity of high-quality datasets, the miss rate of nighttime pedestrian detection (PD) is much higher than that of daytime. Vision-based distance detection (DD) has the advantages of low cost and good interpretability, but the existing methods have low precision, poor robustness, and the DD is mostly performed independently of PD. A narrowband near-infrared (NIR) camera and an NIR lamp to relieve the impact of noisy visible light (VIS) and improve imaging quality have been selected; a LiDAR to obtain distance information has been embedded in the system; a nighttime driving scene pedestrian and its distance joint detection dataset NIRPed has been built. The NIRPed includes 146k pedestrian annotations, which is 3 times that of NightOwls, the largest VIS nighttime pedestrian dataset. Based on Faster-RCNN, a joint PD and DD method using monocular imaging has been proposed. The propo-sed method realizes the joint detection of pedestrian and its distance on the NIRPed dataset with the PD log-average miss rate and DD mean absolute error rate of 6.5% and 5.5%, respectively. For comparison, the joint detection method was implemented on other large-scale VIS pedestrian datasets as well. Moreover, compared with the existing vision-based DD methods, the proposed method is less affected by pedestrian distance and height, exhibits higher accuracy and robustness, and satisfied the ISO requirements for intelligent transportation systems (absolute error rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&lt;$</tex-math> </inline-formula> 15%).