Pedestrian Detection Performance Research Articles

Differential Feature Fusion, Triplet Global Attention, and Web Semantic for Pedestrian Detection

In complex environments and crowded pedestrian scenes, the overlap or loss of local features is a pressing issue. However, existing methods often struggle to strike a balance between eliminating interfering features and establishing feature connections. To address this challenge, we introduce a novel pedestrian detection approach called Differential Feature Fusion under Triplet Global Attention (DFFTGA). This method merges feature maps of the same size from different stages to introduce richer feature information. Specifically, we introduce a pixel-level Triplet Global Attention (TGA) module to enhance feature representation and perceptual range. Additionally, we introduce a Differential Feature Fusion (DFF) module, which optimizes features between similar nodes for filtering. This series of operations helps the model focus more on discriminative features, ultimately improving pedestrian detection performance. Compared to benchmarks, we achieve significant improvements and demonstrate outstanding performance on datasets such as CityPersons and CrowdHuman.

YOLOv8-CB: Dense Pedestrian Detection Algorithm Based on In-Vehicle Camera

Recently, the field of vehicle-mounted visual intelligence technology has witnessed a surge of interest in pedestrian detection. Existing algorithms for dense pedestrian detection at intersections face challenges such as high computational weight, complex models that are difficult to deploy, and suboptimal detection accuracy for small targets and highly occluded pedestrians. To address these issues, this paper proposes an improved lightweight multi-scale pedestrian detection algorithm, YOLOv8-CB. The algorithm introduces a lightweight cascade fusion network, CFNet (cascade fusion network), and a CBAM attention module to improve the characterization of multi-scale feature semantics and location information, and it superimposes a bidirectional weighted feature fusion path BIFPN structure to fuse more effective features and improve pedestrian detection performance. It is experimentally verified that compared with the YOLOv8n algorithm, the accuracy of the improved model is increased by 2.4%, the number of model parameters is reduced by 6.45%, and the computational load is reduced by 6.74%. The inference time for a single image is 10.8 ms. The cascade fusion algorithm YOLOv8-CB has higher detection accuracy and is a lighter model for multi-scale pedestrian detection in complex scenes such as streets or intersections. This proposed algorithm presents a valuable approach for device-side pedestrian detection with limited computational resources.

Robot Real-time Pedestrian Tracking Algorithm based on Deep Learning

In recent years, deep learning models have greatly improved the performance of pedestrian detection and tracking. Although the accuracy of neural network-based modeling methods is high, they often require large computational and storage resource overheads, which makes them difficult to apply to robots with high resource requirements. Pedestrian tracking for robots still faces great challenges in the problems of occlusion, multi-target, and target loss. In this thesis, we focus on solving the problem of real-time pedestrian tracking by lightweight fusion of deep learning target detection models for robots, firstly, through the lightweight YOLO network, we perform pedestrian detection and feature extraction, and then we propose a Gaussian mixture model based feature matching method to construct the target pedestrian tracker, and finally, we use the PID-based control algorithm for real-time control of the robot's motion, and finally realize the real-time pedestrian Tracking. In this thesis, we validate the feature matching method based on the Gaussian mixture model on the ETH dataset, and at the same time, combined with the motion control algorithm, we carry out the actual validation, and the experimental results show that our proposed method can realize the real-time pedestrian tracking.

CFRLA-Net: A Context-Aware Feature Representation Learning Anchor-Free Network for Pedestrian Detection

High resolution and strong semantic representation are both vital for feature extraction networks of pedestrian detection. The existing high-resolution network (HRNet) has presented a promising performance for pedestrian detection. However, we observed that it still has some significant shortcomings for heavily occluded and small-scale pedestrians. In this paper, we propose to address the shortcomings by extracting semantic and spatial context from HRNet. Specifically, we propose a Context-aware Feature Representation Learning Module (CFRL-Module), which combines a Multi-scale Feature Context Extraction Parallel Block for Convolution and Self-attention (CEPCA-Block) with two parallel paths and an Equivalent FFN (EFFN) Block. The core CEPCA-Block adopts a parallel design to integrate convolution and multi-head self-attention (MHSA) with low parameter computational cost, which can obtain the deep semantic context by convolution path and precise context by MHSA path. Furthermore, to overcome the inefficiency of global MHSA in high-resolution pedestrian detection, we propose a novel local window MHSA, which can significantly reduce memory consumption but barely affect the detection performance. Cascading the proposed CFRL-Module with the anchor-free detection head constitutes our Context-aware Feature Representation Learning Anchor-Free Network (CFRLA-Net). The proposed CFRLA-Net can catch a high-level understanding of the heavily occluded and small-scale pedestrian instances based on HRNet, which can effectively solve the limitation of the insufficient feature extraction ability of HRNet for the hard samples. Experimental results show that CFRLA-Net achieves state-of-the-art performance on CityPersons, Caltech, and CrowdHuman benchmarks.

Nighttime pedestrian detection based on Fore-Background contrast learning

The significance of background information is frequently overlooked in contemporary research concerning channel attention mechanisms. This study addresses the issue of suboptimal single-spectral nighttime pedestrian detection performance under low-light conditions by incorporating background information into the channel attention mechanism. Despite numerous studies focusing on the development of efficient channel attention mechanisms, the relevance of background information has been largely disregarded. By adopting a contrast learning approach, we reexamine channel attention with regard to pedestrian objects and background information for nighttime pedestrian detection, resulting in the proposed Fore-Background Contrast Attention (FBCA). FBCA possesses two primary attributes: (1) channel descriptors form remote dependencies with global spatial feature information; (2) the integration of background information enhances the distinction between channels concentrating on low-light pedestrian features and those focusing on background information. Consequently, the acquired channel descriptors exhibit a higher semantic level and spatial accuracy. Experimental outcomes demonstrate that FBCA significantly outperforms existing methods in single-spectral nighttime pedestrian detection, achieving state-of-the-art results on the NightOwls and TJU-DHD-pedestrian datasets. Furthermore, this methodology also yields performance improvements for the multispectral LLVIP dataset. These findings indicate that integrating background information into the channel attention mechanism effectively mitigates detector performance degradation caused by illumination factors in nighttime scenarios.

Exploiting the Potential of Overlapping Cropping for Real-World Pedestrian and Vehicle Detection with Gigapixel-Level Images

Pedestrian and vehicle detection is widely used in intelligent assisted driving, pedestrian counting, drone aerial photography, and other applications. Recently, with the development of gigacameras, gigapixel-level images have emerged. The large field of view and high resolution provide global and local information, which enables object detection in real-world scenarios. Although existing pedestrian and vehicle detection algorithms have achieved remarkable success for standard images, their methods are not suitable for ultra-high-resolution images. In order to improve the performance of existing pedestrian and vehicle detectors in real-world scenarios, we used a sliding window to crop the original images to solve this problem. When fusing the sub-images, we proposed a midline method to reduce the cropped objects that NMS could not eliminate. At the same time, we used varifocal loss to solve the imbalance between positive and negative samples caused by the high resolution. We also found that pedestrians and vehicles were separable in size and comprised more than one target type. As a result, we improved the detector performance with single-class object detection for pedestrians and vehicles, respectively. At the same time, we provided many useful strategies to improve the detector. The experimental results demonstrated that our method could improve the performance of real-world pedestrian and vehicle detection.

A Small Target Pedestrian Detection Model Based on Autonomous Driving

Since small-target pedestrians account for a small proportion of pixels in images and lack texture features, the feature information of small-target pedestrians is often ignored in the feature extraction process, leading to reduced accuracy and poor robustness. To improve the accuracy of small-target pedestrian detection and the anti-interference ability of the model, a small-target pedestrian detection model that fuses residual networks and feature pyramids is proposed. First, a residual block with a discard layer is constructed to replace the standard residual block in the residual network structure to reduce the complexity of the model computation process and solve the problems of gradient disappearance and explosion in the deep network. Then, feature selection and feature alignment modules are added to the lateral connection part of the feature pyramid to enhance important pedestrian features in the input image, and the multiscale feature fusion capability of the model is enhanced for small-target pedestrians, thereby improving the detection accuracy of small-target pedestrians and solving the problems of feature misalignment and ignored multiscale features in the feature pyramid network. Finally, a cascaded autofocus query module is proposed to increase the inference speed of the feature pyramid network through focusing and querying, thus improving the performance and efficiency of small-target pedestrian detection. The experimental results show that the proposed model achieves better detection results than previous models.

Rethinking Cross-Domain Pedestrian Detection: A Background-Focused Distribution Alignment Framework for Instance-Free One-Stage Detectors.

Cross-domain pedestrian detection aims to generalize pedestrian detectors from one label-rich domain to another label-scarce domain, which is crucial for various real-world applications. Most recent works focus on domain alignment to train domain-adaptive detectors either at the instance level or image level. From a practical point of view, one-stage detectors are faster. Therefore, we concentrate on designing a cross-domain algorithm for rapid one-stage detectors that lacks instance-level proposals and can only perform image-level feature alignment. However, pure image-level feature alignment causes the foreground-background misalignment issue to arise, i.e., the foreground features in the source domain image are falsely aligned with background features in the target domain image. To address this issue, we systematically analyze the importance of foreground and background in image-level cross-domain alignment, and learn that background plays a more critical role in image-level cross-domain alignment. Therefore, we focus on cross-domain background feature alignment while minimizing the influence of foreground features on the cross-domain alignment stage. This paper proposes a novel framework, namely, background-focused distribution alignment (BFDA), to train domain adaptive one-stage pedestrian detectors. Specifically, BFDA first decouples the background features from the whole image feature maps and then aligns them via a novel long-short-range discriminator. Extensive experiments demonstrate that compared to mainstream domain adaptation technologies, BFDA significantly enhances cross-domain pedestrian detection performance for either one-stage or two-stage detectors. Moreover, by employing the efficient one-stage detector (YOLOv5), BFDA can reach 217.4 FPS ( 640×480 pixels) on NVIDIA Tesla V100 (7~12 times the FPS of the existing frameworks), which is highly significant for practical applications. The code from this study will be made publicly available.

Pedestrian Detection for Autonomous Cars: Inference Fusion of Deep Neural Networks

Network fusion has been recently explored as an approach for improving pedestrian detection performance. However, most existing fusion methods suffer from runtime efficiency, modularity, scalability, and maintainability due to the complex structure of the entire fused models, their end-to-end training requirements, and sequential fusion process. Addressing these challenges, this paper proposes a novel fusion framework that combines asymmetric inferences from object detectors and semantic segmentation networks for jointly detecting multiple pedestrians. This is achieved by introducing a consensus-based scoring method that fuses pair-wise pixel-relevant information from the object detector and the semantic segmentation network to boost the final confidence scores. The parallel implementation of the object detection and semantic segmentation networks in the proposed framework entails a low runtime overhead. The efficiency and robustness of the proposed fusion framework are extensively evaluated by fusing different state-of-the-art pedestrian detectors and semantic segmentation networks on a public dataset. The generalization of fused models is also examined on new cross pedestrian data collected through an autonomous car. Results show that the proposed fusion method significantly improves detection performance while achieving competitive runtime efficiency.

Pedestrian Detection in Driver Assistance Using SSD and PS-GAN

<p>Pedestrian detection is a critical challenge in the field of general object detection, the performance of object detection has advanced with the development of deep learning. However, considerable improvement is still required for pedestrian detection, considering the differences in pedestrian wears, action, and posture. In the driver assistance system, it is necessary to further improve the intelligent pedestrian detection ability. We present a method based on the combination of SSD and GAN to improve the performance of pedestrian detection. Firstly, we assess the impact of different kinds of methods which can detect pedestrians based on SSD and optimize the detection for pedestrian characteristics. Secondly, we propose a novel network architecture, namely data synthesis PS-GAN to generate diverse pedestrian data for verifying the effectiveness of massive training data to SSD detector. Experimental results show that the proposed manners can improve the performance of pedestrian detection to some extent. At last, we use the pedestrian detector to simulate a specific application of motor vehicle assisted driving which would make the detector focus on specific pedestrians according to the velocity of the vehicle. The results establish the validity of the approach.</p>

Pedestrian Detection in Driver Assistance Using SSD and PS-GAN

<p>Pedestrian detection is a critical challenge in the field of general object detection, the performance of object detection has advanced with the development of deep learning. However, considerable improvement is still required for pedestrian detection, considering the differences in pedestrian wears, action, and posture. In the driver assistance system, it is necessary to further improve the intelligent pedestrian detection ability. We present a method based on the combination of SSD and GAN to improve the performance of pedestrian detection. Firstly, we assess the impact of different kinds of methods which can detect pedestrians based on SSD and optimize the detection for pedestrian characteristics. Secondly, we propose a novel network architecture, namely data synthesis PS-GAN to generate diverse pedestrian data for verifying the effectiveness of massive training data to SSD detector. Experimental results show that the proposed manners can improve the performance of pedestrian detection to some extent. At last, we use the pedestrian detector to simulate a specific application of motor vehicle assisted driving which would make the detector focus on specific pedestrians according to the velocity of the vehicle. The results establish the validity of the approach.</p>

Applying a Deep Learning Neural Network to Gait-Based Pedestrian Automatic Detection and Recognition

Gait recognition is a noncontact biometric procedure that determines the identity or health status of a person by analyzing his or her walking posture and habits, including skeletal and joint movements. The most remarkable feature of this method is the possibility of conducting recognition without demanding much cooperation from participants. Therefore, this recognition technique has attracted much attention from scholars. Additionally, because of the rapid development of graphics processing unit technology, related hardware and computation performance, the applications of deep-learning technology are considerably enhanced. The objective of this study was to apply a deep neural network (DNN), which employs deep-learning technology, to achieve gait-based automatic pedestrian detection and recognition. In contrast to using wearable devices to precisely capture skeletal and joint movements, pedestrian color-image sequences were used as input in this study. Subsequently, a pretraining convolutional neural network (CNN) was employed to capture pedestrian location and extract pedestrian dense optical flow to serve as concrete low-level feature inputs. Then, a finely-tuned DNN based on the wide residual network was employed to extract high-level abstract features. In addition, to overcome the difficulty of obtaining local temporal features by using a 2D CNN, part of the 3D convolutional structure was introduced into the CNN. This design enabled use of limited memory to acquire more effective features and enhance the DNN performance. The experimental results show that the proposed method has exceptional performance for pedestrian detection and recognition.

Near-surface pedestrian detection method based on deep learning for UAVs in low illumination environments

With the development of unmanned aerial vehicles (UAVs) and computer vision, target detection methods based on UAVs have been increasingly applied in military and civilian fields. Considering the adaptability requirements of low illumination environments such as rain, fog, and night, visible and infrared (IR) sensors are often installed on UAVs to perform in all-weather and all-day conditions. To improve the near-surface detection performance of UAVs in low illumination environments, a pedestrian detection method using image fusion and deep learning is proposed. Visible and IR pedestrian images are collected by the UAV. The corresponding aerial images are registered and annotated. These two different types of aerial images are aligned at the time sequence and matched using the scale invariant feature transform. A U-type generative adversarial network (GAN) is first developed to fuse visible and IR images. A convolutional block attention module is introduced to strengthen the pedestrian target information in the GAN. The spatial domain and channel domain attention mechanisms are proposed to generate color fusion images with rich details and solve the problems of feature extraction as well as fusion rules designed manually in the existing image fusion methods. Then, You Only Look Once Version 3 (YOLOv3)-spatial pyramid pooling combined with transfer learning is adopted using the fused images to train the model on our aerial dataset to verify the pedestrian detection performance. In addition, comparison experiments are carried out. The experimental results demonstrate that the YOLOv3 model is successfully transferred to the target dataset. The performance of the proposed detection model using the fused images for transfer training is the best among the different methods. Finally, the accuracy P, recall R, mean average precision, and F1 score reach 0.804, 0.923, 0.928, and 0.859, respectively.

Multi-expert learning for fusion of pedestrian detection bounding box

Performance of pedestrian detection, which is one of the essential tasks in automatic drive, relies heavily on a large number of labels. Some researchers proposed unsupervised domain adaptive frameworks to improve the detection accuracy in wild datasets to reduce the need for labels. However, it is not a down-to-earth and cost-effective way for deploying these frameworks in practical engineering because it needs both source and target data for training. Unlike the former research, this work presents a new fine-tuning method without using source and target data for unsupervised detection. In this work, different well-trained models from the source domain are regarded as less-accurate experts in the wild domain, where a multi-expert learning algorithm is applied to learn from the difference between these models and fuse bounding boxes to present more accurate detection results. Experimental results on three common pedestrian detection datasets show that our method can efficiently improve the detection accuracy under unsupervised settings. Our method can also achieve better performance without source and target data involved compared with state-of-the-art works.

3D Sensor Based Pedestrian Detection by Integrating Improved HHA Encoding and Two-Branch Feature Fusion

Pedestrian detection is vitally important in many computer vision tasks but still suffers from some problems, such as illumination and occlusion if only the RGB image is exploited, especially in outdoor and long-range scenes. Combining RGB with depth information acquired by 3D sensors may effectively alleviate these problems. Therefore, how to utilize depth information and how to fuse RGB and depth features are the focus of the task of RGB-D pedestrian detection. This paper first improves the most commonly used HHA method for depth encoding by optimizing the gravity direction extraction and depth values mapping, which can generate a pseudo-color image from the depth information. Then, a two-branch feature fusion extraction module (TFFEM) is proposed to obtain the local and global features of both modalities. Based on TFFEM, an RGB-D pedestrian detection network is designed to locate the people. In experiments, the improved HHA encoding method is twice as fast and achieves more accurate gravity-direction extraction on four publicly-available datasets. The pedestrian detection performance of the proposed network is validated on KITTI and EPFL datasets and achieves state-of-the-art performance. Moreover, the proposed method achieved third ranking among all published works on the KITTI leaderboard. In general, the proposed method effectively fuses RGB and depth features and overcomes the effects of illumination and occlusion problems in pedestrian detection.

MRPT: Millimeter-Wave Radar-Based Pedestrian Trajectory Tracking for Autonomous Urban Driving

Pedestrians represent agile and low-observable targets, especially under adverse weather conditions, whose trajectory tracking plays a crucial role in determining pedestrian behavior in autonomous urban driving. Thus, this article presents a millimeter-wave radar-based pedestrian trajectory-tracking (MRPT) system that enables all-weather trajectory perception with high precision. More specifically, to improve the tracking performance in the presence of strong background clutters, a track-before-detection-based algorithm is proposed to address pedestrian detection and tracking jointly, instead of regarding them as two separate phases in conventional methods. After that, a continuous detection method based on the integration of target existence probability and the Markov transition matrix is proposed to achieve superior pedestrian detection performance by directly using unthresholded radar data. In addition, based on the binary-phase-multiplex (BPM) strategy, the proposed system is validated using a low-cost automotive frequency-modulated-continuous-waveform (FMCW) multiple-input–multiple-output (MIMO) radar sensor. Consequently, extensive simulation and experimental results demonstrate that MRPT exhibits better pedestrian detection and tracking performance under low signal-noise-ratio (SNR) conditions compared with traditional methods.

Improving the Pedestrian Detection Performance in the Absence of Rich Training Datasets: A UK Case Study

The World Health Organization estimates that well in excess of one million of lives are lost each year due to road traffic accidents. Since the human factor is the preeminent cause behind the traffic accidents, the development of reliable Advanced Driver Assistance Systems (ADASs) and Autonomous Vehicles (AVs) is seen by many as a possible solution to improve road safety. ADASs rely on the car perception system input that consists of camera(s), LIDAR and/or radar to detect pedestrians and other objects on the road. Hardware improvements as well as advances done in employing Deep Learning techniques for object detection popularized the Convolutional Neural Networks in the area of autonomous driving research and applications. However, the availability of quality and large datasets continues to be a most important contributor to the Deep Learning based model’s performance. With this in mind, this work analyses how a YOLO-based object detection architecture responded to limited data available for training and containing low-quality images. The work focused on pedestrian detection, since vulnerable road user’s safety is a major concern within AV and ADAS research communities. The proposed model was trained and tested on data gathered from Coventry, United Kingdom, city streets. The results show that the original YOLOv3 implementation reaches a 42.18% average precision (AP) and the main challenge was in detecting small objects. Network modifications were made and our final model, based on the original YOLOv3 implementation, achieved 51.6% AP. It is also demonstrated that the employed data augmentation approach is responsible for doubling the average precision of the final model.

HQA‐Trans: An end‐to‐end high‐quality‐awareness image translation framework for unsupervised cross‐domain pedestrian detection

Unsupervised cross-domain pedestrian detection has attracted attention in recent years. Although some works adopted unsupervised image translation frameworks to generate an intermediate domain to narrow the gap between source and target domains, the images in the intermediate domain tend to be distorted due to the instability of the generation network. In this work, we propose a new framework to improve the image quality of the generated intermediate domain via an end-to-end translation framework. First, an image quality assessment index is adopted and adjusted appropriately. The part that controls the image quality is kept, and the part that adversely affects the domain style translation is discarded. Secondly, the adjusted image quality assessment index is integrated into the unsupervised image translation framework, where a new loss with the index's weight is proposed. An end-to-end high-quality-awareness image translation framework is constructed to generate a high-quality intermediate domain directly through this process. Finally, the intermediate domain with high-quality images is applied for cross-domain pedestrian detection. Experimental results on benchmark datasets show that the proposed framework can effectively improve unsupervised cross-domain pedestrian detection performance. Compared with some state-of-the-art works, the proposed framework can also achieve superior performance under miss rate metrics.

Design Guidelines on Deep Learning–based Pedestrian Detection Methods for Supporting Autonomous Vehicles

Intelligent transportation systems (ITS) enable transportation participants to communicate with each other by sending and receiving messages, so that they can be aware of their surroundings and facilitate efficient transportation through better decision making. As an important part of ITS, autonomous vehicles can bring massive benefits by reducing traffic accidents. Correspondingly, much effort has been paid to the task of pedestrian detection, which is a fundamental task for supporting autonomous vehicles. With the progress of computational power in recent years, adopting deep learning–based methods has become a trend for improving the performance of pedestrian detection. In this article, we present design guidelines on deep learning–based pedestrian detection methods for supporting autonomous vehicles. First, we will introduce classic backbone models and frameworks, and we will analyze the inherent attributes of pedestrian detection. Then, we will illustrate and analyze representative pedestrian detectors from occlusion handling, multi-scale feature extraction, multi-perspective data utilization, and hard negatives handling these four aspects. Last, we will discuss the developments and trends in this area, followed by some open challenges.

A novel visibility semantic feature-aided pedestrian detection scheme for autonomous vehicles

Intelligent transportation systems (ITS) have become a popular method for enhancing transportation safety and efficiency. As essential participants of ITS, autonomous vehicles need to detect pedestrians accurately. In this paper, we propose a one-stage anchor-free pedestrian detection model named Bi-Center Network (BCNet), which is aided by the semantic features of pedestrians’ visible parts. We perform an ablation study to discover how visibility features could benefit the detector’s performance, including introducing two hyper-parameters and adopting three different attention mechanisms, respectively. The experimental results indicate that the performance of pedestrian detection could be significantly improved, since the visibility semantic could prompt stronger responses on the heatmap. We compare our BCNet variants with state-of-the-art models on the CityPersons dataset and ETH dataset; results indicate that our detector is effective and achieves a promising performance.