CityPersons Dataset Research Articles

An enhanced lightweight model for small-scale pedestrian detection based on YOLOv8s

Autonomous vehicle scenarios often involve occluded and distant pedestrians, leading to missed and false detections or models that are too large to deploy. To address these issues, this study proposed a lightweight model based on Yolov8s. Feature extraction and fusion networks were redesigned to optimize the detection layer for better detection. The Backbone Network incorporated Dual Conv and ELAN to create the EDLAN module. The EDLAN module and optimized SPPF-LSKA improved the small-scale pedestrian feature extraction in complex backgrounds while reducing the parameters and computation. In Neck Network, BiFPN and VoVGSCSP enhance pedestrian features and improve detection. In addition, the WIoU loss function addressed the target imbalance to enhance generalization ability and overall performance. Enhanced Yolov8s was trained and validated using the CityPersons dataset. Compared to Yolov8s, it improved the precision, recall, F1 score, and mAP@50 by 5.2%, 7.2%, 6.8%, and 6.8%, respectively, while reducing the parameters by 68% and compressing the model size by 67%. The validation experiments were conducted on Caltech and BDD100K datasets. The result demonstrated that precision increased by 3.4% and 1.1%, the mAP@50 also increased by 7.6% and 2.8%, respectively. The modified model reduced the model parameters and size while effectively improving the detection accuracy, making it highly valuable for autonomous driving scenarios.

Distilled Center and Scale Prediction: Distilling anchor-free pedestrian detector for edge computing

As an important task of the Internet of Things, pedestrian detection has attained good detection results powered by deep learning. However, common pedestrian detectors based on deep learning usually require lots of computing resources and consume large amounts of energy, making them not suitable for edge devices in the edge computing paradigm. Therefore, for the pedestrian detection task in edge computing, we consider knowledge distillation on the anchor-free detector Center and Scale Prediction to ensure the pedestrian detection performance while reducing the parameter number and inference time of the detector as much as possible. We propose a distillation framework Distilled Center and Scale Prediction, in which we implement feature-based and response-based distillation for transferring knowledge from the larger model to the smaller model. In order to transmit information useful for detection in distillation as much as possible, Multi-Reference Distillation is designed to filter the transferred knowledge. Moreover, Cross-Module Distillation is proposed to enhance the transfer of relational information during distillation. We perform related experiments on the CityPersons dataset. Our proposed distilled detector achieves 10.45% MR−2 with ResNet18 as backbone and 10.27% MR−2 with ResNet50 as backbone, even outperforming the original teacher detectors. At the same time, the inference time per image is reduced by more than 10% compared to the original teacher detector.

Research on Deep Learning Detection Model for Pedestrian Objects in Complex Scenes Based on Improved YOLOv7.

Pedestrian detection is very important for the environment perception and safety action of intelligent robots and autonomous driving, and is the key to ensuring the safe action of intelligent robots and auto assisted driving. In response to the characteristics of pedestrian objects occupying a small image area, diverse poses, complex scenes and severe occlusion, this paper proposes an improved pedestrian object detection method based on the YOLOv7 model, which adopts the Convolutional Block Attention Module (CBAM) attention mechanism and Deformable ConvNets v2 (DCNv2) in the two Efficient Layer Aggregation Network (ELAN) modules of the backbone feature extraction network. In addition, the detection head is replaced with a Dynamic Head (DyHead) detector head with an attention mechanism; unnecessary background information around the pedestrian object is also effectively excluded, making the model learn more concentrated feature representations. Compared with the original model, the log-average miss rate of the improved YOLOv7 model is significantly reduced in both the Citypersons dataset and the INRIA dataset. The improved YOLOv7 model proposed in this paper achieved good performance improvement in different pedestrian detection problems. The research in this paper has important reference significance for pedestrian detection in complex scenes such as small, occluded and overlapping objects.

Ranking-based adaptive query generation for DETRs in crowded pedestrian detection

Variants of DEtection TRansformer (DETRs) have shown promising performance in crowded pedestrian detection. However, we observe that DETRs are sensitive to the hyper-parameter (the number of queries). Adjusting this hyper-parameter is crucial for achieving competitive performance across different crowded pedestrian datasets. Existing query generation methods are limited to generate a fixed number of queries based on this hyper-parameter, which often leads to missed detections and incorrect detections due to the varied number and density of pedestrians in crowded scenes. To address this challenge, we propose an adaptive query generation method called Ranking-based Adaptive Query Generation (RAQG). RAQG comprises three components: a ranking prediction head, a query supplementer, and Soft Gradient L1 Loss (SGL1). Specifically, we leverage the ranking of the lowest confidence score positive training sample to generate queries adaptively. The ranking prediction head predicts this ranking, which guides our query generation. Additionally, to refine the query generation process, we introduce a query supplementer that adjusts the number of queries based on the predicted ranking. Furthermore, we introduce SGL1, a novel loss function for training the ranking prediction head over a wide regression range. Our method is designed to be lightweight and universal, suitable for integration into any DETRs framework for crowded pedestrian detection. Experimental results on Crowdhuman and Citypersons datasets demonstrate that our RAQG method can generate queries adaptively and achieves competitive results. Notably, our approach achieves a state-of-the-art 39.4% MR on Crowdhuman.

YOLOv7-PD: Incorporating DE-ELAN and NWD-CIoU for Advanced Pedestrian Detection Method

In the pedestrian detection task, the excessive depth of the convolutional network in YOLOv7 results in an abundance of background feature information, thereby posing challenges for the model to accurately locate and detect pedestrians, particularly in small-scale or heavily occluded scenarios. To handle this problem, we propose a pedestrian detection model called YOLOv7-PD, to strengthen the accuracy of detecting small-scale pedestrians and occluded pedestrians. First of all, we propose an improved module called DE-ELAN, an improvement on the existing E-ELAN module, which is based on Omni-Dimensional Dynamic Convolution (ODConv). This module leverages four complementary attention types to enhance feature extraction, capturing rich contextual information. Then, we propose a lightweight receptive field enhancement module called light-REFM, which constructs a pyramid structure and acquires fine-grained multi-scale information through dilated convolutions of different sizes. Finally, we propose an improved regression loss function based on the Normalized Wasserstein Distance (NWD) that combines NWD with Complete-IoU (CIoU), enabling precise position and feature capture for small targets. On the Citypersons dataset, YOLOv7-PD outperforms YOLOv7, improving the average precision (AP) by 7% and reducing the miss rate by 2.58%. Experiments on three challenging pedestrian detection datasets demonstrate a balance between precision and speed, achieving excellent performance.

Improved YOLOv3 Integrating SENet and Optimized GIoU Loss for Occluded Pedestrian Detection.

Occluded pedestrian detection faces huge challenges. False positives and false negatives in crowd occlusion scenes will reduce the accuracy of occluded pedestrian detection. To overcome this problem, we proposed an improved you-only-look-once version 3 (YOLOv3) based on squeeze-and-excitation networks (SENet) and optimized generalized intersection over union (GIoU) loss for occluded pedestrian detection, namely YOLOv3-Occlusion (YOLOv3-Occ). The proposed network model considered incorporating squeeze-and-excitation networks (SENet) into YOLOv3, which assigned greater weights to the features of unobstructed parts of pedestrians to solve the problem of feature extraction against unsheltered parts. For the loss function, a new generalized intersection over unionintersection over groundtruth (GIoUIoG) loss was developed to ensure the areas of predicted frames of pedestrian invariant based on the GIoU loss, which tackled the problem of inaccurate positioning of pedestrians. The proposed method, YOLOv3-Occ, was validated on the CityPersons and COCO2014 datasets. Experimental results show the proposed method could obtain 1.2% MR-2 gains on the CityPersons dataset and 0.7% mAP@50 improvements on the COCO2014 dataset.

Multi-Attribute NMS: An Enhanced Non-Maximum Suppression Algorithm for Pedestrian Detection in Crowded Scenes

Removing duplicate proposals is a critical process in pedestrian detection, and is usually performed via Non-Maximum Suppression (NMS); however, in crowded scenes, the detection proposals of occluded pedestrians are hard to distinguish from duplicate proposals, making the detection results inaccurate. In order to address the above-mentioned problem, the authors of this paper propose a Multi-Attribute NMS (MA-NMS) algorithm, which combines density and count attributes in order to adaptively adjust suppression, effectively preserving the proposals of occluded pedestrians while removing duplicate proposals. In order to obtain the density and count attributes, an attribute branch (ATTB), which uses a context extraction module (CEM) to extract the context of pedestrians, and then, concatenates the context with the features of pedestrians in order to predict both the density and count attributes simultaneously, is also proposed. With the proposed ATTB, a pedestrian detector, based on MA-NMS, is constructed for pedestrian detection in crowded scenes. Extensive experiments are conducted using the CrowdHuman and CityPersons datasets, and the results show that the proposed method outperforms mainstream methods on AP (average precision), Recall, and MR−2 (log-average miss rate), sufficiently validating the effectiveness of the proposed MA-NMS algorithm.

THFE: A Triple-hierarchy Feature Enhancement method for tiny boat detection

In boat navigation, especially in complex sea conditions, the detection performance of the tiny boat is related to the safety of boat sailing. However, due to the tiny boat occupying fewer pixels, the effective features of the tiny boat are difficult to obtain. Current tiny object detection methods focus primarily on dataset size matching, feature fusion, and label assignment, lack of attention to texture and detail information loss, and insufficient semantic utilization. Here, we propose a Triple-hierarchy Feature Enhancement (THFE) method to detect tiny boats. The core idea behind THFE is to enhance the semantic information from different layers to supplement the effective features of tiny boats. It consists of three spaces: the super-resolution enhancement space, the semantic enhancement space, and the hierarchical enhancement space. In THFE, sub-pixel convolution, sparse self-attention mechanism, channel attention mechanism, and spatial attention mechanism are adopted to hierarchically enhance each layer’s high-level and low-level semantic features. Finally, each layer’s high-level and low-level semantic features are adaptively fused so that each feature map contains richer high-level and low-level semantic information. Experiments show that our proposed THFE method achieves impressive gains in detection performance. For example, in terms of AP50tiny, our method outperforms state-of-the-art methods by 1.7% on the TinyBoats dataset, 3.1% on the TinyPersons Dataset and 3.9% on the Tiny CityPersons Dataset. To further study the detection of tiny boats, we introduce a tiny boat dataset that will be publicly accessible.

CAPNet: Context and Attribute Perception for Pedestrian Detection

With a focus on practical applications in the real world, a number of challenges impede the progress of pedestrian detection. Scale variance, cluttered backgrounds and ambiguous pedestrian features are the main culprits of detection failures. According to existing studies, consistent feature fusion, semantic context mining and inherent pedestrian attributes seem to be feasible solutions. In this paper, to tackle the prevalent problems of pedestrian detection, we propose an anchor-free pedestrian detector, named context and attribute perception (CAPNet). In particular, we first generate features with consistent well-defined semantics and local details by introducing a feature extraction module with a multi-stage and parallel-stream structure. Then, a global feature mining and aggregation (GFMA) network is proposed to implicitly reconfigure, reassign and aggregate features so as to suppress irrelevant features in the background. At last, in order to bring more heuristic rules to the network, we improve the detection head with an attribute-guided multiple receptive field (AMRF) module, leveraging the pedestrian shape as an attribute to guide learning. Experimental results demonstrate that introducing the context and attribute perception greatly facilitates detection. As a result, CAPNet achieves new state-of-the-art performance on Caltech and CityPersons datasets.

Correlation-and-Correction Fusion Attention Network for Occluded Pedestrian Detection

As a significant issue in computer vision, pedestrian detection has achieved certain achievements with the support of deep learning. However, pedestrian detection in congested scenes still encounters the challenging problem of feature loss and obfuscation. To address the issue, we propose a pedestrian detection network based on a correlation-and-correction fusion attention mechanism. First, a multimask correction attention module is proposed, which generates visible part masks of pedestrians, enhancing the visible region’s features and correcting the false one. Besides, the module preserves the features of multiclass pedestrians by generating multiple masks. Then, we fuse a correlation channel attention module to enhance the correlation of various pedestrians’ body features. Next, we studied three fusion methods of correlation and correction attention mechanisms and found that the serial connection of “correlation first and correction behind” works best. Finally, we extend our method to multiclass pedestrian detection in congested scenes. Experimental results on the CityPersons, Caltech, and CrowdHuman datasets demonstrate the effectiveness of our method. On the CityPersons dataset where more than 70% of pedestrians are occluded, our method outperforms the baseline method by 1.12% on the heavy occlusion subset and surpasses many outstanding methods.

SSAD: Single-Shot Multi-Scale Attentive Detector for Autonomous Driving

A novel Single-shot Multi-scale detection network with feature fusion and multi-scale attention mechanism is proposed for autonomous driving. The proposed model is referred to as a Single-shot Multi-scale Attentive Detector (SSAD) and it would build feature relations of the feature map in the spatial space. The proposed network highlights pedestrian and vehicle regions on the extracted feature map and also suppresses irrelevant regions, from the global relation information and thereby providing reliable guidance for autonomous driving, while detecting smaller and occluded targets. SSAD design is simple, accurate and computationally efficient. Evaluation results of the proposed network on multiple datasets have achieved considerable and promising results. Experimental results show that the SSAD network when tested on Pascal Voc-2007, INRIA, Caltech, and City Persons datasets, outperforms many state-of-the-art (SOTA) detectors.

Pedestrian Detection Using Integrated Aggregate Channel Features and Multitask Cascaded Convolutional Neural-Network-Based Face Detectors.

Pedestrian detection is a challenging task, mainly owing to the numerous appearances of human bodies. Modern detectors extract representative features via the deep neural network; however, they usually require a large training set and high-performance GPUs. For these cases, we propose a novel human detection approach that integrates a pretrained face detector based on multitask cascaded convolutional neural networks and a traditional pedestrian detector based on aggregate channel features via a score combination module. The proposed detector is a promising approach that can be used to handle pedestrian detection with limited datasets and computational resources. The proposed detector is investigated comprehensively in terms of parameter choices to optimize its performance. The robustness of the proposed detector in terms of the training set, test set, and threshold is observed via tests and cross dataset validations on various pedestrian datasets, including the INRIA, part of the ETHZ, and the Caltech and Citypersons datasets. Experiments have proved that this integrated detector yields a significant increase in recall and a decrease in the log average miss rate compared with sole use of the traditional pedestrian detector. At the same time, the proposed method achieves a comparable performance to FRCNN on the INRIA test set compared with sole use of the Aggregated Channel Features detector.

Pedestrian Fall Event Detection in Complex Scenes Based on Attention-Guided Neural Network

To address automatic detection of pedestrian fall events and provide feedback in emergency situations, this paper proposes an attention-guided real-time and robust method for pedestrian detection in complex scenes. First, the YOLOv3 network is used to effectively detect pedestrians in the videos. Then, an improved DeepSort algorithm is used to track by detection. After tracking, the authors extract effective features from the tracked bounding box, use the output of the last convolutional layer, and introduce the attention weight factor into the tracking module for final fall event prediction. Finally, the authors use the sliding window for storing feature maps and SVM classifier to redetect fall events. The experimental results on the CityPersons dataset, Montreal fall dataset, and self-built dataset indicate that this approach has good performance in complex scenes. The pedestrian detection rate is 87.05%, the accuracy of fall event detection reaches 98.55%, and the delay is within 120 ms.

Region Resolution Learning and Region Segmentation Learning with Overall and Body Part Perception for Pedestrian Detection

Pedestrian detection is a great challenge, especially in complex and diverse occlusion environments. When a pedestrian is in an occlusion situation, the pedestrian visible part becomes incomplete, and the body bounding box contains part of the pedestrian, other objects and backgrounds. Based on this, we attempt different methods to help the detector learn more features of the pedestrian under different occlusion situations. First, we propose region resolution learning, which learns the pedestrian regions on the input image. Second, we propose fine-grained segmentation learning to learn the outline and shape of different parts of pedestrians. We propose an anchor-free approach that combines a pedestrian detector CSP, region Resolution learning and Segmentation learning (CSPRS). We help the detector to learn extra features. CSPRS provides another way to perceive pixels, outline and shapes in pedestrian areas. This detector includes region resolution learning, and segmentation learning helps the detector to locate pedestrians. By simply adding the region resolution learning branch and segmentation branch, CSPRS achieves good results. The experimental results show that both methods of learning pedestrian features improve performance. We evaluate our proposed detector CSPRS on the CityPersons benchmark, and the experiments show that CSPRS achieved 42.53% on the heavy subset on the CityPersons dataset.

Cascaded Cross-Layer Fusion Network for Pedestrian Detection

The detection method based on anchor-free not only reduces the training cost of object detection, but also avoids the imbalance problem caused by an excessive number of anchors. However, these methods only pay attention to the impact of the detection head on the detection performance, thus ignoring the impact of feature fusion on the detection performance. In this article, we take pedestrian detection as an example and propose a one-stage network Cascaded Cross-layer Fusion Network (CCFNet) based on anchor-free. It consists of Cascaded Cross-layer Fusion module (CCF) and novel detection head. Among them, CCF fully considers the distribution of high-level information and low-level information of feature maps under different stages in the network. First, the deep network is used to remove a large amount of noise in the shallow features, and finally, the high-level features are reused to obtain a more complete feature representation. Secondly, for the pedestrian detection task, a novel detection head is designed, which uses the global smooth map (GSMap) to provide global information for the center map to obtain a more accurate center map. Finally, we verified the feasibility of CCFNet on the Caltech and CityPersons datasets.

A novel visibility semantic feature-aided pedestrian detection scheme for autonomous vehicles

Intelligent transportation systems (ITS) have become a popular method for enhancing transportation safety and efficiency. As essential participants of ITS, autonomous vehicles need to detect pedestrians accurately. In this paper, we propose a one-stage anchor-free pedestrian detection model named Bi-Center Network (BCNet), which is aided by the semantic features of pedestrians’ visible parts. We perform an ablation study to discover how visibility features could benefit the detector’s performance, including introducing two hyper-parameters and adopting three different attention mechanisms, respectively. The experimental results indicate that the performance of pedestrian detection could be significantly improved, since the visibility semantic could prompt stronger responses on the heatmap. We compare our BCNet variants with state-of-the-art models on the CityPersons dataset and ETH dataset; results indicate that our detector is effective and achieves a promising performance.

Pedestrian Detection by Novel Axis-Line Representation and Regression Pattern.

The pattern of bounding box representation and regression has long been dominant in CNN-based pedestrian detectors. Despite the method’s success, it cannot accurately represent location, and introduces unnecessary background information, while pedestrian features are mainly located in axis-line areas. Other object representations, such as corner-pairs, are not easy to obtain by regression because the corners are far from the axis-line and are greatly affected by background features. In this paper, we propose a novel detection pattern, named Axis-line Representation and Regression (ALR), for pedestrian detection in road scenes. Specifically, we design a 3-d axis-line representation for pedestrians and use it as the regression target during network training. A line-box transformation method is also proposed to fit the widely used box-annotations. Meanwhile, we explore the influence of deformable convolution base-offset on detection performance and propose a base-offset initialization strategy to further promote the gain brought by ALR. Notably, the proposed ALR pattern can be introduced into both anchor-based and anchor-free frameworks. We validate the effectiveness of ALR on the Caltech-USA and CityPersons datasets. Experimental results show that our approach outperforms the baseline significantly through simple modifications and achieves competitive accuracy with other methods without bells and whistles.

KGSNet: Key-Point-Guided Super-Resolution Network for Pedestrian Detection in the Wild.

In real-world scenarios (i.e., in the wild), pedestrians are often far from the camera (i.e., small scale), and they often gather together and occlude with each other (i.e., heavily occluded). However, detecting these small-scale and heavily occluded pedestrians remains a challenging problem for the existing pedestrian detection methods. We argue that these problems arise because of two factors: 1) insufficient resolution of feature maps for handling small-scale pedestrians and 2) lack of an effective strategy for extracting body part information that can directly deal with occlusion. To solve the above-mentioned problems, in this article, we propose a key-point-guided super-resolution network (coined KGSNet) for detecting these small-scale and heavily occluded pedestrians in the wild. Specifically, to address factor 1), a super-resolution network is first trained to generate a clear super-resolution pedestrian image from a small-scale one. In the super-resolution network, we exploit key points of the human body to guide the super-resolution network to recover fine details of the human body region for easier pedestrian detection. To address factor 2), a part estimation module is proposed to encode the semantic information of different human body parts where four semantic body parts (i.e., head and upper/middle/bottom body) are extracted based on the key points. Finally, based on the generated clear super-resolved pedestrian patches padded with the extracted semantic body part images at the image level, a classification network is trained to further distinguish pedestrians/backgrounds from the inputted proposal regions. Both proposed networks (i.e., super-resolution network and classification network) are optimized in an alternating manner and trained in an end-to-end fashion. Extensive experiments on the challenging CityPersons data set demonstrate the effectiveness of the proposed method, which achieves superior performance over previous state-of-the-art methods, especially for those small-scale and heavily occluded instances. Beyond this, we also achieve state-of-the-art performance (i.e., 3.89% MR-2 on the reasonable subset) on the Caltech data set.

Occluded pedestrian detection combined with semantic features

The task of pedestrian detection is to identify the location and size of pedestrians in images or videos. However, occlusions are very common in real-life scenarios, which make pedestrian detection more difficult. In order to solve the occlusion problem in pedestrian detection, a semantic feature enhancement module to acquire more informative and richer semantic features is proposed. The detector enhances semantic features by fusing feature maps of different layers, and detects pedestrians based on their locations and scales. The Experiments performed on Caltech and CityPersons datasets show that the algorithm achieves superior performance for detecting occluded pedestrians, especially heavily occluded ones. 30.6% and 47.9% log-average missing rates are achieved in the heavily occluded subsets of Caltech and Cityperons, respectively. Moreover, this method is robust to the detection of heavily occluded pedestrians, and the module can be easily used by other detection frameworks.

Coupled Network for Robust Pedestrian Detection With Gated Multi-Layer Feature Extraction and Deformable Occlusion Handling.

Pedestrian detection methods have been significantly improved with the development of deep convolutional neural networks. Nevertheless, detecting ismall-scaled pedestrians and occluded pedestrians remains a challenging problem. In this paper, we propose a pedestrian detection method with a couple-network to simultaneously address these two issues. One of the sub-networks, the gated multi-layer feature extraction sub-network, aims to adaptively generate discriminative features for pedestrian candidates in order to robustly detect pedestrians with large variations on scale. The second sub-network targets on handling the occlusion problem of pedestrian detection by using deformable regional region of interest (RoI)-pooling. We investigate two different gate units for the gated sub-network, namely, the channel-wise gate unit and the spatio-wise gate unit, which can enhance the representation ability of the regional convolutional features among the channel dimensions or across the spatial domain, repetitively. Ablation studies have validated the effectiveness of both the proposed gated multi-layer feature extraction sub-network and the deformable occlusion handling sub-network. With the coupled framework, our proposed pedestrian detector achieves promising results on both two pedestrian datasets, especially on detecting small or occluded pedestrians. On the CityPersons dataset, the proposed detector achieves the lowest missing rates (i.e. 40.78% and 34.60%) on detecting small and occluded pedestrians, surpassing the second best comparison method by 6.0% and 5.87%, respectively.