Pedestrian Detection Benchmark Research Articles

Replacing the human driver: An objective benchmark for occluded pedestrian detection

Early detection of vulnerable road users is a crucial requirement for autonomous vehicles to meet and exceed the object detection capabilities of human drivers. One of the most complex outstanding challenges is that of partial occlusion where a target object is only partially available to the sensor due to obstruction by another foreground object. A number of leading pedestrian detection benchmarks provide annotation for partial occlusion, however each benchmark varies greatly in their definition of the occurrence and severity of occlusion. Research demonstrates that a high degree of subjectivity is used to classify occlusion level in these cases and occlusion is typically categorized into 2–3 broad categories such as “partially” and “heavily” occluded. In addition, many pedestrian instances are impacted by multiple inhibiting factors which contribute to non-detection such as object scale, distance from camera, lighting variations and adverse weather. This can lead to inaccurate or inconsistent reporting of detection performance for partially occluded pedestrians depending on which benchmark is used. This research introduces a novel, objective benchmark for partially occluded pedestrian detection to facilitate the objective characterization of pedestrian detection models. Characterization is carried out on seven popular pedestrian detection models for a range of occlusion levels from 0%–99% to demonstrate the impact of progressive levels of partial occlusion on pedestrian detectability. Results show that the proposed benchmark provides more objective, fine grained analysis of pedestrian detection algorithms than the current state of the art.

An objective method for pedestrian occlusion level classification

Pedestrian detection is among the most safety-critical features of driver assistance systems for autonomous vehicles. One of the most complex detection challenges is that of partial occlusion, where a target object is only partially available to the sensor due to obstruction by another foreground object.A number of current pedestrian detection benchmarks provide annotation for partial occlusion to assess algorithm performance in these scenarios, however each benchmark varies greatly in their definition of the occurrence and severity of occlusion. In addition, current occlusion level annotation methods contain a high degree of subjectivity by the human annotator. This can lead to inaccurate or inconsistent reporting of an algorithm’s detection performance for partially occluded pedestrians, depending on which benchmark is used. This research presents a novel, objective method for pedestrian occlusion level classification for ground truth annotation. Occlusion level classification is achieved through the identification of visible pedestrian keypoints and through the use of a novel, effective method of 2D body surface area estimation. Experimental results demonstrate that the proposed method reflects the pixel-wise occlusion level of pedestrians in images and is effective for all forms of occlusion, including challenging edge cases such as self-occlusion, truncation and inter-occluding pedestrians.

Visible part prediction and temporal calibration for pedestrian detection

Despite their great advancement, current pedestrian detection methods focus on single static images, which fail to employ richer information available from the video sequences. Compared with still images, videos can offer temporal information of objects in the time dimension, thus providing the potential to obtain more robust detection performance. Here, a novel pedestrian detection method based on visible part detection and temporal calibration is proposed. Specifically, a part-aware module to predict the visible body part of each pedestrian instance, which enables us to obtain precise motion information of partially occluded pedestrians in a video sequence, is first developed. Then, the temporal coherence for each pedestrian instance based on the predicted motion information is constructed. After that, an adaptive temporal calibration method is introduced to effectively calibrate the final detection result. This method on two video pedestrian detection benchmarks, that is, Caltech-New and MOT17Det, is evaluated. Experimental results show that this method performs favourably against existing pedestrian detection approaches.

Improving Single-Stage Object Detectors for Nighttime Pedestrian Detection

Improving the reliability of nighttime pedestrian detection is a crucial challenge towards the design of robust autonomous systems. Not surprisingly, most pedestrian fatalities occur in low-illumination settings, thus emphasizing the need for new algorithmic advances. This work presents a novel pedestrian detection approach that makes a number of crucial modifications to the state-of-the-art YOLOV5-PANet architecture, in order to improve the reliability of features extracted from nighttime images. More specifically, the proposed architecture systematically incorporates powerful shuffle attention mechanisms and a transformer module to improve the feature learning pipeline. Instead of advocating the use of other sensing modalities that are better suited for nighttime detection, our approach relies only on conventional RGB cameras and is hence broadly applicable. Our empirical studies with nighttime pedestrian detection benchmarks show that with only minimal increase in model complexity, our approach provides significant improvements in detection efficacy over existing solutions. Finally, we explore the impact of post-hoc network pruning on the speed-accuracy trade-off of our approach and demonstrate that it is well suited for reduced memory/compute requirements.

Research and Design of Open Convolutional Neural Network Based on FPGA

In this paper, using the fluidity and parallelism of FPGA, we design a set of memory scheduling mechanism to implement an open convolutional neural network. The self-built vehicle database and the cropped and zoomed Daimler Pedestrian Detection Benchmark dataset are each trained under different network structures, and their recognition rates reached 97.8% and 98.6%. An open feedforward network is implemented on the FPGA platform, and experiments show that for small convolutional networks with different structures, the FPGA platform can increase its recognition speed when resources permit.

Unreliable-to-Reliable Instance Translation for Semi-Supervised Pedestrian Detection

Generating realistic pedestrian instances in a semi-supervised setting is promising but challenging due to the limited labeled data. We propose an unreliable-to-reliable instance translation model (Un2Reliab) conditioned on unreliable instances which poorly align with pedestrians. Un2Reliab mainly consists of an encoder-decoder-like generative network and a discriminative network, which are jointly trained in a minimax game. We adopt regularization to ensure that the synthesized instances are semantically similar to the corresponding ground truth. Furthermore, to preserve the identities of persons, we propose another regularization to ensure that the synthesized instances associated with the same person should be consistent in appearance. As a result, Un2Reliab learns to restore the missing parts of the original instances. As a side benefit, the synthesized instances are brought into better alignment. Inclusion of the synthesized data improves both the diversity and quality of training data, which eventually leads to better generalization performance. Extensive experiments indicate that Un2Reliab is able to synthesize high-fidelity pedestrian instances and improve the previous state-of-the-art results on multiple semi-supervised pedestrian detection benchmarks.

Adaptive Weighted Multi-Level Fusion of Multi-Scale Features: A New Approach to Pedestrian Detection

Great achievements have been made in pedestrian detection through deep learning. For detectors based on deep learning, making better use of features has become the key to their detection effect. While current pedestrian detectors have made efforts in feature utilization to improve their detection performance, the feature utilization is still inadequate. To solve the problem of inadequate feature utilization, we proposed the Multi-Level Feature Fusion Module (MFFM) and its Multi-Scale Feature Fusion Unit (MFFU) sub-module, which connect feature maps of the same scale and different scales by using horizontal and vertical connections and shortcut structures. All of these connections are accompanied by weights that can be learned; thus, they can be used as adaptive multi-level and multi-scale feature fusion modules to fuse the best features. Then, we built a complete pedestrian detector, the Adaptive Feature Fusion Detector (AFFDet), which is an anchor-free one-stage pedestrian detector that can make full use of features for detection. As a result, compared with other methods, our method has better performance on the challenging Caltech Pedestrian Detection Benchmark (Caltech) and has quite competitive speed. It is the current state-of-the-art one-stage pedestrian detection method.

A Shape Transformation-based Dataset Augmentation Framework for Pedestrian Detection

Deep learning-based computer vision is usually data-hungry. Many researchers attempt to augment datasets with synthesized data to improve model robustness. However, the augmentation of popular pedestrian datasets, such as Caltech and Citypersons, can be extremely challenging because real pedestrians are commonly in low quality. Due to the factors like occlusions, blurs, and low-resolution, it is significantly difficult for existing augmentation approaches, which generally synthesize data using 3D engines or generative adversarial networks (GANs), to generate realistic-looking pedestrians. Alternatively, to access much more natural-looking pedestrians, we propose to augment pedestrian detection datasets by transforming real pedestrians from the same dataset into different shapes. Accordingly, we propose the Shape Transformation-based Dataset Augmentation (STDA) framework. The proposed framework is composed of two subsequent modules, i.e. the shape-guided deformation and the environment adaptation. In the first module, we introduce a shape-guided warping field to help deform the shape of a real pedestrian into a different shape. Then, in the second stage, we propose an environment-aware blending map to better adapt the deformed pedestrians into surrounding environments, obtaining more realistic-looking pedestrians and more beneficial augmentation results for pedestrian detection. Extensive empirical studies on different pedestrian detection benchmarks show that the proposed STDA framework consistently produces much better augmentation results than other pedestrian synthesis approaches using low-quality pedestrians. By augmenting the original datasets, our proposed framework also improves the baseline pedestrian detector by up to 38% on the evaluated benchmarks, achieving state-of-the-art performance.

Graininess-Aware Deep Feature Learning for Robust Pedestrian Detection.

In this paper, we propose a graininess-aware deep feature learning method for pedestrian detection. Unlike most existing methods which utilize the convolutional features without explicit distinction, we appropriately exploit multiple convolutional layers and dynamically select most informative features. Specifically, we train a multi-scale pedestrian attention via pixel-wise segmentation supervision to efficiently identify the pedestrian of particular scales. We encodes the fine-grained attention map into the feature maps of the detection layers to guide them to highlight the pedestrians of specific scale and avoid the background interference. The graininess-aware feature maps generated with our attention mechanism are more focused on pedestrians, and in particular on the small-scale and occluded targets. We further introduce a zoom-in-zoom-out module to enhances the features by incorporating local details and context information. Extensive experimental results on five challenging pedestrian detection benchmarks show that our method achieves very competitive or even better performance with the state-of-the-arts and is faster than most existing approaches.

Multi-Grained Deep Feature Learning for Robust Pedestrian Detection

In this paper, we address the challenging problem of detecting pedestrians, which are heavily occluded and/or far from cameras. Unlike most existing pedestrian detection methods which only use coarse-resolution feature maps with fixed receptive fields, our approach exploits multi-grained deep features to make the detector robust to visible parts of occluded pedestrians and small-size targets. Specifically, we jointly train a multi-scale network and a human parsing network in a weakly supervised manner with only bounding box annotations. We carefully design the multi-scale network to predict pedestrians of particular scales with the most appropriate feature maps, by matching their receptive fields with the target sizes. The human parsing network generates a fine-grained attention map, which helps guide the detector to focus on the visible parts of occluded pedestrians and small-size instances. Both networks are computed in parallel and form a unified single stage pedestrian detector, which assures a suitable tradeoff between accuracy and speed. Moreover, we introduce an adversarial hiding network to make our detector more robust to occlusion situations, which generates occlusions on pedestrians with the goal to fool the detector that in turn adapts itself to learn to localize these adversarial instances. Experiments on three challenging pedestrian detection benchmarks show that our proposed method achieves a state-of-the-art performance and executes $2\times $ faster than the competitive methods.

Efficient Single-Stage Pedestrian Detector by Asymptotic Localization Fitting and Multi-Scale Context Encoding.

Though Faster R-CNN based two-stage detectors have witnessed significant boost in pedestrian detection accuracy, they are still slow for practical applications. One solution is to simplify this working flow as a single-stage detector. However, current single-stage detectors (e.g. SSD) have not presented competitive accuracy on common pedestrian detection benchmarks. Accordingly, a structurally simple but effective module called Asymptotic Localization Fitting (ALF) is proposed, which stacks a series of predictors to directly evolve the default anchor boxes of SSD step by step to improve detection results. Additionally, combining the advantages from residual learning and multi-scale context encoding, a bottleneck block is proposed to enhance the predictors' discriminative power. On top of the above designs, an efficient single-stage detection architecture is designed, resulting in an attractive pedestrian detector in both accuracy and speed. A comprehensive set of experiments on two of the largest pedestrian detection datasets (i.e. CityPersons and Caltech) demonstrate the superiority of the proposed method, comparing to the state of the arts on both the benchmarks.

WiderPerson: A Diverse Dataset for Dense Pedestrian Detection in the Wild

Pedestrian detection has achieved significant progress with the availability of existing benchmark datasets. However, there is a gap in the diversity and density between real world requirements and current pedestrian detection benchmarks: 1) most of existing datasets are taken from a vehicle driving through the regular traffic scenario, usually leading to insufficient diversity; 2) crowd scenarios with highly occluded pedestrians are still under represented, resulting in low density. To narrow this gap and facilitate future pedestrian detection research, we introduce a large and diverse dataset named WiderPerson for dense pedestrian detection in the wild. This dataset involves five types of annotations in a wide range of scenarios, no longer limited to the traffic scenario. There are a total of $13,382$ images with $399,786$ annotations, i.e., $29.87$ annotations per image, which means this dataset contains dense pedestrians with various kinds of occlusions. Hence, pedestrians in the proposed dataset are extremely challenging due to large variations in the scenario and occlusion, which is suitable to evaluate pedestrian detectors in the wild. We introduce an improved Faster R-CNN and the vanilla RetinaNet to serve as baselines for the new pedestrian detection benchmark. Several experiments are conducted on previous datasets including Caltech-USA and CityPersons to analyze the generalization capabilities of the proposed dataset and we achieve state-of-the-art performances on these previous datasets without bells and whistles. Finally, we analyze common failure cases and find the classification ability of pedestrian detector needs to be improved to reduce false alarm and miss detection rates. The proposed dataset is available at http://www.cbsr.ia.ac.cn/users/sfzhang/WiderPerson

Real-world pedestrian detection method enhanced by semantic segmentation

Although pedestrian detection has been largely improved with the emergence of convolutional neural networks (CNN), the performance in autonomous driving still faces various challenges, which mainly include large-scale variation, illumination variation, and occlusion of different levels. A robust pedestrian detector enhanced by semantic segmentation is proposed. Inspired by the benefits of multitask learning, our main idea lies in integrating the task of semantic segmentation into the detection framework with auxiliary supervision, inheriting the merits of the two-stream network. Specifically, anchor boxes with various scales are paved on the feature maps of a base CNN; detection is performed based on bounding box classification and regression. On the other stream, semantic segmentation is also performed based on the same feature maps. Extensive experiments on the recently published large-scale pedestrian detection benchmark, i.e., CityPersons, show that the additional supervision from semantic segmentation can significantly improve the detection accuracy without extra computational burdens during inference, which demonstrates the superiority of the proposed method.

Pedestrian detection with motion features via two-stream ConvNets

Motion information can be important for detecting objects, but it has been used less for pedestrian detection, particularly with deep-learning-based methods. We propose a method that uses deep motion features as well as deep still-image features, following the success of two-stream convolutional networks, each of which are trained separately for spatial and temporal streams. To extract motion clues for detection differentiated from other background motions, the temporal stream takes as input the difference in frames that are weakly stabilized by optical flow. To make the networks applicable to bounding-box-level detection, the mid-level features are concatenated and combined with a sliding-window detector. We also introduce transfer learning from multiple sources in the two-stream networks, which can transfer still image and motion features from ImageNet and an action recognition dataset respectively, to overcome the insufficiency of training data for convolutional neural networks in pedestrian datasets. We conducted an evaluation on two popular large-scale pedestrian benchmarks, namely the Caltech Pedestrian Detection Benchmark and Daimler Mono Pedestrian Detection Benchmark. We observed 10% improvement compared to the same method but without motion features.

Real-Time Pedestrian Detection Using Convolutional Neural Networks

Pedestrian detection provides manager of a smart city with a great opportunity to manage their city effectively and automatically. Specifically, pedestrian detection technology can improve our secure environment and make our traffic more efficient. In this paper, all of our work both modification and improvement are made based on YOLO, which is a real-time Convolutional Neural Network detector. In our work, we extend YOLO’s original network structure, and also give a new definition of loss function to boost the performance for pedestrian detection, especially when the targets are small, and that is exactly what YOLO is not good at. In our experiment, the proposed model is tested on INRIA, UCF YouTube Action Data Set and Caltech Pedestrian Detection Benchmark. Experimental results indicate that after our modification and improvement, the revised YOLO network outperforms the original version and also is better than other solutions.

Multi-scale pedestrian detection using skip pooling and recurrent convolution

Detecting pedestrians of different scales is essential for applications like autonomous driving. Recent research progress showed that combining multiple feature maps and contextual information helps detecting objects of different scales. In this paper, we propose a multi-scale pedestrian detector that combines skip pooling from multi-resolution feature maps and recurrent convolutional layers for extracting contextual information. To fully exploit the unique characteristics of the features at different levels for multi-scale pedestrian detection, the multi-scale features and the context features are fused at the fully connected layer. To gather spatial contextual information, we propose a modified recurrent convolutional layer that produces context feature maps with different resolutions. In addition, we construct a set of scale-dependent classification and bounding box regression subnetworks to further improve the performance of multi-scale pedestrian detection. Experiments on Caltech and KITTI pedestrian detection benchmark datasets show that the proposed method achieves the state-of-the-art performance with faster speed.

Deep Convolutional Neural Network for Pedestrian Detection with Multi-Levels Features Fusion

Pedestrian detection aims to localize and recognize every pedestrian instance in an image with a bounding box. The current state-of-the-art method is Faster RCNN, which is such a network that uses a region proposal network (RPN) to generate high quality region proposals, while Fast RCNN is used to classifiers extract features into corresponding categories. The contribution of this paper is integrated low-level features and high-level features into a Faster RCNN-based pedestrian detection framework, which efficiently increase the capacity of the feature. Through our experiments, we comprehensively evaluate our framework, on the Caltech pedestrian detection benchmark and our methods achieve state-of-the-art accuracy and present a competitive result on Caltech dataset.

Pedestrian Detection with Semantic Regions of Interest.

For many pedestrian detectors, background vs. foreground errors heavily influence the detection quality. Our main contribution is to design semantic regions of interest that extract the foreground target roughly to reduce the background vs. foreground errors of detectors. First, we generate a pedestrian heat map from the input image with a full convolutional neural network trained on the Caltech Pedestrian Dataset. Next, semantic regions of interest are extracted from the heat map by morphological image processing. Finally, the semantic regions of interest divide the whole image into foreground and background to assist the decision-making of detectors. We test our approach on the Caltech Pedestrian Detection Benchmark. With the help of our semantic regions of interest, the effects of the detectors have varying degrees of improvement. The best one exceeds the state-of-the-art.

Towards reliable object detection in noisy images

Image noise is a common problem frequently caused by insufficient lighting, low-quality cameras, image compression and other factors. While low image quality is expected to degrade results of visual recognition, most of the current methods and benchmarks for object recognition, such as Pascal Visual Object Classes Challenge and Microsoft Common Objects in Context Challenge, focus on relatively high-quality images. Meanwhile, object recognition in noisy images is a common problem in surveillance and other domains. In this work we address object detection in noisy images and propose a novel low-cost method for image denoising. When combined with the standard Deformable Parts Model and Regions with Convolutional Neural Network object detectors, our method shows improvements of object detection under varying levels of image noise. We present a comprehensive experimental evaluation and compare our method to other denoising techniques as well as to standard detectors re-trained on noisy images. Results are presented for the common Pascal Visual Object Classes benchmark for object detection and KAIST Multispectral Pedestrian Detection Benchmark with the real noise presence in night images.

Discriminative latent semantic feature learning for pedestrian detection

Features act as a key factor in pedestrian detection task. Most widely-used ones like HOG are manually designed and hard to be adaptive, thus now more attention has been paid to the features automatically learned on data. In this paper, a novel approach of learning discriminative features is proposed, addressing two main limitations of the methods in the literature. On one hand, unlike those methods of learning features on low-level pixels, we propose to learn features via a particular sparse coding algorithm enhanced on mid-level image representation, in order to obtain higher-level latent semantics and robustness; On the other hand, those methods usually utilize label information in model training such as deformable part model (DPM) with high computation cost. Instead, we propose to extend the learning process via a maximum margin criterion, in order to better encode discriminative information directly in features by optimizing them to be close to each other if from the same class and far from each other if from different classes. Furthermore, a boosted detection framework rather than the complex DPM is adopted to achieve both high accuracy and efficiency. The proposed approach achieves promising results on several standard pedestrian detection benchmarks.