Nighttime Pedestrian Detection Research Articles

Cross-modality feature fusion for night pedestrian detection

Night pedestrian detection with visible image only suffers from the dilemma of high miss rate due to poor illumination conditions. Cross-modality fusion can ameliorate this dilemma by providing complementary information to each other through infrared and visible images. In this paper, we propose a cross-modal fusion framework based on YOLOv5, which is aimed at addressing the challenges of night pedestrian detection under low-light conditions. The framework employs a dual-stream architecture that processes visible images and infrared images separately. Through the Cross-Modal Feature Rectification Module (CMFRM), visible and infrared features are finely tuned on a granular level, leveraging their spatial correlations to focus on complementary information and substantially reduce uncertainty and noise from different modalities. Additionally, we have introduced a two-stage Feature Fusion Module (FFM), with the first stage introducing a cross-attention mechanism for cross-modal global reasoning, and the second stage using a mixed channel embedding to produce enhanced feature outputs. Moreover, our method involves multi-dimensional interaction, not only correcting feature maps in terms of channel and spatial dimensions but also applying cross-attention at the sequence processing level, which is critical for the effective generalization of cross-modal feature combinations. In summary, our research significantly enhances the accuracy and robustness of nighttime pedestrian detection, offering new perspectives and technical pathways for visual information processing in low-light environments.

NIRPed: A Novel Benchmark for Nighttime Pedestrian and Its Distance Joint Detection

In nighttime driving scenes, due to insufficient and uneven lighting, and the scarcity of high-quality datasets, the miss rate of nighttime pedestrian detection (PD) is much higher than that of daytime. Vision-based distance detection (DD) has the advantages of low cost and good interpretability, but the existing methods have low precision, poor robustness, and the DD is mostly performed independently of PD. A narrowband near-infrared (NIR) camera and an NIR lamp to relieve the impact of noisy visible light (VIS) and improve imaging quality have been selected; a LiDAR to obtain distance information has been embedded in the system; a nighttime driving scene pedestrian and its distance joint detection dataset NIRPed has been built. The NIRPed includes 146k pedestrian annotations, which is 3 times that of NightOwls, the largest VIS nighttime pedestrian dataset. Based on Faster-RCNN, a joint PD and DD method using monocular imaging has been proposed. The propo-sed method realizes the joint detection of pedestrian and its distance on the NIRPed dataset with the PD log-average miss rate and DD mean absolute error rate of 6.5% and 5.5%, respectively. For comparison, the joint detection method was implemented on other large-scale VIS pedestrian datasets as well. Moreover, compared with the existing vision-based DD methods, the proposed method is less affected by pedestrian distance and height, exhibits higher accuracy and robustness, and satisfied the ISO requirements for intelligent transportation systems (absolute error rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&lt;$</tex-math> </inline-formula> 15%).

Nighttime pedestrian detection based on Fore-Background contrast learning

The significance of background information is frequently overlooked in contemporary research concerning channel attention mechanisms. This study addresses the issue of suboptimal single-spectral nighttime pedestrian detection performance under low-light conditions by incorporating background information into the channel attention mechanism. Despite numerous studies focusing on the development of efficient channel attention mechanisms, the relevance of background information has been largely disregarded. By adopting a contrast learning approach, we reexamine channel attention with regard to pedestrian objects and background information for nighttime pedestrian detection, resulting in the proposed Fore-Background Contrast Attention (FBCA). FBCA possesses two primary attributes: (1) channel descriptors form remote dependencies with global spatial feature information; (2) the integration of background information enhances the distinction between channels concentrating on low-light pedestrian features and those focusing on background information. Consequently, the acquired channel descriptors exhibit a higher semantic level and spatial accuracy. Experimental outcomes demonstrate that FBCA significantly outperforms existing methods in single-spectral nighttime pedestrian detection, achieving state-of-the-art results on the NightOwls and TJU-DHD-pedestrian datasets. Furthermore, this methodology also yields performance improvements for the multispectral LLVIP dataset. These findings indicate that integrating background information into the channel attention mechanism effectively mitigates detector performance degradation caused by illumination factors in nighttime scenarios.

Pedestrian Detection by Exemplar-Guided Contrastive Learning.

Typical methods for pedestrian detection focus on either tackling mutual occlusions between crowded pedestrians, or dealing with the various scales of pedestrians. Detecting pedestrians with substantial appearance diversities such as different pedestrian silhouettes, different viewpoints or different dressing, remains a crucial challenge. Instead of learning each of these diverse pedestrian appearance features individually as most existing methods do, we propose to perform contrastive learning to guide the feature learning in such a way that the semantic distance between pedestrians with different appearances in the learned feature space is minimized to eliminate the appearance diversities, whilst the distance between pedestrians and background is maximized. To facilitate the efficiency and effectiveness of contrastive learning, we construct an exemplar dictionary with representative pedestrian appearances as prior knowledge to construct effective contrastive training pairs and thus guide contrastive learning. Besides, the constructed exemplar dictionary is further leveraged to evaluate the quality of pedestrian proposals during inference by measuring the semantic distance between the proposal and the exemplar dictionary. Extensive experiments on both daytime and nighttime pedestrian detection validate the effectiveness of the proposed method.

Nighttime Pedestrian Detection Based on a Fusion of Visual Information and Millimeter-Wave Radar

Nighttime Pedestrian Detection Based on a Fusion of Visual Information and Millimeter-Wave Radar

An Improved YOLOX Model and Domain Transfer Strategy for Nighttime Pedestrian and Vehicle Detection

Aimed at the vehicle/pedestrian visual sensing task under low-light conditions and the problems of small, dense objects and line-of-sight occlusion, a nighttime vehicle/pedestrian detection method was proposed. First, a vehicle/pedestrian detection algorithm was designed based on You Only Look Once X (YOLOX). The model structure was re-parameterized and lightened, and a coordinate-based attention mechanism was introduced into the backbone network to enhance the feature extraction efficiency of vehicle/pedestrian targets. A feature-scale fusion detection branch was added to the feature pyramid, while a loss function was designed, which combines Complete Intersection Over Union (CIoU) for target localization and Varifocal Loss for confidence prediction to improve the feature extraction ability for small, dense, and low-illumination targets. In addition, in order to further improve the detection accuracy of the algorithm under low-light conditions, a training strategy based on data domain transfer was proposed, which fuses the larger-scale daylight dataset with the smaller-scale nighttime dataset after low-illumination degrading. After low-light enhancement, training and testing were performed accordingly. The experimental results show that, compared with the original YOLOX model, the improved algorithm trained by the proposed data domain transfer strategy achieved better performance, and the mean Average Precision (mAP) increased by 5.9% to 82.4%. This research provided effective technical support for autonomous driving safety at night.

Nighttime Pedestrian and Vehicle Detection Based on a Fast Saliency and Multifeature Fusion Algorithm for Infrared Images

In recent years, pedestrian and vehicle detection at night has become an important subject of computer vision applications. Because the environment light is weak at night, the common traditional camera-input algorithms often performs poorly. Pedestrians and vehicles in infrared images are usually brighter than the surrounding environment and have salient features. In this paper, a fast saliency map of pedestrian and vehicle targets in infrared images at night is used to achieve rapid acquisition of areas of interest for pedestrians and vehicles. Based on this, we propose a method to refine and separate the target area to obtain accurate pedestrian and vehicle candidate bounding boxes. Specifically, this paper proposed a multi-feature fusion algorithm of pedestrian and vehicle feature extraction combined with support vector machine (SVM) to determine whether the extracted target area really includes pedestrians and vehicles. Our experimental results show that the proposed method can achieve the expected effect of the classification and detection for pedestrians and vehicles at night, and can meet the real-time requirements of actual road scenarios.

Improving Single-Stage Object Detectors for Nighttime Pedestrian Detection

Improving the reliability of nighttime pedestrian detection is a crucial challenge towards the design of robust autonomous systems. Not surprisingly, most pedestrian fatalities occur in low-illumination settings, thus emphasizing the need for new algorithmic advances. This work presents a novel pedestrian detection approach that makes a number of crucial modifications to the state-of-the-art YOLOV5-PANet architecture, in order to improve the reliability of features extracted from nighttime images. More specifically, the proposed architecture systematically incorporates powerful shuffle attention mechanisms and a transformer module to improve the feature learning pipeline. Instead of advocating the use of other sensing modalities that are better suited for nighttime detection, our approach relies only on conventional RGB cameras and is hence broadly applicable. Our empirical studies with nighttime pedestrian detection benchmarks show that with only minimal increase in model complexity, our approach provides significant improvements in detection efficacy over existing solutions. Finally, we explore the impact of post-hoc network pruning on the speed-accuracy trade-off of our approach and demonstrate that it is well suited for reduced memory/compute requirements.

MAF-YOLO: Multi-modal attention fusion based YOLO for pedestrian detection

Achieving the rapid and precise detection of pedestrians in natural environments is crucial for many applications of artificial intelligence systems. However, accurate pedestrian detection at nighttime is challenging due to the low-luminous density and low-resolution of infrared images, and the detection speed is required. In this paper, a real-time pedestrian detection method using a novel multi-modal attention fusion YOLO (MAF-YOLO) was proposed. Firstly, a multi-modal feature extraction module based on the compressed Darknet53 framework was built to adapt the nighttime pedestrian detection and ensure efficiency. The features were extracted from both modalities and then fused by a modal weighted fusion module. Secondly, we defined a loss function and regenerated the size of the anchor box based on the K-means clustering algorithm to improve the detection speed and the robustness of small objects. Finally, a dual attention module was applied to acquire more semantic features from small objects with low-resolution. The experimental results on KAIST and OSU Color-Thermal datasets corroborated the effectiveness of the proposed MAF-YOLO. The proposed methods could also be utilized to facilitate the performance of other pedestrian detection algorithms.

Multi-task faster R-CNN for nighttime pedestrian detection and distance estimation

Distance estimation and pedestrian detection are critical for safe driving operation decision-making and autonomous vehicle intelligent control strategies. This paper proposes a novel multi-task Faster R-CNN detector which simultaneously realizes distance estimation and pedestrian detection using an improved ResNet-50 architecture. Images were acquired using a near-infrared camera with two near-infrared fill-lights devices during real road nighttime scenarios. Ground truth pedestrian distances used for training were obtained using LIDAR. The data used to optimize the multi-task Faster R-CNN detector were approximately 20 k high-quality near-infrared images with marked pedestrians and tagged distance values. The proposed algorithm including the distance estimation runs at a speed exceeding 7 fps. Pedestrian detection accuracy reached nearly 80% with a total average absolute distance estimation error rate of less than 5%.

Nighttime Pedestrian Detection Based on Thermal Imaging and Convolutional Neural Networks

Nighttime Pedestrian Detection Based on Thermal Imaging and Convolutional Neural Networks

The algorithm of nighttime pedestrian detection in intelligent surveillance for renewable energy power stations

Intelligent surveillance is an important management method for the construction and operation of power stations such as wind power and solar power. The identification and detection of equipment, facilities, personnel, and behaviors of personnel are the key technology for the ubiquitous electricity The Internet of Things. This paper proposes a video solution based on support vector machine and histogram of oriented gradient (HOG) methods for pedestrian safety problems that are common in night driving. First, a series of image preprocessing methods are used to optimize night images and detect lane lines. Second, an image is divided into intelligent regions to be adapted to different road environments. Finally, the HOG and support vector machine methods are used to optimize the pedestrian image on a Linux system, which reduces the number of false alarms in pedestrian detection and the workload of the pedestrian detection algorithm. The test results show that the system can successfully detect pedestrians at night. With image preprocessing optimization, the correct rate of nighttime pedestrian detection can be significantly improved, and the correct rate of detection can reach 92.4%. After the division area is optimized, the number of false alarms decreases significantly, and the average frame rate of the optimized video reaches 28 frames per second.

Near infrared nighttime road pedestrians recognition based on convolutional neural network

Pedestrian recognition is the core technology of pedestrian detection in pedestrian protection systems. This paper compares and analyzes, visible and infrared images obtained via visible-spectrum, near-infrared, short-wave infrared, and long-wave infrared cameras. The results show that near-infrared camera was the best for nighttime pedestrian detection when device cost and pedestrian imaging quality were considered. This paper reports on the first time use of a self-learning softmax with a 9-layer Convolutional Neural Network (CNN) model to identify near-infrared nighttime pedestrians. 267,000 samples obtained from the near-infrared images were employed to optimize the CNN recognition model. Collected near-infrared nighttime samples had 3 categories (background, pedestrian, and cyclist or motorcyclist) and will be made publicly available for researchers use. Testing results indicated that the optimized CNN model using self-learning softmax had a competitive accuracy and potential in real-time pedestrian recognition.

Pedestrian detection based on faster R-CNN in nighttime by fusing deep convolutional features of successive images

Existing studies using visible-light cameras have mainly focused on methods of pedestrian detection during daytime. However, these studies found it difficult to detect pedestrians during nighttime with low external light. The NIR illuminator has limitations in terms of illumination angle and distance, and the illuminator's power needs to be adjusted depending on whether an object is near or distant. Although, thermal cameras were used for nighttime pedestrian detection, thermal cameras are currently expensive and thus difficult to install in many places. To solve these problems, attempts have been made to use visible-light cameras for nighttime pedestrian detection. However, most of these attempts considered an indoor environment where the distance to the object was short. This study proposes a method of pedestrian detection at nighttime using a visible-light camera and faster region-based convolutional neural network (R-CNN). In addition, as pedestrians cannot be reliably detected from a single nighttime image, we combined deep convolutional features in successive frames.Using Korea advanced institute of science and technology (KAIST) open database, we conducted experiments and observed that the proposed method performed better than the baseline methods at all times (day and night). In addition, through the experiments with national ICT Australia Ltd. (NICTA) open database, we confirm that the proposed method is effective for pedestrian detection at all times. Finally, we present theoretical grounds for the proposed fusion.

Nighttime Foreground Pedestrian Detection Based on Three-Dimensional Voxel Surface Model.

Pedestrian detection is among the most frequently-used preprocessing tasks in many surveillance application fields, from low-level people counting to high-level scene understanding. Even though many approaches perform well in the daytime with sufficient illumination, pedestrian detection at night is still a critical and challenging problem for video surveillance systems. To respond to this need, in this paper, we provide an affordable solution with a near-infrared stereo network camera, as well as a novel three-dimensional foreground pedestrian detection model. Specifically, instead of using an expensive thermal camera, we build a near-infrared stereo vision system with two calibrated network cameras and near-infrared lamps. The core of the system is a novel voxel surface model, which is able to estimate the dynamic changes of three-dimensional geometric information of the surveillance scene and to segment and locate foreground pedestrians in real time. A free update policy for unknown points is designed for model updating, and the extracted shadow of the pedestrian is adopted to remove foreground false alarms. To evaluate the performance of the proposed model, the system is deployed in several nighttime surveillance scenes. Experimental results demonstrate that our method is capable of nighttime pedestrian segmentation and detection in real time under heavy occlusion. In addition, the qualitative and quantitative comparison results show that our work outperforms classical background subtraction approaches and a recent RGB-D method, as well as achieving comparable performance with the state-of-the-art deep learning pedestrian detection method even with a much lower hardware cost.

Efficient Pedestrian Detection at Nighttime Using a Thermal Camera

Most of the commercial nighttime pedestrian detection (PD) methods reported previously utilized the histogram of oriented gradient (HOG) or the local binary pattern (LBP) as the feature and the support vector machine (SVM) as the classifier using thermal camera images. In this paper, we propose a new feature called the thermal-position-intensity-histogram of oriented gradient (TPIHOG or THOG) and developed a new combination of the THOG and the additive kernel SVM (AKSVM) for efficient nighttime pedestrian detection. The proposed THOG includes detailed information on gradient location; therefore, it has more distinctive power than the HOG. The AKSVM performs better than the linear SVM in terms of detection performance, while it is much faster than other kernel SVMs. The combined THOG-AKSVM showed effective nighttime PD performance with fast computational time. The proposed method was experimentally tested with the KAIST pedestrian dataset and showed better performance compared with other conventional methods.

Saliency-based Pedestrian Detection in Far Infrared Images

Night time pedestrian detection is more and more important in advanced driver assistant systems (ADAS). Traditional pedestrian detection algorithms in far infrared (FIR) images lack accuracy and have long processing times. Focusing on this issue, in this paper, a visual saliency-based pedestrian detection algorithm is proposed. First, areas that contain suspected pedestrians are detected using a fusion saliency-based method. Then, the sub-image of the suspected pedestrian is used as an input to a histogram of local intensity difference feature and cross kernel-based support vector machine classifier to make a final determination. Experiments performed using a real FIR road image data set demonstrated that the proposed fusion saliency-based region of interest (ROI) detection method has the largest pedestrian inclusion rate and the smallest ROI proportion compared with three other methods. Besides, compared with existing state-of-the-art pedestrian detection algorithms, the proposed method demonstrates a much higher pedestrian detection rate with a comparably short processing time.

基于加速区域卷积神经网络的夜间行人检测研究

基于加速区域卷积神经网络的夜间行人检测研究

Night-Time Pedestrian Detection Based on Temperature and HOGI Feature in Infra-Red Images

Night-Time Pedestrian Detection Based on Temperature and HOGI Feature in Infra-Red Images

Near-Infrared-Based Nighttime Pedestrian Detection Using Grouped Part Models

Pedestrian detection is an important issue in the field of intelligent transportation systems. As a pedestrian is not an apparent object at nighttime, it brings about critical difficulties in effectively detecting a pedestrian for a driving assistant vision system. While using an infrared projector to enhance the illumination contrast, objects in a nighttime environment might reflect the infrared projected by the emitted spotlight. In some cases, however, the clothes on a pedestrian might absorb most of the infrared, thus causing the pedestrian to be partially invisible. To deal with this problem, a nighttime part-based pedestrian detection method is proposed. It divides a pedestrian into parts for a moving vehicle with a camera and a near-infrared lighting projector. Due to a high computation load, selecting effective parts becomes imperative. By analyzing the spatial relationship between every pair of parts, the confidence of the detected parts can be enhanced even when some parts are occluded. At the last stage of this system, the pedestrian detection result is refined by a block-based segmentation method. The system is verified by experiments, and the appealing results are demonstrated.