Nighttime Traffic Scenes Research Articles

Nighttime traffic object detection via adaptively integrating event and frame domains

Intelligent perception is crucial in Intelligent Transportation Systems (ITS), with vision cameras as critical components. However, traditional RGB cameras exhibit a significant decline in performance when capturing nighttime traffic scenes, limiting their effectiveness in supporting ITS. In contrast, event cameras possess a high dynamic range (140 dB vs. 60 dB for traditional cameras), enabling them to overcome frame degradation in low-light conditions. Recently, multimodal learning paradigms have made substantial progress in various vision tasks, such as image-text retrieval. Motivated by this progress, we propose an adaptive selection and fusion detection method that leverages both event and RGB frame domains to optimize nighttime traffic object detection jointly. To address the challenge of unbalanced multimodal data fusion, we design a learnable adaptive selection and fusion module. This module performs feature ranking and fusion in the channel dimension, allowing efficient multimodal fusion. Additionally, we construct a novel multi-level feature pyramid network based on multimodal attention fusion. This network extracts potential features to enhance robustness in detecting nighttime traffic objects. Furthermore, we curate a dataset for nighttime traffic scenarios comprising RGB frames and corresponding event streams. Through experiments, we demonstrate that our proposed method outperforms current state-of-the-art techniques in event-based, frame-based, and event and frame fusion methods. This highlights the effectiveness of integrating the event and frame domains in enhancing nighttime traffic object detection.

EFASPP U-Net for semantic segmentation of night traffic scenes using fusion of visible and thermal images

The development of self-driving cars increases driving safety and accelerates urban transportation. These systems must have robust and real-time understanding of traffic conditions and surroundings, both at day and night. Many semantic image segmentation techniques have been proposed based on deep neural networks to partition the traffic scene images as a substantial step. However, the proposed algorithms and public datasets are mostly based on visible images during the daytime. Also, most of these algorithms are computationally intensive. However, little research has been done to date to address the application of the fusion of thermal and visible images and the high-performance low-volume deep convolutional networks. In this paper, a multispectral Encoder Fused Atrous Spatial Pyramid Pooling (EFASPP) U-Net deep network is proposed to merge the features of the visible and thermal images recorded at night traffic scenes. The proposed network is designed based on the structure of the U-Net, due to its high accuracy and speed of processing, as well as no need for large training datasets. The fusion of visible and thermal features in the encoders of EFASPP U-Net network is performed using standard and atrous convolution layers. Also, a new multispectral dataset is developed in this work for night-time traffic scenes due to the lack of sufficient public dataset in this field. The major contributions of this work include a low-volume high-performance multispectral semantic segmentation network for smart vehicles and a new dataset for this application. The experimental results show the high accuracy and speed of the proposed method.

Thermal Infrared Image Colorization for Nighttime Driving Scenes With Top-Down Guided Attention

Benefitting from insensitivity to light and high penetration of foggy environments, infrared cameras are widely used for sensing in nighttime traffic scenes. However, the low contrast and lack of chromaticity of thermal infrared (TIR) images hinder the human interpretation and portability of high-level computer vision algorithms. Colorization to translate a nighttime TIR image into a daytime color (NTIR2DC) image may be a promising way to facilitate nighttime scene perception. Despite recent impressive advances in image translation, semantic encoding entanglement and geometric distortion in the NTIR2DC task remain under-addressed. Hence, we propose a toP-down attEntion And gRadient aLignment based generative adversarial network, referred to as PearlGAN. A top-down guided attention module and an elaborate attentional loss are first designed to reduce the semantic encoding ambiguity during translation. Then, a structured gradient alignment loss is introduced to encourage edge consistency between the translated and input images. In addition, pixel-level annotation is carried out on a subset of FLIR and KAIST datasets to evaluate the semantic preservation performance of multiple translation methods. Furthermore, a new metric is devised to evaluate the geometric consistency in the translation process. Extensive experiments demonstrate the superiority of the proposed PearlGAN over other image translation methods for the NTIR2DC task. The source code and labeled segmentation masks will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FuyaLuo/PearlGAN/</uri> .

Effective Reflection Suppression Method for Vehicle Detection in Complex Nighttime Traffic Scenes

Abstract Headlight is the most explicit and stable image feature in nighttime scenes. This study proposes a headlight detection and pairing algorithm that adapts to numerous scenes to achieve accurate vehicle detection in the nighttime. This algorithm improved the conventional histogram equalization by using the difference before and after the equalization to suppress the ground reflection and noise. Then, headlight detection was completed based on this difference as a feature. In addition, the authors combined coordinate information, moving distance, symmetry, and stable time to implement headlight pairing, thus enabling vehicle detection in the nighttime. This study effectively overcame complex scenes such as high-speed movement, multi-headlight, and rains. Finally, the algorithm was verified by videos of highway scenes; the detection rate was as high as 96.67%. It can be implemented on the Raspberry Pi embedded platform, and its execution speed can reach 25 frames per second.

A Study on Training Data Selection for Object Detection in Nighttime Traffic Scenes

Grid mapping is widely used to represent the environment surrounding a car or a robot for autonomous navigation. This paper describes an algorithm for evidential occupancy grid (OG) mapping that fuses measurements from different sensors, based on the Dempster-Shafer theory, and is intended for scenes with stationary and moving (dynamic) objects. Conventional OGmapping algorithms tend to struggle in the presence of moving objects because they do not explicitly distinguish between moving and stationary objects. In contrast, evidential OG mapping allows for dynamic and ambiguous states (e.g. a LIDAR measurement: cannot differentiate between moving and stationary objects) that are more aligned with measurements made by sensors. In this paper, we present a framework for fusing measurements as they are received from disparate sensors (e.g. radar, camera and LIDAR) using evidential grid mapping. With this approach, we can form a live map of the environment, and also alleviate the problem of having to synchronize sensors in time. We also designed a new inverse sensor model for radar that allows us to extract more information from object level measurements, by incorporating knowledge of the sensor’s characteristics. We have implemented our algorithm in the OpenVX framework to enable seamless integration into embedded platforms. Test results show compelling performance especially in the presence of moving objects.

Validation methods for geometric camera calibration

We have developed a semi-automatic annotation tool – “CVL Annotator” – for bounding box ground truth generation in videos. Our research is particularly motivated by the need for reference annotations of challenging nighttime traffic scenes with highly dynamic lighting conditions due to reflections, headlights and halos from oncoming traffic. Our tool incorporates a suite of different state-of-the-art tracking algorithms in order to minimize the amount of human input necessary to generate high-quality ground truth data. We focus our user interface on the premise of minimizing user interaction and visualizing all information relevant to the user at a glance. We perform a preliminary user study to measure the amount of time and clicks necessary to produce ground truth annotations of video traffic scenes and evaluate the accuracy of the final annotation results.

BITPNet: Unsupervised Bio-Inspired Two-Path Network for Nighttime Traffic Image Enhancement

Due to the low luminance in nighttime traffic images, image features are not salient, making tasks in intelligent transportation systems such as nighttime vehicle detection challenging. Recently, convolutional neural network based methods have been developed for low-light image enhancement. Most of these methods are supervised and require high-light reference images at the same scenes. However, reference images are difficult to be obtained in nighttime traffic scenes because vehicles always move. In the early visual system the input signals are processed by two parallel visual paths in the retina: one path has small receptive fields (RFs) to process the high frequency information and another path has large RFs to deal with the low frequency information. Inspired by this, we design a novel bio-inspired two-path convolutional neural network (BITPNet) for nighttime traffic image enhancement. The high-frequency path with small convolution kernel size is designed to suppress noises and preserve the details. The low-frequency path with large convolution kernel size is used to enhance the luminance of images. Each path includes an encoder-to-decoder network followed by a new multi-level attention module to combine features of levels with different RFs. The outputs of the two paths are summed by learnt weights for generating the final image enhancement result. Several no-reference image quality metrics are utilized to design a new loss function, resulting in an unsupervised approach. The proposed BITPNet is trained on one nighttime traffic image dataset and evaluated on another nighttime dataset. Experimental results demonstrate that the proposed BITPNet outperforms several state-of-the-art low-light image enhancement methods in terms of visual quality and three no-reference image quality metrics. In addition, when the proposed BITPNet is used as pre-processing for the nighttime multi-class vehicle detection task, it achieves higher detection rate (97.18%) than other methods.

An Adaptive Method for Image Dynamic Range Adjustment

In this paper, we relate the operation of image dynamic range adjustment to the following two tasks: 1) for a high dynamic range (HDR) image, its dynamic range will be mapped to the available dynamic range of display devices and 2) for a low dynamic range (LDR) image, its distribution of intensity will be extended to adequately utilize the full dynamic range of display devices. The common goal of both tasks is to preserve or even enhance the details and improve the visibility of scenes when being matched to the available dynamic range of a display device. In this paper, we propose an efficient method for image dynamic range adjustment with three adaptive steps. First, according to the histogram of the luminance map separated from the given RGB image, two suitable Gamma functions are adaptively selected to separately adjust the luminance of the dark and bright components. Second, an adaptive fusion strategy is proposed to combine the two adjusted luminance maps in order to balance the enhancement of the details in different regions. Third, an adaptive luminance-dependent color restoration method is designed to combine the fused luminance map with the original color components to obtain more consistent color saturation between the images before and after dynamic range adjustment. Extensive experiments show that the proposed method can efficiently compress the dynamic range of HDR scenes with good contrast, clear details, and high structural fidelity of the original image appearance. In addition, the proposed method can also obtain promising performance when being used to enhance LDR nighttime images and greatly facilitates the object (car) detection in nighttime traffic scenes.

Nighttime Motion Vehicle Detection Based on Multiple Instance Learning

In this paper, we propose an effective approach for detecting moving vehicles in nighttime traffic scenes. We use Multiple Instance Learning method to automatically detect vehicle from video sequences by constructing the Multiple Instance Learning model at nighttime. At first, we extract SIFT feature using SIFT feature extraction algorithm, which is used to characterize moving vehicles at nighttime. Then Multiple Instance Learning model is used for the on-road detection of vehicles at nighttime, in order to improve the detection accuracy, the class label information was used for the learning of the Multiple Instance Learning model. Final experiments were performed and evaluate the proposed method at nighttime under urban traffic condition, the experiment results show that the average detection accuracy is over 96.2%, which validates that the proposed vehicle detection approach is feasible and effective for the on-road detection of vehicles at nighttime and identification in various nighttime environments.

Tracking And Counting The Vehicles In Night Scenes

main objective of traffic surveillance system is to reduce the risk caused by accident. Many papers published were concerned only about the vehicle detection during daytime. But we proposed a method to detect the vehicle during night time. The main objective of our paper is to count the vehicles in night time traffic scenes. It consists of three phases, they are vehicle's headlight segmentation, tracking and pairing. We know that in night time, only visible thing will be the headlight of the vehicle therefore we are counting the vehicle based on the headlight information. Initially the headlights of a vehicle is segmented based on the analysis of headlight size, location and area. Then tracke d via a tracking procedure designed to detect vehicle and then headlights are paired using the connected component labeling. Based on the pairing result we can count the vehicles. Here we are using fuzzy hybrid information inference mechanism for error co mpensation. Here by using fuzzy logic we can generate the rules based on the size, color and position. In particular we are generating the rules based on the distance between the headlights of a vehicle.Thereby we can improve the accuracy while counting t he vehicle.

A Region Tracking-Based Vehicle Detection Algorithm in Nighttime Traffic Scenes

The preceding vehicles detection technique in nighttime traffic scenes is an important part of the advanced driver assistance system (ADAS). This paper proposes a region tracking-based vehicle detection algorithm via the image processing technique. First, the brightness of the taillights during nighttime is used as the typical feature, and we use the existing global detection algorithm to detect and pair the taillights. When the vehicle is detected, a time series analysis model is introduced to predict vehicle positions and the possible region (PR) of the vehicle in the next frame. Then, the vehicle is only detected in the PR. This could reduce the detection time and avoid the false pairing between the bright spots in the PR and the bright spots out of the PR. Additionally, we present a thresholds updating method to make the thresholds adaptive. Finally, experimental studies are provided to demonstrate the application and substantiate the superiority of the proposed algorithm. The results show that the proposed algorithm can simultaneously reduce both the false negative detection rate and the false positive detection rate.

A Real-Time Vision System for Nighttime Vehicle Detection and Traffic Surveillance

This paper presents an effective traffic surveillance system for detecting and tracking moving vehicles in nighttime traffic scenes. The proposed method identifies vehicles by detecting and locating vehicle headlights and taillights using image segmentation and pattern analysis techniques. First, a fast bright-object segmentation process based on automatic multilevel histogram thresholding is applied to effectively extract bright objects of interest. This automatic multilevel thresholding approach provides a robust and adaptable detection system that operates well under various nighttime illumination conditions. The extracted bright objects are then processed by a spatial clustering and tracking procedure that locates and analyzes the spatial and temporal features of vehicle light patterns, and identifies and classifies moving cars and motorbikes in traffic scenes. The proposed real-time vision system has also been implemented and evaluated on a TI DM642 DSP-based embedded platform. The system is set up on elevated platforms to perform traffic surveillance on real highways and urban roads. Experimental results demonstrate that the proposed traffic surveillance approach is feasible and effective for vehicle detection and identification in various nighttime environments.

Traffic Sign Conspicuity: The Effects of Clutter, Luminance, and Age

The effects of visual clutter, luminance, and aging on traffic sign conspicuity were examined in two experiments. Experiment 1a evaluated clutter using subjective judgments from 28 older and younger observers. Participants categorized daytime and nighttime traffic scenes into high or low clutter. Using these categorized scenes, in Experiment 1b a new sample of 28 younger and older participants visually searched for traffic signs. Reaction time and eye movement data were analyzed. Results suggested that aging and clutter affect con-spicuity. Moreover, older adults had poorer performance on target-absent trials. No luminance effect or any age by clutter interactions were obtained. Expertise gained through many years of driving may dampen clutter effects experienced by older adults. The nonlinear relationship between fixation frequency and fixation duration is discussed in relation to similar findings in experimental visual search. Results suggest that age differences found in processing simple visual stimuli may not generalize to more naturalistic scenes.

Visibility of Cyclists at Night: Laboratory Evaluation of Three Rear Warning Devices

The relative effectiveness of three bicycle rear warning devices (a red light, a red reflector and amber pedal reflectors) was evaluated using slides of nighttime traffic scenes. Situational variables of target lane position and distance and visual complexity of the scene were also manipulated. Performance was assessed in terms of detection accuracy and response latency. For both measures the red light was shown to be most effective in relatively difficult detection situations (distant target, noisy scene). At closer distances the three devices produced approximately equal performance rates that were all significantly higher than for a cycle that carried no rear warning equipment.