Urban Traffic Scenes Research Articles

A benchmark dataset and semantics-guided detection network for spatial–temporal human actions in urban driving scenes

In real urban driving scenes, human actions are very complex and have the characteristic of multiple concurrent actions. It has a great significance to detect human actions in urban traffic scenes for auxiliary or autonomous driving systems. In this view, we introduce the TITAN-Human Action dataset for the task of multi-person spatial–temporal action detection in urban driving scenes. TITAN-Human Action provides the fine-grained action labels and location coordinates for 17,574 persons in the processed frames from the TITAN dataset. Furthermore, we propose a semantics-guided detection network (SGDNet) based on a semantic inference module (SIM) for spatial–temporal human action detection in urban driving scenes. SIM encodes the category labels into sentence vectors at the semantic level with prompting and embedding, utilizes graphs to represent the directed co-occurrence relations between categories, and adopts the graph convolutional network for semantic inference. SGDNet exploits the inference results of SIM to guide the visual branch in better performing human action detection, thereby achieving the integration of visual and linguistic information. We conducted experiments to evaluate SGDNet and several baseline methods on the TITAN-Human Action dataset, and reveal the generalizability of SIM in spatial–temporal human action detection. The source code and annotation files will be available at https://github.com/yyhbswyn/SGDNet.

Spatio-temporal deep learning framework for pedestrian intention prediction in urban traffic scenes

Pedestrian intent prediction is an essential task for ensuring the safety of pedestrians and vehicles on the road. This task involves predicting whether a pedestrian intends to cross a road or not based on their behavior and surrounding environment. Previous studies have explored feature-based machine learning and vision-based deep learning models for this task but these methods have limitations in capturing the global spatio-temporal context and fusing different features of data effectively. To address these issues, we propose a novel hybrid framework HSTGCN for pedestrian intent prediction that combines spatio-temporal graph convolutional neural networks (STGCN) and long short-term memory (LSTM) networks. The proposed framework utilizes the strengths of both models by fusing multiple features, including skeleton pose, trajectory, height, orientation, and ego-vehicle speed, to predict their intentions accurately. The framework’s performance have been evaluated on the JAAD benchmark dataset and the results show that it outperforms the state-of-the-art methods. The proposed framework has potential applications in developing intelligent transportation systems, autonomous vehicles, and pedestrian safety technologies. The utilization of multiple features can significantly improve the performance of the pedestrian intent prediction task.

Proposing an Efficient Deep Learning Algorithm Based on Segment Anything Model for Detection and Tracking of Vehicles through Uncalibrated Urban Traffic Surveillance Cameras

In this study, we present a novel approach leveraging the segment anything model (SAM) for the efficient detection and tracking of vehicles in urban traffic surveillance systems by utilizing uncalibrated low-resolution highway cameras. This research addresses the critical need for accurate vehicle monitoring in intelligent transportation systems (ITS) and smart city infrastructure. Traditional methods often struggle with the variability and complexity of urban environments, leading to suboptimal performance. Our approach harnesses the power of SAM, an advanced deep learning-based image segmentation algorithm, to significantly enhance the detection accuracy and tracking robustness. Through extensive testing and evaluation on two datasets of 511 highway cameras from Quebec, Canada and NVIDIA AI City Challenge Track 1, our algorithm achieved exceptional performance metrics including a precision of 89.68%, a recall of 97.87%, and an F1-score of 93.60%. These results represent a substantial improvement over existing state-of-the-art methods such as the YOLO version 8 algorithm, single shot detector (SSD), region-based convolutional neural network (RCNN). This advancement not only highlights the potential of SAM in real-time vehicle detection and tracking applications, but also underscores its capability to handle the diverse and dynamic conditions of urban traffic scenes. The implementation of this technology can lead to improved traffic management, reduced congestion, and enhanced urban mobility, making it a valuable tool for modern smart cities. The outcomes of this research pave the way for future advancements in remote sensing and photogrammetry, particularly in the realm of urban traffic surveillance and management.

Infrared multi‐target detection and tracking in dense urban traffic scenes

AbstractInfrared object detection and tracking in dense urban traffic remain a challenge due to factors such as low contrast, small intra‐class differences, and frequent false positives and negatives. To overcome these, the authors introduce YOLO‐IR, an algorithm based on the enhanced YOLOv8s, and YOLO‐DeepOC‐IR, a comprehensive infrared multi‐object tracking method for urban traffic, integrating both detection and tracking. During preprocessing, three infrared image enhancement techniques, local contrast multi‐scale enhancement, non‐local means, and contrast limited adaptive histogram equalization, are applied for better reliability in dense scenes. To further improve the performance, the original YOLOv8s backbone is replaced with MobileVITv3 to enhance detection accuracy and robustness. This infrared feature extraction module, incorporated into the detector, combines canny edge detection, Gabor filtering, and open operation layers, significantly boosting object detection in infrared imagery. The tracker's feature processing capabilities are improved using the learned arrangements of three patch codes descriptor and locality‐sensitive hashing for feature extraction and matching. Experimental results on FLIR ADAS v2 and InfiRay datasets indicate superior performance of this method, achieving 78.6% mAP and 151.1 FPS in detection, and up to 80.8% moving object tracking accuracy, 78.6% identification F1 score, and 62.1% higher order tracking accuracy in multi‐object tracking.

A new CNN-based semantic object segmentation for autonomous vehicles in urban traffic scenes

Semantic segmentation is the most important stage of making sense of the visual traffic scene for autonomous driving. In recent years, convolutional neural networks (CNN)-based methods for semantic segmentation of urban traffic scenes are among the trending studies. However, the methods developed in the studies carried out so far are insufficient in terms of accuracy performance criteria. In this study, a new CNN-based semantic segmentation method with higher accuracy performance is proposed. A new module, the Attentional Atrous Feature Pooling (AAFP) Module, has been developed for the proposed method. This module is located between the encoder and decoder in the general network structure and aims to obtain multi-scale information and add attentional features to large and small objects. As a result of experimental tests with the CamVid data set, an accuracy value of approximately 2% higher was achieved with a mIoU value of 70.59% compared to other state-of-art methods. Therefore, the proposed method can semantically segment objects in the urban traffic scene better than other methods.

Scene-Adaptive 3D Semantic Segmentation Based on Multi-Level Boundary-Semantic-Enhancement for Intelligent Vehicles

3D semantic segmentation is a key technology of scene understanding in the self-driving field, which remains challenging problems. Recent 3D segmentation methods have achieved competitive results in indoor or typical urban traffic scenes. However, in complex and changeable scenarios where structured features are sparse and irregular, few of these methods could achieve well segmentation results, especially causing blurry and inaccurate boundary distinctions between inter-class objects, drivable areas, and backgrounds. In order to fully harvest boundary information and accurately distinguish the category of points on road and object boundaries in real-time, we present an efficient multi-level boundary-semantic-enhanced model for LiDAR semantic segmentation, which comprehensively discover boundary features in three aspects: first, boundary channels are extracted directly from LiDAR range images as the inputs of boundary-branch; second, the boundary attention module is designed to deeply fuse boundary information into the main segmentation branch; third, a modified discriminator is utilized to raise the perception of boundary information by minimizing the gap between the predicted and true boundaries. Besides, we add a semantic-enhanced module using the similar discriminator to optimize semantic segmentation results in the output end. Quantitative and qualitative evaluations are performed on both structured and unstructured real-world datasets including urban dataset SemanticKITTI, off-road dataset Rellis3D and our unstructured test set. The experimental results validate the effectiveness of the proposed methodology in improving efficiency, accuracy and scene-adaptivity.

A distributed multi-vehicle pursuit scheme: generative multi-adversarial reinforcement learning

Multi-vehicle pursuit (MVP) is one of the most challenging problems for intelligent traffic management systems due to multi-source heterogeneous data and its mission nature. While many reinforcement learning (RL) algorithms have shown promising abilities for MVP in structured grid-pattern roads, their lack of dynamic and effective traffic awareness limits pursuing efficiency. The sparse reward of pursuing tasks still hinders the optimization of these RL algorithms. Therefore, this paper proposes a distributed generative multi-adversarial RL for MVP (DGMARL-MVP) in urban traffic scenes. In DGMARL-MVP, a generative multi-adversarial network is designed to improve the Bellman equation by generating the potential dense reward, thereby properly guiding strategy optimization of distributed multi-agent RL. Moreover, a graph neural network-based intersecting cognition is proposed to extract integrated features of traffic situations and relationships among agents from multi-source heterogeneous data. These integrated and comprehensive traffic features are used to assist RL decision-making and improve pursuing efficiency. Extensive experimental results show that the DGMARL-MVP can reduce the pursuit time by 5.47% compared with proximal policy optimization and improve the pursuing average success rate up to 85.67%. Codes are open-sourced in Github.

Behavior-Aware Pedestrian Trajectory Prediction in Ego-Centric Camera Views with Spatio-Temporal Ego-Motion Estimation

With the ongoing development of automated driving systems, the crucial task of predicting pedestrian behavior is attracting growing attention. The prediction of future pedestrian trajectories from the ego-vehicle camera perspective is particularly challenging due to the dynamically changing scene. Therefore, we present Behavior-Aware Pedestrian Trajectory Prediction (BA-PTP), a novel approach to pedestrian trajectory prediction for ego-centric camera views. It incorporates behavioral features extracted from real-world traffic scene observations such as the body and head orientation of pedestrians, as well as their pose, in addition to positional information from body and head bounding boxes. For each input modality, we employed independent encoding streams that are combined through a modality attention mechanism. To account for the ego-motion of the camera in an ego-centric view, we introduced Spatio-Temporal Ego-Motion Module (STEMM), a novel approach to ego-motion prediction. Compared to the related works, it utilizes spatial goal points of the ego-vehicle that are sampled from its intended route. We experimentally validated the effectiveness of our approach using two datasets for pedestrian behavior prediction in urban traffic scenes. Based on ablation studies, we show the advantages of incorporating different behavioral features for pedestrian trajectory prediction in the image plane. Moreover, we demonstrate the benefit of integrating STEMM into our pedestrian trajectory prediction method, BA-PTP. BA-PTP achieves state-of-the-art performance on the PIE dataset, outperforming prior work by 7% in MSE-1.5 s and CMSE as well as 9% in CFMSE.

Fast object detector with center localization confidence based on FCOS for environment perception in urban traffic scene

Environment perception on urban roads is one of the most important tasks in Automatic Driving System, because object detection is necessary in identifying and localizing cars and pedestrians. However, the accuracy and inference speed are still far away from satisfying, and the problem of missed detection occurs frequently. To address the problems, this paper first uniformly scales all dimensions of depth, width, and resolution in feature extraction module, and then fuses the semantic information in bidirectional cross-scale connections, which extract more significant features and detect objects of various scales. To decrease the undetected rate of small objects, this paper further introduces an IoU-based center localization confidence, which predicts centerness and the distance between the center of predicted box and the center of the ground-truth box, so that the predicted point approaches the center of an object. The experimental results on Cityscapes show that the proposed detector brings an improvement of 6.5% Bbox mAP and reaches an inference speed of 36.6 FPS. Compared with the other widely-used detectors such as Cascade R-CNN, FCOS, and YOLOX, it has obvious advantages in both precision and real-time performance. With additional ticks, the detection precision can be further improved by 4.4%. The proposed detector can accurately and quickly identify and localize objects, which is beneficial for the safety of self-driving cars.

A Hybrid Routing Algorithm for V2V Communication in VANETs Based on Blocked Q-Learning

It is of great significance to design a stable and reliable routing protocol for Vehicular Ad Hoc Networks (VANETs) that adopt Vehicle to Vehicle (V2V) communications in the face of frequent network topology changes. In this paper, we propose a hybrid routing algorithm, RCRIQ, based on improved Q-learning. For an established cluster structure, the cluster head is used to select the gateway vehicle according to the gateway utility function to expand the communication range of the cluster further. During the link construction stage, an improved Q-learning algorithm is adopted. The corresponding neighbor vehicle is chosen according to the maximum Q value in the neighbor list. The heuristic algorithm selects the next-hop by the maximum heuristic function value when selecting the next-hop neighbor node. The above two strategies are comprehensively evaluated to determine the next hop. This way ensures the optimal selection of the next hop in terms of reachability and other communication parameters. Simulation experiments show that the algorithm proposed in this article has better performance in terms of routing stability, throughput, and communication delay in the urban traffic scene.

RSG-GCN: Predicting Semantic Relationships in Urban Traffic Scene With Map Geometric Prior

Automated identification of the relationships between traffic actors and surrounding objects, in order to describe their behavior and predict their intentions, has become the focus of increasing attention in the field of autonomous driving. Therefore, in this work, we propose a Road Scene Graphs-Graph Convolutional Network (RSG-GCN) as a novel, graph-based model for predicting the topological graph structure of a given traffic scene. The status of the actors and HD map information are integrated as prior knowledge, allowing the edges linking the actor nodes to capture potential semantic relationships, such as “vehicle approaching pedestrian” and “pedestrian waiting at intersection”. To train this model, we created our own RSG dataset, as well as a relational dataset and benchmark derived from nuScenes. Our extensive range of experiments demonstrate that our model can more accurately predict semantic relationships and behavior in a given traffic scene than other popular traffic scene prediction models. In particular, regarding the use of HD map prior knowledge, we found that the resulting increase in accuracy significantly outweighs performance loss caused by the increase in graph size. The downstream applications of RSG include traffic scene retrieval and synthetic traffic scene generation, which are briefly described.

Research on intelligent vehicle lamp signal recognition in traffic scene

In order to improve the ability of intelligent vehicle to accurately recognize the semantics of the front vehicle, a new method of lamp signal recognition is proposed by combining deep learning with traditional computer vision. Firstly, the method uses YOLOv4 (You Only Look Once) network to detect vehicles and obtain accurate vehicle tail areas; Then, according to the spatial distribution characteristics of the pixels lighting the lamp in hue, saturation, value (HSV), a HSV spatial segmentation method based on region adaptive threshold is proposed to improve the pixel extraction quality; Finally, the deep neural network model is established to train the collected sample data, classify the brake lamp, turn signal lamp and lamp off state according to the information of lighting lamp pixels, and infer the current lamp meaning of the vehicle in front. In this paper, Python3.8, Pytorch1.9 and Opencv3.2 are used as algorithm implementation tools to test the road in the daytime urban traffic scene. The experimental results show that the average accuracy of the algorithm is 81.3%, the accuracy of the left turn signal is 75.8%, and the accuracy of the right turn signal is 76.4%.Article HighlightsThis paper investigates vehicle lamp signal recognition that is rarely explored in intelligent vehicle perception. A convolutional neural network is trained on the autonomous driving dataset for detection of vehicle tail regions.This paper transforms the image color space and proposes a region-based adaptive threshold for semantic segmentation of vehicle taillights with different hue, saturation and value.This paper constructs a deep neural network for lamp signal classification. Combined with the data collected in the actual traffic scene and the optimized network structure, the judgment of turn signals and brake signals is completed.

Urban Traffic Imaging Using Millimeter-Wave Radar

Imaging technology enhances radar environment awareness. Imaging radar can provide richer target information for traffic management systems than conventional traffic detection radar. However, there is still a lack of research on millimeter-wave radar imaging technology for urban traffic surveillance. To solve the above problem, we propose an improved three-dimensional FFT imaging algorithm architecture for radar roadside imaging in urban traffic scenarios, enabling the concurrence of dynamic and static targets imaging. Firstly, by analyzing the target characteristics and background noise in urban traffic scenes, the Monte-Carlo-based constant false alarm detection algorithm (MC-CFAR) and the improved MC-CFAR algorithm are proposed, respectively, for moving vehicles and static environmental targets detection. Then, for the velocity ambiguity solution problem with multiple targets and large velocity ambiguity cycles, an improved Hypothetical Phase Compensation algorithm (HPC-SNR) is proposed and complimented. Further, the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used to remove outliers to obtain a clean radar point cloud image. Finally, traffic targets within the 50 m range are presented as two-dimensional (2D) point cloud imaging. In addition, we also try to estimate the vehicle type by target point cloud size, and its accuracy reaches more than 80% in the vehicle sparse condition. The proposed method is verified by actual traffic scenario data collected by a millimeter-wave radar system installed on the roadside. The work can support further intelligent transportation management and extend radar imaging applications.

Rapid and Ultra-lightweight Semantic Segmentation in Urban Traffic Scene

Rapid and Ultra-lightweight Semantic Segmentation in Urban Traffic Scene

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

With the development of electronic technology, intelligent cars can gradually realize more complex artificial intelligence algorithms. The video caption algorithm is one of them. However, current video caption algorithms only consider single-visual information when applied to urban traffic scenes, which leads to the failure to generate accurate captions of complex sets. The multimodal fusion algorithm based on Transformer is one of the solutions to this problem. However, the existing algorithms have the difficulties of a low fusion performance and high computational complexity. We propose a new video caption Transformer-based model, the MFVC (Multimodal Fusion for Video Caption), to solve these issues. We introduce audio modal data and the attention bottleneck module to increase the available information to describe the generative model and improve the model effect with less operation costs through the attention bottleneck module. Finally, the experiment is conducted on the available datasets, MSR-VTT and MSVD. Meanwhile, to verify the effect of the model on the urban traffic scene, the experiment is carried out on the self-built traffic caption dataset BUUISE, and the evaluation index confirms the model. This model can achieve good results on both available datasets and urban traffic datasets and has excellent application prospects in the intelligent driving industry.

A Comprehensive Review of Vehicle Detection Techniques Under Varying Moving Cast Shadow Conditions Using Computer Vision and Deep Learning

Design of a vision-based traffic analytic system for urban traffic video scenes has a great potential in context of Intelligent Transportation System (ITS). It offers useful traffic-related insights at much lower costs compared to their conventional sensor based counterparts. However, it remains a challenging problem till today due to the complexity factors such as camera hardware constraints, camera movement, object occlusion, object speed, object resolution, traffic flow density, and lighting conditions etc. ITS has many applications including and not just limited to queue estimation, speed detection and different anomalies detection etc. All of these applications are primarily dependent on sensing vehicle presence to form some basis for analysis. Moving cast shadows of vehicles is one of the major problems that affects the vehicle detection as it can cause detection and tracking inaccuracies. Therefore, it is exceedingly important to distinguish dynamic objects from their moving cast shadows for accurate vehicle detection and recognition. This paper provides an in-depth comparative analysis of different traffic paradigm-focused conventional and state-of-the-art shadow detection and removal algorithms. Till date, there has been only one brief survey which highlights the shadow removal methodologies particularly for traffic paradigm. In this paper, a number of research papers containing results of urban traffic scenes have been shortlisted from the last three decades to give a comprehensive overview of the work done in this area. The study reveals that the only way to make a comparative evaluation is to use the existing Highway I, II, and III datasets which are frequently used for qualitative or quantitative analysis of shadow detection or removal algorithms. Furthermore, the paper not only provides cues to solve moving cast shadow problems, but also suggests that even after the advent of Convolutional Neural Network (CNN)-based vehicle detection methods, the problems caused by moving cast shadows persists. Therefore, this paper proposes a hybrid approach which uses a combination of conventional and state-of-the-art techniques as a pre-processing step for shadow detection and removal before using CNN for vehicles detection. The results indicate a significant improvement in vehicle detection accuracies after using the proposed approach.

A Comparison of Bottom-Up Models for Spatial Saliency Predictions in Autonomous Driving.

Bottom-up saliency models identify the salient regions of an image based on features such as color, intensity and orientation. These models are typically used as predictors of human visual behavior and for computer vision tasks. In this paper, we conduct a systematic evaluation of the saliency maps computed with four selected bottom-up models on images of urban and highway traffic scenes. Saliency both over whole images and on object level is investigated and elaborated in terms of the energy and the entropy of the saliency maps. We identify significant differences with respect to the amount, size and shape-complexity of the salient areas computed by different models. Based on these findings, we analyze the likelihood that object instances fall within the salient areas of an image and investigate the agreement between the segments of traffic participants and the saliency maps of the different models. The overall and object-level analysis provides insights on the distinctive features of salient areas identified by different models, which can be used as selection criteria for prospective applications in autonomous driving such as object detection and tracking.

Traffic Flow Parameters Collection under Variable Illumination Based on Data Fusion

Accurate traffic flow parameters are the supporting data for analyzing traffic flow characteristics. Vehicle detection using traffic surveillance pictures is a typical method for gathering traffic flow characteristics in urban traffic scenes. In complicated lighting conditions at night, however, neither classical nor deep-learning-based image processing algorithms can provide adequate detection results. This study proposes a fusion technique combining millimeter-wave radar data with image data to compensate for the lack of image-based vehicle detection under complicated lighting to complete all-day parameters collection. The proposed method is based on an object detector named CenterNet. Taking this network as the cornerstone, we fused millimeter-wave radar data into it to improve the robustness of vehicle detection and reduce the time-consuming postcalculation of traffic flow parameters collection. We collected a new dataset to train the proposed method, which consists of 1000 natural daytime images and 1000 simulated nighttime images with a total of 23094 vehicles counted, where the simulated nighttime images are generated by a style translator named CycleGAN to reduce labeling workload. Another four datasets of 2400 images containing 20161 vehicles were collected to test the proposed method. The experimental results show that the method proposed has good adaptability and robustness at natural daytime and nighttime scenes.

Vehicle Trajectory Extraction in Traffic Scene Based on Monocular Vision

Accurately extracting the trajectory information of multiple vehicles in urban traffic videos is of great significance in applications. It can help analyze and predict drivers’ behavior, count road flow, and restore accident scenarios. However, due to the complexity of the interaction between vehicles and the environment, it is still a challenging problem to accurately extract the trajectory of multi-target vehicles in urban traffic scenes, which often requires the assistance of a variety of different devices, which increases the practicality and difficulty of application. A method is proposed based on the combination of detection and tracking. This method only needs to use a video stream captured by a fixed panoramic camera and the coordinate information of several feature points in the scene to extract the speed and trajectory information of the vehicle in the scene. Different optimization methods are used in each link to reduce errors, and build a simulation environment for the test scenario, reconstruct and verify the extracted trajectory information in the simulation environment. Experiments have proved that the effect of the method is basically satisfactory. It has the advantages of fewer constraints, convenient deployment, and accurate restoration while ensuring a certain real-time performance, which has practical engineering significance.

Trajectory Prediction in Autonomous Driving With a Lane Heading Auxiliary Loss

Predicting a vehicle's trajectory is an essential ability for autonomous vehicles navigating through complex urban traffic scenes. Bird's-eye-view roadmap information provides valuable information for making trajectory predictions, and while state-of-the-art models extract this information via image convolution, auxiliary loss functions can augment patterns inferred from deep learning by further encoding common knowledge of social and legal driving behaviors. Since human driving behavior is inherently multimodal, models which allow for multimodal output tend to outperform single-prediction models on standard metrics. We propose a loss function which enhances such models by enforcing expected driving rules on all predicted modes. Our contribution to trajectory prediction is twofold; we propose a new metric which addresses failure cases of the off-road rate metric by penalizing trajectories that oppose the ascribed heading (flow direction) of a driving lane, and we show this metric to be differentiable and therefore suitable as an auxiliary loss function. We then use this auxiliary loss to extend the the standard multiple trajectory prediction (MTP) and MultiPath models, achieving improved results on the nuScenes prediction benchmark by predicting trajectories which better conform to the lane-following rules of the road.