Signal Control Method Research Articles

Multi-Agent Deep Reinforcement Learning with Graph Attention Network for Traffic Signal Control in Multiple-Intersection Urban Areas

Deep reinforcement learning has seen significant progress in traffic signal control. However, existing research still lacks the ability to effectively capture the correlation of road network information and the perception capability of traffic signal states. To address this gap, we propose a multi-intersection traffic signal control method that integrates a graph attention network, named the graph attention network-deep deterministic polcy gradient (GAT-DDPG) algorithm. This algorithm incorporates the restart random walk into the attention mechanism, exploring graph information through global random walks, reducing reliance on local nodes, and enhancing the model’s comprehensive understanding of graph structure features, thereby improving the modeling capability of traffic network structures. Moreover, the algorithm can automatically identify and extract key features from the complex data of the traffic network without manual intervention, adapting to different traffic network topologies, and can update and adjust the traffic signal control system in real-time to accommodate actual traffic flow and congestion situations. Experimental results indicate that the GAT-DDPG algorithm reduces average vehicle travel time significantly across three real road networks (Hangzhou and Jinan in China, and New York, U.S.) and two synthetic road network datasets. Additionally, it demonstrates optimal convergence speed and performance in these real datasets, attributed to its capability to capture global information and deeply comprehend the intricate structures of traffic networks. The research proves that this model has significant advantages in the field of traffic signal control, improving the operational efficiency of urban area intersections. Future work will incorporate additional road environment factors to better adapt to complex urban traffic.

Pri-DDQN: learning adaptive traffic signal control strategy through a hybrid agent

Adaptive traffic signal control is the core of the intelligent transportation system (ITS), which can effectively reduce the pressure on traffic congestion and improve travel efficiency. Methods based on deep Q-leaning network (DQN) have become the mainstream to solve single-intersection traffic signal control. However, most of them neglect the important difference of samples and the dependence of traffic states, and cannot quickly respond to randomly changing traffic flows. In this paper, we propose a new single-intersection traffic signal control method (Pri-DDQN) based on reinforcement learning and model the traffic environment as a reinforcement learning environment, and the agent chooses the best action to schedule the traffic flow at the intersection based on the real-time traffic states. With the goal of minimizing the waiting time and queue length at intersections, we use double DQN to train the agent, incorporate traffic state and reward into the loss function, and update the target network parameters asynchronously, to improve the agent’s learning ability. We try to use the power function to dynamically change the exploration rate to accelerate convergence. In addition, we introduce a priority-based dynamic experience replay mechanism to increase the sampling rate of important samples. The results show that Pri-DDQN achieves better performance, compared to the best baseline, it reduces the average queue length is reduced by 13.41%, and the average waiting time by 32.33% at the intersection.

Dynamic Control Method for CAV-Shared Lanes at Intersections in Mixed Traffic Flow

The existing signal control methods for mixed traffic related to connected automated vehicles (CAVs) and connected human-driven vehicles (CHVs) at intersections fail to tap the traffic potential of CAV-dedicated lanes. Accordingly, a dynamic allocation method of CAV-shared lanes is proposed, and the method of traffic flow scheduling and CAV trajectory optimization for multilane intersections with CAV-shared lanes is constructed to improve the traffic performance. The simulation results show that the optimization strategy proposed in this study can reduce the average delay at the intersection to varying degrees compared with the control strategy, using (a) the dynamic CAV-dedicated lane allocation method and (b) the shared-phase dedicated-lane method. Although the stops of CAVs will increase, the time utilization rate of most approach lanes is considerably improved, particularly CAV-shared lanes that can effectively improve the intersection performance. Further analysis shows that the number of CAV-shared lanes is closely dependent on the CAV penetration rate. The method proposed in this study is suitable for multilane intersections with a high CAV penetration rate.

Adaptive Traffic Signal Control Method Based on Offline Reinforcement Learning

The acceleration of urbanization has led to increasingly severe traffic congestion, creating an urgent need for effective traffic signal control strategies to improve road efficiency. This paper proposes an adaptive traffic signal control method based on offline reinforcement learning (Offline RL) to address the limitations of traditional fixed-time signal control methods. By monitoring key parameters such as real-time traffic flow and queue length, the proposed method dynamically adjusts signal phases and durations in response to rapidly changing traffic conditions. At the core of this research is the design of a model named SD3-Light, which leverages advanced offline reinforcement learning to predict the optimal signal phase sequences and their durations based on real-time intersection state features. Additionally, this paper constructs a comprehensive offline dataset, which enables the model to be trained without relying on real-time traffic data, thereby reducing costs and improving the model’s generalization ability. Experiments conducted on real-world traffic datasets demonstrate the effectiveness of the proposed method in reducing the average travel time. Comparisons with several existing methods highlight the clear advantages of our approach in enhancing traffic management efficiency.

Coordination of distributed adaptive signal control and advisory speed optimization based on shockwave theory

AbstractThis paper presents a distributed adaptive signal control and advisory speed coordination method based on shockwave theory, which accommodates diverse traffic conditions. In order to assess signal control efficiency under various scenarios, an innovative evaluation index termed synthetic delay is introduced based on the analysis of traffic dynamics at intersections. Considering the formation and dissipation of queue, and flow fluctuation of incoming traffic, it automatically evaluates control delay and throughput with distinctive significances. The distributed adaptive control method calculates the optimal green time in real time to minimize total synthetic delay at intersections. Furthermore, the coordination of advisory speed with the signal control schemes is addressed to ensure smooth progressions for vehicles. The proposed method considers the saturation of traffic and upstream traffic flow changes, leading to adaptability to changing traffic scenarios and effective coordination of traffic control. Several simulations were conducted and compared with the proposed method with other control methods. The results demonstrate that the proposed methods reduce the control delay and increase intersection throughput remarkably under different traffic saturations, confirming their effectiveness.

CLlight: Enhancing representation of multi-agent reinforcement learning with contrastive learning for cooperative traffic signal control

Multi-agent reinforcement learning has shown great potential for coordinating multi-intersection traffic signals due to its powerful adaptive capabilities, treating each intersection as an agent. However, in the real world, different intersections possess differentiating characteristics such as unique vehicle distributions and traffic patterns. Most existing methods directly add neighboring intersection states to local intersections and optimize the cooperative policy network based on synthesized global features. This indirect optimization approach makes it difficult to thoroughly explore the mutual interactions among different intersection agents, preventing agents from truly learning features with cooperative awareness. To resolve these challenges, we introduce contrastive learning as representation task to the multi-intersection traffic signal control approach named CLlight for two-stage policy network updating. In the first stage, we utilize policy-based or actor-critic-based reinforcement learning methods such as A2C, SAC, and PPO to train policy networks with certain representational capabilities. In the second stage, by extracting pre- and post-masked features and reconstructing the post-masked features, the agents are encouraged to learn the similarities and differences between different intersection policies, which in turn enhances the cooperative and individual representation capabilities of the policy network. To the best of our knowledge, this is the first application of contrastive learning in the field of traffic signal control. Experimental results demonstrate, compared to other state-of-the-art traffic signal control methods, superior average travel time and average waiting time performance under various scenarios, tested on synthetic and real-world datasets.

Dynamic Network-Level Traffic Speed and Signal Control in Connected Vehicle Environment.

The advent of connected vehicles holds significant promise for enhancing existing traffic signal and vehicle speed control methods. Despite this potential, there has been a lack of concerted efforts to address issues related to vehicle fuel consumption and emissions during travel across multiple intersections controlled by traffic signals. To bridge this gap, this research introduces a novel technique aimed at optimizing both traffic signals and vehicle speeds within transportation networks. This approach is designed to contribute to the improvement of transportation networks by simultaneously addressing issues related to fuel consumption and pollutant emissions. Simulation results vividly illustrate the pronounced the effectiveness of the proposed traffic signal and vehicle speed control methods of alleviating vehicle delay, reducing stops, lowering fuel consumption, and minimizing CO2 emissions. Notably, these benefits are particularly prominent in scenarios characterized by moderate traffic density, emphasizing the versatility and positive impact of the method across varied traffic conditions.

A cooperative perception based adaptive signal control under early deployment of connected and automated vehicles

Connected vehicle-based adaptive traffic signal control requires certain market penetration rates (MPRs) to be effective, usually exceeding 10%. Cooperative perception based on connected and automated vehicles (CAVs) can effectively improve overall data collection efficiency and reduce required MPR. However, the distribution of observed vehicles under cooperative perception is highly skewed and imbalanced, especially under very low CAV MPRs (e.g., 1%). To address this challenge, this paper proposes a novel deep reinforcement learning-based adaptive traffic signal control (RL-TSC) method that integrates a traffic flow model, known as the cell transmission model (CTM), denoted as CAVLight. Traffic states estimated from the CTM are integrated with the data collected from the cooperative perception environment to update the states in the CAVLight model. The design of reward function aims for reducing total vehicle delays and stabilizing agent behaviors. Extensive numerical experiments under a real-world intersection with varying traffic demand levels and CAV MPRs are conducted to compare the performance of CAVLight and other benchmark algorithms, including a fixed-time controller, an actuated controller, the max pressure model, and an optimization-based adaptive TSC. Results demonstrate the superiority of CAVLight in performance and generalizability over benchmarks, especially under 1% CAV MPR scenario with high traffic demands. The influence of CTM integration on CAVLight is further explored through RL agent policy visualization and sensitivity analysis in CTM parameters and CAV perception capabilities (i.e., detection range and detection accuracy).

A Carbon Benefits-Based Signal Control Method in a Connected Environment

This study proposes an innovative carbon benefits-based signal control method for connected vehicle (CV) environments, aiming to reduce carbon emissions at urban intersections. By integrating a Carbon Inclusion Mechanism (CIM), the proposed approach offers carbon rewards to vehicles adhering to speed guidance. The method exhibits the following features: (i) higher ceiling of carbon emissions reduction at signal control intersection; (ii) higher compliance rate (CR) of vehicles by taking advantage of carbon economic incentives; (iii) a method for calculating carbon emissions reduction at the intersection. To validate the effectiveness, performance evaluations of emissions, stop frequencies, and delays were conducted through microscopic simulation. Sensitivity analysis encompassed various traffic demands, different CRs of carbon-benefit connected vehicles (CBCVs), and unbalanced traffic demand. The results demonstrated that the proposed method excels in reducing traffic emissions, stop frequencies, and delays. Specifically, carbon emissions were reduced by 5.24% to 17.60%, stop frequencies decreased by 14.8% to 75.4%, and delays were reduced by 22.82% to 52.62%. By utilizing connected vehicle technology and CIM, this study contributes to sustainable urban traffic management, laying a foundation for future research and the practical implementation of emission reduction strategies.

Traffic Signal Control Optimization Based on Neural Network in the Framework of Model Predictive Control

To improve the effectiveness of model predictive control (MPC) in dynamic traffic signal control strategies, it has been combined with graph convolutional networks (GCNs) and deep reinforcement learning (DRL) technologies. In this study, a neural-network-based traffic signal control optimization method under the MPC framework is proposed. A dynamic correlation matrix is introduced in the predictive model to adapt to the dynamic changes in correlations between nodes over time. The signal control optimization strategy is solved using DRL, where the agent explores the optimal control strategy based on pre-set constraints in the future road environment. The geometric structure and traffic flow data of a real intersection were selected as the simulation validation environment, and a joint simulation was conducted using Python and SUMO. The experimental results indicate that in low-traffic scenarios, the queue length is reduced by more than 2 vehicles compared to the selected comparison methods; in high-traffic scenarios, the queue length is reduced by an average of 17 vehicles. Under the actual traffic data of the intersection, the average speed is increased by 6.4% compared to the fixed timing method; compared to the inductive signal control method, it increases from 9.76 m/s to 11.69 m/s, an improvement of 19.7%, effectively enhancing the intersection signal control performance.

Digital Twin-Enhanced Adaptive Traffic Signal Framework under Limited Synchronization Conditions

Unmanned traffic signal control is regarded as a sustainable intelligent management methodology. However, it faces the challenge of unpredictable traffic flow due to stochastic arrivals. The digital twin (DT) has emerged as a promising approach to address the challenges of time-varying traffic demand in urban transportation. Previous studies of DT-based adaptive traffic signal control (ATSC) methods all assume ideal synchronization conditions between the DT and the physical twin (PT). It means that the DT can immediately figure out the next neglecting limitation of realistic conditions, i.e., discrepancies between the DT and PT and computational ability. This paper proposes a DT-ATSC framework aimed at reducing the total delay at isolated intersections under limited synchronization conditions. The framework contains two parts: (1) a cell transmission model-based intersection simulation model featuring less computational consumption and the parameter self-calibration mechanism; and (2) scheme searching algorithms that can guide the DT to create optimization-oriented signal timing scheme candidates. Three options are provided for the scheme searching algorithms, i.e., grid search (GS), the genetic algorithm (GA), and Bayesian optimization (BO). A testing platform is created to validate the effectiveness of the proposed DT-ATSC. Experimental results indicate that the proposed DT-ATSC-BO outperforms the DT-ATSC-GA and DT-ATSC-GS. Meanwhile, the average vehicle delay of the DT-ATSC-BO is up to 53% lower than that of the current adaptive signal control method, which indicates that the proposed DT-ATSC has achieved the expected effect. Moreover, compared to the previous related work, the proposed DT-ATSC framework is more likely to be able to be applied in realistic situations because synchronization issues are incorporated in the design of the DT-ATSC by assuming a limited margin time for a decision.

Explicit coordinated signal control using soft actor–critic for cycle length determination

AbstractExplicit signal coordination carries prior knowledge of traffic engineering and is widely accepted for global implementation. With the recent popularity of reinforcement learning, numerous researchers have turned to implicit signal coordination. However, these methods inevitably require learning coordination from scratch. To maximize the use of prior knowledge, this study proposes an explicit coordinated signal control (ECSC) method using a soft actor–critic for cycle length determination. This method can fundamentally solve the challenges encountered by traditional methods in determining the cycle length. Soft actor–critic was selected among various reinforcement learning methods. A single agent was administered to the arterials. An action is defined as the selection of a cycle length from among the candidates. The state is represented as a feature vector, including the cycle length and features of each leg at every intersection. The reward is defined as departures that indirectly minimize system vehicle delays. Simulation results indicate that ECSC significantly outperforms the baseline methods, as evident in system vehicle delay across nearly all demand scenarios and throughput in high demand scenarios. The ECSC revitalizes explicit signal coordination and introduces new perspectives on the application of reinforcement learning methods in signal coordination.

Signal control for overflow prevention at intersections using partial connected vehicle data

Various signal control methods assumed that the bottleneck capacity and traffic volume are known. However, both may be unknown in nonrecurrent congestion cases, such as traffic accidents. In this study, a signal control optimisation method is developed for overflow prevention considering unknown traffic volume and bottleneck dropped capacity. The proposed method predicts remaining space at the junction exits and arrival–departure curves using partial connected vehicle data. Subsequently, the signal control is reallocated using the model predictive control technique. The results of case study show that the model is valid to prevent overflow when the connected vehicle penetration rate exceeds 10%. The average delay of the proposed method is reduced by 48.56% and 24.49% compared with those of the adaptive signal control method considering the queue length of only the approach lanes and that considering the queue lengths of both the approach and exit lanes but without prediction, respectively.

Deep Reinforcement Learning for Ecological and Distributed Urban Traffic Signal Control with Multi-Agent Equilibrium Decision Making

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent’s decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network.

F rugal L ight : Symmetry-Aware Cyclic Heterogeneous Intersection Control using Deep Reinforcement Learning with Model Compression, Distillation and Domain Knowledge

Developing countries need to better manage fast increasing traffic flows, owing to rapid urbanization. Else, increasing traffic congestion would increase fatalities due to reckless driving, as well as keep vehicular emissions and air pollution critically high in cities like New Delhi. State-of-the-art traffic signal control methods in developed countries, however, use expensive sensing, computation, and communication resources. How far can control algorithms go, under resource constraints, is explored through the design and evaluation of FrugalLight (FL) in this article. We also captured and processed a real traffic dataset at a busy intersection in New Delhi, India, using efficient techniques on low cost embedded devices. This dataset ( https://delhi-trafficdensity-dataset.github.io ) contains traffic density information at fine time granularity of one measurement every second, from all approaches of the intersection for 40 days. FrugalLight ( https://github.com/sachin-iitd/FrugalLight ) is evaluated on the collected traffic dataset from New Delhi and another open source traffic dataset from New York. FrugalLight matches the performance of state-of-the-art Convolutional Neural Network (CNN) based sensing and Deep Reinforcement Learning (DRL) based control algorithms, while utilizing resources less by an order of magnitude. We further explore improvements using a careful combination of knowledge distillation and domain knowledge based DRL model compression, with employing Model-Agnostic Meta-Learning to quickly adapt to traffic at new intersections. The collected real dataset and FrugalLight therefore opens up opportunities for resource efficient RL based intersection control design for the ML research community, where the controller should have limited carbon footprint. Such intelligent, green, intersection controllers can help reduce traffic congestion and associated vehicular emissions, even if compute and communication infrastructure is limited in low resource regions. This is a critical step toward achieving two of the United Nations Sustainable Development Goals (SDG), namely sustainable cities and communities and climate action.

Edge AI-Based Smart Intersection and Its Application for Traffic Signal Coordination: A Case Study in Pyeongtaek City, South Korea

Recently, smart intersections have emerged as a novel intelligent transportation system (ITS) solution that integrates traffic monitoring, optimal signal control, and even traffic safety. Although smart intersections have been prevalent in many cities, there are a few drawbacks in their practical operations. First, there are inevitable delays in transmitting and processing the video data. Second, there is still a need to develop a real-time signal control method leveraging the acquired data from smart intersections. Thus, this study aims to construct edge AI-based smart intersections and to provide their application for traffic signal coordination. To this end, we install smart intersections on three consecutive intersections of Route 45 in Pyeongtaek city, South Korea. The real-time traffic data are collected by an edge AI video analysis model which is compressed and optimized for its operation in on-site edge devices. The optimized model maintains a similar level of accuracy (93.64%), even if the size is reduced by 97.8% compared to the original. Next, we utilize the LT2 model to treat the coordination failure problem in nonpeak hours occurring unnecessary delays of the side-streets with relatively high demands. We complement some constraint conditions in order to consider the compatibility with the current legacy system. The experiment is conducted on a virtual environment of which geometry and traffic demand are configured based on the features of the study site. The numerical results conclude that the optimal offsets calculated by the LT2 model effectively manage bandwidths for multidirectional flows based on the real-time traffic demands collected from the edge AI-based smart intersections. This study contributes to serve high-resolution real-time traffic data using edge AI on smart intersections and to provide a case study for signal coordination.

Traffic Light Control System

The traffic light control system is a key component of urban traffic management, and its performance directly impacts the smooth flow of traffic and traffic safety. Based on some issues existing in the current traffic signal control systems, this paper proposes a novel traffic light control system based on intelligent algorithms. This system combines artificial intelligence technology with traditional traffic signal control methods to achieve intelligent control of traffic signals, thereby enhancing traffic efficiency and road safety. The paper provides a detailed explanation of the system's design principles, algorithm implementation, and experimental results. Comparative experiments demonstrate that the system has significant advantages in reducing traffic congestion and improving vehicle passage efficiency. Finally, the paper discusses possible future improvement directions and application scenarios, envisioning the development prospects of traffic light control systems in the future.

CVDMARL: A Communication-Enhanced Value Decomposition Multi-Agent Reinforcement Learning Traffic Signal Control Method

Effective traffic signal control (TSC) plays an important role in reducing vehicle emissions and improving the sustainability of the transportation system. Recently, the feasibility of using multi-agent reinforcement learning technology for TSC has been widely verified. However, the process of mapping road network states onto actions has encountered many challenges, due to the limited communication between agents and the partial observability of the traffic environment. To address this problem, this paper proposes a communication-enhancement value decomposition, multi-agent reinforcement learning TSC method (CVDMARL). The model combines two communication methods: implicit and explicit communication, decouples the complex relationships among the multi-signal agents through the centralized-training and decentralized-execution paradigm, and uses a modified deep network to realize the mining and selective transmission of traffic flow features. We compare and analyze CVDMARL with six different baseline methods based on real datasets. The results show that compared to the optimal method MN_Light, among the baseline methods, CVDMARL’s queue length during peak hours was reduced by 9.12%, the waiting time was reduced by 7.67%, and the convergence algebra was reduced by 7.97%. While enriching the information content, it also reduces communication overhead and has better control effects, providing a new idea for solving the collaborative control problem of multi-signalized intersections.

Study of medium and long-term free flow capacity and queue discharge rates on roads.

With the rise in vehicle ownership, traffic congestion has emerged as a major barrier to urban progress, making the study and optimization of urban road capacity exceedingly crucial. The research on the medium and long-term free-flowing capacity and queue emission rate of roads takes an in-depth exploration of this issue from a cutting-edge perspective, aiming to find solutions adaptable to the progression of the times. The purpose of this study is to understand and predict the road capacity and queue emission rate more accurately, thus improving the urban traffic condition. Existing literature primarily focuses on short-term forecasts of road capacity, leaving a notable void in the research of medium and long-term road capacity and queue emission rate. This gap often results in a lack of sufficient foresight when urban traffic planning faces practical issues. To fill this void, this study undertook an in-depth examination of the road capacity and queue emission rate over the medium and long term (10 years) based on big data analysis and artificial intelligence theories. This paper employs a Radial Basis Function (RBF) neural network, combined with twelve other parameters that could potentially impact road capacity, such as traffic volume, road width, number of lanes, traffic signal control methods, etc., to analyze the relationship between each parameter and free-flow traffic and queue emission rate. These analyses are grounded in extensive road data, encompassing not only the city's main roads but also secondary roads and community roads. The study results show a continuous downward trend in the free-flowing capacity of roads and a slight upward trend in the queue emission rate over the past decade. Further analysis reveals the extent of impact each factor has on the free-flow traffic and queue emission rate, providing a scientific basis for future urban traffic planning.

Cooperative traffic signal control through a counterfactual multi-agent deep actor critic approach

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput.