Traffic Signal Control Problem Research Articles

Optimizing green splits in high‐dimensional traffic signal control with trust region Bayesian optimization

AbstractCentralized traffic signal control has long been a challenging, high‐dimensional optimization problem. This study establishes a simulation‐based optimization framework and develops a novel optimization algorithm based on trust region Bayesian optimization (TuRBO), which can efficiently obtain an approximate optimal solution to the high‐dimensional traffic signal control problem. Local Gaussian process (GP), trust region, and Thompson sampling are employed in the TuRBO and contribute considerably to performance in terms of computational speed, solution quality, and scalability. Empirical studies are carried out using data from Mudanjiang and Chengdu, China. The performance of TuRBO is compared with that of Bayesian optimization (BO), genetic algorithm and random sampling. The results show that TuRBO converges the fastest because of its ability to balance exploration and exploitation through the trust region and Thompson sampling. Meanwhile, because TuRBO enables more efficient exploitation through the local GP, the solution quality of TuRBO outperforms others significantly. The average waiting time achieved by TuRBO was 2.84% lower than that achieved by BO. Finally, the method has been successfully extended to a large network with 233‐dimensional spaces and 122 signalized intersections, demonstrating that the developed methodology can deal with high‐dimensional traffic signal control effectively for real case applications.

A Comparative Study of Traffic Signal Control Based on Reinforcement Learning Algorithms

In recent years, the increasing production and sales of automobiles have led to a notable rise in congestion on urban road traffic systems, particularly at ramps and intersections with traffic signals. Intelligent traffic signal control represents an effective means of addressing traffic congestion. Reinforcement learning methods have demonstrated considerable potential for addressing complex traffic signal control problems with multidimensional states and actions. In this research, the team propose Q-learning and Deep Q-Network (DQN) based signal control frameworks that use variable phase sequences and cycle times to adjust the order and the duration of signal phases to obtain a stable traffic signal control strategy. Experiments are simulated using the traffic simulator Simulation of Urban Mobility (SUMO) to test the average speed and the lane occupancy rate of vehicles entering the ramp to evaluate its safety performance and test the vehicle’s traveling time to assess its stability. The simulation results show that both reinforcement learning algorithms are able to control cars in dynamic traffic environments with higher average speed and lower lane occupancy rate than the no-control method and that the DQN control model improves the average speed by about 10% and reduces the lane occupancy rate by about 30% compared to the Q-learning control model, providing a higher safety performance.

Cooperative traffic signal control through a counterfactual multi-agent deep actor critic approach

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput.

BlindLight: High Robustness Reinforcement Learning Method to Solve Partially Blinded Traffic Signal Control Problem

BlindLight: High Robustness Reinforcement Learning Method to Solve Partially Blinded Traffic Signal Control Problem

Enhancing traffic signal control with composite deep intelligence

Traffic signal control has always been a hot topic in the field of intelligent transportation. With the increasing complexity of urban traffic conditions due to urbanization, how to develop effective scheduling strategies to adapt to the changing traffic demands has become a key problem in current intelligent transportation. In light of this, this paper focuses on the traffic signal control problem at intersections and proposes a composite intelligent traffic signal control model based on heterogeneous graph neural networks with dual attention mechanisms and deep reinforcement learning. For the first time, the model incorporates the dual attention mechanism in graph neural networks into the traffic signal control, integrating graph neural networks with deep reinforcement learning techniques and traffic intersection scenarios. This allows for the construction of traffic condition models and the scheduling control of traffic resources, catering to the perception and decision-making needs in complex traffic environments. Firstly, the graph relationship representation of intersection resources is established, constructing the graph information structure for traffic flow and signal states. Then, a heterogeneous graph neural network is designed, incorporating both node-level and semantic-level dual attention mechanisms to characterize the traffic state and explore the relationships, enabling the extraction of explicit and implicit information in traffic intersections. Lastly, a deep reinforcement learning algorithm that combines Double Deep Q-Network (DDQN) and Dueling DQN is implemented to improve the algorithm's generalization and execution efficiency, enhancing the adaptability and stability of traffic signal scheduling in complex environments. Simulation tests are conducted on the SUMO simulation platform using real-world application datasets. Compared to four other similar traffic control model, the proposed model demonstrates performance advantages of more than 13% in terms of average reward, average delay, queue length, and waiting time. This validates the effectiveness of the proposed model.

Mean Field Multi-Agent Reinforcement Learning Method for Area Traffic Signal Control

Reinforcement learning is an effective method for adaptive traffic signal control in urban transportation networks. As the number of training rounds increases, the optimal control strategy is learned, and the learning capabilities of deep neural networks are further enhanced, thereby avoiding the limitations of traditional signal control methods. However, when faced with the sequential decision tasks of regional signal control, it encounters issues such as the curse of dimensionality and environmental non-stationarity. To address the limitations of traditional reinforcement learning algorithms applied to multiple intersections, the mean field theory is applied. This models the traffic signal control problem at multiple intersections within a region as interactions between individual intersections and the average effects of neighboring intersections. By decomposing the Q-function through bilateral estimation between the agent and its neighbors, this method reduces the complexity of interactions between agents while preserving global interactions between the agents. A traffic signal control model based on Mean Field Multi-Agent Reinforcement Learning (MFMARL) was constructed, containing two algorithms: Mean Field Q-Network Area Traffic Signal Control (MFQ-ATSC) and Mean Field Actor-Critic Network Area Traffic Signal Control (MFAC-ATSC). The model was validated using the SUMO simulation platform. The experimental results indicate that across different metrics, such as average speed, the mean field reinforcement learning method outperforms classical signal control methods and several existing approaches.

A stochastic dynamic network loading model for mixed traffic with autonomous and human-driven vehicles

In this study, we develop a stochastic dynamic network loading (DNL) model for the mixed traffic with autonomous vehicles (AVs) and human-driven vehicles (HVs). The source of stochasticity is the uncertainty inherent in the arrival process of the two classes of vehicular flow. The developed model captures both within-link and between-link traffic flow dependencies and evaluates the network state distribution in an analytical manner. The model has two main components, a probabilistic link model and a probabilistic node model. The link model is a stochastic formulation of the link transmission model (LTM), which captures the boundary conditions of a link and approximates the evolution of link state distribution. The node model, on the other hand, characterizes the flow transmissions across a network node. It reflects the between-link dependency by evaluating the expected transmission flow through an iterative algorithm, with an explicit consideration of the interactions between supply and demand constraints associated with a node. The developed model is validated versus replicated running of the deterministic LTM as well as microscopic traffic simulations, and the results reveal that it yields relatively accurate estimations. We also present two applications of the proposed model, including a traffic signal control problem and a class-based ramp metering problem, to demonstrate its practical value.

Traffic Signal Optimization Based on Fuzzy Control and Differential Evolution Algorithm

Urban traffic congestion is often concentrated at urban intersections. An urban road traffic signal control system is needed to prevent problems such as driving delays caused by frequent traffic congestions on trunk lines, exhaust emissions owing to frequent start and stop of vehicles, and fuel wastage due to long idling times. Maximizing the traffic capacity of an intersection and reducing the delay rate of vehicles has always been a problem for traffic control research. The coordinated control of urban traffic signals is regarded as a multi-objective optimization problem. A mathematical model for urban trunk traffic is studied herein. An average delay model, average queue length model, total delay calculation model for vehicles at intersections, and vehicle exhaust emission model are established to obtain an optimization model for a new traffic trunk coordinated control system. In addition, our study combines the fuzzy control theory with the adaptive sequencing mutation multi-objective differential evolution algorithm (FASM-MDEA). This new optimization method for traffic signal control at urban intersections is proposed as a solution for the traffic flow optimization model to solve the problem of traffic signal coordination and control of urban trunk lines. The simulation results demonstrate the effectiveness of the model optimization algorithm proposed in this study.

Distributed Stochastic MPC Traffic Signal Control for Urban Networks

In this paper, we design a stochastic model predictive control (MPC)-based traffic signal control method for urban networks when the uncertainties of the traffic model parameters (including the exogenous traffic flows and the turning ratios of downstream traffic flows) are taken into account. Considering that the traffic model parameters are random variables with known expectations and variances, the traffic signal control and coordination problem is formulated as a quadratic program with linear and second-order cone constraints. In order to reduce computational complexity, we suggest a way to decompose the optimization problem corresponding to the whole network into multiple subproblems. By applying an Alternating Direction Method of Multipliers (ADMM) scheme, the optimal stochastic traffic signal splits are found in a distributed manner. The effectiveness of the designed control method is validated via some simulations using VISSIM and MATLAB.

Cycle-based signal timing with traffic flow prediction for dynamic environment

This article studies adaptive traffic signal control problem of single intersection in dynamic environment. A novel cycle-based signal timing method with traffic flow prediction (CycleRL) is proposed to improve the traffic efficiency under dynamic traffic flow. Firstly, the empirical mode decomposition is applied to denoise the flow data. Then a data-model hybrid driven traffic flow prediction strategy is designed to predict the traffic flow, which combines a model-based Kalman filter and an LSTM network-based predictor and adopts another Kalman filter to fuse both prediction results to improve the prediction precision. Besides, a robust signal cycle timing strategy based on human-machine collaboration is developed to deal with dynamic traffic flow, which firstly designs a rule-based signal cycle scheme according to the predicted flow data as the preliminary scheme, and then finetunes the preliminary scheme based on Soft Actor-Critic (SAC) algorithm according to the real-time traffic dynamics. The experiments in both synthetic scenario and real-world scenario show that the proposed data-model hybrid driven traffic flow prediction algorithm has better prediction performance and the proposed CycleRL method outperforms rule-based methods, flow-based allocation methods and traditional reinforcement learning method. Moreover, it is also shown that the proposed CycleRL method has better transferability to bridge the discrepancy across domains.

Data-Driven RBFNN-Enhanced Model-Free Adaptive Traffic Symmetrical Signal Control for a Multi-Phase Intersection with Fast-Changing Traffic Flow

Fast-changing demand in real traffic systems always leads to asymmetrical traffic flow and queues, which aggravates congestion and energy waste. In this paper, the traffic signal control problem of multi-phase intersections was studied with fast-changing traffic flows. First, a novel model-free adaptive control-based symmetrical queuing balancing method was designed by using the full-format dynamic linearization (FFDL) technique. Second, in order to deal with the fast-changing traffic flow, a radial basis function neural network (RBFNN) was added to adjust parameters in a two-layer structure. Moreover, a variable cycle tuning algorithm was introduced to further reduce the time loss. Using the simulation, the proposed algorithm was compared with three other control strategies under low and high traffic demand, respectively, and the results showed the capability of the proposed algorithm.

Traffic Signal Control with Successor Feature-Based Deep Reinforcement Learning Agent

In this paper, we study the problem of traffic signal control in general intersections by applying a recent reinforcement learning technique. Nowadays, traffic congestion and road usage are increasing significantly as more and more vehicles enter the same infrastructures. New solutions are needed to minimize travel times or maximize the network capacity (throughput). Recent studies embrace machine learning approaches that have the power to aid and optimize the increasing demands. However, most reinforcement learning algorithms fail to be adaptive regarding goal functions. To this end, we provide a novel successor feature-based solution to control a single intersection to optimize the traffic flow, reduce the environmental impact, and promote sustainability. Our method allows for flexibility and adaptability to changing circumstances and goals. It supports changes in preferences during inference, so the behavior of the trained agent (traffic signal controller) can be changed rapidly during the inference time. By introducing the successor features to the domain, we define the basics of successor features, the base reward functions, and the goal preferences of the traffic signal control system. As our main direction, we tackle environmental impact reduction and support prioritized vehicles’ commutes. We include an evaluation of how our method achieves a more effective operation considering the environmental impact and how adaptive it is compared to a general Deep-Q-Network solution. Aside from this, standard rule-based and adaptive signal-controlling technologies are compared to our method to show its advances. Furthermore, we perform an ablation analysis on the adaptivity of the agent and demonstrate a consistent level of performance under similar circumstances.

Multi-Agent Reinforcement Learning for Traffic Signal Control: A Cooperative Approach

The rapid growth of urbanization and the constant demand for mobility have put a great strain on transportation systems in cities. One of the major challenges in these areas is traffic congestion, particularly at signalized intersections. This problem not only leads to longer travel times for commuters, but also results in a significant increase in local and global emissions. The fixed cycle of traffic lights at these intersections is one of the primary reasons for this issue. To address these challenges, applying reinforcement learning to coordinating traffic light controllers has become a highly researched topic in the field of transportation engineering. This paper focuses on the traffic signal control problem, proposing a solution using a multi-agent deep Q-learning algorithm. This study introduces a novel rewarding concept in the multi-agent environment, as the reward schemes have yet to evolve in the following years with the advancement of techniques. The goal of this study is to manage traffic networks in a more efficient manner, taking into account both sustainability and classic measures. The results of this study indicate that the proposed approach can bring about significant improvements in transportation systems. For instance, the proposed approach can reduce fuel consumption by 11% and average travel time by 13%. The results of this study demonstrate the potential of reinforcement learning in improving the coordination of traffic light controllers and reducing the negative impacts of traffic congestion in urban areas. The implementation of this proposed solution could contribute to a more sustainable and efficient transportation system in the future.

Deep Reinforcement Learning Based Traffic Signal Control: A Comparative Analysis

More recently, the advancement of deep learning techniques enables reinforcement learning (RL) to handle high-dimensional decision-making problems. It brings increasing attention in transport areas. Some work has applied them to solve the intractable traffic signal control (TSC) problem and achieved promising results. However, very few research comprehensively investigates the impacts of key design elements, including state definitions, reward functions and deep reinforcement learning (DRL) methods, on TSC performance. To fill this research gap, this paper first selects commonly used design elements from existing literature. Then, we compare their learning stability and control performance at an isolated intersection under different scenarios via simulation experiments. The experimental results show that the quantitative state (e.g., the number of vehicles on lanes) and image-like state (e.g., vehicle position and speed) have no significant impact on the performance under different traffic demands. However, the impact of various reward functions on performance is vivid, especially under high traffic demand. Also, high-resolution vehicular network data may not be possible to improve control performance versus ordinary camera data. In addition, the value-based DRL algorithms outperform the policy-based algorithm and traditional TSC control methods. The findings of this research would provide insights and guidance for transport engineers to design an efficient DRL-based TSC system in a real traffic environment.

Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control

Traffic signal control (TSC) is a challenging problem within intelligent transportation systems and has been tackled using multi-agent reinforcement learning (MARL). While centralized approaches are often infeasible for large-scale TSC problems, decentralized approaches provide scalability but introduce new challenges, such as partial observability. Communication plays a critical role in decentralized MARL, as agents must learn to exchange information using messages to better understand the system and achieve effective coordination. Deep MARL has been used to enable inter-agent communication by learning communication protocols in a differentiable manner. However, many deep MARL communication frameworks proposed for TSC allow agents to communicate with all other agents at all times, which can add to the existing noise in the system and degrade overall performance. In this study, we propose a communication-based MARL framework for large-scale TSC. Our framework allows each agent to learn a communication policy that dictates “which” part of the message is sent “to whom”. In essence, our framework enables agents to selectively choose the recipients of their messages and exchange variable length messages with them. This results in a decentralized and flexible communication mechanism in which agents can effectively use the communication channel only when necessary. We designed two networks, a synthetic <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4 \times 4$ </tex-math></inline-formula> grid network and a real-world network based on the Pasubio neighborhood in Bologna. Our framework achieved the lowest network congestion compared to related methods, with agents utilizing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 47-65 \%$ </tex-math></inline-formula> of the communication channel. Ablation studies further demonstrated the effectiveness of the communication policies learned within our framework.

Neighborhood Cooperative Multiagent Reinforcement Learning for Adaptive Traffic Signal Control in Epidemic Regions

Nowadays, multiagent reinforcement learning (MARL) have shared significant advances in the adaptive traffic signal control (ATSC) problems. For most of the researches, agents are all isomorphic, which disregards the situation in which isomerous intersections cooperative together in a real ATSC scenario, especially in epidemic regions where different intersections have quite different levels of importance. To this end, this paper models the ATSC problem as a networked Markov game (NMG), in which agents take into account information, including traffic conditions of it and its connected neighbors. A cooperative MARL framework named neighborhood cooperative hysteretic DQN (NC-HDQN) is proposed. Specifically, for each NC-HDQN agent in the NMG, first, the framework analyses correlation degrees with their connected neighbors and weighs observations and rewards by these correlations. Second, NC-HDQN agents independently optimize their strategies on the weighted information using hysteretic DQN (HDQN), which is designed to learn optimal joint strategies in cooperative multiagent games. Third, a rule-based NC-HDQN method and a Pearson correlation coefficient based NC-HDQN method, i.e., empirical NC-HDQN (ENC-HDQN) and Pearson NC-HDQN (PNC-HDQN), respectively, are designed. The first method maps the correlation degree between two connected agents according to vehicle numbers on roads between the two agents. In contrast, the second method uses the Pearson correlation coefficient to calculate the correlation degree adaptively. Our methods are empirically evaluated in both a synthetic scenario and two real-world traffic scenarios and give better performances in almost every standard test metric for ATSC.

Deep Reinforcement Learning for Intersection Signal Control Considering Pedestrian Behavior

Using deep reinforcement learning to solve traffic signal control problems is a research hotspot in the intelligent transportation field. Researchers have recently proposed various solutions based on deep reinforcement learning methods for intelligent transportation problems. However, most signal control optimization takes the maximization of traffic capacity as the optimization goal, ignoring the concerns of pedestrians at intersections. To address this issue, we propose a pedestrian-considered deep reinforcement learning traffic signal control method. The method combines a reinforcement learning network and traffic signal control strategy with traffic efficiency and safety aspects. At the same time, the waiting time of pedestrians and vehicles passing through the intersection is considered, and the Discrete Traffic State Encoding (DTSE) method is applied and improved to define the more comprehensive states and rewards. In the training of the neural network, the multi-process operation method is adopted, and multiple environments are run for training simultaneously to improve the model’s training efficiency. Finally, extensive simulation experiments are conducted on actual intersection scenarios using the simulation software Simulation of Urban Mobility (SUMO). The results show that compared to Dueling DQN, the waiting time due to our method decreased by 58.76% and the number of people waiting decreased by 51.54%. The proposed method can reduce both the number of people waiting and the waiting time at intersections.

A meta–reinforcement learning algorithm for traffic signal control to automatically switch different reward functions according to the saturation level of traffic flows

AbstractReinforcement learning (RL) algorithms have been widely applied in solving traffic signal control problems. Traffic environments, however, are intrinsically nonstationary, which creates a convergence problem that RL algorithms struggle to overcome. Basically, as a target problem for an RL algorithm, the Markov decision process (MDP) can be solved only when both the transition and reward functions do not vary. Unfortunately, the environment for traffic signal control is not stationary since the goal of traffic signal control varies according to congestion levels. For unsaturated traffic conditions, the objective of traffic signal control should be to minimize vehicle delay. On the other hand, the objective must be to maximize the throughput when traffic flow is saturated. A multiregime analysis is possible for varying conditions, but classifying the traffic regime creates another complex task. The present study provides a meta‐RL algorithm that embeds a latent vector to recognize the different contexts of an environment in order to automatically classify traffic regimes and apply a customized reward for each context. In simulation experiments, the proposed meta‐RL algorithm succeeded in differentiating rewards according to the saturation level of traffic conditions.

Bayesian optimization techniques for high-dimensional simulation-based transportation problems

Bayesian optimization (BO) is an attractive method for tackling transportation optimization problems due to its ability to balance exploitation and exploration. However, scaling BO to solve high-dimensional problems is a major challenge which has remained unsolved. Since transportation problems can be high-dimensional, the use of BO to solve transportation problems has been limited. This paper explores the use of BO with Gaussian process (GP) models to tackle high-dimensional transportation problems. It proposes formulations of the prior mean function and covariance function of the GP that enable BO to incorporate problem-specific transportation information, while remaining computationally tractable. This is done through the use of an analytical surrogate model. We validate the method with the 1-D and 100-D Griewank functions. The impact of different forms of bias in the surrogate model was also evaluated using the 100-D Griewank function example. We then apply the method to a high-dimensional traffic signal control problem in New York City. The results indicate the added value of using the problem-specific information in the prior mean and/or the covariance function. Importantly, for surrogate models that are not accurate approximations of, but have significant (anti-)correlation with the true objective function, the better approach is to embed the information in the covariance function, rather than in the prior mean function. More generally, the use of problem-specific information in the covariance function is robust to the accuracy of the surrogate model. This opens the way for a variety of low-resolution analytical transportation models to be used to tackle high-dimensional simulation-based optimization problems.

Hybrid Prediction-Based Approach for Traffic Signal Control Problem

Hybrid Prediction-Based Approach for Traffic Signal Control Problem