Adaptive Traffic Signal Research Articles

Neighborhood Cooperative Multiagent Reinforcement Learning for Adaptive Traffic Signal Control in Epidemic Regions

Nowadays, multiagent reinforcement learning (MARL) have shared significant advances in the adaptive traffic signal control (ATSC) problems. For most of the researches, agents are all isomorphic, which disregards the situation in which isomerous intersections cooperative together in a real ATSC scenario, especially in epidemic regions where different intersections have quite different levels of importance. To this end, this paper models the ATSC problem as a networked Markov game (NMG), in which agents take into account information, including traffic conditions of it and its connected neighbors. A cooperative MARL framework named neighborhood cooperative hysteretic DQN (NC-HDQN) is proposed. Specifically, for each NC-HDQN agent in the NMG, first, the framework analyses correlation degrees with their connected neighbors and weighs observations and rewards by these correlations. Second, NC-HDQN agents independently optimize their strategies on the weighted information using hysteretic DQN (HDQN), which is designed to learn optimal joint strategies in cooperative multiagent games. Third, a rule-based NC-HDQN method and a Pearson correlation coefficient based NC-HDQN method, i.e., empirical NC-HDQN (ENC-HDQN) and Pearson NC-HDQN (PNC-HDQN), respectively, are designed. The first method maps the correlation degree between two connected agents according to vehicle numbers on roads between the two agents. In contrast, the second method uses the Pearson correlation coefficient to calculate the correlation degree adaptively. Our methods are empirically evaluated in both a synthetic scenario and two real-world traffic scenarios and give better performances in almost every standard test metric for ATSC.

Leveraging autonomous vehicles in mixed-autonomy traffic networks with reinforcement learning-controlled intersections

ABSTRACT Development of new approaches to adaptive traffic signal control has received significant attention; an example is the reinforcement learning (RL), where training and implementation of an RL agent can allow adaptive signal control in real time, considering the agent’s past experiences. Furthermore, autonomous vehicle (AV) technology has shown promise to enhancing the traffic mobility at highways and intersections. In this paper, delayed action deep Q-learning is developed for a vehicle network with signalized intersections to control the signal phase. A model predictive control (MPC) scheme is proposed to allow AVs to adapt their speed. Several case studies that consider mixed autonomy are examined aiming at reducing network traffic and fuel consumption in the traffic network with multiple intersections. Simulation studies reveal that even with a few AVs in the network, the waiting time, fuel consumption, and the number of stop-and-go movements are significantly reduced, while the travel time is increased.

Real-Time multi-objective optimization of safety and mobility at signalized intersections

ABSTRACT Adaptive Traffic Signal Control (ATSC) is becoming a popular dynamic traffic management technique, especially with the emerging connected vehicles (CVs) technology. ATSC algorithms have been extensively considered in the literature for enhancing traffic mobility at signalized intersections. However, improving safety has rarely been used as an objective in existing ATSC algorithms. To fill this gap, this paper proposes a multi-criteria reinforcement learning based ATSC algorithm with two optimization objectives: real-time safety and mobility. The algorithm was trained on both objectives using traffic simulation. The safety objective was considered using extreme value theory (EVT) real-time crash risk evaluation models. Reducing the total intersection delay was the mobility objective. Different weights were considered in the training to account for both objectives simultaneously. The performance of the trained algorithm was then validated using real-world video data. Results show that the proposed multi-objective algorithm can improve both safety and mobility even under lower weights.

Adaptive Traffic Signal Control With Deep Reinforcement Learning and High Dimensional Sensory Inputs: Case Study and Comprehensive Sensitivity Analyses

Despite the constant rise in global urban populations and subsequent rise in transportation demand, significant expansion of infrastructure has been hampered by the constraints of space, cost, and environmental concerns. Therefore, optimizing the efficiency of existing infrastructure is becoming increasingly important. Adaptive traffic signal controllers aim to provide demand-responsive strategies to minimize motorists’ delay and achieve higher throughput at signalized intersections. With the advent of new sensory technologies and more intelligent control methods, the contribution of this paper is an adaptive traffic signal controller able to receive un-preprocessed high-dimensional sensory information such as GPS traces from connected vehicles and self-learn to minimize intersection delays. We use deep neural networks to operate directly on detailed sensory inputs and feed them into a reinforcement learning-based optimal control agent. The integration of these two components is known as deep learning. Using deep learning, we achieve two goals: (1) We eliminate the need for handcrafting a feature extraction process such as determining queue lengths, which is challenging and location-specific, and (2) we achieve better performance and faster training times compared to conventional tabular reinforcement learning approaches. We test our proposed controller against a tabular reinforcement learning agent, a reinforcement learning agent with a fully-connected Neural Network as a function approximator, and a state-of-practice, actuated traffic signal controller.

Real-Time Adaptive Traffic Signal Control in a Connected and Automated Vehicle Environment: Optimisation of Signal Planning with Reinforcement Learning under Vehicle Speed Guidance.

Adaptive traffic signal control (ATSC) is an effective method to reduce traffic congestion in modern urban areas. Many studies adopted various approaches to adjust traffic signal plans according to real-time traffic in response to demand fluctuations to improve urban network performance (e.g., minimise delay). Recently, learning-based methods such as reinforcement learning (RL) have achieved promising results in signal plan optimisation. However, adopting these self-learning techniques in future traffic environments in the presence of connected and automated vehicles (CAVs) remains largely an open challenge. This study develops a real-time RL-based adaptive traffic signal control that optimises a signal plan to minimise the total queue length while allowing the CAVs to adjust their speed based on a fixed timing strategy to decrease total stop delays. The highlight of this work is combining a speed guidance system with a reinforcement learning-based traffic signal control. Two different performance measures are implemented to minimise total queue length and total stop delays. Results indicate that the proposed method outperforms a fixed timing plan (with optimal speed advisory in a CAV environment) and traditional actuated control, in terms of average stop delay of vehicle and queue length, particularly under saturated and oversaturated conditions.

Adaptive Traffic Signal Control Based on Neural Network Prediction of Weighted Traffic Flow

Adaptive Traffic Signal Control Based on Neural Network Prediction of Weighted Traffic Flow

An Intelligent IoT Based Traffic Light Management System: Deep Reinforcement Learning

Traffic is one of the indispensable problems of modern societies, which leads to undesirable consequences such as time wasting and greater possibility of accidents. Adaptive Traffic Signal Control (ATSC), as a key part of Intelligent Transportation Systems (ITS), plays a key role in reducing traffic congestion by real-time adaptation to dynamic traffic conditions. Moreover, these systems are integrated with Internet of Things (IoT) devices. IoT can lead to easy implementation of traffic management systems. Recently, the combination of Artificial Intelligence (AI) and the IoT has attracted the attention of many researchers and can process large amounts of data that are suitable for solving complex real-world problems about traffic control. In this paper, we worked on the real-world scenario of Shiraz City, which currently does not use any intelligent method and works based on fixed-time traffic signal scheduling. We applied IoT approaches and AI techniques to control traffic lights more efficiently, which is an essential part of the ITS. Specifically, sensors such as surveillance cameras were used to capture real-time traffic information for the intelligent traffic signal control system. In fact, an intelligent traffic signal control system is provided by utilizing distributed Multi-Agent Reinforcement Learning (MARL) and applying the traffic data of adjacent intersections along with local information. By using MARL, our goal was to improve the overall traffic of six signalized junctions of Shiraz City in Iran. We conducted numerical simulations for two synthetic intersections by simulated data and for a real-world map of Shiraz City with real-world traffic data received from the transportation and municipality traffic organization and compared it with the traditional system running in Shiraz. The simulation results show that our proposed approach performs more efficiently than the fixed-time traffic signal control scheduling implemented in Shiraz in terms of average vehicle queue lengths and waiting times at intersections.

A Distributed Multi-Agent Reinforcement Learning With Graph Decomposition Approach for Large-Scale Adaptive Traffic Signal Control

With the emerging connected-vehicle technologies and smart roadways, the need for intelligent adaptive traffic signal controls (ATSC) is more than ever before. This paper first proposes an Accumulated Exponentially Weighted Waiting Time-based Adaptive Traffic Signal Control (AEWWT-ATSC) model to calculate priorities of roadways for signal scheduling. As the size of the traffic network grows, it adds great complexities and challenges to computational efficiencies. Considering this, we propose a novel Distributed Multi-agent Reinforcement Learning (DMARL) with a graph decomposition approach for large-scale ATSC problems. The decomposition clusters intersections by the level of connectivity (LoC), defined by the average residual capacities (ARC) between connected intersections, enabling us to train subgraphs instead of the entire network in a synchronized way. The problem is formulated as a Markov Decision Process (MDP), and the Double Dueling Deep Q Network with Prioritized Experience Replay is utilized to solve it. Under the optimal policy, the agents can select the optimal signal durations to minimize the waiting time and queue size. In evaluation, we show the superiority of the AEWWT-ATSC based RL methods in different densities and demonstrate the DMARL with a graph decomposition approach on a large graph in Manhattan, NYC. The approach is generic and can be extended to various types of use cases.

Dynamic capacity estimation of mixed traffic flows with application in adaptive traffic signal control

Intersection control plays a vital role in influencing transportation efficiency inside urban areas. Connected and Automated Vehicle (CAV) technology enables frequent traffic information sharing through vehicular networks, which emerges as a promising way to reduce vehicle traveling time and improve intersection capacity. Meanwhile, with the gradual deployment of CAVs, the traditional traffic flow consisting of human driven vehicles will evolve into a mixed traffic flow composed of traffic participants with differing intelligence capabilities. Considering the evolution towards the mixed traffic environment, we propose a dynamic capacity-aware traffic signal control method: max-pressure for mixed traffic flow (MPMF) traffic signal controller. Firstly, the lane capacity is modeled and approximated based on the saturation flow rate in mixed traffic flow considering the penetration rate of CAVs. Then the dynamic capacity is involved in calculating pressure in the max-pressure method to indicate the importance of a path in mixed traffic flow. A modification strategy for the max-pressure input value is also proposed. An isolated intersection scenario was simulated first to assess the proposed method locally. A multi-intersection network experiment was also conducted to verify the network-level performance of the proposed MPMF method. Comparative results between the proposed MPMF method, the classic max-pressure control, and the existing fixed time control method demonstrate that the MPMF can effectively improve the performance of intersections and be suitable for the multi-intersection road network.

An inductive heterogeneous graph attention-based multi-agent deep graph infomax algorithm for adaptive traffic signal control

Adaptive traffic signal control (ATSC) facilitates alleviating traffic congestion. Multi-agent deep reinforcement learning (MDRL) is a new promising algorithm for ATSC, and Graph Neural Networks (GNNs) further promote its learning ability. However, there are some drawbacks in the state-of-the-art MDRL algorithms. (1) These algorithms cannot effectively fuse diverse heterogeneous information of traffic networks due to adopting homogeneous GNNs; (2) These algorithms cannot be effectively trained due to merely adopting MDRL loss functions. In this paper, we propose an Inductive Heterogeneous graph Attention-based Multi-agent Deep Graph Infomax (IHA-MDGI) algorithm for ATSC. The proposed IHA-MDGI algorithm conducts both feature fusion via a proposed Inductive Heterogeneous graph Attention (IHA) algorithm and training via a proposed Multi-agent Deep Graph Infomax (MDGI) framework. Specifically, (1) Unlike the algorithms which adopt homogeneous GNNs, in the IHA algorithm, heterogeneous GNNs are designed to fuse both heterogeneous structural information and heterogeneous features of traffic networks, which aims to acquire heterogeneous information embeddings of traffic networks. (2) In the MDGI framework, the acquired embeddings are used to calculate the signal-control policies and Q-value for each agent, and then a mutual-information loss function is designed, which combines with the MDRL loss function to jointly train the whole algorithm. The designed mutual-information loss function focuses on maximizing mutual information between input (i.e., heterogeneous information embeddings) and output (i.e., Q-value), which can produce cooperative signal-control policies and maximize Q-value. We conduct the experiments in both real-traffic and synthetic-traffic networks under the time-varying traffic flows, and the results demonstrate that IHA-MDGI algorithm outperforms the state-of-the-art MDRL algorithms about multiple metrics.

Priority-Based Adaptive Traffic Signal Control System for Smart Cities

Priority-Based Adaptive Traffic Signal Control System for Smart Cities

A Novel Adaptive Traffic Signal Control Based on Cloud/Fog/Edge Computing

A Novel Adaptive Traffic Signal Control Based on Cloud/Fog/Edge Computing

Multi-Agent Transfer Reinforcement Learning With Multi-View Encoder for Adaptive Traffic Signal Control

Multi-agent reinforcement learning (MARL) based methods for adaptive traffic signal control (ATSC) have shown promising potentials to solve the heavy traffic problems. The existing MARL methods adopt centralized or distributed strategies. The former only models the environment as an agent and suffers from the exponential growth of action and state space. The latter extends the independent reinforcement learning methods, such as DQN, to multiple interactions directly or propagates information, such as state and policy, without taking their qualities into account. In this paper, we propose a multi-agent transfer reinforcement learning method to enhance the performance of MARL for ATSC, which is termed as multi-agent transfer soft actor-critic with the multi-view encoder (MT-SAC). The MT-SAC combines centralized and distributed strategies. In MT-SAC, we propose a multi-view state encoder and a transfer learning paradigm with guidance. The encoder processes input states from multiple perspectives and uses an attention mechanism to weigh the neighborhood information. While the paradigm enables the agents to handle different conditions for improving generalization abilities by transfer learning. Experimental studies on different scale road networks show that the MT-SAC outperforms the state-of-the-art algorithms and makes the traffic signal controllers more collaborative and robust.

Economic adaptive traffic signal system: design and operation

Economic adaptive traffic signal system: design and operation

A deep reinforcement learning-based cooperative approach for multi-intersection traffic signal control

Recently, Adaptive Traffic Signal Control (ATSC) in the multi-intersection system is considered as one of the most critical issues in Intelligent Transportation Systems (ITS). Among the proposed AI-based approaches, Deep Reinforcement Learning (DRL) has been largely applied while showing better performances. This paper proposes a new DRL-based cooperative approach for controlling multiple intersections. The problem is modelled as a Multi-Agent Reinforcement Learning (MARL) system, while each agent is trained to select the best action to control an intersection by obtaining information about its local lanes state. The cooperation aspect is manifested in this approach by considering the effect of the state, action and reward of neighbour agents in the process of policy learning. An intersection controller applies a Deep Q-Network (DQN) method, while transferring state, action and reward received from their neighbour agents to its own loss function during the learning process. The experimental results under different scenarios shows that the proposed approach outperforms many state-of-the-art approaches in terms of three metrics: Average Waiting Time (AWT), Average Queue Length (AQL) and Average Emission CO2 (AEC). In addition, the cooperation between the different trained DRL-based controllers allows the system to continuously learn and improve its performance by interacting with the environment, particularly when the traffic is congested.

Efficient Intersection Management Based on an Adaptive Fuzzy-Logic Traffic Signal

Traffic signals may generate bottlenecks due to an unfair timing balance. Facing this problem, adaptive traffic signal controllers have been proposed to compute the phase durations according to conditions monitored from on-road sensors. However, high hardware requirements, as well as complex setups, make the majority of these approaches infeasible for most cities. This paper proposes an adaptive traffic signal fuzzy-logic controller which uses the flow rate, retrieved from simple traffic counters, as a unique input requirement. The controller dynamically computes the cycle duration according to the arrival flow rates, executing a fuzzy inference system guided by the reasoning: the higher the traffic flow, the longer the cycle length. The computed cycle is split into different phases proportionally to the arrival flow rates according to Webster’s method for signalization. Consequently, the controller only requires determining minimum/maximum flow rates and cycle lengths to establish if–then mappings, allowing the reduction of technical requirements and computational overhead. The controller was tested through a microsimulation model of a real isolated intersection, which was calibrated with data collected from a six-month traffic study. Results revealed that the proposed controller with fewer input requirements and lower computational costs has a competitive performance compared to the best and most used approaches, being a feasible solution for many cities.

A review of reinforcement learning applications in adaptive traffic signal control

In urban areas, the problem of recurring daily congestion is constantly increasing. A possible solution is seen in the application of adaptive traffic signal control (ATSC) systems for the control of signalized intersections. While available ATSC systems can achieve an increase in the Level of Service, the focus of ATSC research has shifted towards the application of reinforcement learning (RL) techniques, which allow the controller to learn the optimal control policy by direct interaction with the environment. This paper describes the fundamentals of traffic signal control, RL algorithms and approaches and their application to ATSC, with a discussion on the impact of connected and autonomous vehicles on future traffic signal control. In conclusion, a summary of open research questions and possible directions for future research in the domain of RL-based traffic signal control is given.

Adaptive traffic signal management method combining deep learning and simulation

Adaptive traffic signal management method combining deep learning and simulation

Meta-learning based spatial-temporal graph attention network for traffic signal control

Traffic signal control is of great importance to the urban transportation systems and public travel, yet it becomes challenging because of two essential factors. First, spatial–temporal correlations are crucial to an intersection scenario. However, existing works have either considered only one of these features or a simple fusion of spatial and temporal information without adequately exploiting the potential correlations. Second, some works using graph neural network treats static graph nodes among adjacent intersections, ignoring the fact that intersection traffic is changing dynamically. These dynamically changing characteristics of an intersection are likewise significant for traffic signal prediction. If these problems are not resolved, the traffic pressure will increase and people’s time will be wasted.To resolve these challenges, we put forward a novel meta-learning spatial–temporal graph attention network (MetaSTGAT) for adaptive traffic signal control. Specifically, we design a graph neural framework with a graph attention network (GAT) and long short-term memory (LSTM) network to obtain spatial and temporal information. The spatial–temporal features are elaborately merged to improve its performance. Besides, to adapt the graph network to the dynamic traffic flow, i.e., the dynamics of the nodes, we propose a meta-learning method for graph neural network’s weights generation. This dynamic weight generation process captures the dynamic changes of graph nodes, and thus the dynamic changes of intersections. In this way, the neighboring nodes in the graph network get new weights in advance by additional features when they influence each other. Comprehensive experiments performed in the multi-intersection scenario on synthetic and real-world datasets demonstrate the effectiveness of MetaSTGAT against other state-of-the-art methods. Our method reduces travel time by 12.23%, 19.30%, 13.84%, 10.91%, 8.24%, and 8.74% over the graph-level method CoLight on four synthetic datasets and two real-world datasets, respectively.

CVLight: Decentralized learning for adaptive traffic signal control with connected vehicles

This paper develops a decentralized reinforcement learning (RL) scheme for multi-intersection adaptive traffic signal control (TSC), called “CVLight”, that leverages data collected from connected vehicles (CVs). The state and reward design facilitates coordination among agents and considers travel delays collected by CVs. A novel algorithm, Asymmetric Advantage Actor-critic (Asym-A2C), is proposed where both CV and non-CV information is used to train the critic network, while only CV information is used to execute optimal signal timing. Comprehensive experiments show the superiority of CVLight over state-of-the-art algorithms under a 2-by-2 synthetic road network with various traffic demand patterns and penetration rates. The learned policy is then visualized to further demonstrate the advantage of Asym-A2C. A pre-train technique is applied to improve the scalability of CVLight, which significantly shortens the training time and shows the advantage in performance under a 5-by-5 road network. A case study is performed on a 2-by-2 road network located in State College, Pennsylvania, USA, to further demonstrate the effectiveness of the proposed algorithm under real-world scenarios. Compared to other baseline models, the trained CVLight agent can efficiently control multiple intersections solely based on CV data and achieve the best performance, especially under low CV penetration rates.