Transferable traffic signal control: Reinforcement learning with graph centric state representation

Abstract

Reinforcement learning (RL) has emerged as an alternative approach for optimizing the traffic signal control system. However, there is a restricted exploration problem encountered when a signal control model is trained with a predefined demand scenario in the traffic simulation. With the restricted exploration, the model learns a policy based only on partial experiences in the search space, which yields a partially-trained policy. Partially-trained policy fails to adapt to some unexperienced (‘unexplored’, ‘never-before-seen’) dataset that have different distributions from the training dataset. Although this issue has critical effects on training a signal control model, it has not been considered in the literature. Therefore, this research aims to obtain a transferable policy to enhance the model’s applicability on unexperienced traffic states. The key idea is to represent the state as graph-structured data, and train it using a graph neural network (GNN). Since this approach enables to learn the relationship between the features resulting from the spatial structure of the intersection, it is able to transfer the already-learned knowledge of the relationship to the unexperienced data. In order to investigate the transferability, an experiment is conducted on five unexperienced test demand scenarios. For the evaluation, the performance of the proposed GNN model is compared with the conventional DQN model that is based on vector-valued state. At first, the models are trained with only a single dataset (training demand scenario). Then, they are tested with different unexperienced dataset (test demand scenarios) without additional trainings. The results show that the proposed GNN model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional DQN model fails.