Impact of Deep Reinforcement Learning on Variable Speed Limit strategies in connected vehicles environments

Abstract

The Variable Speed Limit (VSL) control is considered in the context of connected vehicles acting as moving sensors, while their obedience to speed limit is enforced by a mandatory Intelligent Speed Adaptation (ISA) system. The objective of this study is to extend spatially static differential VSL control by introducing novel VSL strategies that are based on spatially dynamic speed limit zones. The spatial configuration of speed limit zones requires a novel traffic state representation based on the set of consecutive matrices which encode each vehicle position and speed within the controlled motorway during the control time step. The actions for all proposed VSL strategies are computed by the same Deep Reinforcement Learning (DRL) approach based on the Deep Deterministic Policy Gradient (DDPG) architecture. The DDPG learning models contain integration of Convolution and Long Short-Term Memory (LSTM) known as ConvLSTM layers, along with the Convolution and Fully Connected layers. Thus, those models can learn complex spatio-temporal traffic dynamics based on proposed traffic state representation. The results show that proposed VSL strategies have achieved higher overall motorway throughput compared to one which is based on the static speed limit zones and baseline cases (no-control and Simple Proportional Speed Controller algorithm). Simultaneously, they achieved a minimal increase in the number of aggressive braking, while the average headway is increased. All proposed VSL control approaches are simulated and analyzed by using a synthetic microscopic motorway model and characteristic traffic scenario for urban motorways.