Generating merging strategies for connected autonomous vehicles based on spatiotemporal information extraction module and deep reinforcement learning

Abstract

A major challenge concerning a mixed traffic flow system, composed of connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs), is how to improve overall efficiency and safety by assigning appropriate control strategies to CAVs. Deep reinforcement learning (DRL) is a promising approach to address this challenge. It enables the joint training of multiple CAVs by fusing CAV sensing information and does not need compliance of HDVs. However, the fusion of CAV sensing information is non-trivial. Traditional DRL models usually fail to take advantage of connectivity among CAVs and time series characteristics of vehicle sensing information, leading to insufficient awareness of the traffic environment. Aimed at tackling these issues, this study proposes a DRL framework named spatiotemporal deep Q network (STDQN), by integrating a double deep Q network (DDQN) and a spatiotemporal information extraction module. A long–short term memory neural network with an attention mechanism (AttenLSTMNN) is leveraged to extract temporal dependencies from vehicle perceptive information. In addition, a graph convolution network (GCN) is employed to model the spatial correlations among vehicles in a local range, as well as the connectivity of multiple CAVs in a global range. Simulation experiments are conducted in an onramp merging scenario, which is one of the most important and commonly seen scenarios in highway or city expressway systems. Experimental results prove that as compared to baseline DRL and rule-based methods, the proposed STDQN can improve the overall traffic efficiency, safety, and driving comfort. The proposed framework is promised to be deployed into real CAVs, to realize cooperative, safe, and efficient autonomous driving.