Lane-changing system based on deep Q-learning with a request–respond mechanism

Abstract

Lane-changing control is a crucial task to regulate the traffic flow in the intelligent transportation system efficiently, and manipulating the lane-changing maneuvers for many vehicles optimally and simultaneously in a congested traffic environment is a challenge. This study formulates a lane-changing model where the traffic lanes are discretized into cells, considers both mandatory lane-changing and discretionary lane-changing maneuvers and proposes a novel approach based on a deep Q-network with a request–respond mechanism. The modeled lane-changing system is divided into isolated and symmetry groups, then independent Q-learning and the concept “central agent” are integrated to simplify the training process. In the proposed approach, these groups can be classified into two categories: request group and respond group. The request group and respond group are trained separately during the training process. Specifically, the request group trains agents with only considering the states of the group, while the respond group also evaluates the superimposing actions (i.e., the request message) from the request group besides the states of the group. The execution process is also treated the same decentralized way as the training process. Then, the baselines such as a rule-based method, a game theory-based decomposition algorithm and a simple deep Q-learning method are compared with the proposed approach. Results reveal that the proposed approach performs as well as the game theory-based decomposition algorithm and outperforms the other two in terms of lane-changing efficiency. However, the computational time of the proposed approach in the execution process is far less than the game theory-based decomposition algorithm, especially in a congested traffic environment.