Automated Driving Highway Traffic Merging using Deep Multi-Agent Reinforcement Learning in Continuous State-Action Spaces

Abstract

Achieving the highest levels of automated driving will require effective solutions to the key challenging maneuver of highway on-ramp merging. This paper extends our previous work on a multi-agent reinforcement-learning (MARL) approach to study the problem of highway on-ramp merging, with particular emphasis on the study of the behavior of the vehicle that is on the on-ramp with approaching traffic. Our previous model was based on a discretized space of states and actions. Here, we present results on a more sophisticated model based on a continuous space of states and actions. We exploit recent advances on deep reinforcement learning (deep RL) to train controllers for this task in an idealized environment using an implementation of our MARL approach. We specifically employ artificial neural network architectures for policy and function approximation within our multiagent Q-learning approach. We show the effectiveness of our trained controllers by demonstrating their collision-avoidance performance on interaction scenarios with different in-traffic behavior. We compare their performance to those obtained using a similar deep RL single-agent approach. We argue why the resulting MARL-based controllers are essentially optimal within the context, conditions, and parameters of the evaluation environment that we employ and our previously established fundamental performance limitations governing the highway on-ramp merging maneuver.