On the Explainability and Expressiveness of Function Approximation Methods in RL-Based Traffic Signal Control

Abstract

With fast increasing urbanization levels, adaptive traffic signal control methods have great potential for optimizing traffic jams. In particular, deep reinforcement learning (RL) approaches have been shown to be able to outperform classic control methods. However, deep RL algorithms are often employed as black boxes, which limits their use in the real-world as the decisions made by the agents can not be properly explained. In this paper, we compare different function approximations methods used to estimate de action-value function of RL-based traffic controllers. In particular, we compare (i) their expressiveness, based on the resulting performance of the learned policies, and (ii) their explainability capabilities. To explain the decisions of each method, we use Shapley Additive Explanations (SHAP) to show the impact of the agent's state features on each possible action. This allows us to explain the learned policies with a single image, enabling an understanding of how the agent behaves in the face of different traffic conditions. In addition, we discuss the application of post-hoc explainability models in the context of adaptive traffic signal control, noting their potential and pointing out some of their limitations. Comparing our resulting methods to state-of-the-art adaptive traffic signal controllers, we saw significant improvements in travel time, speed score, and throughput in two different scenarios based on real traffic data.