Automated function development for emission control with deep reinforcement learning

Abstract

The conventional automotive development process for embedded systems today is still time- and data-inefficient, and requires highly experienced software developers and calibration engineers. Consequently, it is cost-intensive and at the same time prone to sub-optimal solutions. Reinforcement Learning offers a promising approach to address these challenges. The evolved agents have proven their ability to master complex control tasks in a close-to-optimal manner without any human intervention, but the training procedures are hardly compatible with current development processes. As a result, Reinforcement Learning has rarely been used in powertrain development until now. This work describes an integration of Reinforcement Learning in the embedded system development process to automatically train and deploy agents in transient driving cycles. Using the example of exhaust gas re-circulation control for a Diesel engine, an agent is successfully trained in a fully virtualized environment, achieving emission reductions of up to 10% in comparison to a state-of-the-art controller. Further investigations are carried out to quantify the impact of the driving cycle and ambient conditions on the agent’s performance. To demonstrate the transferability between different levels of virtualization, the experienced agent is then tested in closed-loop with a real hardware controller to operate the physical actuator. By confirming the reproducibility of the learned strategy on real hardware, this article serves as proof-of-concept for a sustainable, Reinforcement Learning based path to automatically develop embedded controllers for complex control problems.