Control of a nonlinear active suspension system based on deep reinforcement learning and expert demonstrations

Abstract

A well-controlled active suspension system has the potential to provide better ride comfort. Benefiting from its powerful feature extraction and nonlinear generalization capabilities, the deep reinforcement learning (DRL), such as deep deterministic policy gradient (DDPG), has shown great potential to make decisions adaptively and intelligently in the control of active suspension system. However, the DDPG is troubled by the problem of low training efficiency due to the high proportion of illegal strategies. This paper proposed a novel DDPG controller for a nonlinear uncertain active suspension system by combining DRL with expert demonstrations. Specifically, the improved training method integrated with both a pre-training mechanism based on PID expert samples and an adaptive experience replay mechanism, is put forward for the DDPG to achieve both the goals of imitating the expert and improving the training efficiency. Moreover, considering the ride comfort and the state constraints as targets, a mixed reward function is designed to guide RL agents for learning effective actions. It is shown that the proposed training methods effectively accelerate the convergence of the DDPG. Furthermore, the comparison experiments demonstrate that the proposed controller provides great vibration attenuation, and has better adaptiveness to various working conditions and parametric uncertainty.