Deep reinforcement learning with reward shaping for tracking control and vibration suppression of flexible link manipulator

Abstract

This paper puts forward a novel deep reinforcement learning control using deep deterministic policy gradient (DRLC-DDPG) framework to address the reference tracking and vibration suppression problem of rotary flexible link (RFL) manipulator. Specifically, this study attempts to address the continuous action space DRLC problem through DDPG algorithm and presents a Lyapunov function based reward shaping approach for guaranteed deep reinforcement learning (DRL) convergence and enhanced speed of training. The proposed approach synthesizes the hard and soft constraints of the flexible manipulator as a constrained Markov decision problem (MDP) and evaluates the performance of DRLC-DDPG framework through hardware in loop (HIL) testing to realize precise servo tracking and suppressed vibration of the flexible manipulator. For identifying the dynamical model of the RFL, an empirical Auto-Regressive eXogenous (ARX) model using the closed loop identification technique is built. Moreover, to extract the true states (servo angle and deflection angle) from the actual measurements, which typically have the influence of sensor noise, an adaptive Kalman filter (AKF) is augmented with the DRLC scheme. The experimental results of DRLC-DDPG scheme compared with those of the model predictive control (MPC) for several test cases reveal that the proposed scheme is superior to MPC both in terms of trajectory tracking and robustness against the external disturbances and model uncertainty.