Comparing actor-critic deep reinforcement learning controllers for enhanced performance on a ball-and-plate system

Abstract

The optimization of controller parameters remains an ongoing challenge in the field of control system applications. This study introduces a novel approach involving the creation of custom actor-critic deep reinforcement learning (DRL) based PID controllers. These controllers are designed with the goal of achieving adaptive tuning, precise trajectory tracking, and stability in a ball-and-plate system. To achieve this objective, multiple actor-critic reinforcement learning agents were developed using different learning algorithms: Soft actor critic (SAC), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3). These agents incorporate multilayer-perceptron (MLP) policy learning algorithms in both the actor and critic network architectures, employing non-linear activation functions. This enables them to fine-tune PID control parameters within an infinite search space. Additionally, a custom reward function derived from the system’s environment was integrated into the learning process. The performance of the proposed methods was compared against a benchmark method, specifically, an existing deep reinforcement learning controllers reported in the literature. The evaluation of these controllers and other approaches was based on error metrics and time response analysis. Results demonstrate that the proposed controller denoted as SAC-PID(5) excelled in trajectory tracking and outperformed other methods. It exhibited minimal predictive errors and the shortest time responses in the majority of experiments. This highlights the significance of designing a customized SAC agents with appropriate network architecture, which positively impacts the learning process for intelligent tuning of controllers for classical control systems.