Local Update Dynamic Policy Programming in reinforcement learning of pneumatic artificial muscle-driven humanoid hand control

Abstract

Pneumatic Artificial Muscle (PAM) is an attractive device to be used as an actuator for humanoid robots because of its high power-to-weight ratio and good flexibility. However, both the modeling and the controlling of PAM-driven robots are challenging due to the high nonlinearities of a PAM's air pressure dynamics and its mechanical structure. This paper focuses on applying Reinforcement Learning (RL) to the control of a PAM-driven robots without our knowledge of its model. We propose a new RL algorithm, Local Update Dynamic Policy Programming (LUDPP), as an extension of Dynamic Policy Programming (DPP). This algorithm exploits the nature of smooth policy update of DPP to considerably reduce the computational complexity in both time and space: at each iteration, this algorithm only updates the value function locally throughout the whole state-action space. We applied LUDPP to control one finger (2 DOFs with a 12-dimensional state-action space) of Shadow Dexterous Hand, a PAM-driven humanoid robot hand. Experimental results suggest that our method can achieve successful control of such a robot with a limited computational resource whereas other conventional value function based RL algorithms (DPP, LSPI) cannot.