Graph and dynamics interpretation in robotic reinforcement learning task

Abstract

Robot control tasks are typically solved by reinforcement learning approaches in a circular way of trial and learn. A recent trend of the research on robotic reinforcement learning is the employment of the deep learning methods. Existing deep learning methods achieve the control by training the approximation models of the dynamic function, value function or the policy function in the control algorithms. However, these methods usually handle the modeling from a statistical perspective without considering the physics characteristics of the robot’s motion. One of the typical problems is the force transmission through different parts of a robot and the calculations of the robotic dynamics quantities are prone to be ignored. To this end, we propose to use the force transmission graph to interpret the force transmission mechanism obeyed by the motion of the robot and estimate the dynamics quantities of the robot’s motion with a quadratic model. Following this concern, we propose a model-based reinforcement learning framework for robotic control in which the dynamic model comprises two components, i.e. the Graph Convolution Network (GCN) and the Two-Layer Perception (TLP) network. The GCN serves as a parameter estimator of the force transmission graph and a structural feature extractor. The TLP network approximates the quadratic model that should be able to estimate the dynamics quantities of the robot’s motion. For this reason, the proposed framework is named as GCN of Dynamics estimation in Reinforcement Learning method (GDRL for short). The deployed method interprets the intrinsic mechanism of robotic force transmissions through robot limbs, therefore the model is highly interpretable. Experimental results show that GDRL can predict the gesture and location of the robot for the next move well such that the performance of our method surpasses that of the previous methods in robot control task in our task setting of multiple types of robots. We also designed to compare with the previous model-free methods in our task setting, and the results are outstanding, which is owing to the interpretation of the physical characteristics.