Transferring Generalizable Motor Primitives From Simulation to Real World

Abstract

Reinforcement learning provides robots with an autonomous learning framework where a skill can be learned by exploration. Exploration in real world is, however, inherently unsafe and time consuming, and causes wear and tear. To address these, learning policies in simulation and then transferring them to physical systems has been proposed. In this letter, we propose a novel sample-efficient transfer approach, which is agnostic to the dynamics of a simulated system and combines it with incremental learning. Instead of transferring a single control policy, we transfer a generalizable contextual policy generated in simulation using one or few samples from real world to a target global model, which can generate policies across parameterized real-world situations. We studied the generalization capability of the incremental transfer framework using MuJoCo physics engine and KUKA LBR 4+. Experiments with ball-in-a-cup and basketball tasks demonstrated that the target model improved the generalization capability beyond the direct use of the source model indicating the effectiveness of the proposed framework. Experiments also indicated that the transfer capability depends on the generalization capability of the corresponding source model, similarity between source and target environment, and number of samples used for transferring.