Residual Learning From Demonstration: Adapting DMPs for Contact-Rich Manipulation

Todor Davchev,Kevin Sebastian Luck,Michael Burke,Franziska Meier,Subramanian Ramamoorthy,Stefan Schaal

doi:10.1109/lra.2022.3150024

Abstract

Manipulation skills involving contact and friction are inherent to many robotics tasks. Using the class of motor primitives for peg-in-hole like insertions, we study how robots can learn such skills. Dynamic Movement Primitives (DMP) are a popular way of extracting such policies through behaviour cloning (BC) but can struggle in the context of insertion. Policy adaptation strategies such as residual learning can help improve the overall performance of policies in the context of contact-rich manipulation. However, it is not clear how to best do this with DMPs. As a result, we consider several possible ways for adapting a DMP formulation and propose ``residual Learning from Demonstration`` (rLfD), a framework that combines DMPs with Reinforcement Learning (RL) to learn a residual correction policy. Our evaluations suggest that applying residual learning directly in task space and operating on the full pose of the robot can significantly improve the overall performance of DMPs. We show that rLfD offers a gentle to the joints solution that improves the task success and generalisation of DMPs \rb{and enables transfer to different geometries and frictions through few-shot task adaptation}. The proposed framework is evaluated on a set of tasks. A simulated robot and a physical robot have to successfully insert pegs, gears and plugs into their respective sockets. Other material and videos accompanying this paper are provided at https://sites.google.com/view/rlfd/.

Highlights

Contributions The key contributions of this paper are: C1 An extensive comparison of the adaptation of different parts of the Dynamic Movement Primitives (DMP) formulation using a range of adaptation and exploration strategies for contact-rich insertion; C2 A framework for applying full pose residual learning on DMPs applied directly in task space, and the demonstration of its utility to three types of physical insertion tasks; C3 Showing that using full-pose residual nonlinear policies (e.g. Reinforcement Learning (RL)-driven) to adapt DMPs results in more accurate, gentle and more generalisable DMP solutions
It remains unclear how best to do so for contact-rich insertion tasks. This paper explores this and finds that a framework bridging reinforcement learning and DMP learning from demonstration in task space is most effective when paired with orientation aware corrections
Results show that residual Learning from Demonstration“ (rLfD) allows for successful policy transfer on both tasks, requiring eight times less training