Deep Imitation Learning with Memory for Robocup Soccer Simulation

Abstract

Imitation learning is a field that is rapidly gaining attention due to its relevance to many autonomous agent applications. Providing demonstrations of effective behaviour to teach the agent is useful in real world challenges such as sparse rewards and dynamic environments. However, most imitation learning approaches don’t retain a memory of previous actions and treat the demonstrations as independent and identically distributed samples. This neglects the temporal dependency between low-level actions that are performed in sequence to achieve the desired behaviour. This paper proposes an imitation learning method to learn sequences of actions by utilizing memory in deep neural networks. Long short-term memory networks are utilized to capture the temporal dependencies in a teacher’s demonstrations. This way, past states and actions provide context for performing following actions. The network is trained using raw low-level features and directly maps the input to low-level parametrized actions in real-time. This minimizes the need for task specific knowledge to be manually employed in the learning process compared to related approaches. The proposed methods are evaluated on a benchmark soccer simulator and compared to supervised learning and data-aggregation approaches. The results show that utilizing memory while learning significantly improves the performance and generalization of the agent and can provide a stationary policy than can produce robust predictions at any point in the sequence.