Implicit Estimation of Another's Intention Based on Modular Reinforcement Learning

Abstract

When we try to accomplish a collaborative task, e.g., playing football or carrying large tables, we have to share a goal and a way of achieving the goal. Although people accomplish such tasks, achieveing such cooperation is not so easy in the context of a computational multi-agent learning system because participating agents cannot observe another person’s intention directly. We cannot know directly what other participants intend to do and how they intend to achieve that. Therefore, we have to notice another participant’s intention by utilizing other hints or information. In other words, we have to estimate another’s intention to accomplish collaborative tasks. In particular, in multi-agent reinforcement learning tasks, when another’s intention is unobservable the learning process is fatally harmed. When a participating agent of a collaborative task changes its intention and switches or modifies its controller, system dynamics for each agent will inevitably change. If other agents learn on the basis of simple reinforcement learning architecture, they cannot keep up with changes in the task environment because most reinforcement learning architectures assume that environmental dynamics are fixed. To overcome the problem, each agent must have a simple reinforcement learning architecture and some additional capability, which solves the problem. We take the capability of “estimation of another’s intention” as an example of such a capability. Human beings can perform several kinds of collaborative tasks. This means that we have some computational skills, which enable us to estimate another’s intention to some extent even if we cannot observe another’s intention directly. The computational model for implicit communication is described in this chapter on the basis of a framework of modular reinforcement learning. The computational model is called situation-sensitive reinforcement learning (SSRL), which is a type of modular reinforcement learning architecture. We assumed that such a distributed learning architecture would be essential for an autonomous agent to cope with a physically dynamic environment and a socially dynamic environment that included changes in another agent’s intentions. The skill, estimation of another’s intention, seems to be a social skill. However, human adaptability, which we believe our selves to be equipped with to deal with a physically dynamic environment, enables an agent to deal with such a dynamic social environment, including O pe n A cc es s D at ab as e w w w .in te ch w eb .o rg