Learning of Joint Visual Attention by Reinforcement Learning

Abstract

In this paper, we propose a neural network model of joint visual attention learning that plays an important role in infant development, and we discuss previous studies of experimental psychology on joint visual attention based on simulation results using the model. We assumed an imaginary experiment to develop the model. A mother and an infant are sitting face to face with a table between them. Some objects familiar to the infant are placed on the table, and toys operated by remote control are put outside of the infant’s view. The infant is given a reward of seeing something interesting only when the infant follows the mother’s gaze after eye contact. We constructed the model of this experiment with a reinforcement learning algorithm, and simulated the experiment on a computer. As a result, it was revealed that the infant could learn a series of joint-visual-attention-like actions by receiving rewards from an environment, although it initially has little knowledge of the environment. This result suggests that infants can acquire joint visual attention without comprehension of the nature of joint attention, i.e., ”I’m looking at the same thing that others are looking at.” Introduction Modeling the development of infant intelligence is one of the strategies for understanding human intelligence. We focus on development in infancy from the viewpoint of engineering. Neonatal babies have little knowledge of the environment, nevertheless they acquire new knowledge and behavior suitable for the environment step by step in their developmental stage. Although the whole brain system of adults is very complicated, we believe that we can create a model of intelligence relatively easily by pursuing those developmental steps one by one. In this study, we focus on joint visual attention as one of those developmental processes. In an engineering sense, joint attention can be defined as the sharing of attention with others, and joint visual attention is defined as looking at what others are looking at. Although this definition may cause some objections, we adopt it in this paper. The detailed study of joint visual attention began with Scaife and Bruner’s work (1975). They observed that a child in early and middle infancy follows an adult’s gaze, and stated that this behavior is an important factor in early development. However, it is not yet clear how we acquire joint visual attention. There are two theories, nature and nurture, at present (Baron-Cohen, 1995; Butterworth & Jarrett, 1991; Corkum & Moore, 1995). In this paper, we propose an engineering model of joint visual attention learning by conditioning with signals from the environment, and examine this behavior by computer simulation. Based on the results, we discuss the requirements of fundamental parts that are necessary for such learning. Behavior Acquisition by Reinforcement Learning Imaginary Experiment We contrived the following imaginary experiment for our study based on the behavioral experiment of Corkum and Moore (1995). A mother and an infant are sitting face to face with a table between them. Some objects familiar to the infant are placed on the table. In the early stage of learning, the infant randomly directs its attention to the objects including the mother’s face. The mother, however, is always gazing at the infant’s eyes. Toys are set outside of the infant’s view, and the observer can operate them by remote control (Figure 1). When the infant looks at the mother’s face and they make eye contact, the mother moves her eyes to gaze toward any one of the toys. Furthermore, when the infant follows the direction of her gaze , the observer operates the toy, arousing pleasure in the infant’s mind. In other words, the infant can obtain a reward of seeing something interesting only when it perform the two consecutive actions of looking at the mother’s face and following her gaze. Temporal Difference Learning In this study, we used the temporal difference (TD) method (Sutton & Barto, 1998) for the learning of joint visual attention. TD learning is an algorithm that learns the value function V (st) of each state st based on a reward r that is given later from the environment. An agent learns behavior strategy so that the value function may increase (0 < γ < 1).