Mutual Adaptive Meaning Acquisition by Paralanguage Information: Experimental Analysis of Communication Establishing Process

Abstract

Mutual Adaptive Meaning Acquisition by Paralanguage Information: Experimental Analysis of Communication Establishing Process Takanori Komatsu (komatsu@cs.c.u-tokyo.ac.jp) Kentaro Suzuki (suzuki@cs.c.u-tokyo.ac.jp) Kazuhiro Ueda (ueda@gregorio.c.u-tokyo.ac.jp) Kazuo Hiraki (khiraki@idea.c.u-tokyo.ac.jp) Department of System Sciences, The University of Tokyo 3-8-1 Komaba, Meguro-ku, Tokyo, 153-8902 JAPAN Natsuki Oka(oka@mrit.mei.co.jp) Advanced Technology Research Laboratories, Matsushita Electric Industrial Co., Ltd. 3-4 Hikaridai, Seika, Souraku, Kyoto 619-0237 JAPAN Abstract The effects of adjustments in teaching strategy and of paralanguage information in speech sound on the mean- ing acquisition process were determined by means of an experiment in which the game “Pong” was played by a team of two subjects. One subject (the teacher) coached the other one (the operator), instructing the operator in which direction to move the game paddle and when to hit the “ball.” However, the teacher’s speech was ren- dered linguistically incomprehensible. Three phenomena were observed. First, the use of a high-pitched voice by the teacher caused the operator to pay more attention to her/his actions. Second, meaning acquisition could be regarded as a reinforcement learning process based on a multi-reward system (i.e., one for successful game action and a different one from the teacher’s high-pitched voice for the wrong action). Third, the subjects adapted to each other; that is, they learned to respond more appropriately to each other’s behaviors (we call this mutual adapta- tion). These three phenomena are thought to play impor- tant roles in the acquisition of meaning from incompre- hensible speech. Introduction How do people who speak different languages learn to communicate? How do they acquire the meanings of each other’s speech? One way to interpret the meaning- acquisition process is to view it as statistical learning in which a certain speech sound is linked to the situ- ation in which it was given. Several research groups have constructed general meaning-acquisition models based on this simple interpretation (for instance, Siskind, 1996). Testing of these models demonstrated that the word meanings can be acquired using statistical learning methodologies. Some groups have investigated the teaching of word meanings to robot agents, which can move by themselves (Billard et al., 1998; Roy, 1999). For example, Kaplan (2000) showed that a four-legged robot could learn the names of a dozen objects shown in front of its “eyes” (camera) by the experimenter during its action experi- ences. One study in particular demonstrated that an au- tonomous agent could learn not only the meanings of instruction words but also the meanings of evaluation in- structions such as “good” and “bad”. Suzuki et al. (2002) developed a learning agent model that could learn the meanings of words and evaluation instructions during its action experiences. A large variety of learner models have thus been demonstrated. Siskind’s learner model was a computer that processed string inputs; Kaplan’s was a four-legged robot that could learn object names; and Suzuki’s was a computer agent that could move by itself in a virtual en- vironment. In contrast, only one type of “teacher” has been demonstrated, one who gave instruction based on unchangeable rules. In real life, however, good teach- ers adjust their teaching strategy to fit the learner’s mode of learning. For example, a caregiver will speak to a preverbal infant using only simple words. Then, when the infant starts to speak, the caregiver will start to use more complex words and speak in different ways. So far, there have been no studies on how dynamically adjust- ing the teaching strategy affects the meaning acquisition process. Investigating the process involved in acquiring mean- ings from speech sounds requires investigating the ef- fects of not only phoneme information, which can be expressed using characters and text, but also of paralan- guage information, such as prosody, speech speed and loudness, which cannot be expressed using characters and text. The effects of paralanguages information have been studied in various acoustic studies. For example, many studies have focused on the turn-taking mechanism (Pirrehumbert & Hirschberg, 1990) and emotional recog- nition (Hirose et al., 1997). However, so far, there have been none on the effect of paralanguage information on the meaning acquisition process. We investigated the effects of adjustments in teach- ing strategy and of paralanguage information in speech sounds on the meaning acquisition process. We carried out a communication experiment, in which a team of two subjects played “Pong”. One subject (the teacher) coached the other one (the operator), instructing her/him in which direction to move the game paddle and when to hit the “ball.” However, the teacher’s speech was ren- dered linguistically incomprehensible. We observed how the listener acquired the meanings of the given instruc- tions. For the paralanguage information, we focused on prosodic information, because data extraction should be easier and the possibility of the engineering realization is higher. The results provide a new point of view about meaning acquisition process and can be used to develop