Effects of hand gesture and lip movements on auditory learning of second language speech sounds

Abstract

Previous research found that auditory training helps native English speakers to perceive phonemic vowel length distinction in Japanese, but that their performance has never reached native levels (Hirata et al., 2007). Given that multimodal information, such as hand gesture and lip movements, influences semantic aspects of language processing and development (Kelly et al., 2002), we examined whether this multimodal information helps to improve native English speakers' ability to perceive Japanese vowel length distinction. Forty-five native English speakers participated in one of three types of training: (1) audio alone; (2) audio with hand gestures; and (3) audio with lip movements and hand gestures. Before and after training, participants were given phoneme perception tests that measured their ability to distinguish between short and long vowels in Japanese, e.g., /kato/ versus /kato:/. Our original prediction was that more modalities in training would result in greater learning. Although all three groups improved from pre- to post-test, there were no significant differences among the three training groups. Unlike the original prediction, hand gestures and lip movements did not seem to augment learning of difficult sound distinctions. We will discuss possible benefits and limitations of using multimodal information in second language speech learning.