Speech gestures and facial expression in speechreading

Abstract

Two experiments investigated the role of facial expression in speechreading. In Experiment 1 sentences were presented either with only the actor's face shown or with the actor shown in half-size. One happy script with happy, sad and neutral sentences was used, and either a content cue or an emotional cue was presented prior to each sentence. In Experiment 2 happy and sad sentences from one happy and one sad script were first recorded by the actor with a neutral face (face and shoulder shown) and then with a congruent facial expression. The results revealed that there was no difference in performance between half-size and face presentation, but content cues increased performance compared to emotional cues. As expected (Abelson, 1981), happy sentences were easier to speechread than sad in the happy script and vice versa. Also, facial expression raised performance for happy sentences in the happy script. Positive emotions are suggested to be associated with approach behavior in part relying on increased non-pantomimic speech gestures and articulation of the speaker (e.g., Davidson et al., 1990).