Combining Facial Expression and Touch for Perceiving Emotional Valence

Abstract

We conducted an experimental study to investigate how the participants combine touch and facial expression to evaluate emotional valence. In this study, visual and haptic stimuli were presented separately and then presented together. The visual stimuli comprised pictures of facial expressions with different emotional levels, and the touch stimuli consisted of air jet tactile stimulation performed on the arms of the participants. The participants were asked to evaluate the communicated emotional valence on a continuous scale. Information Integration Theory was used to model the algebraic rule (addition, multiplication, averaging, and so on) that underlies the multimodal perception of emotional valence. Analyses showed that the participants usually integrated both visual and touch information to evaluate the emotional valence. The main integration rule was averaging, with nonsystematic predominance of each modality over the other modality.