Abstract
Emotion is communicated via the integration of concurrently presented information from multiple information channels, such as voice, face, gesture and touch. This study investigated the neural and perceptual correlates of emotion perception as influenced by facial and vocal information by measuring changes in oxygenated hemoglobin (HbO) using functional near-infrared spectroscopy (fNIRS) and acquiring psychometrics. HbO activity was recorded from 103 channels while participants (hbox {n} = 39, hbox {age} = 20.37 hbox {years}) were presented with vocalizations produced in either a happy, angry or neutral prosody. Voices were presented alone or paired with an emotional face and compared with a face-only condition. Behavioral results indicated that when voices were paired with faces, a bias in the direction of the emotion of the voice was present. Subjects’ responses also showed greater variance and longer reaction times when responding to the bimodal conditions when compared to the face-only condition. While both the happy and angry prosody conditions exhibited right lateralized increases in HbO compared to the neutral condition, these activations were segregated into posterior-anterior subdivisions by emotion. Specific emotional prosodies may therefore differentially influence emotion perception, with happy voices exhibiting posterior activity in receptive emotion areas and angry voices displaying activity in anterior expressive emotion areas.
Highlights
Emotion is communicated via the integration of concurrently presented information from multiple information channels, such as voice, face, gesture and touch
Findings from functional magnetic resonance imaging and electroencephalography (EEG) have shown that the posterior superior temporal sulcus exhibits increased activity when viewing lip, eye, and cheek movements[10,11,12,13,14,15]. These dynamic facial features are essential to all aspects of speech production as findings from EEG and fMRI have shown that the pSTS exhibits increased activity when viewing dynamic faces, listening to affective voices[11,16,17], and during silent lip reading[18]
Findings from lesion studies in aprosodia patients have shown that damage to structural homologues of Wernicke’s and Broca’s area in the right hemisphere produce deficits in comprehending and reproducing prosody similar to those seen in patients with receptive and expressive a phasias19,20. fMRI studies have shown that these areas may possess distinct representations of emotion as areas of the pSTS and superior temporal gyrus (STG) exhibit unique, emotion-specific patterns of activation when presented with nonverbal affective voices[21,22,23,24]
Summary
Emotion is communicated via the integration of concurrently presented information from multiple information channels, such as voice, face, gesture and touch. The pSTS appears to play an essential role in audiovisual speech perception by integrating the emotional content of a speaker’s voice with their concomitant facial expression to aid in speaker identification and affect r ecognition[26] Together, these findings indicate that socially relevant audiovisual stimuli converge and engage partially overlapping regions in the pSTS/G1,27–29. The neural correlate of this supramodal representation appears to occur in the pSTS, wherein patterns of activity change between emotions, but not between different sensory modalities conveying an emotion[27,32,33] This emotional specificity can even detect incongruencies between affective vocal and facial input, as the pSTS exhibits increased activity to mismatched happy and angry stimuli[34]. This overlap may represent the common engagement of several neural structures, which support the perception, experience, and expression of emotion[35]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
