Face-voice Stimuli Research Articles

Confrontation with others' emotions changes bodily resonance differently in those with low and high levels of depersonalization.

We present novel research on the cortical dynamics of atypical perceptual and emotional processing in people with symptoms of depersonalization-derealization disorder (DP-DR). We used electroencephalography (EEG)/event-related potentials (ERPs) to delineate the early perceptual mechanisms underlying emotional face recognition and mirror touch in adults with low and high levels of DP-DR symptoms (low-DP and high-DP groups). Face-sensitive visual N170 showed markedly less differentiation for emotional versus neutral face-voice stimuli in the high- than in the low-DP group. This effect was related to self-reported bodily symptoms like disembodiment. Emotional face-voice primes altered mirror touch at somatosensory cortical components P45 and P100 differently in the two groups. In the high-DP group, mirror touch occurred only when seeing touch after being confronted with angry face-voice primes. Mirror touch in the low-DP group, however, was unaffected by preceding emotions. Modulation of mirror touch following angry others was related to symptoms of self-other confusion. Results suggest that others' negative emotions affect somatosensory processes in those with an altered sense of bodily self. Our findings are in line with the idea that disconnecting from one's body and self (core symptom of DP-DR) may be a defence mechanism to protect from the threat of negative feelings, which may be exacerbated through self-other confusion. This article is part of the theme issue 'Sensing and feeling: an integrative approach to sensory processing and emotional experience'.

Simultaneous face and voice processing in schizophrenia

While several studies have consistently demonstrated abnormalities in the unisensory processing of face and voice in schizophrenia (SZ), the extent of abnormalities in the simultaneous processing of both types of information remains unclear. To address this issue, we used event-related potentials (ERP) methodology to probe the multisensory integration of face and non-semantic sounds in schizophrenia.EEG was recorded from 18 schizophrenia patients and 19 healthy control (HC) subjects in three conditions: neutral faces (visual condition-VIS); neutral non-semantic sounds (auditory condition-AUD); neutral faces presented simultaneously with neutral non-semantic sounds (audiovisual condition-AUDVIS).When compared with HC, the schizophrenia group showed less negative N170 to both face and face–voice stimuli; later P270 peak latency in the multimodal condition of face–voice relative to unimodal condition of face (the reverse was true in HC); reduced P400 amplitude and earlier P400 peak latency in the face but not in the voice–face condition.Thus, the analysis of ERP components suggests that deficits in the encoding of facial information extend to multimodal face–voice stimuli and that delays exist in feature extraction from multimodal face–voice stimuli in schizophrenia. In contrast, categorization processes seem to benefit from the presentation of simultaneous face–voice information. Timepoint by timepoint tests of multimodal integration did not suggest impairment in the initial stages of processing in schizophrenia.

Multisensory perception of the six basic emotions is modulated by attentional instruction and unattended modality.

Previous studies have shown that the perception of facial and vocal affective expressions interacts with each other. Facial expressions usually dominate vocal expressions when we perceive the emotions of face–voice stimuli. In most of these studies, participants were instructed to pay attention to the face or voice. Few studies compared the perceived emotions with and without specific instructions regarding the modality to which attention should be directed. Also, these studies used combinations of the face and voice which expresses two opposing emotions, which limits the generalizability of the findings. The purpose of this study is to examine whether the emotion perception is modulated by instructions to pay attention to the face or voice using the six basic emotions. Also we examine the modality dominance between the face and voice for each emotion category. Before the experiment, we recorded faces and voices which expresses the six basic emotions and orthogonally combined these faces and voices. Consequently, the emotional valence of visual and auditory information was either congruent or incongruent. In the experiment, there were unisensory and multisensory sessions. The multisensory session was divided into three blocks according to whether an instruction was given to pay attention to a given modality (face attention, voice attention, and no instruction). Participants judged whether the speaker expressed happiness, sadness, anger, fear, disgust, or surprise. Our results revealed that instructions to pay attention to one modality and congruency of the emotions between modalities modulated the modality dominance, and the modality dominance is differed for each emotion category. In particular, the modality dominance for anger changed according to each instruction. Analyses also revealed that the modality dominance suggested by the congruency effect can be explained in terms of the facilitation effect and the interference effect.

Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains.

In everyday life, multiple sensory channels jointly trigger emotional experiences and one channel may alter processing in another channel. For example, seeing an emotional facial expression and hearing the voice’s emotional tone will jointly create the emotional experience. This example, where auditory and visual input is related to social communication, has gained considerable attention by researchers. However, interactions of visual and auditory emotional information are not limited to social communication but can extend to much broader contexts including human, animal, and environmental cues. In this article, we review current research on audiovisual emotion processing beyond face-voice stimuli to develop a broader perspective on multimodal interactions in emotion processing. We argue that current concepts of multimodality should be extended in considering an ecologically valid variety of stimuli in audiovisual emotion processing. Therefore, we provide an overview of studies in which emotional sounds and interactions with complex pictures of scenes were investigated. In addition to behavioral studies, we focus on neuroimaging, electro- and peripher-physiological findings. Furthermore, we integrate these findings and identify similarities or differences. We conclude with suggestions for future research.

Emotion Judgment of Face-voice Stimuli: Cultural difference found by using face-voice stimuli in which different emotions are expressed

Emotion Judgment of Face-voice Stimuli: Cultural difference found by using face-voice stimuli in which different emotions are expressed

Emotional Voice and Emotional Body Postures Influence Each Other Independently of Visual Awareness

Multisensory integration may occur independently of visual attention as previously shown with compound face-voice stimuli. We investigated in two experiments whether the perception of whole body expressions and the perception of voices influence each other when observers are not aware of seeing the bodily expression. In the first experiment participants categorized masked happy and angry bodily expressions while ignoring congruent or incongruent emotional voices. The onset between target and mask varied from −50 to +133 ms. Results show that the congruency between the emotion in the voice and the bodily expressions influences audiovisual perception independently of the visibility of the stimuli. In the second experiment participants categorized the emotional voices combined with masked bodily expressions as fearful or happy. This experiment showed that bodily expressions presented outside visual awareness still influence prosody perception. Our experiments show that audiovisual integration between bodily expressions and affective prosody can take place outside and independent of visual awareness.

Preattentive processing of audio-visual emotional signals

Previous research has shown that redundant information in faces and voices leads to faster emotional categorization compared to incongruent emotional information even when attending to only one modality. The aim of the present study was to test whether these crossmodal effects are predominantly due to a response conflict rather than interference at earlier, e.g. perceptual processing stages. In Experiment 1, participants had to categorize the valence and rate the intensity of happy, sad, angry and neutral unimodal or bimodal face–voice stimuli. They were asked to rate either the facial or vocal expression and ignore the emotion expressed in the other modality. Participants responded faster and more precisely to emotionally congruent compared to incongruent face–voice pairs in both the Attend Face and in the Attend Voice condition. Moreover, when attending to faces, emotionally congruent bimodal stimuli were more efficiently processed than unimodal visual stimuli. To study the role of a possible response conflict, Experiment 2 used a modified paradigm in which emotional and response conflicts were disentangled. Incongruency effects were significant even in the absence of response conflicts. The results suggest that emotional signals available through different sensory channels are automatically combined prior to response selection.

Depressed mothers' newborns are less responsive to animate and inanimate stimuli

AbstractData from our research are reviewed showing limited attentiveness and responsivity to animate stimuli in newborns of depressed mothers. These include face–voice stimuli of mothers and strangers, newborn cry sounds and instrumental and vocal music. Newborns of depressed mothers are also noted to be less attentive and responsive when presented with inanimate stimuli, including different texture nipples, different temperature nipples and different weight tubes. Potential underlying mechanisms are suggested by research showing negative effects of prenatal depression, elevated prenatal cortisol and lower prenatal dopamine and serotonin on the newborns of depressed mothers. Additional risk factors are explored, including sleep disturbances and other prenatal stressors. Early interventions, including pregnancy massage, massage therapy for infants and demonstration of the Brazelton Assessment, have been noted to enhance the neonates' attentiveness and responsivity to animate and inanimate stimuli. Limitations of these studies are discussed, including the sample being limited to lower SES women who have many other problems that potentially affect newborns. More controlled studies are needed to assess potential mediating factors for the inferior performance of these newborns on perceptual tasks such as limited attentiveness and/or greater arousal as well as longitudinal follow‐up studies to determine whether these limited perceptual abilities continue into infancy. Copyright © 2010 John Wiley & Sons, Ltd.