Vocal Emotion Research Articles

Lateralized behavior and cardiac activity of dogs in response to human emotional vocalizations

Over the recent years, the study of emotional functioning has become one of the central issues in dog cognition. Previous studies showed that dogs can recognize different emotions by looking at human faces and can correctly match the human emotional state with a vocalization having a negative emotional valence. However, to this day, little is known about how dogs perceive and process human non-verbal vocalizations having different emotional valence. The current research provides new insights into emotional functioning of the canine brain by studying dogs’ lateralized auditory functions (to provide a first insight into the valence dimension) matched with both behavior and physiological measures of arousal (to study the arousal dimension) in response to playbacks related to the Ekman’s six basic human emotions. Overall, our results indicate lateralized brain patterns for the processing of human emotional vocalizations, with the prevalent use of the right hemisphere in the analysis of vocalizations with a clear negative emotional valence (i.e. “fear” and “sadness”) and the prevalent use of the left hemisphere in the analysis of positive vocalization (“happiness”). Furthermore, both cardiac activity and behavior response support the hypothesis that dogs are sensitive to emotional cues of human vocalizations.

Hearing Aids Benefit Recognition of Words in Emotional Speech but Not Emotion Identification

Vocal emotion perception is an important part of speech communication and social interaction. Although older adults with normal audiograms are known to be less accurate at identifying vocal emotion compared to younger adults, little is known about how older adults with hearing loss perceive vocal emotion or whether hearing aids improve the perception of emotional speech. In the main experiment, older hearing aid users were presented with sentences spoken in seven emotion conditions, with and without their own hearing aids. Listeners reported the words that they heard as well as the emotion portrayed in each sentence. The use of hearing aids improved word-recognition accuracy in quiet from 38.1% (unaided) to 65.1% (aided) but did not significantly change emotion-identification accuracy (36.0% unaided, 41.8% aided). In a follow-up experiment, normal-hearing young listeners were tested on the same stimuli. Normal-hearing younger listeners and older listeners with hearing loss showed similar patterns in how emotion affected word-recognition performance but different patterns in how emotion affected emotion-identification performance. In contrast to the present findings, previous studies did not find age-related differences between younger and older normal-hearing listeners in how emotion affected emotion-identification performance. These findings suggest that there are changes to emotion identification caused by hearing loss that are beyond those that can be attributed to normal aging, and that hearing aids do not compensate for these changes.

Speech Emotion Analysis of Different Age Groups Using Clustering Techniques

The shape, length, and size of the vocal tract and vocal folds vary with the age of the human being. The variation may be of different age or sickness or some other conditions. Arguably, the features extracted from the utterances for the recognition task may differ for different age group. It complicates further for different emotions. The recognition system demands suitable feature extraction and clustering techniques that can separate their emotional utterances. Psychologists, criminal investigators, professional counselors, law enforcement agencies and a host of other such entities may find such analysis useful. In this article, the emotion study has been evaluated for three different age groups of people using the basic age- dependent features like pitch, speech rate, and log energy. The feature sets have been clustered for different age groups by utilizing K-means and Fuzzy c-means (FCM) algorithm for the boredom, sadness, and anger states. K-means algorithm has outperformed the FCM algorithm in terms of better clustering and lower computation time as the authors' results suggest.

Multimodal and Spectral Degradation Effects on Speech and Emotion Recognition in Adult Listeners.

For cochlear implant (CI) users, degraded spectral input hampers the understanding of prosodic vocal emotion, especially in difficult listening conditions. Using a vocoder simulation of CI hearing, we examined the extent to which informative multimodal cues in a talker’s spoken expressions improve normal hearing (NH) adults’ speech and emotion perception under different levels of spectral degradation (two, three, four, and eight spectral bands). Participants repeated the words verbatim and identified emotions (among four alternative options: happy, sad, angry, and neutral) in meaningful sentences that are semantically congruent with the expression of the intended emotion. Sentences were presented in their natural speech form and in speech sampled through a noise-band vocoder in sound (auditory-only) and video (auditory–visual) recordings of a female talker. Visual information had a more pronounced benefit in enhancing speech recognition in the lower spectral band conditions. Spectral degradation, however, did not interfere with emotion recognition performance when dynamic visual cues in a talker’s expression are provided as participants scored at ceiling levels across all spectral band conditions. Our use of familiar sentences that contained congruent semantic and prosodic information have high ecological validity, which likely optimized listener performance under simulated CI hearing and may better predict CI users’ outcomes in everyday listening contexts.

Comparison of Two Music Training Approaches on Music and Speech Perception in Cochlear Implant Users

In normal-hearing (NH) adults, long-term music training may benefit music and speech perception, even when listening to spectro-temporally degraded signals as experienced by cochlear implant (CI) users. In this study, we compared two different music training approaches in CI users and their effects on speech and music perception, as it remains unclear which approach to music training might be best. The approaches differed in terms of music exercises and social interaction. For the pitch/timbre group, melodic contour identification (MCI) training was performed using computer software. For the music therapy group, training involved face-to-face group exercises (rhythm perception, musical speech perception, music perception, singing, vocal emotion identification, and music improvisation). For the control group, training involved group nonmusic activities (e.g., writing, cooking, and woodworking). Training consisted of weekly 2-hr sessions over a 6-week period. Speech intelligibility in quiet and noise, vocal emotion identification, MCI, and quality of life (QoL) were measured before and after training. The different training approaches appeared to offer different benefits for music and speech perception. Training effects were observed within-domain (better MCI performance for the pitch/timbre group), with little cross-domain transfer of music training (emotion identification significantly improved for the music therapy group). While training had no significant effect on QoL, the music therapy group reported better perceptual skills across training sessions. These results suggest that more extensive and intensive training approaches that combine pitch training with the social aspects of music therapy may further benefit CI users.

Investigating Duration Effects of Emotional Speech Stimuli in a Tonal Language by Using Event-Related Potentials

Studying event-related potentials (ERPs) is considered as an effective method for investigating cerebral mechanisms of processing emotional speech. It has been shown that the amplitudes of ERP components in the cognitive processing of emotional speech are modulated by acoustic characteristics, such as valence and arousal. However, whether the duration of emotional speech stimuli impacts emotion-related cognitive processing remains unclear. To better understand the effect of emotional speech stimulus duration on emotion-related cognitive processing, we explored whether emotional speech ERPs were influenced by the duration of stimuli presented. Specifically, this paper focused on the ERP investigation of different durations (short: 0.50–1.00 s; medium: 1.50–2.00 s; and long: 2.50–3.00 s) of Chinese emotional speech stimuli. Chinese is a typical tonal language, and the stimuli were excerpted from radio plays in order to make emotions more obvious and easier to distinguish. We investigated the three different stages of the emotional speech processing: sensory processing, salience detection, and cognition. During the experiment, participants passively listened to emotional utterances matched for semantics and prosody with four emotions (sadness, anger, happiness, and surprise). Our results showed significant differences in the amplitudes of ERP components for different emotions during short-duration emotional speech stimuli. These findings suggest that shorter duration emotional speech stimuli may be more effective for separating the ERP components representing different emotions (N100, P200, and N300).

From shrieks to “Stupid poo”: emotive language in a developmental perspective

AbstractThe aim of this paper is to highlight and describe the forms of verbal emotive utterances that appeared in a longitudinal corpus of 11 Swedish children interacting with parents, siblings and friends. The children were in the ages 0;9 to 5;10 and were recorded four to six times during a two-year period. The verbal emotive expressions of the material are divided into the categories Descriptive versus Accompanying utterances. Descriptive utterances are emotive mainly from semantic conventions, whereas Accompanying utterances are emotive due to prosodic and contextual traits. The categories are illustrated and related to conventions, language development and cognitive growth. By classifying and labeling verbal expressions as emotive in different ways, it is argued that we can gain a better understanding of how language is used when intertwined with emotions, but also that we access a way to compare and investigate emotive language in a more thorough manner.

Biased and unbiased perceptual decision-making on vocal emotions

Perceptual decision-making on emotions involves gathering sensory information about the affective state of another person and forming a decision on the likelihood of a particular state. These perceptual decisions can be of varying complexity as determined by different contexts. We used functional magnetic resonance imaging and a region of interest approach to investigate the brain activation and functional connectivity behind two forms of perceptual decision-making. More complex unbiased decisions on affective voices recruited an extended bilateral network consisting of the posterior inferior frontal cortex, the orbitofrontal cortex, the amygdala, and voice-sensitive areas in the auditory cortex. Less complex biased decisions on affective voices distinctly recruited the right mid inferior frontal cortex, pointing to a functional distinction in this region following decisional requirements. Furthermore, task-induced neural connectivity revealed stronger connections between these frontal, auditory, and limbic regions during unbiased relative to biased decision-making on affective voices. Together, the data shows that different types of perceptual decision-making on auditory emotions have distinct patterns of activations and functional coupling that follow the decisional strategies and cognitive mechanisms involved during these perceptual decisions.

Is the voice an auditory face? An ALE meta-analysis comparing vocal and facial emotion processing.

This meta-analysis compares the brain structures and mechanisms involved in facial and vocal emotion recognition. Neuroimaging studies contrasting emotional with neutral (face: N = 76, voice: N = 34) and explicit with implicit emotion processing (face: N = 27, voice: N = 20) were collected to shed light on stimulus and goal-driven mechanisms, respectively. Activation likelihood estimations were conducted on the full data sets for the separate modalities and on reduced, modality-matched data sets for modality comparison. Stimulus-driven emotion processing engaged large networks with significant modality differences in the superior temporal (voice-specific) and the medial temporal (face-specific) cortex. Goal-driven processing was associated with only a small cluster in the dorsomedial prefrontal cortex for voices but not faces. Neither stimulus- nor goal-driven processing showed significant modality overlap. Together, these findings suggest that stimulus-driven processes shape activity in the social brain more powerfully than goal-driven processes in both the visual and the auditory domains. Yet, whereas faces emphasize subcortical emotional and mnemonic mechanisms, voices emphasize cortical mechanisms associated with perception and effortful stimulus evaluation (e.g. via subvocalization). These differences may be due to sensory stimulus properties and highlight the need for a modality-specific perspective when modeling emotion processing in the brain.

Altered emotional recognition and expression in patients with Parkinson's disease.

BackgroundParkinson’s disease (PD) patients exhibit deficits in emotional recognition and expression abilities, including emotional faces and voices. The aim of this study was to explore emotional processing in pre-deep brain stimulation (pre-DBS) PD patients using two sensory modalities (visual and auditory).MethodsFifteen PD patients who needed DBS surgery and 15 healthy, age- and gender-matched controls were recruited as participants. All participants were assessed by the Karolinska Directed Emotional Faces database 50 Faces Recognition test. Vocal recognition was evaluated by the Montreal Affective Voices database 50 Voices Recognition test. For emotional facial expression, the participants were asked to imitate five basic emotions (neutral, happiness, anger, fear, and sadness). The subjects were required to express nonverbal vocalizations of the five basic emotions. Fifteen Chinese native speakers were recruited as decoders. We recorded the accuracy of the responses, reaction time, and confidence level.ResultsFor emotional recognition and expression, the PD group scored lower on both facial and vocal emotional processing than did the healthy control group. There were significant differences between the two groups in both reaction time and confidence level. A significant relationship was also found between emotional recognition and emotional expression when considering all participants between the two groups together.ConclusionThe PD group exhibited poorer performance on both the recognition and expression tasks. Facial emotion deficits and vocal emotion abnormalities were associated with each other. In addition, our data allow us to speculate that emotional recognition and expression may share a common system.

The Effects of the Literal Meaning of Emotional Phrases on the Identification of Vocal Emotions.

This study investigates the discrepancy between the literal emotional content of speech and emotional tone in the identification of speakers' vocal emotions in both the listeners' native language (Japanese), and in an unfamiliar language (random-spliced Japanese). Both experiments involve a "congruent condition," in which the emotion contained in the literal meaning of speech (words and phrases) was compatible with vocal emotion, and an "incongruent condition," in which these forms of emotional information were discordant. Results for Japanese indicated that performance in identifying emotions did not differ significantly between the congruent and incongruent conditions. However, the results for random-spliced Japanese indicated that vocal emotion was correctly identified more often in the congruent than in the incongruent condition. The different results for Japanese and random-spliced Japanese suggested that the literal meaning of emotional phrases influences the listener's perception of the speaker's emotion, and that Japanese participants could infer speakers' intended emotions in the incongruent condition.

Age differences in vocal emotion perception: on the role of speaker age and listener sex

ABSTRACTOlder adults have greater difficulty than younger adults perceiving vocal emotions. To better characterise this effect, we explored its relation to age differences in sensory, cognitive and emotional functioning. Additionally, we examined the role of speaker age and listener sex. Participants (N = 163) aged 19–34 years and 60–85 years categorised neutral sentences spoken by ten younger and ten older speakers with a happy, neutral, sad, or angry voice. Acoustic analyses indicated that expressions from younger and older speakers denoted the intended emotion with similar accuracy. As expected, younger participants outperformed older participants and this effect was statistically mediated by an age-related decline in both optimism and working-memory. Additionally, age differences in emotion perception were larger for younger as compared to older speakers and a better perception of younger as compared to older speakers was greater in younger as compared to older participants. Last, a female perception benefit was less pervasive in the older than the younger group. Together, these findings suggest that the role of age for emotion perception is multi-faceted. It is linked to emotional and cognitive change, to processing biases that benefit young and own-age expressions, and to the different aptitudes of women and men.

One- to four-year-olds connect diverse positive emotional vocalizations to their probable causes

The ability to understand why others feel the way they do is critical to human relationships. Here, we show that emotion understanding in early childhood is more sophisticated than previously believed, extending well beyond the ability to distinguish basic emotions or draw different inferences from positively and negatively valenced emotions. In a forced-choice task, 2- to 4-year-olds successfully identified probable causes of five distinct positive emotional vocalizations elicited by what adults would consider funny, delicious, exciting, sympathetic, and adorable stimuli (Experiment 1). Similar results were obtained in a preferential looking paradigm with 12- to 23-month-olds, a direct replication with 18- to 23-month-olds (Experiment 2), and a simplified design with 12- to 17-month-olds (Experiment 3; preregistered). Moreover, 12- to 17-month-olds selectively explored, given improbable causes of different positive emotional reactions (Experiments 4 and 5; preregistered). The results suggest that by the second year of life, children make sophisticated and subtle distinctions among a wide range of positive emotions and reason about the probable causes of others' emotional reactions. These abilities may play a critical role in developing theory of mind, social cognition, and early relationships.

I Think We're Alone Now: Solitary Social Behaviors in Adolescents with Autism Spectrum Disorder.

Research into emotional responsiveness in Autism Spectrum Disorder (ASD) has yielded mixed findings. Some studies report uniform, flat and emotionless expressions in ASD; others describe highly variable expressions that are as or even more intense than those of typically developing (TD) individuals. Variability in findings is likely due to differences in study design: some studies have examined posed (i.e., not spontaneous expressions) and others have examined spontaneous expressions in social contexts, during which individuals with ASD-by nature of the disorder-are likely to behave differently than their TD peers. To determine whether (and how) spontaneous facial expressions and other emotional responses are different from TD individuals, we video-recorded the spontaneous responses of children and adolescents with and without ASD (between the ages of 10 and 17 years) as they watched emotionally evocative videos in a non-social context. Researchers coded facial expressions for intensity, and noted the presence of laughter and other responsive vocalizations. Adolescents with ASD displayed more intense, frequent and varied spontaneous facial expressions than their TD peers. They also produced significantly more emotional vocalizations, including laughter. Individuals with ASD may display their emotions more frequently and more intensely than TD individuals when they are unencumbered by social pressure. Differences in the interpretation of the social setting and/or understanding of emotional display rules may also contribute to differences in emotional behaviors between groups.

Impaired facial and vocal emotion decoding in schizophrenia is underpinned by basic perceptivo-motor deficits

ABSTRACTIntroduction: Emotional decoding impairments have been largely demonstrated in schizophrenia for facial and prosodic stimuli, when presented separately. Nevertheless, the exploration of crossmodal integration has been far less considered, despite its omnipresence in daily social interactions. Moreover, the role played by basic visuo-motor impairments in unimodal and crossmodal decoding remains unexplored.Methods: Thirty-two patients were compared with 32 matched controls in an emotional decoding task including unimodal (visual and auditory) and crossmodal (congruent and incongruent) conditions. A control perceptive task was also conducted to take potential low-level perceptual deficits into account.Results: Schizoprenic patients presented lower performance and higher reaction times for both unimodal tasks (visual and auditory) and crossmodal conditions. Moreover, reaction times for the visuo-perceptive task were also significantly longer for patients compared to controls.Conclusions: The consistency of the results across unimodal and crossmodal tasks suggests a globalised emotional impairment in schizophrenia, independent of the sensorial modality and crossmodal nature of the stimuli. Centrally, given the results in the visuo-perceptive task, the impairments observed for emotional recognition appears at least partly explained by primary cognitive deficits, namely reduced processing speed.

Neural Processing of Musical and Vocal Emotions Through Cochlear Implants Simulation.

Cochlear implants (CIs) partially restore the sense of hearing in the deaf. However, the ability to recognize emotions in speech and music is reduced due to the implant's electrical signal limitations and the patient's altered neural pathways. Electrophysiological correlations of these limitations are not yet well established. Here we aimed to characterize the effect of CIs on auditory emotion processing and, for the first time, directly compare vocal and musical emotion processing through a CI-simulator. We recorded 16 normal hearing participants' electroencephalographic activity while listening to vocal and musical emotional bursts in their original form and in a degraded (CI-simulated) condition. We found prolonged P50 latency and reduced N100-P200 complex amplitude in the CI-simulated condition. This points to a limitation in encoding sound signals processed through CI simulation. When comparing the processing of vocal and musical bursts, we found a delay in latency with the musical bursts compared to the vocal bursts in both conditions (original and CI-simulated). This suggests that despite the cochlear implants' limitations, the auditory cortex can distinguish between vocal and musical stimuli. In addition, it adds to the literature supporting the complexity of musical emotion. Replicating this study with actual CI users might lead to characterizing emotional processing in CI users and could ultimately help develop optimal rehabilitation programs or device processing strategies to improve CI users' quality of life.

Laughter catches attention!

In social interactions, emotionally salient and sudden changes in vocal expressions attract attention. However, only a few studies examined how emotion and attention interact in voice processing. We investigated neutral, happy (laughs) and angry (growls) vocalizations in a modified oddball task.Participants silently counted the targets in each block and rated the valence and arousal of the vocalizations. A combined event-related potential and time-frequency analysis focused on the P3 and pre-stimulus alpha power to capture attention effects in response to unexpected events.Whereas an early differentiation between emotionally salient and neutral vocalizations was reflected in the P3a response, the P3b was selectively enhanced for happy voices. The P3b modulation was predicted by pre-stimulus frontal alpha desynchronization, and by the perceived pleasantness of the targets.These findings indicate that vocal emotions may be differently processed based on task relevance and valence. Increased anticipation and attention to positive vocal cues (laughter) may reflect their high social relevance.

The right touch: Stroking of CT-innervated skin promotes vocal emotion processing

Research has revealed a special mechanoreceptor, called C-tactile (CT) afferent, that is situated in hairy skin and that seems relevant for the processing of social touch. We pursued a possible role of this receptor in the perception of other social signals such as a person’s voice. Participants completed three sessions in which they heard surprised and neutral vocal and nonvocal sounds and detected rare sound repetitions. In a given session, participants received no touch or soft brushstrokes to the arm (CT innervated) or palm (CT free). Event-related potentials elicited to sounds revealed that stroking to the arm facilitated the integration of vocal and emotional information. The late positive potential was greater for surprised vocal relative to neutral vocal and nonvocal sounds, and this effect was greater for arm touch relative to both palm touch and no touch. Together, these results indicate that stroking to the arm facilitates the allocation of processing resources to emotional voices, thus supporting the possibility that CT stimulation benefits social perception cross-modally.

Significance of incorporating excitation source parameters for improved emotion recognition from speech and electroglottographic signals

The work presented in this paper explores the effectiveness of incorporating the excitation source parameters such as strength of excitation and instantaneous fundamental frequency ( $$F_0$$ ) for emotion recognition task from speech and electroglottographic (EGG) signals. The strength of excitation (SoE) is an important parameter indicating the pressure with which glottis closes at the glottal closure instants (GCIs). The SoE is computed by the popular zero frequency filtering (ZFF) method which accurately estimates the glottal signal characteristics by attenuating or removing the high frequency vocaltract interactions in speech. The arbitrary impulse sequence, obtained from the estimated GCIs, is used to derive the instantaneous $$F_0$$ . The SoE and the instantaneous $$F_0$$ parameters are combined with the conventional mel frequency cepstral coefficients (MFCC) to improve the recognition rates of distinct emotions (Anger, Happy and Sad) using Gaussian mixture models as classifier. The performances of the proposed combination of SoE and instantaneous $$F_0$$ and their dynamic features with MFCC coefficients are compared with the emotion utterances (4 emotions and neutral) from classical German full blown emotion speech database (EmoDb) having simultaneous speech and EGG signals and Surrey Audio Visual Expressed Emotion database (3 emotions and neutral) for both speaker dependent and speaker independent emotion recognition scenarios. To reinforce the effectiveness of the proposed features and for better statistical consistency of the emotion analysis, a fairly large emotion speech database of 220 utterances per emotion in Tamil language with simultaneous EGG recordings, is used in addition to EmoDb. The effectiveness of SoE and instantaneous $$F_0$$ in characterizing different emotions is also confirmed by the improved emotion recognition performance in Tamil speech-EGG emotion database.

Right-ear precedence and vocal emotion contagion: The role of the left hemisphere

ABSTRACTMuch evidence suggests that the processing of emotions is lateralized to the right hemisphere of the brain. However, under some circumstances the left hemisphere might play a role, particularly for positive emotions and emotional experiences. We explored whether emotion contagion was right-lateralized, lateralized valence-specifically, or potentially left-lateralized. In two experiments, right-handed female listeners rated to what extent emotionally intoned pseudo-sentences evoked target emotions in them. These sound stimuli had a 7 ms ear lead in the left or right channel, leading to stronger stimulation of the contralateral hemisphere. In both experiments, the results revealed that right ear lead stimuli received subtly but significantly higher evocation scores, suggesting a left hemisphere dominance for emotion contagion. A control experiment using an emotion identification task showed no effect of ear lead. The findings are discussed in relation to prior findings that have linked the processing of emotional prosody to left-hemisphere brain regions that regulate emotions, control orofacial musculature, are involved in affective empathy processing areas, or have an affinity for processing emotions socially. Future work is needed to eliminate alternative interpretations and understand the mechanisms involved. Our novel binaural asynchrony method may be useful in future work in auditory laterality.