Vocal Emotion Research Articles

Restricting movements of lower face leaves recognition of emotional vocalizations intact but introduces a valence positivity bias

Blocking facial mimicry can disrupt recognition of emotion stimuli. Many previous studies have focused on facial expressions, and it remains unclear whether this generalises to other types of emotional expressions. Furthermore, by emphasizing categorical recognition judgments, previous studies neglected the role of mimicry in other processing stages, including dimensional (valence and arousal) evaluations. In the study presented herein, we addressed both issues by asking participants to listen to brief non-verbal vocalizations of four emotion categories (anger, disgust, fear, happiness) and neutral sounds under two conditions. One of the conditions included blocking facial mimicry by creating constant tension on the lower face muscles, in the other condition facial muscles remained relaxed. After each stimulus presentation, participants evaluated sounds’ category, valence, and arousal. Although the blocking manipulation did not influence emotion recognition, it led to higher valence ratings in a non-category-specific manner, including neutral sounds. Our findings suggest that somatosensory and motor feedback play a role in the evaluation of affect vocalizations, perhaps introducing a directional bias. This distinction between stimulus recognition, stimulus categorization, and stimulus evaluation is important for understanding what cognitive and emotional processing stages involve somatosensory and motor processes.

Transformer-Based Multilingual Speech Emotion Recognition Using Data Augmentation and Feature Fusion

In recent years data science has been applied in a variety of real-life applications such as human-computer interaction applications, computer gaming, mobile services, and emotion evaluation. Among the wide range of applications, speech emotion recognition (SER) is also an emerging and challenging research topic. For SER, recent studies used handcrafted features that provide the best results but failed to provide accuracy while applied in complex scenarios. Later, deep learning techniques were used for SER that automatically detect features from speech signals. Deep learning-based SER techniques overcome the issues of accuracy, yet there are still significant gaps in the reported methods. Studies using lightweight CNN failed to learn optimal features from composite acoustic signals. This study proposed a novel SER model to overcome the limitations mentioned earlier in this study. We focused on Arabic vocal emotions in particular because they received relatively little attention in research. The proposed model performs data augmentation before feature extraction. The 273 derived features were fed as input to the transformer model for emotion recognition. This model is applied to four datasets named BAVED, EMO-DB, SAVEE, and EMOVO. The experimental findings demonstrated the robust performance of the proposed model compared to existing techniques. The proposed SER model achieved 95.2%, 93.4%, 85.1%, and 91.7% accuracy on BAVED, EMO-DB, SAVEE, and EMOVO datasets respectively. The highest accuracy was obtained using BAVED dataset, indicating that the proposed model is well suited to Arabic vocal emotions.

Real-life and posed vocalizations to lottery wins differ fundamentally in their perceived valence.

A basic premise of classic emotion theories is that distinct emotional experiences yield distinct emotional vocalizations-each informative of its situational context. Furthermore, it is commonly assumed that emotional vocalizations become more distinct and diagnostic as their intensity increases. Critically, these theoretical assumptions largely rely on research utilizing posed vocal reactions of actors, which may be overly simplified and stereotypical. While recent work suggests that intense, real-life vocalizations may be nondiagnostic, the exact way in which increasing degrees of situational intensity affect the perceived valence of real-life versus posed expressions remains unknown. Here we compared real-life and posed vocalizations to winning increasing amounts of money in the lottery. Results show that while posed vocalizations are perceived as positive for both low- and high-sum wins, real-life vocalizations are perceived as positive only for low-sum wins, but as negative for high-sum wins. These findings demonstrate the potential gaps between real-life and posed expressions and highlight the role of situational intensity in driving perceptual ambiguity for real-life emotional expressions. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Validation of Affective Sentences: Extending Beyond Basic Emotion Categories

We use nonverbal and verbal emotion cues to determine how others are feeling. Most studies in vocal emotion perception do not consider the influence of verbal content, using sentences with nonsense words or words that carry no emotional meaning. These online studies aimed to validate 95 sentences with verbal content intended to convey 10 emotions. Participants were asked to select the emotion that best described the emotional meaning of the sentence. Study 1 included 436 participants and Study 2 included 193. The Simpson diversity index was applied as a measure of dispersion of responses. Across the two studies, 38 sentences were labelled as representing 10 emotion categories with a low degree of diversity in participant responses. Expanding current databases beyond basic emotion categories is important for researchers exploring the interaction between tone of voice and verbal content, and/or people’s capacity to make subtle distinctions between their own and others’ emotions.

Disentangling emotional signals in the brain: an ALE meta-analysis of vocal affect perception.

Recent advances in neuroimaging research on vocal emotion perception have revealed voice-sensitive areas specialized in processing affect. Experimental data on this subject is varied, investigating a wide range of emotions through different vocal signals and task demands. The present meta-analysis was designed to disentangle this diversity of results by summarizing neuroimaging data in the vocal emotion perception literature. Data from 44 experiments contrasting emotional and neutral voices was analyzed to assess brain areas involved in vocal affect perception in general, as well as depending on the type of voice signal (speech prosody or vocalizations), the task demands (implicit or explicit attention to emotions), and the specific emotion perceived. Results reassessed a consistent bilateral network of Emotional Voices Areas consisting of the superior temporal cortex and primary auditory regions. Specific activations and lateralization of these regions, as well as additional areas (insula, middle temporal gyrus) were further modulated by signal type and task demands. Exploring the sparser data on single emotions also suggested the recruitment of other regions (insula, inferior frontal gyrus, frontal operculum) for specific aspects of each emotion. These novel meta-analytic results suggest that while the bulk of vocal affect processing is localized in the STC, the complexity and variety of such vocal signals entails functional specificities in complex and varied cortical (and potentially subcortical) response pathways.

Enhanced recognition of vocal emotions in individuals with naturally good musical abilities.

Music training is widely assumed to enhance several nonmusical abilities, including speech perception, executive functions, reading, and emotion recognition. This assumption is based primarily on cross-sectional comparisons between musicians and nonmusicians. It remains unclear, however, whether training itself is necessary to explain the musician advantages, or whether factors such as innate predispositions and informal musical experience could produce similar effects. Here, we sought to clarify this issue by examining the association between music training, music perception abilities and vocal emotion recognition. The sample (N = 169) comprised musically trained and untrained listeners who varied widely in their musical skills, as assessed through self-report and performance-based measures. The emotion recognition tasks required listeners to categorize emotions in nonverbal vocalizations (e.g., laughter, crying) and in speech prosody. Music training was associated positively with emotion recognition across tasks, but the effect was small. We also found a positive association between music perception abilities and emotion recognition in the entire sample, even with music training held constant. In fact, untrained participants with good musical abilities were as good as highly trained musicians at recognizing vocal emotions. Moreover, the association between music training and emotion recognition was fully mediated by auditory and music perception skills. Thus, in the absence of formal music training, individuals who were "naturally" musical showed musician-like performance at recognizing vocal emotions. These findings highlight an important role for factors other than music training (e.g., predispositions and informal musical experience) in associations between musical and nonmusical domains. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Acoustic Parameters Related to Behavioral Discrimination of Emotional Vocalizations in Java Sparrows

Animals emit vocalizations related to internal or external states, such as hormone levels or predation risks. The Java Sparrow Lonchura oryzivora, a songbird, produces similar trill-structured calls in both aggressive and affiliative situations. A previous study found that trills produced in aggressive situations (AG trills) are faster, louder, and have a wider range of frequencies than trills produced in affiliative situations (AF trills). To assess whether Java Sparrows perceive these trills as different calls, we conducted a playback experiment using a habituation-dishabituation paradigm. In this paradigm, we first played a stimulus set for ten minutes (habituation stimulus), and after that we played either a test stimulus set or a control stimulus set. We analyzed behavioral changes in the study bird that took place between the last minute of the habituation stimulus and the next stimulus set (test or control). As trill stimuli, we used natural AG and AF trills (Experiment 1) and trills in which the trill rates and syllables types were modified (Experiment 2). The results showed that Java Sparrows distinguish between AG and AF trills (Experiment 1). However, when the trill acoustic structure was modified, they distinguished between the different syllable types, but not between different trill rates. As we used a natural behavior as an index, we could not measure the perceptual limit of trill discrimination in these birds. Instead, here, we asked whether the difference between AG and AF trills is meaningful or not, and which acoustic features were important for discrimination. To fully understand how birds perceive vocalizations in the context of communication, more experiments are required, both controlled experiments utilizing operant conditioning and electrophysiology, and multimodal experiments that reflect natural social relationships.

One-shot emotional voice conversion based on feature separation

The task of emotional voice conversion (EVC) aims to convert speech from one emotional state into another, while keeping linguistic content, speaker identity and other emotion-independent information unchanged. Because previous studies were limited to a specific set of emotions, it is challenging to realize the conversion of emotions never seen in training stage. In this paper, we propose a one-shot emotional voice conversion model based on feature separation. The proposed method could control emotional characteristics with Global Emotion Embeddings (GEEs), and introduce activation guidance (AG) and mutual information (MI) minimization to reduce the correlations between emotion embedding and emotion-independent representation. At run-time conversion, it could produce the desired emotional utterance from a single pairwise utterance without any emotion labels, whether the target emotion appears in the training set or not. The subjective and objective evaluations validate the effectiveness of our proposed model for both the degree of feature separation and emotion expression, even it could achieve unseen emotion conversion.

Expressive Speech Acts in Political Discourse: An Integrative Approach Perspective

The article aims at analyzing of the contemporary political discourse modality that has acquired specific characteristic features of highly emotional utterances based on deliberate or unintended violation of political etiquette principles recently. The article’s generalized theorizing is illustrated by a sample of case study of invectives in political discourse. This analysis aims at distinguishing “agonal” signs (a deliberate use of invectives in speech) and pragmatic borrowings (inadvertent use of invectives) in their functioning, their pragma-semantic characteristics and discursive markers, which helps us in identification of both types of political discourse linguistic items. This research represents an integrative approach combining the Critical Discourse Analysis, the Political Discourse Semiotics Theory, the Role Theory, the Communication Theory, and others, in order to discover the actual reasons and consequences of these changes in the society in general, and in political discourse in particular.

Design of Interactive Vocal Guidance and Artistic Psychological Intervention System Based on Emotion Recognition.

The research on artistic psychological intervention to judge emotional fluctuations by extracting emotional features from interactive vocal signals has become a research topic with great potential for development. Based on the interactive vocal music instruction theory of emotion recognition, this paper studies the design of artistic psychological intervention system. This paper uses the vocal music emotion recognition algorithm to first train the interactive recognition network, in which the input is a row vector composed of different vocal music characteristics, and finally recognizes the vocal music of different emotional categories, which solves the problem of low data coupling in the artistic psychological intervention system. Among them, the vocal music emotion recognition experiment based on the interactive recognition network is mainly carried out from six aspects: the number of iterative training, the vocal music instruction rate, the number of emotion recognition signal nodes in the artistic psychological intervention layer, the number of sample sets, different feature combinations, and the number of emotion types. The input data of the system is a training class learning video, and actions and expressions need to be recognized before scoring. In the simulation process, before the completion of the sample indicators is unbalanced, the R language statistical analysis tool is used to balance the existing unbalanced data based on the artificial data synthesis method, and 279 uniformly classified samples are obtained. The 279∗7 dataset was used for statistical identification of the participants. The experimental results show that under the guidance of four different interactive vocal music, the vocal emotion recognition rate is between 65.85%-91.00%, which promotes the intervention of music therapy on artistic psychological intervention.

Hyper-Sensitivity to Pitch and Poorer Prosody Processing in Adults With Autism: An ERP Study.

Individuals with autism typically experience a range of symptoms, including abnormal sensory sensitivities. However, there are conflicting reports on the sensory profiles that characterize the sensory experience in autism that often depend on the type of stimulus. Here, we examine early auditory processing to simple changes in pitch and later auditory processing of more complex emotional utterances. We measured electroencephalography in 24 adults with autism and 28 controls. First, tones (1046.5Hz/C6, 1108.7Hz/C#6, or 1244.5Hz/D#6) were repeated three times or nine times before the pitch changed. Second, utterances of delight or frustration were repeated three or six times before the emotion changed. In response to the simple pitched tones, the autism group exhibited larger mismatch negativity (MMN) after nine standards compared to controls and produced greater trial-to-trial variability (TTV). In response to the prosodic utterances, the autism group showed smaller P3 responses when delight changed to frustration compared to controls. There was no significant correlation between ERPs to pitch and ERPs to prosody. Together, this suggests that early auditory processing is hyper-sensitive in autism whereas later processing of prosodic information is hypo-sensitive. The impact the different sensory profiles have on perceptual experience in autism may be key to identifying behavioral treatments to reduce symptoms.

Contributions of fundamental frequency and timbre to vocal emotion perception and their electrophysiological correlates.

Our ability to infer a speaker's emotional state depends on the processing of acoustic parameters such as fundamental frequency (F0) and timbre. Yet, how these parameters are processed and integrated to inform emotion perception remains largely unknown. Here we pursued this issue using a novel parameter-specific voice morphing technique to create stimuli with emotion modulations in only F0 or only timbre. We used these stimuli together with fully modulated vocal stimuli in an event-related potential (ERP) study in which participants listened to and identified stimulus emotion. ERPs (P200 and N400) and behavioral data converged in showing that both F0 and timbre support emotion processing but do so differently for different emotions: Whereas F0 was most relevant for responses to happy, fearful and sad voices, timbre was most relevant for responses to voices expressing pleasure. Together, these findings offer original insights into the relative significance of different acoustic parameters for early neuronal representations of speaker emotion and show that such representations are predictive of subsequent evaluative judgments.

Crossed functional specialization between the basal ganglia and cerebellum during vocal emotion decoding: Insights from stroke and Parkinson’s disease

There is growing evidence that both the basal ganglia and the cerebellum play functional roles in emotion processing, either directly or indirectly, through their connections with cortical and subcortical structures. However, the lateralization of this complex processing in emotion recognition remains unclear. To address this issue, we investigated emotional prosody recognition in individuals with Parkinson’s disease (model of basal ganglia dysfunction) or cerebellar stroke patients, as well as in matched healthy controls (n = 24 in each group). We analysed performances according to the lateralization of the predominant brain degeneration/lesion. Results showed that a right (basal ganglia and cerebellar) hemispheric dysfunction was likely to induce greater deficits than a left one. Moreover, deficits following left hemispheric dysfunction were only observed in cerebellar stroke patients, and these deficits resembled those observed after degeneration of the right basal ganglia. Additional analyses taking disease duration / time since stroke into consideration revealed a worsening of performances in patients with predominantly right-sided lesions over time. These results point to the differential, but complementary, involvement of the cerebellum and basal ganglia in emotional prosody decoding, with a probable hemispheric specialization according to the level of cognitive integration.

Vocal emotion recognition by native Turkish children with normal hearing and with hearing aids

Development of vocal emotion recognition in children with normal hearing takes many years before reaching adult-like levels. In children with hearing loss, decreased audibility and potential loss of sensitivity to relevant acoustic cues may additionally affect vocal emotion perception. Hearing aids (HAs) are traditionally optimized for speech understanding, and it is not clear how children with HAs are performing in perceiving vocal emotions. In this study, we investigated vocal emotion recognition in native Turkish normal hearing children (NHC, age range: 5–18 years), normal hearing adults (NHA, age range: 18–45 years), and children with HAs (HAC, age range: 5–18 years), using pseudo-speech sentences expressed in one of the three emotions, happy, sad, or angry (Geneva Multimodal Emotion Portrayal (GEMEP) Corpus by Banziger and Scherer, 2010; EmoHI Test by Nagels et al., 2021). Visual inspection of the preliminary data suggests that performance increases with increasing age for NHC and that in general, HAC have lower recognition scores compared to NHC. Further analyses will be presented, along with acoustical analysis of the stimuli and an exploration of effects of HA settings. In addition, for cross-language comparison, these data will be compared to previously collected data with the same paradigm in children from the UK and the Netherlands.

Differences in dogs' event-related potentials in response to human and dog vocal stimuli; a non-invasive study.

Recent advances in the field of canine neuro-cognition allow for the non-invasive research of brain mechanisms in family dogs. Considering the striking similarities between dog's and human (infant)'s socio-cognition at the behavioural level, both similarities and differences in neural background can be of particular relevance. The current study investigates brain responses of n = 17 family dogs to human and conspecific emotional vocalizations using a fully non-invasive event-related potential (ERP) paradigm. We found that similarly to humans, dogs show a differential ERP response depending on the species of the caller, demonstrated by a more positive ERP response to human vocalizations compared to dog vocalizations in a time window between 250 and 650 ms after stimulus onset. A later time window between 800 and 900 ms also revealed a valence-sensitive ERP response in interaction with the species of the caller. Our results are, to our knowledge, the first ERP evidence to show the species sensitivity of vocal neural processing in dogs along with indications of valence sensitive processes in later post-stimulus time periods.

Longitudinal change in neural response to vocal emotion in adolescence.

Adolescence is associated with maturation of function within neural networks supporting the processing of social information. Previous longitudinal studies have established developmental influences on youth’s neural response to facial displays of emotion. Given the increasing recognition of the importance of non-facial cues to social communication, we build on existing work by examining longitudinal change in neural response to vocal expressions of emotion in 8- to 19-year-old youth. Participants completed a vocal emotion recognition task at two timepoints (1 year apart) while undergoing functional magnetic resonance imaging. The right inferior frontal gyrus, right dorsal striatum and right precentral gyrus showed decreases in activation to emotional voices across timepoints, which may reflect focalization of response in these areas. Activation in the dorsomedial prefrontal cortex was positively associated with age but was stable across timepoints. In addition, the slope of change across visits varied as a function of participants’ age in the right temporo-parietal junction (TPJ): this pattern of activation across timepoints and age may reflect ongoing specialization of function across childhood and adolescence. Decreased activation in the striatum and TPJ across timepoints was associated with better emotion recognition accuracy. Findings suggest that specialization of function in social cognitive networks may support the growth of vocal emotion recognition skills across adolescence.

The time course of emotional authenticity detection in nonverbal vocalizations

Previous research has documented perceptual and brain differences between spontaneous and volitional emotional vocalizations. However, the time course of emotional authenticity processing remains unclear. We used event-related potentials (ERPs) to address this question, and we focused on the processing of laughter and crying. We additionally tested whether the neural encoding of authenticity is influenced by attention, by manipulating task focus (authenticity versus emotional category) and visual condition (with versus without visual deprivation). ERPs were recorded from 43 participants while they listened to vocalizations and evaluated their authenticity (volitional versus spontaneous) or emotional meaning (sad versus amused). Twenty-two of the participants were blindfolded and tested in a dark room, and 21 were tested in standard visual conditions. As compared to volitional vocalizations, spontaneous ones were associated with reduced N1 amplitude in the case of laughter, and increased P2 in the case of crying. At later cognitive processing stages, more positive amplitudes were observed for spontaneous (versus volitional) laughs and cries (1000–1400 msec), with earlier effects for laughs (700–1000 msec). Visual condition affected brain responses to emotional authenticity at early (P2 range) and late processing stages (middle and late LPP ranges). Task focus did not influence neural responses to authenticity. Our findings suggest that authenticity information is encoded early and automatically during vocal emotional processing. They also point to a potentially faster encoding of authenticity in laughter compared to crying.

Emorec: a new approach for detecting and improving the emotional state of learners in an e-learning environment

ABSTRACT In e-learning environments, several activities are offered to learners, including learning and assessment activities. During these activities, the learner may encounter difficulties, such as blocking situations or lack of motivation. This paper presents a new approach to detect these difficulties based on the learner’s emotional states and recommends solutions to motivate him/her. Our first contribution is recognizing the learner’s emotional state using two modules: facial expression recognition using a Gabor filter bank and vocal emotion recognition using MFCC features. Concerning the recommendations in these cases of difficulties, a psychological and/or pedagogical recommendation generator is proposed in this study; it is our second contribution. Three experiments to validate our approach were conducted on two groups of students: test and control. The results indicate that the proposed method improves the learner’s emotional state, motivation, and engagement time in the system.

Voice emotion recognition by Mandarin-speaking pediatric cochlear implant users in Taiwan.

ObjectivesTo explore the effects of obligatory lexical tone learning on speech emotion recognition and the cross‐culture differences between United States and Taiwan for speech emotion understanding in children with cochlear implant.MethodsThis cohort study enrolled 60 cochlear‐implanted (cCI) Mandarin‐speaking, school‐aged children who underwent cochlear implantation before 5 years of age and 53 normal‐hearing children (cNH) in Taiwan. The emotion recognition and the sensitivity of fundamental frequency (F0) changes for those school‐aged cNH and cCI (6–17 years old) were examined in a tertiary referred center.ResultsThe mean emotion recognition score of the cNH group was significantly better than the cCI. Female speakers' vocal emotions are more easily to be recognized than male speakers' emotion. There was a significant effect of age at test on voice recognition performance. The average score of cCI with full‐spectrum speech was close to the average score of cNH with eight‐channel narrowband vocoder speech. The average performance of voice emotion recognition across speakers for cCI could be predicted by their sensitivity to changes in F0.ConclusionsBetter pitch discrimination ability comes with better voice emotion recognition for Mandarin‐speaking cCI. Besides the F0 cues, cCI are likely to adapt their voice emotion recognition by relying more on secondary cues such as intensity and duration. Although cross‐culture differences exist for the acoustic features of voice emotion, Mandarin‐speaking cCI and their English‐speaking cCI peer expressed a positive effect for age at test on emotion recognition, suggesting the learning effect and brain plasticity. Therefore, further device/processor development to improve presentation of pitch information and more rehabilitative efforts are needed to improve the transmission and perception of voice emotion in Mandarin.Level of evidence3.

Effects of mild-to-moderate sensorineural hearing loss and signal amplification on vocal emotion recognition in middle-aged–older individuals

Previous research has shown deficits in vocal emotion recognition in sub-populations of individuals with hearing loss, making this a high priority research topic. However, previous research has only examined vocal emotion recognition using verbal material, in which emotions are expressed through emotional prosody. There is evidence that older individuals with hearing loss suffer from deficits in general prosody recognition, not specific to emotional prosody. No study has examined the recognition of non-verbal vocalization, which constitutes another important source for the vocal communication of emotions. It might be the case that individuals with hearing loss have specific difficulties in recognizing emotions expressed through prosody in speech, but not non-verbal vocalizations. We aim to examine whether vocal emotion recognition difficulties in middle- aged-to older individuals with sensorineural mild-moderate hearing loss are better explained by deficits in vocal emotion recognition specifically, or deficits in prosody recognition generally by including both sentences and non-verbal expressions. Furthermore a, some of the studies which have concluded that individuals with mild-moderate hearing loss have deficits in vocal emotion recognition ability have also found that the use of hearing aids does not improve recognition accuracy in this group. We aim to examine the effects of linear amplification and audibility on the recognition of different emotions expressed both verbally and non-verbally. Besides examining accuracy for different emotions we will also look at patterns of confusion (which specific emotions are mistaken for other specific emotion and at which rates) during both amplified and non-amplified listening, and we will analyze all material acoustically and relate the acoustic content to performance. Together these analyses will provide clues to effects of amplification on the perception of different emotions. For these purposes, a total of 70 middle-aged-older individuals, half with mild-moderate hearing loss and half with normal hearing will perform a computerized forced-choice vocal emotion recognition task with and without amplification.