Vocal Emotional Expressions Research Articles

A lightweight and privacy preserved federated learning ecosystem for analyzing verbal communication emotions in identical and non-identical databases

The lack of vocal emotional expression is a major deficit in social communication disorders. The current scenario of artificial intelligence focuses on collaborative training of deep learning models without losing data privacy. The primary objective of this paper is to propose a federated learning-based classification model to identify and analyze the emotional capabilities of individuals with vocal emotion deficits. The methodology has developed a collaborative and privacy-preserved approach using federated learning for training the deep learning models. The proposed methodology utilizes Mel-frequency Cepstral Coefficients (MFCC) to preprocess audio recordings. The four datasets (RAVDESS, CREMA, TESS, SAVEE) including emotion-based classified audio recordings were collected from open sources. The collected audio recordings are 3 s each and the total data set has 668376 audio files with happy - 175119 files, sad – 172611 files, angry – 176346 files, and normal - 144300 files. Further, the input audio was pre-processed to generate MFCC features. The study began with extracting features from multiple pre-trained DL models as its base model. Then, the performance of the federated learning (FL) model was tested on independent and identically distributed (IID) and non-IID data. Further, this paper presents a federated deep learning-based multimodal system for verbal communication emotions classification that uses audio datasets to meet data privacy requirements by DL on the FL ecosystem. As per the findings, the federated learning trained model provides nearly similar parametric results in comparison to base model training. For IID data, the model had 99.71 % validation accuracy, precision (99.73 %), recall (99.69 %), and validation loss (0.01). The FL architecture with non-IID data outperformed these measures with validation accuracy (99.97 %), precision (99.97 %), recall (99.97 %), and least loss (0). Hence the acquired results support the utilization of federated learning ecosystem-based trained models with identically and non-identically distributed audio features from emotion identification without losing parametric results. In conclusion, the proposed techniques could be applied to identify verbal emotional deficits in individuals and could support developing emerging technological interventions for their well-being.

Maternal childhood emotional abuse increases cardiovascular responses to children's emotional facial expressions.

Parents with a history of childhood maltreatment may be more likely to respond inadequately to their child's emotional cues, such as crying or screaming, due to previous exposure to prolonged stress. While studies have investigated parents' physiological reactions to their children's vocal expressions of emotions, less attention has been given to their responses when perceiving children's facial expressions of emotions. The present study aimed to determine if viewing facial expressions of emotions in children induces cardiovascular changes in mothers (hypo- or hyper-arousal) and whether these differ as a function of childhood maltreatment. A total of 104 mothers took part in this study. Their experiences of childhood maltreatment were measured using the Childhood Trauma Questionnaire (CTQ). Participants' electrocardiogram signals were recorded during a task in which they viewed a landscape video (baseline) and images of children's faces expressing different intensities of emotion. Heart rate variability (HRV) was extracted from the recordings as an indicator of parasympathetic reactivity. Participants presented two profiles: one group of mothers had a decreased HRV when presented with images of children's facial expressions of emotions, while the other group's HRV increased. However, HRV change was not significantly different between the two groups. The interaction between HRV groups and the severity of maltreatment experienced was marginal. Results suggested that experiences of childhood emotional abuse were more common in mothers whose HRV increased during the task. Therefore, more severe childhood experiences of emotional abuse could be associated with mothers' cardiovascular hyperreactivity. Maladaptive cardiovascular responses could have a ripple effect, influencing how mothers react to their children's facial expressions of emotions. That reaction could affect the quality of their interaction with their child. Providing interventions that help parents regulate their physiological and behavioral responses to stress might be helpful, especially if they have experienced childhood maltreatment.

An Interactive Feeder to Induce and Assess Emotions from Vocalisations of Chickens.

Understanding the emotional states of animals is a long-standing research endeavour that has clear applications in animal welfare. Vocalisations are emerging as a promising way to assess both positive and negative emotional states. However, the vocal expression of emotions in birds is a relatively unexplored research area. The goal of this study was to develop an interactive feeding system that would elicit positive and negative emotional states, and collect recordings of the vocal expression of these emotions without human interference. In this paper, the mechatronic design and development of the feeder is described. Design choices were motivated by the desire for the hens to voluntarily interact with the feeder and experience the different stimuli that were designed to induce (1) positive low-arousal, (2) positive high-arousal, (3) negative low-arousal, and (4) negative high-arousal states. The results showed that hens were motivated to engage with the feeder despite the risk of receiving negative stimuli and that this motivation was sustained for at least 1 week. The potential of using the interactive feeder to analyse chicken vocalisations related to emotional valence and arousal is being explored, offering a novel proof of concept in animal welfare research. Preliminary findings suggest that hens vocalised in response to all four stimulus types, with the number of vocalisations, but not the probability of vocalising, distinguishing between low- and high-arousal states. Thus, the proposed animal-computer interaction design has potential to be used as an enrichment device and for future experiments on vocal emotions in birds.

Vocal Emotional Expressions in Mothers with and without a History of Major Depressive Disorder

Vocal Emotional Expressions in Mothers with and without a History of Major Depressive Disorder

Acoustic analysis of clients' expression of self-compassion, self-criticism, and self-protection within emotion focused therapy video sessions.

When it comes to the non-verbal communication of emotions, it is apparent that the human voice is one of the main ways of expressing emotion and is increasingly important in psychotherapeutic dialog. There is ample research focusing on the vocal expression of emotions. However, to date the analysis of the vocal quality of clients' in-sessional emotional experience remains largely unexplored. Moreover, there is generally a gap within the psychotherapy literature in the understanding of the vocal character of self-compassion, self-criticism, and protective anger. In this study we investigated how clients vocally convey self-compassion, self-protection and self-criticism in Emotion Focused therapy sessions. For this purpose we investigated 12 commercially available Emotion Focused Therapy videos that employed a two chair or empty chair dialog. Praat software was used for the acoustic analysis of the most common features - pitch (known as fundamental frequency or F0) and intensity (voice amplitude, i.e., loudness). Results showed that intensity was significantly higher for self-criticism and self-protection than for self-compassion. Regarding pitch the findings showed no significant differences between the three states. More research analyzing acoustic features in a larger number of cases is required to obtain a deeper understanding of clients' vocal expression of self-compassion, self-protection and self-criticism in Emotion Focused Therapy.

Vocal expression of emotions discriminates dementia with Lewy bodies from Alzheimer's disease.

DLB showed distinctive reduction in vocal expression of emotions.Cognitive impairment was associated with reduced vocal emotional expression in DLB.Insular atrophy was associated with reduced vocal emotional expression in DLB.Emotional expression measures successfully differentiated DLB from AD or controls.

Principal dimensions of laryngeal vocal control in a computational model of voice production

Voice production is constrained by laryngeal physiology and physics. Such constraints may present themselves as principal dimensions that are shared among speakers in how they produce and perceive voice. In this study, we attempt to identify such principal dimensions in voice outcome measures and the underlying laryngeal control mechanisms in a three-dimensional computational model of voice production. A large-scale voice simulation was performed with parametric variations in vocal fold geometry, stiffness, glottal gap, and subglottal pressure. Principal component analysis was applied to data combining both the laryngeal control parameters and acoustic and aerodynamic voice outcome measures. The results showed two dominant dimensions of vocal control. The first dimension describes interaction between respiration and vocal fold adduction in a way that increases glottal flow amplitude and vocal intensity at the cost of decreasing high-frequency harmonic production. The second dimension mainly describes control of medial surface thickness and glottal closure, which allows simultaneous increase in vocal intensity and high-frequency harmonic production but also increases risk of vocal fold injury. A third dimension with a reduced weight describes control of the fundamental frequency. The importance of these principal control dimensions to vocal expression of emotion and vocal health is discussed.

Vision plays a calibrating role in discriminating threat-related vocal emotions.

The ability to reliably discriminate vocal expressions of emotion is crucial to engage in successful social interactions. This process is arguably more crucial for blind individuals, since they cannot extract social information from faces and bodies, and therefore chiefly rely on voices to infer the emotional state of their interlocutors. Blind have demonstrated superior abilities in several aspects of auditory perception, but research on their ability to discriminate vocal features is still scarce and has provided unclear results. Here, we used a gating psychophysical paradigm to test whether early blind people would differ from individually matched sighted controls at the recognition of emotional expressions. Surprisingly, blind people showed lower performance than controls in discriminating specific vocal emotions. We presented segments of nonlinguistic emotional vocalizations of increasing duration (100-400 ms), portraying five basic emotions (fear, happy, sad, disgust, and angry), and we asked our participants for an explicit emotion categorization task. We then calculated sensitivity indices and confusion patterns of their performance. We observed better performance of the sighted group in the discrimination of angry and fearful expression, with no between-group differences for other emotions. This result supports the view that vision plays a calibrating role for specific threat-related emotions specifically. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The role of the age and gender, and the complexity of the syntactic unit in the perception of affective emotions in voice.

The study aimed to identify (1) whether the age and gender of listeners and the length of vocal stimuli affect emotion discrimination accuracy in voice; and (2) whether the determined level of expression of perceived affective emotions is age and gender-dependent. Thirty-two age-matched listeners listened to 270 semantically neutral voice samples produced in neutral, happy, and angry intonation by ten professional actors. The participants were required to categorize the auditory stimulus based on three options and judge the intensity of emotional expression in the sample using a customized tablet web interface. The discrimination accuracy of happy and angry emotions decreased with age, while accuracy in discriminating neutral emotions increased with age. Females rated the intensity level of perceived affective emotions higher than males across all linguistic units. These were: for angry emotions in words (z = -3.599, p < .001), phrases (z = -3.218, p = .001), and texts (z = -2.272, p = .023), for happy emotions in words (z = -5.799, p < .001), phrases (z = -4.706, p < .001), and texts (z = -2.699, p = .007). Accuracy in perceiving vocal expressions of emotions varies according to age and gender. Young adults are better at distinguishing happy and angry emotions than middle-aged adults, while middle-aged adults tend to categorize perceived affective emotions as neutral. Gender also plays a role, with females rating expressions of affective emotions in voices higher than males. Additionally, the length of voice stimuli impacts emotion discrimination accuracy.

Automatic Brain Categorization of Discrete Auditory Emotion Expressions.

Seamlessly extracting emotional information from voices is crucial for efficient interpersonal communication. However, it remains unclear how the brain categorizes vocal expressions of emotion beyond the processing of their acoustic features. In our study, we developed a new approach combining electroencephalographic recordings (EEG) in humans with a frequency-tagging paradigm to 'tag' automatic neural responses to specificcategories of emotion expressions. Participants were presented with a periodic stream of heterogeneous non-verbal emotional vocalizations belonging to five emotion categories: anger, disgust, fear, happiness and sadness at 2.5Hz (stimuli length of 350ms with a 50ms silent gap between stimuli). Importantly, unknown to the participant, a specific emotion category appeared at a target presentation rate of 0.83Hz that would elicit an additional response in the EEG spectrum only if the brain discriminates the target emotion category from other emotion categories and generalizes across heterogeneous exemplars of the target emotion category. Stimuli were matched across emotion categories for harmonicity-to-noise ratio, spectral center of gravity and pitch. Additionally, participants were presented with a scrambled version of the stimuli with identical spectral content and periodicity but disrupted intelligibility. Both types of sequences had comparable envelopes and early auditory peripheral processing computed via the simulation of the cochlear response. We observed that in addition to the responses at the general presentation frequency (2.5Hz) in both intact and scrambled sequences, a greater peak in the EEG spectrum at the target emotion presentation rate (0.83Hz) and its harmonics emerged in the intact sequence in comparison to the scrambled sequence. The greater response at the target frequency in the intact sequence, together with our stimuli matching procedure, suggest that the categorical brain response elicited by a specific emotion is at least partially independent from the low-level acoustic features of the sounds. Moreover, responses at the fearful and happy vocalizations presentation rates elicited different topographies and differenttemporal dynamics, suggesting that different discrete emotions are represented differently in the brain. Our paradigm revealed the brain's ability to automatically categorize non-verbal vocal emotion expressions objectively (at a predefined frequency of interest), behavior-free, rapidly (in few minutes of recording time) and robustly (with a high signal-to-noise ratio), making it a useful tool to study vocal emotion processing and auditory categorization in generaland in populations where behavioral assessments are more challenging.

Recognition of Facial and Vocal Emotional Expressions by SOAR Model

Recognition of Facial and Vocal Emotional Expressions by SOAR Model

Vocal Emotional Expression in Parkinson’s Disease: Roles of Sex and Emotions

Introduction: Parkinson’s disease (PD) commonly causes speech impairments, including difficulties in expressing emotions through voice. Method: The objective of this study was to investigate gendered vocal expressions of fear, anger, sadness, and happiness for mild to moderate PD. Prosodic features (related to fundamental frequency (F0), intensity (I), speech rate, articulation rate, and number and duration of pauses) and acoustic correlates of voice quality (CPPS, jitter, shimmer, and HNR) were collected from 14 patients with PD (mean age = 69.93; SD = 7.12; 8 males, 6 females) and 13 healthy controls (HC) (mean age = 68.13; SD = 8.27; 5 males, 8 females) matched for age, sex, and years of education. The utterances were extracted from four emotional and one neutral text. The neutral utterance and the emotional utterances were compared. Intra-sex comparison (female with PD vs. female HC and male with PD vs. male HC) and inter-sex comparison (female vs. male both for patients with PD and for HC), were performed with the Mann–Whitney test. A Mann–Whitney test was also used to compare the different emotional conditions, considering sex and PD diagnosis as well. Results: No significant intra-sex differences were found for the neutral speech, but inter-sex differences emerged. Regarding emotional speech, females with PD featured lower MaxF0 than female HCs for happiness and higher intensity variability (SD I) for sadness. Utterances by females with PD had lower CPPS than utterances by HCs for anger and fear. Utterances by males with PD had lower minimum intensity (MinI) than utterances by male HCs when expressing fear. Conclusions: Emotional vocal expression in individuals with PD was found to be impaired and showed sex differences. These findings have the potential to significantly impact the quality of life of PD patients.

Facial and Vocal Emotion Recognition in Adolescence: A Systematic Review

AbstractThe ability to recognize emotion is important to wellbeing and building relationships with others, making this skill important in adolescence. Research investigating adolescents’ ability to recognize facial and vocal emotion expressions has reported differing conclusions about the pattern of emotion recognition across this developmental period. This systematic review aimed to clarify the pattern of recognition for facial and vocal emotion expressions, and the relationship of performance to different task and emotion expression characteristics. A comprehensive and systematic search of the literature was conducted using six databases. To be eligible for inclusion, studies had to report data for adolescents between 11 and 18 years of age and measure accuracy of the recognition of emotion cues in either the face or voice. A total of 2333 studies were identified and 47 met inclusion criteria. The majority of studies focused on facial emotion recognition. Overall, early, mid-, and late-adolescents showed a similar pattern of recognition for both facial and vocal emotion expressions with the exception of Sad facial expressions. Sex of the participant also had minimal impact on the overall recognition of different emotions. However, analysis showed considerable variability according to task and emotion expression characteristics. Future research needs to increase focus on recognition of complex emotions, and low-intensity emotion expressions as well as the influence of the inclusion of Neutral as a response option.

The priming effects of emotional vocal expressions on face encoding and recognition: An ERP study.

Previous studies have suggested that emotional primes, presented as visual stimuli, influence face memory (e.g., encoding and recognition). However, due to stimulus-associated issues, whether emotional primes affect face encoding when the priming stimuli are presented in an auditory modality remains controversial. Moreover, no studies have investigated whether the effects of emotional auditory primes are maintained in later stages of face memory, such as face recognition. To address these issues, participants in the present study were asked to memorize angry and neutral faces. The faces were presented after a simple nonlinguistic interjection expressed with angry or neutral prosodies. Subsequently, participants completed an old/new recognition task in which only faces were presented. Event-related potential (ERP) results showed that during the encoding phase, all faces preceded by an angry vocal expression elicited larger N170 responses than faces preceded by a neutral vocal expression. Angry vocal expression also enhanced the late positive potential (LPP) responses specifically to angry faces. In the subsequent recognition phase, preceding angry vocal primes reduced early LPP responses to both angry and neutral faces and late LPP responses specifically to neutral faces. These findings suggest that the negative emotion of auditory primes influenced face encoding and recognition.

Cultural differences in vocal expression analysis: Effects of task, language, and stimulus-related factors.

Cultural context shapes the way that emotions are expressed and socially interpreted. Building on previous research looking at cultural differences in judgements of facial expressions, we examined how listeners recognize speech-embedded emotional expressions and make inferences about a speaker’s feelings in relation to their vocal display. Canadian and Chinese participants categorized vocal expressions of emotions (anger, fear, happiness, sadness) expressed at different intensity levels in three languages (English, Mandarin, Hindi). In two additional tasks, participants rated the intensity of each emotional expression and the intensity of the speaker’s feelings from the same stimuli. Each group was more accurate at recognizing emotions produced in their native language (in-group advantage). However, Canadian and Chinese participants both judged the speaker’s feelings to be equivalent or more intense than their actual display (especially for highly aroused, negative emotions), suggesting that similar inference rules were applied to vocal expressions by the two cultures in this task. Our results provide new insights on how people categorize and interpret speech-embedded vocal expressions versus facial expressions and what cultural factors are at play.

Vocal expression of emotions in farmed spotted paca (Cuniculus paca)

Vocal emissions in non-human mammals can be used as non-invasive indicators of animal emotion and welfare. Therefore, we aimed to validate the use of acoustic parameters as indicators of affective states and welfare in farmed spotted paca (Cuniculus paca). We recorded the vocalizations of 36 pacas, living in 12 groups (two females and one male/group), during four distinct experiences that were assumed to generate different affective states (negative, positive, ambiguous, and highly positive). Spotted pacas emitted more snorts, either as single elements or phrases, when experiencing a negative affective state. Four acoustic parameters of snorts, roars, and barks, which were recorded during all experiences, differed according to the affective valence assumed to be experienced by the spotted pacas. During the negative condition – morning cleaning routine spotted pacas emitted many more snort calls, which are characterized by a shift in the mean amplitude towards a higher level of intensity and as well as by a shift in the energy distribution (Q75) towards a higher frequency than those emitted in the positive condition compared to those emitted in the assumed positive valence condition – feeding time with the regular diet. These call-characteristics, along with the increase in the mean amplitude of snort calls from the lowest to the highest arousal levels, may thus reflect negative affective state and arousal in spotted paca. Therefore, the increase in the overall emission of snorts together with changes in their acoustic parameters may be valuable as indicators of negative emotions of C. paca, and thus have potential as an automated and non-invasive tool for welfare assessment in this species.

Differences of people with visual disabilities in the perceived intensity of emotion inferred from speech of sighted people in online communication settings

Purpose As humans convey information about emotions by speech signals, emotion recognition via auditory information is often employed to assess one’s affective states. There are numerous ways of applying the knowledge of emotional vocal expressions to system designs that accommodate users’ needs adequately. Yet, little is known about how people with visual disabilities infer emotions from speech stimuli, especially via online platforms (e.g., Zoom). This study focussed on examining the degree to which they perceive emotions strongly or weakly, i.e., perceived intensity but also investigating the degree to which their sociodemographic backgrounds affect them perceiving different intensity levels of emotions when exposed to a set of emotional speech stimuli via Zoom. Materials and methods A convenience sample of 30 individuals with visual disabilities participated in zoom interviews. Participants were given a set of emotional speech stimuli and reported the intensity level of the perceived emotions on a rating scale from 1 (weak) to 8 (strong). Results When the participants were exposed to the emotional speech stimuli, calm, happy, fearful, sad, and neutral, they reported that neutral was the dominant emotion they perceived with the greatest intensity. Individual differences were also observed in the perceived intensity of emotions, associated with sociodemographic backgrounds, such as health, vision, job, and age. Conclusions The results of this study are anticipated to contribute to the fundamental knowledge that will be helpful for many stakeholders such as voice technology engineers, user experience designers, health professionals, and social workers providing support to people with visual disabilities. IMPLICATIONS FOR REHABILITATION Technologies equipped with alternative user interfaces (e.g., Siri, Alexa, and Google Voice Assistant) meeting the needs of people with visual disabilities can promote independent living and quality of life. Such technologies can also be equipped with systems that can recognize emotions via users’ voice, such that users can obtain services customized to fit their emotional needs or adequately address their emotional challenges (e.g., early detection of onset, provision of advice, and so on). The results of this study can be beneficial to health professionals (e.g., social workers) who work closely with clients who have visual disabilities (e.g., virtual telehealth sessions) as they could gain insights or learn how to recognize and understand the clients’ emotional struggle by hearing their voice, which is contributing to enhancement of emotional intelligence. Thus, they can provide better services to their clients, leading to building a strong bond and trust between health professionals and clients with visual disabilities even they meet virtually (e.g., Zoom).

Emotion expression through spoken language in Huntington disease

Patients with Huntington's disease suffer from disturbances in the perception of emotions; they do not correctly read the body, vocal and facial expressions of others. With regard to the expression of emotions, it has been shown that they are impaired in expressing emotions through face but up until now, little research has been conducted about their ability to express emotions through spoken language.To better understand emotion production in both voice and language in Huntington's Disease (HD), we tested 115 individuals: 68 patients (HD), 22 participants carrying the mutant HD gene without any motor symptoms (pre-manifest HD), and 25 controls in a single-centre prospective observational follow-up study. Participants were recorded in interviews in which they were asked to recall sad, angry, happy, and neutral stories. Emotion expression through voice and language was investigated by comparing the identifiability of emotions expressed by controls, preHD and HD patients in these interviews. To assess separately vocal and linguistic expression of emotions in a blind design, we used machine learning models instead of a human jury performing a forced-choice recognition test. Results from this study showed that patients with HD had difficulty expressing emotions through both voice and language compared to preHD participants and controls, who behaved similarly and above chance. In addition, we did not find any differences in expression of emotions between preHD and healthy controls. We further validated our newly proposed methodology with a human jury on the speech produced by the controls. These results are consistent with the hypothesis that emotional deficits in HD are caused by impaired sensori-motor representations of emotions, in line with embodied cognition theories. This study also shows how machine learning models can be leveraged to assess emotion expression in a blind and reproducible way.

An acoustic study on the voice of emotion in Beijing Chinese adults

Although Mandarin and Beijing Chinese are both tone language, Beijing Chinese, a dialect spoken in Beijing, is considered more colloquial. This study examines the acoustic properties of the vocal expression of emotions in Beijing Chinese. Using an emotional-stories elicitation method (Anolli etal., 2008), 30 native speakers of Beijing Chinese produced one target sentence in eight emotions, including anger, contempt, fear, guilt, joy, pride, sadness, and shame. The same sentence produced in a neutral condition was used as the baseline. The results showed that mean f0 values were high for shame and fear, while low for pride and anger. Pitch variation was high for shame and guilt, but low in sad and pride. Shame and sad also yielded high intensity variation. For speech rate, guilt was expressed with the fastest speech rate, while sad with the slowest. Longer pauses were found for sad, and shorter pauses for anger and guilt. Unlike Anolli et al. (2008), in which Mandarin speakers showed low f0 variation and slow speech rate for guilt, the current results showed that pitch variation was high for guilt, and it was characterized by the fastest speech rate in Beijing. It suggests that for tone language, regardless of the similar phonological systems, emotions were expressed differently in a more colloquial situation.

Longitudinal change in neural response to vocal emotion in adolescence.

Adolescence is associated with maturation of function within neural networks supporting the processing of social information. Previous longitudinal studies have established developmental influences on youth’s neural response to facial displays of emotion. Given the increasing recognition of the importance of non-facial cues to social communication, we build on existing work by examining longitudinal change in neural response to vocal expressions of emotion in 8- to 19-year-old youth. Participants completed a vocal emotion recognition task at two timepoints (1 year apart) while undergoing functional magnetic resonance imaging. The right inferior frontal gyrus, right dorsal striatum and right precentral gyrus showed decreases in activation to emotional voices across timepoints, which may reflect focalization of response in these areas. Activation in the dorsomedial prefrontal cortex was positively associated with age but was stable across timepoints. In addition, the slope of change across visits varied as a function of participants’ age in the right temporo-parietal junction (TPJ): this pattern of activation across timepoints and age may reflect ongoing specialization of function across childhood and adolescence. Decreased activation in the striatum and TPJ across timepoints was associated with better emotion recognition accuracy. Findings suggest that specialization of function in social cognitive networks may support the growth of vocal emotion recognition skills across adolescence.