Emotional Voice Research Articles

Perceived prolonged stress leads to difficulties in recognizing sadness from voice cues in men but not women.

It has long been known that stress has detrimental effects on cognition (e.g., Alderson & Novack, 2002; Lupien & Lepage, 2001), most notably documented for memory functions (e.g., Schwabe & Wolf, 2013). Interestingly, less is known about the effects of stress on other cognitive functions including language processing. Here, we have examined the effects of self-reported prolonged stress on recognition of emotional language content with a particular emphasis on gender differences. We tested how well 399 participants with different perceived stress levels recognized emotional voice cues. Findings confirm previous results from the emotional prosody literature by demonstrating that women generally outperform men in the vocal emotion recognition task. Crucially, results also revealed that medium levels of perceived stress impair the ability to detect sadness from voice cues in men but not women. These findings were not modulated by task demands (e.g., speeded response) or better acoustic discrimination abilities in women. Results are in line with the idea that perceived stress has a different impact on men versus women and that women have a higher level of experience in voice sadness recognition, potentially due to their predominant role as primary caretakers. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Adopting a languaging approach for teaching about the climate crisis in English language arts

Purpose This paper posits the need for English language arts (ELA) teachers to foster students’ use of languaging about their relations with ecosystems and peers, leading to their engaging in collective action to critique and transform status-quo systems impacting the climate crisis. Design/methodology/approach This paper reviews the current theory of languaging theory and research that focuses on the use of languaging to enact relations with ecosystems and others and voice emotions for transforming communities and reducing emissions contributing to climate change. Findings This review of languaging theory/research leads to identifying examples of teachers having students critique the use of languaging constituting status quo energy and community/transportation systems, respond to examples of characters using languaging in literary texts, using languaging in discussing or writing about the need to address climate change, critiquing languaging in media promoting consumption, using media to interact with audiences and using languaging through engaging in role-play activities. Originality/value This focus on languaging in ELA classrooms is a unique perspective application of languaging theory, leading students to engage in collective, communal action to address the climate crisis.

A Nanoparticle-Based Artificial Ear for Personalized Classification of Emotions in the Human Voice Using Deep Learning.

Artificial intelligence and human-computer interaction advances demand bioinspired sensing modalities capable of comprehending human affective states and speech. However, endowing skin-like interfaces with such intricate perception abilities remains challenging. Here, we have developed a flexible piezoresistive artificial ear (AE) sensor based on gold nanoparticles, which can convert sound signals into electrical signals through changes in resistance. By testing the sensor's performance at both frequency and sound pressure level (SPL), the AE has a frequency response range of 20 Hz to 12 kHz and can sense sound signals from up to 5 m away at a frequency of 1 kHz and an SPL of 126 dB. Furthermore, through deep learning, the device achieves up to 96.9% and 95.0% accuracy in classification and recognition applications for seven emotional and eight urban environmental noises, respectively. Hence, on one hand, our device can monitor the patient's emotional state by their speech, such as sudden yelling and screaming, which can help healthcare workers understand patients' condition in time. On the other hand, the device could also be used for real-time monitoring of noise levels in aircraft, ships, factories, and other high-decibel equipment and environments.

A Combined CNN Architecture for Speech Emotion Recognition.

Emotion recognition through speech is a technique employed in various scenarios of Human-Computer Interaction (HCI). Existing approaches have achieved significant results; however, limitations persist, with the quantity and diversity of data being more notable when deep learning techniques are used. The lack of a standard in feature selection leads to continuous development and experimentation. Choosing and designing the appropriate network architecture constitutes another challenge. This study addresses the challenge of recognizing emotions in the human voice using deep learning techniques, proposing a comprehensive approach, and developing preprocessing and feature selection stages while constructing a dataset called EmoDSc as a result of combining several available databases. The synergy between spectral features and spectrogram images is investigated. Independently, the weighted accuracy obtained using only spectral features was 89%, while using only spectrogram images, the weighted accuracy reached 90%. These results, although surpassing previous research, highlight the strengths and limitations when operating in isolation. Based on this exploration, a neural network architecture composed of a CNN1D, a CNN2D, and an MLP that fuses spectral features and spectogram images is proposed. The model, supported by the unified dataset EmoDSc, demonstrates a remarkable accuracy of 96%.

Emotion recognition using machine learning: Opportunities and challenges for supporting those with autism or depression

Facial recognition technology, driven by advancements in machine learning, has become integral to various applications, from mobile security to access control. Recent developments in deep learning, particularly through convolutional and recurrent neural networks, have paved the way for enhanced emotion recognition capabilities. This paper explores the potential of facial emotion recognition (FER) and voice emotion recognition technologies, assessing their current state and future prospects. While FER techniques have evolved from traditional methods to deep learning approaches, and voice emotion recognition has shown promising results, challenges remain due to the inherent variability in human facial expressions and vocal characteristics. The paper further examines the impact of emotion recognition technologies on mental health, highlighting their potential to provide significant support to individuals with mental health disorders. By addressing the accuracy and application of these technologies, the research underscores the importance of continued innovation and careful implementation to maximize their benefits and mitigate potential risks. Ultimately, emotion recognition provides promising avenues for supporting those with mental illnesses. Aimed at fostering more personalized and effective interventions, these technologies hold the potential to revolutionize mental health care and improve the overall well-being of individuals.

The interactive effects of pedagogical agent role and voice emotion design on children’s learning

The interactive effects of pedagogical agent role and voice emotion design on children’s learning

Facial Animation Strategies for Improved Emotional Expression in Virtual Reality

The portrayal of emotions by virtual characters is crucial in virtual reality (VR) communication. Effective communication in VR relies on a shared understanding, which is significantly enhanced when virtual characters authentically express emotions that align with their spoken words. While human emotions are often conveyed through facial expressions, existing facial animation techniques have mainly focused on lip-syncing and head movements to improve naturalness. This study investigates the influence of various factors in facial animation on the emotional representation of virtual characters. We conduct a comparative and analytical study using an audio-visual database, examining the impact of different animation factors. To this end, we utilize a total of 24 voice samples, representing 12 different speakers, with each emotional voice segment lasting approximately 4–5 s. Using these samples, we design six perceptual experiments to investigate the impact of facial cues—including facial expression, lip movement, head motion, and overall appearance—on the expression of emotions by virtual characters. Additionally, we engaged 20 participants to evaluate and select appropriate combinations of facial expressions, lip movements, head motions, and appearances that align with the given emotion and its intensity. Our findings indicate that emotional representation in virtual characters is closely linked to facial expressions, head movements, and overall appearance. Conversely, lip-syncing, which has been a primary focus in prior studies, seems less critical for conveying emotions, as its accuracy is difficult to perceive with the naked eye. The results of our study can significantly benefit the VR community by aiding in the development of virtual characters capable of expressing a diverse range of emotions.

No Pain, No Gain—Giving Real-Time Emotional Feedback in a Virtual Mirror Improves Collaboration in Virtual Teamwork

This study investigates the impact of real-time emotional feedback on the quality of teamwork conducted over videoconferencing. We developed a framework that provides real-time feedback through a virtual mirror based on facial and voice emotion recognition. In an experiment with 28 teams (84 participants), teams collaborated over Zoom (version 5.16.6) to set up a virtual Mars station using custom simulation software (Mars Star City, version 4.0). Participants were divided into 14 experimental teams, which were shown the virtual mirror, and 14 control teams without it. Team performance was measured by the improvement in the Mars simulation output quality. Our analysis using correlation, multi-level regression, and machine learning revealed that fewer interruptions but an increasing number over time correlated with higher performance. Higher vocal arousal and happiness also enhanced performance. We confirmed that female presence in teams boosts performance. SHAP values indicated that high variability in happiness, head movement, and positive facial valence—an “emotional rollercoaster”—positively predicted team performance. The experimental group outperformed the control group, suggesting that virtual mirroring improves virtual teamwork and that interrupting each other more while speaking less, leads to better results.

Scalability and diversity of StarGANv2-VC in Arabic emotional voice conversion: Overcoming data limitations and enhancing performance

Emotional Voice Conversion (EVC) for under-resourced languages like Arabic faces challenges due to limited emotional speech data. This study explored strategies to mitigate dataset scarcity and improve Arabic EVC performance. Fundamental experiments (Speaker-Dependent, Gender-Dependent, Gender-Independent) were conducted using the KSUEmotions dataset to analyze speaker, gender, and model impacts. Data augmentation techniques like time stretching and phase shuffling artificially increased data diversity. Attention mechanisms integrated into StarGANv2-VC aimed to better capture emotional cues. Transfer learning leveraged the larger English Emotional Speech Database (ESD) to enhance the Arabic system. A novel “Reordering Speaker-Emotion Data” approach treated each emotion as a separate speaker to expand the emotional variability. Our comprehensive approach, combining transfer learning, data augmentation, and architectural modifications, demonstrates the potential to overcome dataset limitations and enhance the performance of Arabic EVC systems.

Using Vocal-Based Emotions as a Human Error Prevention System with Convolutional Neural Networks

Human error is a mark assigned to an event that has negative effects or does not produce a desired result, with emotions playing an important role in how humans think and behave. If we detect feelings early, it may decrease human error. The human voice is one of the most powerful tools that can be used for emotion recognition. This study aims to reduce human error by building a system that detects positive or negative emotions of a user like (happy, sad, fear, and anger) through the analysis of the proposed vocal emotion component using Convolutional Neural Networks. By applying the proposed method to an emotional voice database (RAVDESS) using Librosa for voice processing and PyTorch, with the emotion classification of (happy/angry), the results show a better accuracy (98%) in comparison to the literature with regard to making a decision to deny or allow a user to access sensitive operations or send a warning to the system administrator prior to accessing system resources.

Style Generative Adversarial Network Combined with Dynamic Fundamental Frequency Difference Compensation: A Practical and Efficient Method for Emotional Voice Conversion

This paper explores a novel emotional voice conversion (EVC) method based on the style generative adversarial network combined with dynamic fundamental frequency difference compensation. On the one hand, in terms of spectrum mapping, based on the method of EVC using star generative adversarial network, we propose to replace one-hot vectors with the emotion style features extracted from spectrum to represent emotional information. It enables the conversion not only to fully learn emotion style from the target speech, but also to transfer unseen emotion styles to a new utterance, that is, one-shot EVC. On the other hand, given that the traditional logarithm Gaussian normalization transformation and its some variants for prosody transfer are not enough to reflect the fine-grained fundamental frequency distribution difference between the different emotional speech, we propose the improved strategy of dynamic fundamental frequency difference compensation. The experimental results show that our proposed method can accomplish high quality one-shot EVC, significantly outperforming the baseline in terms of speech quality and emotional saturation in both objective and subjective evaluations.

Emotional voice conversion using DBiLSTM-NN with MFCC and LogF0 features

Emotional voice conversion using DBiLSTM-NN with MFCC and LogF0 features

Enhancements in Immediate Speech Emotion Detection: Harnessing Prosodic and Spectral Characteristics

Speech is essential to human communication for expressing and understanding feelings. Emotional speech processing has challenges with expert data sampling, dataset organization, and computational complexity in large-scale analysis. This study aims to reduce data redundancy and high dimensionality by introducing a new speech emotion recognition system. The system employs Diffusion Map to reduce dimensionality and includes Decision Trees and K-Nearest Neighbors(KNN)ensemble classifiers. These strategies are suggested to increase voice emotion recognition accuracy. Speech emotion recognition is gaining popularity in affective computing for usage in medical, industry, and academics. This project aims to provide an efficient and robust real-time emotion identification framework. In order to identify emotions using supervised machine learning models, this work makes use of paralinguistic factors such as intensity, pitch, and MFCC. In order to classify data, experimental analysis integrates prosodic and spectral information utilizing methods like Random Forest, Multilayer Perceptron, SVM, KNN, and Gaussian Naïve Bayes. Fast training times make these machine learning models excellent for real-time applications. SVM and MLP have the highest accuracy at 70.86% and 79.52%, respectively. Comparisons to benchmarks show significant improvements over earlier models.

The Impact of Portraits, Product Pricing, and Voice Emotion on Crowdfunding Success Rates Across Different Genders

In recent years, crowdfunding as an internet entrepreneurial method has gradually entered the public's attention. Crowdfunding platforms have witnessed significant development across various sectors, including charity, cultural creativity, and technological innovation. This study explores the impact of gender, visibility, product pricing, and voice emotion on crowdfunding success rates within the American platform Kickstarter. Emphasis is placed on understanding how gender factors influence crowdfunding in different contexts, aiming to assist project founders to enhance their project success rates. The study analyzes success rates of founders of different genders in various situations. The findings under face visibility conditions, the success rate of male funders is significantly lower than the mean success rate of female funders. Crowdfunding success rates are generally higher when product prices exceed the average, with female funders consistently achieving higher success rates than male funders in this scenario. In crowdfunding video voiceovers, for founders of different genders, the impact of energy levels in crowdfunding videos on success rates is not significant. However, higher upset values lead to a greater success rate, with a similar increase for founders of different genders. The study concludes that in crowdfunding videos where faces are shown, female fundraisers significantly enhance success rates. Higher crowdfunding project prices lead to higher success rates, and female fundraisers are more likely to succeed in these projects. While the degree of discouragement in voiceovers significantly influences projects, gender is not a crucial influencing factor.

Family Emotional Decorative Paintings: Interaction Design for Communication

This paper describes an approach applied to decorative home paintings: an emotional interaction approach. The study utilizes the theory of constructed emotion and the Wizard of Oz research methodology to extend the user's emotional experience in home scenarios with emotional self-reflection and regulation. Family Emotion Decorative Painting uses the family-shared photo library as the image base data, obtains the emotion type through facial expression and voice emotion recognition, and correlates the moment's emotion with the emotion reacted in the picture. Through qualitative research, we found that family emotional decorative painting can bring emotional arousal and soothing to participants. Family Emotional Decorative Paintings also provoked reflection on technology in and around the home. These findings suggest several opportunities, such as designing for anticipation, better supporting reflection on the past. The study also allowed the team to question the traditional interaction design view that more frequent and prolonged human-computer interactions are a measure of good design. The team will next focus on ways for users to engage in novel and meaningful ways outside of the application.

Self-Rated Confidence in Vocal Emotion Recognition Ability: The Role of Gender.

We studied the role of gender in metacognition of voice emotion recognition ability (ERA), reflected by self-rated confidence (SRC). To this end, we guided our study in two approaches: first, by examining the role of gender in voice ERA and SRC independently and second, by looking for gender effects on the ERA association with SRC. We asked 100 participants (50 men, 50 women) to interpret a set of vocal expressions portrayed by 30 actors (16 men, 14 women) as defined by their emotional meaning. Targets were 180 repetitive lexical sentences articulated in congruent emotional voices (anger, sadness, surprise, happiness, fear) and neutral expressions. Trial by trial, the participants were assigned retrospective SRC based on their emotional recognition performance. A binomial generalized linear mixed model (GLMM) estimating ERA accuracy revealed a significant gender effect, with women encoders (speakers) yielding higher accuracy levels than men. There was no significant effect of the decoder's (listener's) gender. A second GLMM estimating SRC found a significant effect of encoder and decoder genders, with women outperforming men. Gamma correlations were significantly greater than zero for women and men decoders. In spite of varying interpretations of gender in each independent rating (ERA and SRC), our results suggest that both men and women decoders were accurate in their metacognition regarding voice emotion recognition. Further research is needed to study how individuals of both genders use metacognitive knowledge in their emotional recognition and whether and how such knowledge contributes to effective social communication.

Hybrid model for speech emotion recognition of normal and autistic children (SERNAC)

Since the last decade, autism spectrum disorder (ASD) has been used as a general term to describe a wide range of conditions, including autistic syndrome, Asperger's disorder, and pervasive developmental disability. This problem emerges as a decreased ability to share emotions and a greater difficulty understanding others' feelings, leading to increased social communication difficulties. To assist patients with ASD, we proposed a concept that incorporates speech emotion detection technologies, which are widely used in the field of human-computer interaction (particularly youngsters). An algorithm based on a novel method for classifying normal and autistic children's speech emotions is implemented in this article. The training data set is treated to a new approach after all features have been extracted. The technique discussed in this study is the creation of a hybrid algorithm that serves as a classifier for normal and autistic children's speech emotions. Voice emotion recognition can be identified accurately and with a lower error rate. The data collection includes speech samples from 200 normal and 250 autistic groups in four moods (Angry, Happy, Neutral and Sad). As per research findings, the implemented hybrid algorithm for Normal and Autistic Children Speech Emotions (SERNAC) outperformed the existing classifiers by increasing accuracy.

You better listen to yourself: studying metacognitive efficiency in emotion recognition by voice

PurposeMetacognition, or the ability to monitor the performance of oneself, is known for its fundamental importance for human behavior adjustments. However, studies of metacognition in social behaviors focused on emotion recognition are relatively scarce. In the current study, we aimed to examine the effectiveness of metacognition, measured by self-rated confidence in voice emotion recognition tasks within healthy individuals.MethodsWe collected 180 audio-recorded lexical sentences portraying discrete emotions: anger, happiness, sadness, fear, surprise, and neutrality expressions. Upon listening to voice stimuli, participants (N = 100; 50 females, 50 males) completed the perception task of recognition of emotion. After each trial, a confidence rating (CR) was assigned.ResultsA series of one-tailed t-tests showed that the differences in mean CRs assigned to correct and incorrect performances were significant for all emotions and neutral expression.ConclusionsOur preliminary results demonstrate efficiency in metacognition of emotion recognition by voice. Theoretically, our results support the difference between accuracy in metacognition, measured by CR, and efficiency in metacognition, as it specified the CR between correct and incorrect performance. To gain better insights into practical issues, further studies are needed to examine whether and what are the differences between accuracy and efficiency in metacognition, as part of social communication.

THE IMPACTS OF NATIVE SPEAKER TEACHERS’ NONVERBAL COMMUNICATION IN EFL CLASSROOMS

ABSTRACT This research tries to analyze nonverbal communication types employed by native speaker teachers when teaching English as a Foreign Language (EFL) classes and the effect of those nonverbal communications on students. This study employed descriptive methodology. Two native-speaking teachers in English and fifteen students comprise the study's participants. Observation and interviews were utilized to acquire data. This study found that teachers frequently employ six types of nonverbal communication. These included the application of kinesics, proxemics, ocelusics, chronemics, paravocalics, and physical appearance. The data analysis also revealed that the teachers’ nonverbal communication had both good and bad effects on the pupils. The teachers’ use of appropriate hand gestures, eye contact, and voice emotion when teaching the topic significantly helped to develop healthy teacher-student relations, particularly in motivating and appreciating students. However, eye contact can induce anxiety in students. Keywords: Nonverbal Communication, Effects of Nonverbal Communication, EFL Classroom, Native Speaker Teacher

Acoustic divergence from the training sample determines talker identification accuracy for emotional voices

Different emotional states introduce substantial acoustic variations in talkers’ voices. It remains unclear how within-talker variability across emotional states affects listeners' ability to maintain perceptual constancy during talker identification. Here, we investigated (1) how changes in talkers’ emotional state affected talker identification accuracy, (2) how emotional state affected key features of voice acoustics, and (3) how emotion-related changes in these acoustic features affected listeners’ talker identification performance. Forty-eight listeners learned to identify talkers from speech expressing one emotional state (neutral, fearful, or angry) and then attempted to generalize that knowledge to speech expressing another emotional state. Talker identification accuracy was significantly worse in untrained emotions. Changes in voice acoustics across emotions were characterized for mean F0, F0 variability, jitter, HNR, speaking rate, and mean F2. To determine how emotion-related acoustic changes affected talker identification, we modeled talker identification accuracy as a function of difference in these features between training and test stimuli. Accuracy decreased as acoustic differences increased, regardless of talkers’ emotion. Thus, perceptual constancy depends on acoustic similarity to prior experience with a talker’s voice. Larger acoustic deviations, like those introduced by changes in emotional state, are more likely to cause a listener to misidentify a talker.