Emotional Speech Research Articles

EmoBone: A Multinational Audio Dataset of Emotional Bone Conducted Speech

AbstractThis paper introduces EmoBone, a comprehensive audio‐only emotional bone‐conducted speech dataset featuring speakers from various countries. The dataset comprises speeches from 28 individuals representing 10 different nations, with each participant delivering 10 sentences designed to evoke distinct emotions. In addition to an air‐conducted microphone, the recordings utilized bone conduction technology, transmitting sound directly to the speakers' inner ears, ensuring high‐quality emotional speech recordings. To assess the validity of the dataset, 80 university students from Bangladesh listened to the recordings and successfully identified the expressed emotions with an accuracy exceeding 76%. Statistical methods were also employed to evaluate the reliability of the dataset, revealing a high level of agreement among raters. EmoBone, with a cumulative duration surpassing 19 h and 15 680 unique utterances, stands as the most extensive emotional speech dataset available. This makes it a valuable tool for studying how emotional speech varies across cultures. Furthermore, due to its utilization of bone conduction technology, EmoBone facilitates the study of acoustic features in emotional speech from diverse dimensions. The data that supports the findings of this study is available upon reasonable request. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Self-supervised Learning for Speech Emotion Recognition Task Using Audio-visual Features and Distil Hubert Model on BAVED and RAVDESS Databases

Self-supervised Learning for Speech Emotion Recognition Task Using Audio-visual Features and Distil Hubert Model on BAVED and RAVDESS Databases

Speech Emotion Recognition Using a Multi-Time-Scale Approach to Feature Aggregation and an Ensemble of SVM Classifiers

Due to its relevant real-life applications, the recognition of emotions from speech signals constitutes a popular research topic. In the traditional methods applied for speech emotion recognition, audio features are typically aggregated using a fixed-duration time window, potentially discarding information conveyed by speech at various signal durations. By contrast, in the proposed method, audio features are aggregated simultaneously using time windows of different lengths (a multi-time-scale approach), hence, potentially better utilizing information carried at phonemic, syllabic, and prosodic levels compared to the traditional approach. A genetic algorithm is employed to optimize the feature extraction procedure. The features aggregated at different time windows are subsequently classified by an ensemble of support vector machine (SVM) classifiers. To enhance the generalization property of the method, a data augmentation technique based on pitch shifting and time stretching is applied. According to the obtained results, the developed method outperforms the traditional one for the selected datasets, demonstrating the benefits of using a multi-time-scale approach to feature aggregation.

Emotion recognition for human–computer interaction using high-level descriptors

Recent research has focused extensively on employing Deep Learning (DL) techniques, particularly Convolutional Neural Networks (CNN), for Speech Emotion Recognition (SER). This study addresses the burgeoning interest in leveraging DL for SER, specifically focusing on Punjabi language speakers. The paper presents a novel approach to constructing and preprocessing a labeled speech corpus using diverse social media sources. By utilizing spectrograms as the primary feature representation, the proposed algorithm effectively learns discriminative patterns for emotion recognition. The method is evaluated on a custom dataset derived from various Punjabi media sources, including films and web series. Results demonstrate that the proposed approach achieves an accuracy of 69%, surpassing traditional methods like decision trees, Naïve Bayes, and random forests, which achieved accuracies of 49%, 52%, and 61% respectively. Thus, the proposed method improves accuracy in recognizing emotions from Punjabi speech signals.

Speech Emotion Recognition

Speech emotion recognition is a vital area of research with applications starting from human-computer interaction to mental health monitoring. This paper will provide a comprehensive survey of the techniques, methods, applications, and challenges in speech emotion recognition. It begins by checking the significance of recognizing emotions from speech and its diverse applications across various fields. The paper then shows the method employed for emotion speech recognition, encompassing traditional machine learning techniques, such as support vector machines and Gaussian mixture models, as well as contemporary approaches, including deep learning and multimodal fusion. Moreover, it examines benchmark datasets commonly used for training and evaluation purposes in emotion speech recognition research. Speech Emotion Recognition (SER) has a wide range of applications and there has been a lot of research going on in this fascinating area in recent years. However, the entertainment sector suffers from a lack of study in this research. Many of use Neural Network (NN) and Long Short-Term Memory (LSTM) architectures to categorize the emotions in audio recordings captured by actors expressing various emotions. Moreover, our survey explores real-world applications of emotion speech recognition like virtual assistants, health sector, market sector, education, Mental health diagnosis. The paper discusses the challenge associated with emotion speech recognition, including the variability of emotional expressions, cultural influences, humor and privacy concerns. In future it will help others to deal with many Noisy dataset and various cultural effects of emotion.

“Love looks not with the eyes”: supranormal processing of emotional speech in individuals with late-blindness versus preserved processing in individuals with congenital-blindness

ABSTRACT Processing of emotional speech in the absence of visual information relies on two auditory channels: semantics and prosody. No study to date has investigated how blindness impacts this process. Two theories, Perceptual Deficit, and Sensory Compensation, yiled different expectations about the role of visual experience (or its lack thereof) in processing emotional speech. To test the effect of vision and early visual experience on processing of emotional speech, we compared individuals with congenital blindness (CB, n = 17), individuals with late blindness (LB, n = 15), and sighted controls (SC, n = 21) on identification and selective-attention of semantic and prosodic spoken-emotions. Results showed that individuals with blindness performed at least as well as SC, supporting Sensory Compensation and the role of cortical reorganisation. Individuals with LB outperformed individuals with CB, in accordance with Perceptual Deficit, supporting the role of early visual experience. The LB advantage was moderated by executive functions (working-memory). Namely, the advantage was erased for individuals with CB who showed higher levels of executive functions. Results suggest that vision is not necessary for processing of emotional speech, but early visual experience could improve it. The findings support a combination of the two aforementioned theories and reject a dichotomous view of deficiencies/enhancements of blindness.

Speech emotion recognition in real static and dynamic human-robot interaction scenarios

The use of speech-based solutions is an appealing alternative to communicate in human-robot interaction (HRI). An important challenge in this area is processing distant speech which is often noisy, and affected by reverberation and time-varying acoustic channels. It is important to investigate effective speech solutions, especially in dynamic environments where the robots and the users move, changing the distance and orientation between a speaker and the microphone. This paper addresses this problem in the context of speech emotion recognition (SER), which is an important task to understand the intention of the message and the underlying mental state of the user. We propose a novel setup with a PR2 robot that moves as target speech and ambient noise are simultaneously recorded. Our study not only analyzes the detrimental effect of distance speech in this dynamic robot-user setting for speech emotion recognition but also provides solutions to attenuate its effect. We evaluate the use of two beamforming schemes to spatially filter the speech signal using either delay-and-sum (D&S) or minimum variance distortionless response (MVDR). We consider the original training speech recorded in controlled situations, and simulated conditions where the training utterances are processed to simulate the target acoustic environment. We consider the case where the robot is moving (dynamic case) and not moving (static case). For speech emotion recognition, we explore two state-of-the-art classifiers using hand-crafted features implemented with the ladder network strategy and learned features implemented with the wav2vec 2.0 feature representation. MVDR led to a signal-to-noise ratio higher than the basic D&S method. However, both approaches provided very similar average concordance correlation coefficient (CCC) improvements equal to 116 % with the HRI subsets using the ladder network trained with the original MSP-Podcast training utterances. For the wav2vec 2.0-based model, only D&S led to improvements. Surprisingly, the static and dynamic HRI testing subsets resulted in a similar average concordance correlation coefficient. Finally, simulating the acoustic environment in the training dataset provided the highest average concordance correlation coefficient scores with the HRI subsets that are just 29 % and 22 % lower than those obtained with the original training/testing utterances, with ladder network and wav2vec 2.0, respectively.

Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture

ABSTRACT In the field of emotion recognition, analyzing emotions from speech alone (single-modal speech emotion recognition) has several limitations, including limited data volume and low accuracy. Additionally, single-task models lack generalization and fail to fully utilize relevant information. To address these issues, this paper proposes a new bi-modal bi-task emotion recognition model. The proposed model introduces multi-task learning on the Transformer architecture. On one hand, unsupervised contrastive predictive coding is used to extract denser features from the data while preserving self-information and context-related information. On the other hand, model robustness against interfering information is enhanced by employing self-supervised contrastive learning. Furthermore, the proposed model utilizes a modality fusion module to incorporate textual and audio information to implicitly align features from both modalities. The proposed model achieved accuracy rates of 82.3% and 83.5% on the IEMOCAP and RAVDESS datasets, respectively, when considering weighted accuracy (WA). When weight is not considered (unweighted accuracy (UA)), the model achieved 83.0% and 82.4% accuracy. Compared to the existing methods, the performance is further improved.

Speech Emotion Recognition Based on Temporal-Spatial Learnable Graph Convolutional Neural Network

The Graph Convolutional Neural Networks (GCN) method has shown excellent performance in the field of deep learning, and using graphs to represent speech data is a computationally efficient and scalable approach. In order to enhance the adequacy of graph neural networks in extracting speech emotional features, this paper proposes a Temporal-Spatial Learnable Graph Convolutional Neural Network (TLGCNN) for speech emotion recognition. TLGCNN firstly utilizes the Open-SMILE toolkit to extract frame-level speech emotion features. Then, a bidirectional long short-term memory (Bi LSTM) network is used to process the long-term dependencies of speech features which can further extract deep frame-level emotion features. The extracted frame-level emotion features are then input into subsequent network through two pathways. Finally, one pathway constructs the extracted frame-level deep emotion feature vectors into a graph structure applying an adaptive adjacency matrix to catch latent spatial connections, while the other pathway concatenates emotion feature vectors with graph-level embedding obtained from learnable graph convolutional neural network for prediction and classification. Through these two pathways, TLGCNN can simultaneously obtain temporal speech emotional information through Bi-LSTM and spatial speech emotional information through Learnable Graph Convolutional Neural (LGCN) network. Experimental results demonstrate that this method achieves weighted accuracy of 66.82% and 58.35% on the IEMOCAP and MSP-IMPROV databases, respectively.

Speech emotion recognition systems and their security aspects

Speech emotion recognition (SER) systems leverage information derived from sound waves produced by humans to identify the concealed emotions in utterances. Since 1996, researchers have placed effort on improving the accuracy of SER systems, their functionalities, and the diversity of emotions that can be identified by the system. Although SER systems have become very popular in a variety of domains in modern life and are highly connected to other systems and types of data, the security of SER systems has not been adequately explored. In this paper, we conduct a comprehensive analysis of potential cyber-attacks aimed at SER systems and the security mechanisms that may prevent such attacks. To do so, we first describe the core principles of SER systems and discuss prior work performed in this area, which was mainly aimed at expanding and improving the existing capabilities of SER systems. Then, we present the SER system ecosystem, describing the dataflow and interactions between each component and entity within SER systems and explore their vulnerabilities, which might be exploited by attackers. Based on the vulnerabilities we identified within the ecosystem, we then review existing cyber-attacks from different domains and discuss their relevance to SER systems. We also introduce potential cyber-attacks targeting SER systems that have not been proposed before. Our analysis showed that only 30% of the attacks can be addressed by existing security mechanisms, leaving SER systems unprotected in the face of the other 70% of potential attacks. Therefore, we also describe various concrete directions that could be explored in order to improve the security of SER systems.

A Multimodal Fusion Behaviors Estimation Method for Public Dangerous Monitoring

At the present stage, the identification of dangerous behaviors in public places mostly relies on manual work, which is subjective and has low identification efficiency. This paper proposes an automatic identification method for dangerous behaviors in public places, which analyzes group behavior and speech emotion through deep learning network and then performs multimodal information fusion. Based on the fusion results, people can judge the emotional atmosphere of the crowd, make early warning, and alarm for possible dangerous behaviors. Experiments show that the algorithm adopted in this paper can accurately identify dangerous behaviors and has great application value.

Speech Emotion Recognition using Machine Learning

This research paper presents a Speech Emotion Recognition (SER) system utilizing a Multilayer Perceptron (MLP) classifier and real-time audio analysis. The system records audio samples, extracts relevant features, and employs machine learning techniques to predict emotions in spoken language. The study focuses on the development of an intuitive Graphical User Interface (GUI) using the Kivy framework, providing a user-friendly platform for real-time emotion analysis. In shortly In this project, we attempt to detect underlying emotions such as (sad, happy, neutral, angry, disgust, surprised, fearful and calm) in recorded speech by analysing the acoustic features of the audio data of recordings and Created an application to implement the same on user input.

Age-related differences in processing of emotions in speech disappear with babble noise in the background

ABSTRACT Older adults process emotional speech differently than young adults, relying less on prosody (tone) relative to semantics (words). This study aimed to elucidate the mechanisms underlying these age-related differences via an emotional speech-in-noise test. A sample of 51 young and 47 older adults rated spoken sentences with emotional content on both prosody and semantics, presented on the background of wideband speech-spectrum noise (sensory interference) or on the background of multi-talker babble (sensory/cognitive interference). The presence of wideband noise eliminated age-related differences in semantics but not in prosody when processing emotional speech. Conversely, the presence of babble resulted in the elimination of age-related differences across all measures. The results suggest that both sensory and cognitive-linguistic factors contribute to age-related changes in emotional speech processing. Because real world conditions typically involve noisy background, our results highlight the importance of testing under such conditions.

Joint enhancement and classification constraints for noisy speech emotion recognition

In the natural environment, the received speech signal is often interfered by noise, which reduces the performance of speech emotion recognition (SER) system. To this end, a noisy SER method based on joint constraints, including enhancement constraint and arousal-valence classification constraint (EC-AVCC), is proposed. This method introduces speech enhancement features and constraints into emotion recognition for the first time, which helps to improve the performance of noisy SER. It does not require denoising pre-processing for noisy speech, thus saving testing time. Specifically, it uses multi-domain statistical features (MDSF) composed of emotional and enhanced features as the input of the SER model based on convolution neural network (CNN) and long short-term memory-attention (ALSTM). In addition, the model is constrained jointly by speech enhancement and arousal-valence classification to obtain the robust and discriminative deep emotion features. Furthermore, in the auxiliary speech enhancement task, a joint loss function that simultaneously constrains the error of the ideal ratio mask and the error of the corresponding MDSF is introduced to obtain more robust features. The experimental results on the CASIA Chinese emotional dataset and the EMO-DB German emotional database show that compared with the CNN baseline, the proposed method improves the accuracy of SER in white noise and babble noise by 4.7%-9.9%.

Real-time Speech Emotion Recognition using Machine Learning

This project is centred around examining spoken language emotion, a critical aspect of human-computer interaction and artificial intelligence. Leveraging advanced techniques such as Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models, we aim to decipher and understand the nuanced expressions of emotions conveyed through speech. The investigation draws from diverse datasets, including RAVDESS, SAVEE, TESS, and CREAM-D, each carefully chosen to encapsulate a broad spectrum of emotions and real-world speech scenarios. The project unfolds by elucidating the methodologies employed in dataset selection, the prepossessing of raw audio data, and the intricacies of utilizing CNN and LSTM techniques for speech emotion analysis. The main objective is to create a robust and reliable outcome model capable of accurately classifying and interpreting emotions in speech across various contexts. Practical applications of this Research extend to fields such as sentiment analysis and potential contributions to mental health monitoring. Through this project, we contribute valuable insights to the evolving landscape of speech emotion analysis, addressing the inherent challenges and exploring opportunities for enhancing human-computer interaction and emotional intelligence in artificial systems.

Modeling and Sentiment Analysis of Social Relationships in Elderly Smart Homes Based on Graph Neural Networks

With the expansion of Speech Emotion Recognition in the consumer domain, several devices, particularly those designed for managing smart home personal assistants for the elderly, have been widely available on the market. The increasing processing power and connection, together with the growing need to facilitate longer residency through technological interventions, highlight the potential benefits of smart home assistants. Enabling these assistants to recognize human emotions would greatly improve user-assistant communication, allowing the assistant to deliver more constructive and customized feedback to the user. In this research work, Modeling and Sentiment Analysis of Social Relationships in Elderly Smart Homes Based on Graph Neural Networks (SASR-MBHNN-BBOA) is proposed. The input data are collected from Social Recommendation Dataset. Then, input data are pre-processed utilizing Inverse Optimal Safety Filters (IOSF) for cleaning the data and removing the background noise. Then the pre-processed data are given to Memristive Bi-neuron Hopfield Neural Network (MBHNN) for predicting the sentiments like positive, negative and neutral. In general, MBHNN doesn’t express some adoption of optimization approaches for determining optimal parameters to predicting the sentiments accurately. Hence BBOA is proposed to optimize MBHNN classifier which precisely predicts the sentiments in elderly smart home. The proposed SASR-MBHNN-BBOA method is implemented in Python, and it assessed with numerous performance metrics such as accuracy, precision, recall, F1-score, ROC. The outcomes show SASR-MBHNN-BBOA attains 20.8%, 19.5%, and 29.6% higher Accuracy, 28.8%, 22.5%, and 32.6% higher Precision, 15.5%, 27.4%, and 18.2% higher Recall are analysed with existing methods such as, Emotional speech analysis in real time for smart home assistants.(SASR-CNN-SHA), Machine Learning to Investigate Elderly Care Requirements in China via the Lens of Family Caregivers (SASR-ML-IECR),Identifying User Emotions via Audio Conversations with Smart Assistants (SASR-DNN-EASA) methods respectively.

A State-of-arts Review of Deep Learning Techniques for Speech Emotion Recognition

In sophisticated Human-Computer Interfaces (HCI), the emotional state of the user is becoming a crucial component that is closely linked to emotional speech recognition. Spoken expressions, which can be a part of human-machine interaction, are an important source of emotional information. Speech emotion recognition (SER) in deep learning (DL) continues to be a hot topic, especially in the field of emotional computing. Current deep learning (DL) and neural network methods are applied in this highly active field of research. This is as a result of its expanding potential, advancements in algorithms, and practical uses. Quantitative factors such as pitch, intensity, accent and Mel-Frequency Cepstral Coefficients (MFCC) can be employed to model the paralinguistic data contained in human speech. To achieve SER, three key procedures are usually followed: data processing, feature selection/extraction, and classification based on the underlying emotional qualities. The nature of these processes and the peculiarities of human speech lend support to the employment of DL techniques for SER implementation. A variety of DL methods have been used for SER tasks in recent affective computing research works; however, only a small number of them capture the underlying ideas and methodologies that can be used to facilitate the three main steps of SER implementation. With a focus on the three SER implementation processes, we provide a state-of-the-art assessment of research conducted over the last ten years that tackled SER tasks from DL perspectives in this work. Various issues are covered in detail, including the problem of low classification accuracy of Speaker-Independent experiments and the related remedies. The review offers principles for SER evaluation as well, emphasizing indicators that can be experimented with and common baselines.

Chinese Emotional Speech Audiometry Project (CESAP): Establishment and Validation of a New Material Set With Emotionally Neutral Disyllabic Words.

The Chinese Emotional Speech Audiometry Project (CESAP) aims to establish a new material set for Chinese speech audiometry tests, which can be used in both neutral and emotional prosody settings. As the first endeavor of CESAP, this study demonstrates the development of the material foundation and reports its validation in neutral prosody. In the development step, 40 phonetically balanced word lists consisting of 30 Chinese disyllabic words with neutral valence were first generated. In a following affective rating experiment, 35 word lists were qualified for validation based on the familiarity and valence ratings from 30 normal-hearing (NH) participants. For validation, performance-intensity functions of each word list were fitted with responses from 60 NH subjects under six presentation levels (-1, 3, 5, 7, 11, and 20 dB HL). The final material set was determined by the intelligibility scores at each decibel level and the mean slopes. First, 35 lists satisfied the criteria of phonetic balance, limited repetitions, high familiarity, and neutral valence and were selected for validation. Second, 15 lists were compiled in the final material set based on the pairwise differences in intelligibility scores and the fitted 20%-80% slopes. The established material set had high reliability and validity and was sensitive to detect intelligibility changes (50% slope: 6.20%/dB; 20%-80% slope: 5.45%/dB), with small covariance of variation for thresholds (15%), 50% slope (12%), and 20%-80% slope (12%). Our final material set of 15 word lists takes the initiative to control the emotional aspect of audiometry tests, which enriches available Mandarin speech recognition materials and warrants future assessments in emotional prosody among populations with hearing impairments. https://doi.org/10.23641/asha.25742814.

Speech Emotion Recognition Based on Machine Learning

The speech is the most effective means of communication, to recognize the emotions in speech is the most crucial task. In this paper we are using the Artificial Neural Network to recognize the emotions in speech. Hence, providing an efficient and accurate technique for speech based emotion recognition is also an important task. This study is focused on seven basic human emotions (angry, disgust, fear, happy, neutral, surprise, sad). The training and validating accuracy and also lose can be seen in a graph while training the dataset.According to it confusion matrix for model is created. The features frequency, pitch, amplitude and format of speech is used ti recognize seven basic emotions from speech. Keywords: Emotion detection, Emotion classification, verbal emotion classification.

Enhancing speech emotion recognition through deep learning and handcrafted feature fusion

In this paper, we introduce an innovative investigation in speech emotion recognition (SER). The proposed model combines deep learning-based and handcrafted audio features to achieve optimal accuracy. The proposed model employs an iterative feature selection and majority voting pipeline to obtain better results by fusing deep learning-based and handcrafted features. The wav2vec2 model and the openSmile audio processing library are used in order to extract audio features from audio data. Then the feature selection and majority voting techniques are used to identify the optimal feature selection methods for diverse feature sets and to combine their strengths. The experiments are performed using a diverse and extensive corpus to ensure the robustness of the proposed method. In the construction of this multi-corpus dataset, we used four well-known benchmark datasets, namely Ravdess, Savee, Crema-D, and Tess. All of these datasets are publicly available. These datasets are combined on six common emotions: sadness, happiness, fear, anger, surprise, and disgust. The resultant dataset comprises 11,511 samples across these categories. The proposed method has been shown to achieve results comparable to those reported in the existing literature. The experimental results indicate that the proposed pipeline leads to a 3 % improvement in classification accuracy. The highest achieved accuracy on the multi-corpus dataset is 92.55 %.