Speech Emotion Recognition Model Research Articles

A Systematic Evaluation of Adversarial Attacks against Speech Emotion Recognition Models

A Systematic Evaluation of Adversarial Attacks against Speech Emotion Recognition Models

Hybrid model for speech emotion recognition of normal and autistic children (SERNAC)

Since the last decade, autism spectrum disorder (ASD) has been used as a general term to describe a wide range of conditions, including autistic syndrome, Asperger's disorder, and pervasive developmental disability. This problem emerges as a decreased ability to share emotions and a greater difficulty understanding others' feelings, leading to increased social communication difficulties. To assist patients with ASD, we proposed a concept that incorporates speech emotion detection technologies, which are widely used in the field of human-computer interaction (particularly youngsters). An algorithm based on a novel method for classifying normal and autistic children's speech emotions is implemented in this article. The training data set is treated to a new approach after all features have been extracted. The technique discussed in this study is the creation of a hybrid algorithm that serves as a classifier for normal and autistic children's speech emotions. Voice emotion recognition can be identified accurately and with a lower error rate. The data collection includes speech samples from 200 normal and 250 autistic groups in four moods (Angry, Happy, Neutral and Sad). As per research findings, the implemented hybrid algorithm for Normal and Autistic Children Speech Emotions (SERNAC) outperformed the existing classifiers by increasing accuracy.

A novel concatenated 1D-CNN model for speech emotion recognition

Speech Emotion Recognition (SER) has conventionally been executed solely through acoustic data. Convolutional Neural Network (CNN) in Deep Learning (DL) is a cutting-edge technique that has been effectively used in diverse fields, such as analyzing speech emotion. This study utilizes the Mel Frequency Magnitude Coefficient (MFMC), a modified variant of the Mel Frequency Cepstral Coefficient (MFCC) feature, in conjunction with a one-dimensional (1D) CNN to increase the SER performance. The accuracy of SER is evaluated utilizing the proposed models. Model 1 refers to MFCC-1D-CNN, model 2 refers to MFMC-1D-CNN, and model 3 refers to the concatenated model of model 1 and model 2. The models were evaluated on four datasets comprising eight distinct emotions, specifically anger, happiness, sadness, boredom, fear, surprise, disgust, neutral, and calm. The proposed models, namely model 1, model 2, and model 3, were evaluated on the four datasets using coefficients of 12, 24, and 30. The Proposed Model 1 achieved accuracy rates for EMO-DB, EMOVO, SAVEE, and RAVDESS of 88.1%, 87.5%, 75%, and 97.7% for the respective coefficients. While the proposed model 2 achieved accuracy rates of 93.8%, 98.3%, 90.3%, and 97.5% for the same coefficients. However, the proposed concatenated model 3 outperformed the accuracy of 95.6%, 99.4%, 91.7%, and 98.1% for the four datasets EMO-DB, EMOVO, SAVEE, and RAVDESS compared to model 1 and model 2. Experimental results indicate that the proposed concatenated model 3 improves accuracy and prevents the model from overfitting.

Fuzzy speech emotion recognition considering semantic awareness

Speech emotion recognition is of great significance in the industry such as social robots, health care, and intelligent education. Due to the obscurity of emotional expression in speech, most works on speech emotion recognition (SER) ignore the consistency of speech emotion recognition, leading to fuzzy expression and low accuracy in emotional recognition. In this paper, we propose a semantic aware speech emotion recognition model to alleviate this issue. Specifically, a speech feature extraction module based on CNN and Transformer is designed to extract local and global information from the speech. Moreover, a semantic embedding support module is proposed to use text semantic information as auxiliary information to assist the model in extracting emotional features of speech, and can effectively overcome the problem of low recognition rate caused by emotional ambiguity. In addition, the model uses a key-value pair attention mechanism to fuse the features, which makes the fusion of speech and text features preferable. In experiments on two benchmark corpora IEMOCAP and EMO-DB, the recognition rates of 74.3% and 72.5% were obtained under respectively, which show that the proposed model can significantly improve the accuracy of emotion recognition.

An Effective Speech Emotion Recognition Model for Multi-Regional Languages Using Threshold-based Feature Selection Algorithm

An Effective Speech Emotion Recognition Model for Multi-Regional Languages Using Threshold-based Feature Selection Algorithm

Optimal trained ensemble of classification model for speech emotion recognition: Considering cross-lingual and multi-lingual scenarios

Optimal trained ensemble of classification model for speech emotion recognition: Considering cross-lingual and multi-lingual scenarios

Applying software‐defined ambient intelligence techniques for cognitive and emotional awareness in the Internet of Things

SummarySmart “discrete” technologies are taking over everyday activities with an intended shift towards complete automation. With this, ambient intelligence (AmI) has come under the limelight, to make these routines subtle and automated, enabled through the widespread adoption of the ubiquitous and pervasive Internet of Things (IoT). AmI is context‐driven depending largely on user input through various interfaces, which is nothing but the interplay between IoT and artificial intelligence (AI). Despite extensive incorporation of personal digital assistants and smart home systems, AI is not yet capable of accurately interpreting and responding to human emotions. This paper aims to present one such application to gauge the emotional and cognitive state of the user and respond appropriately, be it ambient lighting or an emergency SOS. The proposed model extracts network packets to work with audio messages on the real‐time transport protocol (RTP) layer and presents a lightweight multilayer perceptron (MLP)‐based speech emotion recognition (SER) model for emotional analysis with an accuracy of 80.52%. It also features a simple confidence rating system for further optimization.

Attention-based Speech Emotion Recognition Approach for Medical Application

Abstract: Presently AI is used in various medical fields. Mental health is an important part of the overall health of a person and speech is the primary form for expression of emotion. Thus, Speech Emotion Recognition can be used to understand emotions of the person and help doctors focus on the cure. Speech Emotion recognition is analysis and classification of speech signals to detect the underlying emotions. This paper proposes a model for speech emotion recognition using an attention mechanism. The model is developed using Mel-frequency cepstral coefficients (MFCCs), and a combination of 2D CNN layers and LSTM recurrent layers for temporal aggregation. The proposed model is evaluated using a dataset of speech recordings containing eight emotion categories. The results show that the model achieves 89% accuracy. The attention mechanism is found to improve the recognition performance by focusing on relevant emotional information and ignoring irrelevant information. This research has potential applications in clinical settings in detection as well as treatment for mental health issues.

Performance Improvement of Speech Emotion Recognition Systems by Combining 1D CNN and LSTM with Data Augmentation

In recent years, the increasing popularity of smart mobile devices has made the interaction between devices and users, particularly through voice interaction, more crucial. By enabling smart devices to better understand users’ emotional states through voice data, it becomes possible to provide more personalized services. This paper proposes a novel machine learning model for speech emotion recognition called CLDNN, which combines convolutional neural networks (CNN), long short-term memory neural networks (LSTM), and deep neural networks (DNN). To design a system that closely resembles the human auditory system in recognizing audio signals, this article uses the Mel-frequency cepstral coefficients (MFCCs) of audio data as the input of the machine learning model. First, the MFCCs of the voice signal are extracted as the input of the model. Local feature learning blocks (LFLBs) composed of one-dimensional CNNs are employed to calculate the feature values of the data. As audio signals are time-series data, the resulting feature values from LFLBs are then fed into the LSTM layer to enhance learning on the time-series level. Finally, fully connected layers are used for classification and prediction. The experimental evaluation of the proposed model utilizes three databases: RAVDESS, EMO-DB, and IEMOCAP. The results demonstrate that the LSTM model effectively models the features extracted from the 1D CNN due to the time-series characteristics of speech signals. Additionally, the data augmentation method applied in this paper proves beneficial in improving the recognition accuracy and stability of the systems for different databases. Furthermore, according to the experimental results, the proposed system achieves superior recognition rates compared to related research in speech emotion recognition.

The design of intelligent fuzzy cognitive system of music emotion by product supply chain management

AbstractThe classification of musical emotion is a challenging problem due to the subjectivity of human perception. In order to analyse the emotion contained in music from the perspective of product supply chain management, the psychological model and characteristics of music emotion are introduced. Additionally, the current product supply chain and the composition of the two contracts are analysed based on the characteristics of online music products. Finally, a speech emotion recognition model based on a Fuzzy Cognitive Map (FCM) is proposed. Two corpora containing five emotions (sadness, anger, happiness, surprise, and boredom) are selected to test the performance and characteristics of the network. In order to evaluate the system performance, different acoustic features are extracted, including prosodic features, formant features, marginal spectral features, and Mel‐Frequency Cepstral Coefficients (MFCC). The results show that the average recognition rate of MFCC features on the Berlin emotional speech database is 70.36%, which is 0.75% higher than the second‐highest edge spectrum features and achieves better results. Meanwhile, on this database, the Musical Emotion Recognition System based on FCM (MERS‐FCM) is improved by 20.36%, 7.34%, and 4.12% compared with Back Propagation, K‐nearest neighbour, and support vector machine, respectively, which shows that MERS‐FCM has better performance. The research of the music emotion recognition system based on FCM can provide more reference paths for the use and development of music emotion information.

Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS.

The existing research on emotion recognition commonly uses mel spectrogram (MelSpec) and Geneva minimalistic acoustic parameter set (GeMAPS) as acoustic parameters to learn the audio features. MelSpec can represent the time-series variations of each frequency but cannot manage multiple types of audio features. On the other hand, GeMAPS can handle multiple audio features but fails to provide information on their time-series variations. Thus, this study proposes a speech emotion recognition model based on a multi-input deep neural network that simultaneously learns these two audio features. The proposed model comprises three parts, specifically, for learning MelSpec in image format, learning GeMAPS in vector format, and integrating them to predict the emotion. Additionally, a focal loss function is introduced to address the imbalanced data problem among the emotion classes. The results of the recognition experiments demonstrate weighted and unweighted accuracies of 0.6657 and 0.6149, respectively, which are higher than or comparable to those of the existing state-of-the-art methods. Overall, the proposed model significantly improves the recognition accuracy of the emotion "happiness", which has been difficult to identify in previous studies owing to limited data. Therefore, the proposed model can effectively recognize emotions from speech and can be applied for practical purposes with future development.

An Analysis of Emotional Responses of Students in Bilingual Classes and Adjustment Strategies

Students’ willingness to participate in bilingual communication is greatly influenced by their positive emotions in the bilingual class. The automatic recognition of students’ emotional state in bilingual class can assist teachers to correctly master the laws of students’ emotional changes during the bilingual learning process as fast as possible. However, the speech emotion features extracted by existing speech emotion recognition models are not universal and not suitable for bilingual speech emotion recognition, and the accuracy needs to be improved. To cope with these issues, this paper aims to study the emotional responses of students in bilingual class based on emotional analysis and provide adjustment strategies for it. At first, the signals of students’ speech records in bilingual class were pre-processed, the one-dimensional features of the signals were converted into 2-dimensional speech spectrum features so as to attain more useful information to facilitate the emotional recognition of students’ speech records in the bilingual class. Then this paper combined the bi-linear Convolution Neural Network (CNN) with capsule network to construct a bilingual class student emotion recognition model, and experimental results verified the effectiveness of the constructed model.

The Efficacy of Deep Learning-Based Mixed Model for Speech Emotion Recognition

Human speech indirectly represents the mental state or emotion of others. The use of Artificial Intelligence (AI)-based techniques may bring revolution in this modern era by recognizing emotion from speech. In this study, we introduced a robust method for emotion recognition from human speech using a well-performed preprocessing technique together with the deep learning-based mixed model consisting of Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN). About 2800 audio files were extracted from the Toronto emotional speech set (TESS) database for this study. A high pass and Savitzky Golay Filter have been used to obtain noise-free as well as smooth audio data. A total of seven types of emotions; Angry, Disgust, Fear, Happy, Neutral, Pleasant-surprise, and Sad were used in this study. Energy, Fundamental frequency, and Mel Frequency Cepstral Coefficient (MFCC) have been used to extract the emotion features, and these features resulted in 97.5% accuracy in the mixed LSTM + CNN model. This mixed model is found to be performed better than the usual state-of-the-art models in emotion recognition from speech. It also indicates that this mixed model could be effectively utilized in advanced research dealing with sound processing.

A Discriminative Feature Representation Method Based on Cascaded Attention Network With Adversarial Strategy for Speech Emotion Recognition

Currently, speech emotion recognition models still could not show satisfactory performance due to the complexity of emotions. In most of the previous studies, there is a common problem that some of the particular emotions are severely misclassified. In this paper, we propose a novel framework integrating cascaded attention network and adversarial joint loss strategy for speech emotion recognition, aiming at discriminating the confusions by emphasizing more on the emotions which are difficult to be correctly classified. First, we extract log-Mels, deltas and delta-deltas of log-Mels as 3D features to effectively reduce the interference of external factors. Next, we introduce a cascaded attention network to extract effective emotional features, where spatiotemporal attention selectively locates the targeted emotional regions from the input features. In these targeted regions, the self attention with head fusion captures the long-distance dependence of temporal features. Finally, an adversarial joint loss strategy is proposed to distinguish the emotional embeddings with high similarity by the generated hard triplets in an adversarial fashion. To evaluate our proposed method, experiments are performed with the IEMOCAP, CASIA, and EMODB corpora. The experimental results demonstrate that our proposed method significantly outperforms the state-of-the-art approaches on all datasets.

Learning Speech Emotion Representations in the Quaternion Domain

The modeling of human emotion expression in speech signals is an important, yet challenging task. The high resource demand of speech emotion recognition models, combined with the general scarcity of emotion-labelled data are obstacles to the development and application of effective solutions in this field. In this paper, we present an approach to jointly circumvent these difficulties. Our method, named RH-emo, is a novel semi-supervised architecture aimed at extracting quaternion embeddings from real-valued monoaural spectrograms, enabling the use of quaternion-valued networks for speech emotion recognition tasks. RH-emo is a hybrid real/quaternion autoencoder network that consists of a real-valued encoder in parallel to a real-valued emotion classifier and a quaternion-valued decoder. On the one hand, the classifier permits to optimization of each latent axis of the embeddings for the classification of a specific emotion-related characteristic: valence, arousal, dominance, and overall emotion. On the other hand, quaternion reconstruction enables the latent dimension to develop intra-channel correlations that are required for an effective representation as a quaternion entity. We test our approach on speech emotion recognition tasks using four popular datasets: IEMOCAP, RAVDESS, EmoDB, and TESS, comparing the performance of three well-established real-valued CNN architectures (AlexNet, ResNet-50, VGG) and their quaternion-valued equivalent fed with the embeddings created with RH-emo. We obtain a consistent improvement in the test accuracy for all datasets, while drastically reducing the resources' demand of models. Moreover, we performed additional experiments and ablation studies that confirm the effectiveness of our approach. The RH-emo repository is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ispamm/rhemo</uri> .

Strong Generalized Speech Emotion Recognition Based on Effective Data Augmentation

The absence of labeled samples limits the development of speech emotion recognition (SER). Data augmentation is an effective way to address sample sparsity. However, there is a lack of research on data augmentation algorithms in the field of SER. In this paper, the effectiveness of classical acoustic data augmentation methods in SER is analyzed, based on which a strong generalized speech emotion recognition model based on effective data augmentation is proposed. The model uses a multi-channel feature extractor consisting of multiple sub-networks to extract emotional representations. Different kinds of augmented data that can effectively improve SER performance are fed into the sub-networks, and the emotional representations are obtained by the weighted fusion of the output feature maps of each sub-network. And in order to make the model robust to unseen speakers, we employ adversarial training to generalize emotion representations. A discriminator is used to estimate the Wasserstein distance between the feature distributions of different speakers and to force the feature extractor to learn the speaker-invariant emotional representations by adversarial training. The simulation experimental results on the IEMOCAP corpus show that the performance of the proposed method is 2-9% ahead of the related SER algorithm, which proves the effectiveness of the proposed method.

Modeling Speech Emotion Recognition via Attention-Oriented Parallel CNN Encoders

Meticulous learning of human emotions through speech is an indispensable function of modern speech emotion recognition (SER) models. Consequently, deriving and interpreting various crucial speech features from raw speech data are complicated responsibilities in terms of modeling to improve performance. Therefore, in this study, we developed a novel SER model via attention-oriented parallel convolutional neural network (CNN) encoders that parallelly acquire important features that are used for emotion classification. Particularly, MFCC, paralinguistic, and speech spectrogram features were derived and encoded by designing different CNN architectures individually for the features, and the encoded features were fed to attention mechanisms for further representation, and then classified. Empirical veracity executed on EMO-DB and IEMOCAP open datasets, and the results showed that the proposed model is more efficient than the baseline models. Especially, weighted accuracy (WA) and unweighted accuracy (UA) of the proposed model were equal to 71.8% and 70.9% in EMO-DB dataset scenario, respectively. Moreover, WA and UA rates were 72.4% and 71.1% with the IEMOCAP dataset.

Speech emotion recognition using 2D-convolutional neural network

&lt;span lang="EN-US"&gt;This research proposes a speech emotion recognition model to predict human emotions using the convolutional neural network (CNN) by learning segmented audio of specific emotions. Speech emotion recognition utilizes the extracted features of audio waves to learn speech emotion characteristics; one of them is mel frequency cepstral coefficient (MFCC). Dataset takes a vital role to obtain valuable results in model learning. Hence this research provides the leverage of dataset combination implementation. The model learns a combined dataset with audio segmentation and zero padding using 2D-CNN. Audio segmentation and zero padding equalize the extracted audio features to learn the characteristics. The model results in 83.69% accuracy to predict seven emotions: neutral, happy, sad, angry, fear, disgust, and surprise from the combined dataset with the segmentation of the audio files.&lt;/span&gt;

Research on Feature Fusion Speech Emotion Recognition Technology for Smart Teaching

At present, there are numerous intelligent teaching applications based on various advanced pattern recognition technologies, such as face sign-in, classroom action recognition, student facial expression recognition, and other systems which have been gradually applied to major schools. Speech emotion recognition can analyze the characteristics of current teaching emotions, so as to discover the rules and details of teachers’ grasp of emotions. In order to realize the task of emotion classification in the process of intelligent teaching through machine learning technology, a teaching speech emotion recognition method with multifeature fusion and deep learning is proposed. The proposed method front-end processes the speech spectrum features in a multifeature fusion manner and combines them with an artificial neural network classifier to form the final speech emotion recognition model. First, after speech preprocessing, three features are selected and fused features using a network structure trained by parallel subnetworks. Then, the classifier used is a hybrid neural network classifier combining a convolutional neural network and a recurrent neural network. Finally, 100 open web courses were used to train and test the model. The test results show that the hybrid neural network sound model using multifeature fusion has good teaching speech emotion recognition capability.

Innovative Mechanism of Rural Finance: Risk Assessment Methods and Impact Factors of Agricultural Loans Based on Personal Emotion and Artificial Intelligence.

Agricultural finance is in an embarrassing position in the current financial environment, especially during the process of COVID-19. Based on a small-scale peasant economy, it can no longer meet the rapidly rising demand of farmers for agricultural finance. Moreover, there has been a serious disconnection between the financial system of secondary and tertiary industries, and the quality of development needs to be improved urgently. The agricultural loan risk assessment has always been the main problem that we pay great attention to in the innovation of agricultural finance. Agricultural loans are an indispensable element in supporting agricultural development and promoting rural revitalization strategy. However, financial institutions have certain credit risks in reviewing and issuing agricultural loans. This article studies the speech emotion recognition of farmers in loan review to assess loan risk. As for emotional confusion caused by speech segmentation, a special method of data connection between Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (Bi-LSTM) Networks is designed, and a variable-length speech emotion recognition model including CNN and Bi-LSTM is designed. Experimental results show that the proposed algorithm can effectively improve the risk assessment of farmers in loan review.