Accuracy Of Speech Emotion Recognition Research Articles

Joint enhancement and classification constraints for noisy speech emotion recognition

In the natural environment, the received speech signal is often interfered by noise, which reduces the performance of speech emotion recognition (SER) system. To this end, a noisy SER method based on joint constraints, including enhancement constraint and arousal-valence classification constraint (EC-AVCC), is proposed. This method introduces speech enhancement features and constraints into emotion recognition for the first time, which helps to improve the performance of noisy SER. It does not require denoising pre-processing for noisy speech, thus saving testing time. Specifically, it uses multi-domain statistical features (MDSF) composed of emotional and enhanced features as the input of the SER model based on convolution neural network (CNN) and long short-term memory-attention (ALSTM). In addition, the model is constrained jointly by speech enhancement and arousal-valence classification to obtain the robust and discriminative deep emotion features. Furthermore, in the auxiliary speech enhancement task, a joint loss function that simultaneously constrains the error of the ideal ratio mask and the error of the corresponding MDSF is introduced to obtain more robust features. The experimental results on the CASIA Chinese emotional dataset and the EMO-DB German emotional database show that compared with the CNN baseline, the proposed method improves the accuracy of SER in white noise and babble noise by 4.7%-9.9%.

Speech emotion recognition using a combination of variational mode decomposition and Hilbert transform

The objective of automated speech emotion recognition (SER) is to recognize each emotion of the speech signal uniquely and efficiently using machines like computers, mobile devices, etc. The popularity of SER has been widely acknowledged among researchers due to its extensive applicability in practical contexts. The use of SER has shown to be advantageous in several domains, including medical treatment, enhancement of security systems, surveillance operations, online marketing strategies, online educational platforms, online search engines, personal communication, customer relationship management, reinforcement of machine and human connection, and numerous other areas. Numerous authors have used various techniques, including the combination of multiple features (acoustic, non-acoustic, or both) and classifiers (machine learning, deep learning, or both), in order to enhance the efficacy of emotion categorization. In the present work, our objective is to enhance the performance of emotion classification (PECL) by using a combined approach of the variational mode decomposition (VMD) and Hilbert transform (HT) techniques. Using the VMD method, we decomposed the speech signal frame into many sub-signals, or intrinsic mode functions (IMFs). Then HT is applied to each VMD-based IMF signal to find the mode instantaneous amplitude (MIA) and mode instantaneous frequency (MIF) signal vectors. We extracted proposed features such as HT-based approximate entropy (HTAE), HT-based permutation entropy (HTPE), HT-based increment entropy (HTIE), and HT-based sample entropy (HTSE) using each MIA and MIF signal vector. The combination of the proposed HT-based features is called HT-based entropy (HTE) features. Then, we accessed the PECL using the HTE and MFCC features alone and in conjunction with a deep neural network (DNN) classifier. The experiment's results showed that the combinations of the proposed feature (MFCC + HTE) using a DNN classifier outperformed the individual features and obtained a SER accuracy of 86.92% for the EMOVO dataset and 91.63% for the EMO-DB dataset.

A novel concatenated 1D-CNN model for speech emotion recognition

Speech Emotion Recognition (SER) has conventionally been executed solely through acoustic data. Convolutional Neural Network (CNN) in Deep Learning (DL) is a cutting-edge technique that has been effectively used in diverse fields, such as analyzing speech emotion. This study utilizes the Mel Frequency Magnitude Coefficient (MFMC), a modified variant of the Mel Frequency Cepstral Coefficient (MFCC) feature, in conjunction with a one-dimensional (1D) CNN to increase the SER performance. The accuracy of SER is evaluated utilizing the proposed models. Model 1 refers to MFCC-1D-CNN, model 2 refers to MFMC-1D-CNN, and model 3 refers to the concatenated model of model 1 and model 2. The models were evaluated on four datasets comprising eight distinct emotions, specifically anger, happiness, sadness, boredom, fear, surprise, disgust, neutral, and calm. The proposed models, namely model 1, model 2, and model 3, were evaluated on the four datasets using coefficients of 12, 24, and 30. The Proposed Model 1 achieved accuracy rates for EMO-DB, EMOVO, SAVEE, and RAVDESS of 88.1%, 87.5%, 75%, and 97.7% for the respective coefficients. While the proposed model 2 achieved accuracy rates of 93.8%, 98.3%, 90.3%, and 97.5% for the same coefficients. However, the proposed concatenated model 3 outperformed the accuracy of 95.6%, 99.4%, 91.7%, and 98.1% for the four datasets EMO-DB, EMOVO, SAVEE, and RAVDESS compared to model 1 and model 2. Experimental results indicate that the proposed concatenated model 3 improves accuracy and prevents the model from overfitting.

Assessing the effectiveness of ensembles in Speech Emotion Recognition: Performance analysis under challenging scenarios

Speech Emotion Recognition (SER) is an important application in areas such as online gaming, e-learning, and medical care. However, recognizing emotion in speech is computationally difficult since it necessitates a thorough search for feature selection, algorithm hyperparameter tuning, or algorithm combinations, making ensemble use interesting. Although ensembles are frequently employed in SER, their application has not been greatly explored, and their potential benefits for enhancing recognition accuracy and robustness to variability in speech signals have not been fully realized. The purpose of this article is to assess the effectiveness of ensembles in SER by analyzing their performance under challenging scenarios. The experiment made in this study involved evaluating speech samples from various languages, using an out-of-date set of features, and using simple algorithms with default hyperparameters. For classifier set selection, a basic ensemble technique with decision-level voting and a rudimentary heuristic were applied.The results indicated that basic classifiers significantly improved the SER rate, with an absolute improvement ranging from 0.57% to 9.89%. The suggested ensemble approach outperformed state of the art SER methods, including deep learning-based ones, in terms of recognition rates. The findings justify the use of ensembles in SER applications, particularly in circumstances with insufficient data or out-of-date features and algorithms. The work recommends further investigation of ensembles to enhance recognition accuracy and improve robustness in the face of voice signal variability. Finally, the results of the experiment show that ensembles have the potential to increase SER accuracy, and future research in this field can benefit from the study’s conclusions.

Feature Aggregation with Two-Layer Ensemble Framework for Multilingual Speech Emotion Recognition

In this study, we present a framework for improving the accuracy of speech emotion recognition in a multilingual environment. In our prior experiments, where machine learning (ML) models were trained to predict emotions in Korean and then tested in English, as well as vice versa, we observed a dependency on language in emotion recognition, resulting in poor accuracy. We suspect that this may be related to the spectral differences in certain emotions between Korean and English and to the tendency for different formant values to have different acoustic frequencies. For this study, we investigated several different methods, including models with mixed databases, a single database, and bagging, boosting, and voting ML algorithms. Finally, we developed a framework consisting of two branches: one for the aggregation of high-dimensional features from multilingual data and one for a two-layered ensemble framework for emotion classification. In the ensemble framework for Korean and English (EF-KEN), features are extracted and ensemble models are trained, boosted, and evaluated by applying them to different spoken languages (English and Korean). The final experimental result demonstrates a meaningful improvement in an environment with two different languages.

A Primary task driven adaptive loss function for multi-task speech emotion recognition

Although Speech Emotion Recognition (SER) is becoming an active research area, the state-of-the-art performance is limited by the scarcity of emotional datasets. The introduction of Multi-Task Learning (MTL) which jointly trains multiple relative tasks, can utilize more emotion-related information within limited data, enhancing the SER performance but also increasing the training difficulty. In this paper, we propose a primary-task driven adaptive loss function named as PTDA-Loss to adaptively adjust the task weights in the loss function, aiming to more reasonably allocate the learning resources in the training process and further improve the SER performance. We observe that excessive concern with the auxiliary task can lead to insufficient learning resources for SER, not conducive to SER performance improvement. Inspired by this finding, in PTDA-Loss, we retain the advantage of SER in the whole training process. Meanwhile, the learning resources for the auxiliary task will be appropriately but not over compressed, ensuring the effective learning of useful information from the auxiliary task. To improve the model’s learning ability, furthermore, we first employ several dynamic convolutional layers as the backbone network. Regarding gender recognition and speaker recognition as the auxiliary task respectively, ablation experiments on the IEMOCAP, MSP-IMPROV and MELD datasets prove the rationality and effectiveness of PTDA-Loss. Additionally, comparative experiments demonstrate that our method achieves state-of-the-art performance on the three datasets, further enhancing the SER accuracy.

Cross-Corpus Training Strategy for Speech Emotion Recognition Using Self-Supervised Representations

Speech Emotion Recognition (SER) plays a crucial role in applications involving human-machine interaction. However, the scarcity of suitable emotional speech datasets presents a major challenge for accurate SER systems. Deep Neural Network (DNN)-based solutions currently in use require substantial labelled data for successful training. Previous studies have proposed strategies to expand the training set in this framework by leveraging available emotion speech corpora. This paper assesses the impact of a cross-corpus training extension for a SER system using self-supervised (SS) representations, namely HuBERT and WavLM. The feasibility of training systems with just a few minutes of in-domain audio is also analyzed. The experimental results demonstrate that augmenting the training set with EmoDB (German), RAVDESS, and CREMA-D (English) datasets leads to improved SER accuracy on the IEMOCAP dataset. By combining a cross-corpus training extension and SS representations, state-of-the-art performance is achieved. These findings suggest that the cross-corpus strategy effectively addresses the scarcity of labelled data and enhances the performance of SER systems.

Speech emotion recognition using overlapping sliding window and Shapley additive explainable deep neural network

ABSTRACT Speech emotion recognition (SER) has several applications, such as e-learning, human-computer interaction, customer service, and healthcare systems. Although researchers have investigated lots of techniques to improve the accuracy of SER, it has been challenging with feature extraction, classifier schemes, and computational costs. To address the aforementioned problems, we propose a new set of 1D features extracted by using an overlapping sliding window (OSW) technique for SER in this study. In addition, a deep neural network-based classifier scheme called the deep Pattern Recognition Network (PRN) is designed to categorize emotional states from the new set of 1D features. We evaluate the proposed method on the Emo-DB and the AESSD datasets that contain several different emotional states. The experimental results show that the proposed method achieves an accuracy of 98.5% and 87.1% on the Emo-DB and AESSD datasets, respectively. It is also more comparable with accuracy to and better than the state-of-the-art and current approaches that use 1D features on the same datasets for SER. Furthermore, the SHAP (SHapley Additive exPlanations) analysis is employed for interpreting the prediction model to assist system developers in selecting the optimal features to integrate into the desired system.

A Deep Learning Method Using Gender-Specific Features for Emotion Recognition.

Speech reflects people's mental state and using a microphone sensor is a potential method for human-computer interaction. Speech recognition using this sensor is conducive to the diagnosis of mental illnesses. The gender difference of speakers affects the process of speech emotion recognition based on specific acoustic features, resulting in the decline of emotion recognition accuracy. Therefore, we believe that the accuracy of speech emotion recognition can be effectively improved by selecting different features of speech for emotion recognition based on the speech representations of different genders. In this paper, we propose a speech emotion recognition method based on gender classification. First, we use MLP to classify the original speech by gender. Second, based on the different acoustic features of male and female speech, we analyze the influence weights of multiple speech emotion features in male and female speech, and establish the optimal feature sets for male and female emotion recognition, respectively. Finally, we train and test CNN and BiLSTM, respectively, by using the male and the female speech emotion feature sets. The results show that the proposed emotion recognition models have an advantage in terms of average recognition accuracy compared with gender-mixed recognition models.

Speech emotion recognition based on convolutional neural network with attention-based bidirectional long short-term memory network and multi-task learning

Speech emotion recognition (SER) is a challenging task since the distribution of the features is different among various people. To improve generalization performance and accuracy of SER, we employ balanced augmented sampling on the triple-channel log-Mel spectrograms to improve the imbalance of the sample distribution among emotional categories and provide sufficient inputs for the deep neural network model. Time-domain filter and frequency-domain filter are used to process the triple-channel log-Mel spectrograms respectively in order to increase the diversity of features. After that, a deep neural network composed of convolutional neural network (CNN) and attention-based bidirectional long short-term memory network (ABLSTM) is employed for feature extraction, in which multi-task learning is adopted to improve the performance of the deep neural network. We select seven auxiliary tasks and determine the optimal auxiliary tasks through experimental comparison. Finally, our method is evaluated on IEMOCAP and MSP-IMPROV database, and it achieves 70.27% and 66.27% in terms of WAR and UAR on IEMOCAP database, while the WAR and UAR are 60.90% and 61.83% on MSP-IMPROV database respectively, which demonstrates its better performance than other works.

A novel speech emotion recognition method based on feature construction and ensemble learning.

In the field of Human-Computer Interaction (HCI), speech emotion recognition technology plays an important role. Facing a small number of speech emotion data, a novel speech emotion recognition method based on feature construction and ensemble learning is proposed in this paper. Firstly, the acoustic features are extracted from the speech signal and combined to form different original feature sets. Secondly, based on Light Gradient Boosting Machine (LightGBM) and Sequential Forward Selection (SFS) method, a novel feature selection method named L-SFS is proposed. And then, the softmax regression model is used to learn automatically the weights of the four single weak learners including Support Vector Machine (SVM), K-Nearest Neighbor (KNN), Extreme Gradient Boosting (XGBoost) and LightGBM. Lastly, based on the learned automatically weights and the weighted average probability voting strategy, an ensemble classification model named Sklex is constructed, which integrates the above four single weak learners. In conclusion, the method reflects the effectiveness of feature construction and the superiority and stability of ensemble learning, and gets good speech emotion recognition accuracy.

Speech Emotion Recognition Enhanced Traffic Efficiency Solution for Autonomous Vehicles in a 5G-Enabled Space–Air–Ground Integrated Intelligent Transportation System

Speech emotion recognition (SER) is becoming the main human–computer interaction logic for autonomous vehicles in the next generation of intelligent transportation systems (ITSs). It can improve not only the safety of autonomous vehicles but also the personalized in-vehicle experience. However, current vehicle-mounted SER systems still suffer from two major shortcomings. One is the insufficient service capacity of the vehicle communication network, which is unable to meet the SER needs of autonomous vehicles in next-generation ITSs in terms of the data transmission rate, power consumption, and latency. Second, the accuracy of SER is poor, and it cannot provide sufficient interactivity and personalization between users and vehicles. To address these issues, we propose an SER-enhanced traffic efficiency solution for autonomous vehicles in a 5G-enabled space–air–ground integrated network (SAGIN)-based ITS. First, we convert the vehicle speech information data into spectrograms and input them into an AlexNet network model to obtain the high-level features of the vehicle speech acoustic model. At the same time, we convert the vehicle speech information data into text information and input it into the Bidirectional Encoder Representations from Transformers (BERT) model to obtain the high-level features of the corresponding text model. Finally, these two sets of high-level features are cascaded together to obtain fused features, which are sent to a softmax classifier for emotion matching and classification. Experiments show that the proposed solution can improve not only the SAGIN’s service capabilities, resulting in a large capacity, high bandwidth, ultralow latency, and high reliability, but also the accuracy of vehicle SER as well as the performance, practicality, and user experience of the ITS

ATDA: Attentional temporal dynamic activation for speech emotion recognition

Speech emotion recognition (SER) plays a vital role in intelligent human–computer interaction (HCI). The Convolutional Neural Network (CNN) is widely used in SER, effectively capturing the static local features but ignoring the temporal dynamic features necessary for SER. To solve this problem, we place an Attentional Temporal Dynamic Activation (ATDA) module into the CNN-based model to empower it to learn the static and dynamic features simultaneously. In particular, the ATDA module comprises a Temporal Dynamic Activation (TDA) block followed by a Multi-view and Multi-granularity Attention (MMA) block. The TDA block calculates the temporal difference at the feature level to activate the dynamic information and generate the fundamental dynamic feature. The MMA block further detects and amplifies the emotion-related dynamic features based on multiple attention views and granularities. These two blocks within the ATDA module cooperate to activate and extract the dynamic emotional features. Meanwhile, the static features are obtained by a convolutional layer, which are then combined with the dynamic features to generate the final emotional representations. Finally, experiments on the IEMOCAP, MSP-IMPROV, and MELD datasets reveal that the proposed ATDA-CNN model achieves competitive results and enhances SER accuracy by learning meaningful emotional representations.

Two-level discriminative speech emotion recognition model with wave field dynamics: A personalized speech emotion recognition method

Presently available speech emotion recognition (SER) methods generally rely on a single SER model. Getting a higher accuracy of SER involves feature extraction method and model design scheme in the speech. However, the generalization performance of models is typically poor because the emotional features of different speakers can vary substantially. The present work addresses this issue by applying a two-level discriminative model to the SER task. The first level places an individual speaker within a specific speaker group according to the speaker’s characteristics. The second level constructs a personalized SER model for each group of speakers using the wave field dynamics model and a dual-channel general SER model. Two-level discriminative model are fused for implementing an ensemble learning scheme to achieve effective SER classification. The proposed method is demonstrated to provide higher SER accuracy in experiments based on interactive emotional dynamic motion capture (IEMOCAP) corpus and a custom-built SER corpus. In IEMOCAP corpus, the proposed model improves the recognition accuracy by 7%. In custom-built SER corpus, both masked and unmasked speakers is employed to demonstrate that the proposed method maintains higher SER accuracy.

Speech emotion recognition based on multi‐feature and multi‐lingual fusion

A speech emotion recognition algorithm based on multi-feature and Multi-lingual fusion is proposed in order to resolve low recognition accuracy caused bylack of large speech dataset and low robustness of acoustic features in the recognition of speech emotion. First, handcrafted and deep automatic features are extractedfrom existing data in Chinese and English speech emotions. Then, the various features are fused respectively. Finally, the fused features of different languages are fused again and trained in a classification model. Distinguishing the fused features with the unfused ones, the results manifest that the fused features significantly enhance the accuracy of speech emotion recognition algorithm. The proposedsolution is evaluated on the two Chinese corpus and two English corpus, and isshown to provide more accurate predictions compared to original solution. As a result of this study, the multi-feature and Multi-lingual fusion algorithm can significantly improve the speech emotion recognition accuracy when the dataset is small.

Application of Improved Neural Network Strategy in Emotion Information Discrimination

Emotional information recognition is an important application in the field of artificial intelligence. This paper analyzes the existing speech emotion recognition system, based on the improved neural network strategy to improve the accuracy of speech emotion recognition, through the combination of convolution neural network (CNN) and long-term and short-term neural network (LSTM), the emotional data information is successfully recognized, and the better accuracy is obtained.

Adaptive algorithm for feature selection of speech emotion recognition based on genetic algorithm and SVM

As a key carrier for people to obtain information in daily life, speech not only provides text signals, but also contains emotional signals. As an important factor of semantics, emotion signals are the focus of improving the intelligent level of human-computer interaction. Therefore, it is important to improving the accuracy of speech emotion recognition. This paper proposes to incorporate derivative feature parameters into the voice emotion recognition feature selection, and to improve the feature selection algorithm based on genetic algorithm and SVM. This article may provide technical support to promote the development of artificial intelligence.

Head Fusion: Improving the Accuracy and Robustness of Speech Emotion Recognition on the IEMOCAP and RAVDESS Dataset

Speech Emotion Recognition (SER) refers to the use of machines to recognize the emotions of a speaker from his (or her) speech. SER benefits Human-Computer Interaction(HCI). But there are still many problems in SER research, e.g., the lack of high-quality data, insufficient model accuracy, little research under noisy environments, etc. In this paper, we proposed a method called Head Fusion based on the multi-head attention mechanism to improve the accuracy of SER. We implemented an attention-based convolutional neural network(ACNN) model and conducted experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) data set. The accuracy is improved to 76.18% (weighted accuracy, WA) and 76.36% (unweighted accuracy, UA). To the best of our knowledge, compared with the state-of-the-art result on this dataset (76.4% of WA and 70.1% of WA), we achieved a UA improvement of about 6% absolute while achieving a similar WA. Furthermore, We conducted empirical experiments by injecting speech data with 50 types of common noises. We inject the noises by altering the noise intensity, time-shifting the noises, and mixing different noise types, to identify their varied impacts on the SER accuracy and verify the robustness of our model. This work will also help researchers and engineers properly add their training data by using speech data with the appropriate types of noises to alleviate the problem of insufficient high-quality data.

Unsupervised feature selection and NMF de-noising for robust Speech Emotion Recognition

Speech feature fusion is the most commonly used phenomenon for improving the accuracy in Speech Emotion Recognition (SER). But in this, there is a disadvantage of increasing the complexity in SER system in terms of processing time. Besides this, some of the features could be redundant and do not contribute for SER and lead to incorrect emotion prediction and reduction in SER accuracy. To overcome this problem, in this paper, unsupervised feature selection is applied to the feature set with the combination of INTERSPEECH 2010 paralinguistic features, Gammatone Cepstral Coefficients (GTCC) and Power Normalized Cepstral Coefficients (PNCC). The Feature Selection with Adaptive Structure Learning (FSASL), Unsupervised Feature Selection with Ordinal Locality (UFSOL) and a novel Subset Feature Selection (SuFS) algorithm are the feature dimension reduction techniques used to acquire better SER performance in this work. The proposed SER system is analyzed in both clean and noisy environments. The EMO-DB and IEMOCAP emotion databases are considered for evaluating the proposed SER performance. For noise analysis, the clean speech is corrupted with different noises of Aurora noise database and white Gaussian noise at different Signal to Noise Ratio (SNR) levels from −5dB to 20 dB. Support Vector Machine (SVM) classifier with linear and Radial Basis Function (RBF) kernels using 10-fold cross-validation and hold-out validation is used in this analysis with classification accuracy and computation time as the performance metrics. The results show that the proposed SER system outperforms the baseline SER system as well as many of the existing literature works both in clean and noisy conditions. For SNR levels >15 dB, the proposed SER system in presence of different noises performs same as the SER in clean environments. Whereas, for lower SNRs <15 dB the performance is likely to be reduced. Therefore, to overcome this drawback and improve the SER performance in noisy conditions, a dense Non-Negative Matrix Factorization (denseNMF) method is adopted for de-noising the noisy speech signal prior to SER achieving noise robustness.

PAD 3-D Speech Emotion Recognition Based on Feature Fusion

In order to establish the mutual fusion relationship between the global and timing features of speech and achieve better speech emotion recognition, this paper proposes a PAD 3-D space emotion recognition method based on feature fusion. This paper uses the Opensmile to extract four speech global feature sets: IS09_emotion, IS10_paraling, IS11_speaker_state, IS12_speaker_trait, and establishes a deep learning model combined with CNN, LSTM, and attention mechanism to extract the timing features of each frame simultaneously. Finally, the global features and timing features are fused through the Stacking model. The experimental results show that compared with using global features or timing features alone for speech emotion recognition, the method of using the Stacking model to fuse global features and temporal features can effectively improve the accuracy of speech emotion recognition.