Emotional Speech Corpus Research Articles

Optimization Research on Integrating Mental Health Into Ideological and Political Theory Courses Through Deep Learning Network Applications

This study aims to evaluate the effectiveness of incorporating mental health education into ideological and political theory classrooms using speech emotion recognition technology from the field of human-computer interaction. Mental health education is essential for students' well-being, and this study establishes a theoretical framework for its integration into the curriculum. Utilizing recurrent neural networks (RNNs) augmented with attention mechanisms, the research demonstrates the model's proficiency in identifying subtle emotional cues within speech data. The model achieves an average recognition accuracy of 87.21% on the RAVDESS speech emotion corpus, showing particular strengths in detecting emotions like happiness and boredom. The findings suggest that this technology can be effectively employed to gauge student satisfaction with different teaching methodologies post-integration, offering valuable insights for refining educational practices in ideological and political theory classes through improved mental health education initiatives.

Speech Emotion Recognition Algorithm of Intelligent Robot Based on ACO-SVM

Despite the significant advancements in computer speech emotion recognition technology, the deployment of intelligent robots in this domain continues to encounter challenges related to inefficiency and emotional ambiguity. Aiming at the shortcomings of existing algorithms in the accuracy and processing of complex emotion states, a novel emotion recognition model of intelligent robot speech based on ACO algorithm and multi-level support vector machine is proposed. This paper first identifies the efficiency and accuracy problems existing in the current emotion recognition system, and clarified the research gap, that is, how to effectively improve the recognition performance of the model under diverse emotion states. To solve this problem, genetic algorithm is used to fine-tune SVM model parameters, and multi-level SVM classification architecture is constructed to enhance the accuracy of emotion recognition. In the feature selection stage, ACO algorithm is introduced to explore the optimal combination of emotion features, aiming at improving the efficiency and robustness of emotion recognition. The experimental results showed that the average recognition rate of the multilevel SVM model optimized by genetic algorithm was as high as 81.26%, which was 8.78 percentage points higher than that before optimization. When the ACO algorithm was combined with the optimized multi-stage SVM, the average recognition rate of German and Chinese speech emotion corpus was further improved to 85.62%, with an improvement of 4.38 percentage points. In particular, the optimized algorithm performed significantly better than other emotion categories in recognizing complex emotions such as "anger" and "sadness." This paper not only offers an innovative solution for the emotion recognition technology within the realm of intelligent robots, but also extends its methodology to have significant implications for related fields, thereby strongly advancing the in-depth application and development of intelligent bionic algorithms in speech emotion recognition. In addition, through this series of optimization strategies, this paper significantly improves the accuracy of emotion recognition, laying a solid foundation for intelligent robots to achieve more natural and efficient human-computer interaction.

Emotion recognition from Burmese speech based on fused features and deep learning method

Burmese language is challenging for speech emotion classification. Moreover, it is lack of resource and few research was made in this topic. To solve the challenging problem, novel feature extraction for Burmese language is proposed. For lack of resource, Burmese speech emotion corpus called BMISEC is built. To support the challenging problem, the advantages of feature extractions are fused to create a robust feature. Four features are fused. Novel text-tone feature, local binary pattern, mel-frequency cepstral coefficient and discrete wavelet transform are fused. To progress the performance, deep learning method called DenseNet-Emotion is used for classification. Support vector machine is used in DenseNet’s classifier layer. To show the robustness of the proposed system, three types of experiments are made on Tensorflow framework. They are ablation study, experiments with three publicly available datasets and experiments with the previous research methods and they are compared with the proposed method. It is found that feature fusion is superior to only one feature in emotion recognition. BMISEC gets better performance than other datasets. Moreover, the proposed method gets the superior result than previous research methods. The proposed method gets the accuracy of 88.388% for 50 epochs.

Creation of a diverse mixed-lingual emotional speech corpus with a framework for enhanced emotion detection

Speech Emotion Recognition (SER) has advanced considerably during the past 20 years. Till date, various SER systems have been developed for monolingual, multilingual and cross corpus contexts. However, in a country like India where numerous languages are spoken and often humans converse in more than one language, a dedicated SER system for mixed-lingual scenario is more crucial to be established which is the focus of this work. A self-recorded database that includes speech emotion samples with 11 diverse Indian languages has been developed. In parallel, a mixed-lingual database is formed with three popular standard databases of Berlin, Baum and SAVEE to represent mixed-lingual environment for western background. A detailed investigation of GeMAPS (Geneva Minimalistic Acoustic Parameter Set) feature set for mixed-lingual SER is performed. A distinct set of MFCC (Mel Frequency Cepstral Coefficients) coefficients derived from sine and cosine-based filter banks enriches the GeMAPS feature set and are proven to be robust for mixed-lingual emotion recognition. Various Machine Learning (ML) and Deep Learning (DL) algorithms have been applied for emotion recognition. The experimental results demonstrate GeMAPS features classified from ML has been quite robust for recognizing all the emotions across the mixed-lingual database of the western languages. However, with diverse recording conditions and languages of the Indian self-recorded database the GeMAPS with enriched features and classified using DL are proven to be significant for mixed-lingual emotion recognition.

JVNV: A Corpus of Japanese Emotional Speech With Verbal Content and Nonverbal Expressions

JVNV: A Corpus of Japanese Emotional Speech With Verbal Content and Nonverbal Expressions

Speech Emotion Recognition Based on Voice Fundamental Frequency

The human voice is one of the basic means of communication, thanks to which one also can easily convey the emotional state. This paper presents experiments on emotion recognition in human speech based on the fundamental frequency. AGH Emotional Speech Corpus was used. This database consists of audio samples of seven emotions acted by 12 different speakers (6 female and 6 male). We explored phrases of all the emotions – all together and in various combinations. Fast Fourier Transformation and magnitude spectrum analysis were applied to extract the fundamental tone out of the speech audio samples. After extraction of several statistical features of the fundamental frequency, we studied if they carry information on the emotional state of the speaker applying different AI methods. Analysis of the outcome data was conducted with classifiers: K-Nearest Neighbours with local induction, Random Forest, Bagging, JRip, and Random Subspace Method from algorithms collection for data mining WEKA. The results prove that the fundamental frequency is a prospective choice for further experiments.

Cross Corpus Speech Emotion Recognition using transfer learning and attention-based fusion of Wav2Vec2 and prosody features

Speech Emotion Recognition (SER) performance degrades when their training and test conditions or corpora differ. Cross-corpus SER (CCSER) is a research branch that discusses adapting an SER system to identify speech emotions on a corpus that has different recording conditions or language from the training corpus. For CCSER, adaption can be performed in the feature extraction module or emotion classifier, which are the two main components of the SER system. In this paper, we propose AFTL method (attention-based feature fusion along with transfer learning), including methods in both feature extraction and classification for CCSER. In the feature extraction part, we use Wav2Vec 2.0 transformer blocks and prosody features, and we propose an attention method for fusing them. In the classifier part, we use transfer learning for transferring the knowledge of a model trained on source emotional speech corpus to recognize emotions on a target corpus. We performed experiments on numerous speech emotional datasets as target corpora, where we used IEMOCAP as the source corpus. For instance, we achieve 92.45% accuracy on the EmoDB dataset, where we only use 20% of speakers for adapting the source model. In addition, for other target corpora, we obtained admissible results.

DESCU: Dyadic emotional speech corpus and recognition system for Urdu language

Speech signal contains the emotional state of a speaker along with the message. The recognition of the emotional state of a speaker helps in determining the true meaning of a message and allows for more natural communication between humans and machines. This paper presents the design and development of a dyadic emotional speech corpus for the Urdu language. The corpus is developed by recording dialog scenarios for anger, happy, neutral, and sad emotions. The performance of frame-level features, utterance-level features, and spectrograms have been evaluated in this work. Emotion recognition experiments have been conducted using classifiers including Support Vector Machine, Hidden Markov Models and Convolutional Neural Networks. Experimental results show that the utterance-level features outperform the frame-level features and spectrograms. The combined feature set of cepstral, spectral, prosodic, and voice quality features performs better than the individual feature sets. The unweighted average recalls of 84.1%, 80.2%, 84.7% have been achieved for speaker-dependent and speaker-independent and text-independent emotion recognition, respectively.

An Efficient Framework for Constructing Speech Emotion Corpus Based on Integrated Active Learning Strategies

Speech emotion recognition has been developed rapidly in recent decades because of the appearance of machine learning. Nevertheless, lack of corpus remains a significant issue. For actual speech emotion corpus construction, many professional actors are required to perform voices with various emotions in specific scenes. In the process of data labelling, since the number of samples of different emotion categories is extremely imbalanced, it is difficult to efficiently label the samples. Hence, we proposed an integrated active learning sampling strategy and designed an efficient framework for constructing speech emotion corpora in order to address the problems presented above. Comparing experiments with other active learning algorithms on 13 datasets, our method was shown to improve sampling efficiency. In addition, it is able to select small category samples to be labelled with preference in imbalanced datasets. During the actual corpus construction experiments, our method can prioritize selecting small class emotion samples. As even when the amount of labelled data is less than 50%, the accuracy rate still can reach 90%. This greatly enhances the efficiency of constructing the speech emotion corpus and fills in the gaps.

Rethinking Auditory Affective Descriptors Through Zero-Shot Emotion Recognition in Speech

Zero-shot speech emotion recognition (SER) endows machines with the ability of sensing unseen-emotional states in speech, compared with conventional SER endeavors on supervised cases. On addressing the zero-shot SER task, auditory affective descriptors (AADs) are typically employed to transfer affective knowledge from seen- to unseen-emotional states. However, it remains unknown which types of AADs can well describe emotional states in speech during the transfer. In this regard, we define and research on three types of AADs, namely, per-emotion semantic-embedding, per-emotion manually annotated, and per-sample manually annotated AADs, through zero-shot emotion recognition in speech. This leads to a systematic design including prototype- and annotation-based zero-shot SER modules, relying on the input from per-emotion and per-sample AADs, respectively. We then perform extensive experimental comparisons between human and machines’ AADs on the French emotional speech corpus CINEMO for positive-negative (PN) and within-negative (WN) tasks. The experimental results indicate that semantic-embedding prototypes from pretrained models can outperform manually annotated emotional dimensions in zero-shot SER. The results further demonstrate that it is possible for machines to understand and describe affective information in speech better than human beings, with the help of sufficient pretrained models.

An Urdu speech corpus for emotion recognition.

Emotion recognition from acoustic signals plays a vital role in the field of audio and speech processing. Speech interfaces offer humans an informal and comfortable means to communicate with machines. Emotion recognition from speech signals has a variety of applications in the area of human computer interaction (HCI) and human behavior analysis. In this work, we develop the first emotional speech database of the Urdu language. We also develop the system to classify five different emotions: sadness, happiness, neutral, disgust, and anger using different machine learning algorithms. The Mel Frequency Cepstrum Coefficient (MFCC), Linear Prediction Coefficient (LPC), energy, spectral flux, spectral centroid, spectral roll-off, and zero-crossing were used as speech descriptors. The classification tests were performed on the emotional speech corpus collected from 20 different subjects. To evaluate the quality of speech emotions, subjective listing tests were conducted. The recognition of correctly classified emotions in the complete Urdu emotional speech corpus was 66.5% with K-nearest neighbors. It was found that the disgust emotion has a lower recognition rate as compared to the other emotions. Removing the disgust emotion significantly improves the performance of the classifier to 76.5%.

Emotion Estimation Method with Mel-frequency Spectrum, Voice Power Level and Pitch Frequency of Human Voices through CNN Learning Processes

Emotion estimation method with Mel-frequency spectrum, voice power level and pitch frequency of human voices through CNN (Convolutional Neural Network) learning processes is proposed. Usually, frequency spectra are used for emotion estimation. The proposed method utilizes not only Mel-frequency spectrum, but also voice pressure level (voice power level) and pitch frequency to improve emotion estimation accuracy. These components are used through CNN learning processes using training samples which are provided by Keio University (emotional speech corpus) together with our own training samples which are collected by our students in emotion estimation processes. In these processes, the target emotion is divided into two categories, confident and non-confident. Through experiments, it is found that the proposed method is superior to the traditional method with only Mel-frequency by 15%.

Factors in Emotion Recognition With Deep Learning Models Using Speech and Text on Multiple Corpora

Emotion recognition performance of deep learning models is influenced by multiple factors such as acoustic condition, textual content, style of emotion expression (e.g. acted, natural), etc. In this paper, multiple factors are analysed by training and evaluating state-of-the-art deep learning models using the input modalities speech, text, and their combination across 6 emotional speech corpora. A novel deep learning model architecture is presented that further improves the state-of-the-art in multimodal emotion recognition with speech and text on the IEMOCAP corpus. Results from models trained on individual corpora show that combining speech and text improves performance only on corpora where the text of utterances varies across different emotions, while it reduced performance on corpora with fixed text expressed in different emotions, where the speech-only models performed better. Further, cross-corpus investigations are presented to understand the robustness to changing acoustic and textual content. Results show that models perform significantly better in matched conditions in particular single corpus models perform better than multi-corpus models, with the latter showing a tendency to be more robust to acoustic variations, while performance still depends on characteristics of both training corpora and test corpus.

MsEmoTTS: Multi-Scale Emotion Transfer, Prediction, and Control for Emotional Speech Synthesis

Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model and with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audio-based and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detailed analysis shows the effectiveness of each module and the good design of the proposed method.

Bangla Speech Emotion Recognition and Cross-Lingual Study Using Deep CNN and BLSTM Networks

In this study, we have presented a deep learning-based implementation for speech emotion recognition (SER). The system combines a deep convolutional neural network (DCNN) and a bidirectional long-short term memory (BLSTM) network with a time-distributed flatten (TDF) layer. The proposed model has been applied for the recently built audio-only Bangla emotional speech corpus SUBESCO. A series of experiments were carried out to analyze all the models discussed in this paper for baseline, cross-lingual, and multilingual training-testing setups. The experimental results reveal that the model with a TDF layer achieves better performance compared with other state-of-the-art CNN-based SER models which can work on both temporal and sequential representation of emotions. For the cross-lingual experiments, cross-corpus training, multi-corpus training, and transfer learning were employed for the Bangla and English languages using the SUBESCO and RAVDESS datasets. The proposed model has attained a state-of-the-art perceptual efficiency achieving weighted accuracies (WAs) of 86.9%, and 82.7% for the SUBESCO and RAVDESS datasets, respectively.

A generalizable speech emotion recognition model reveals depression and remission.

Affective disorders are associated with atypical voice patterns; however, automated voice analyses suffer from small sample sizes and untested generalizability on external data. We investigated a generalizable approach to aid clinical evaluation of depression and remission from voice using transfer learning: We train machine learning models on easily accessible non-clinical datasets and test them on novel clinical data in a different language. A Mixture of Experts machine learning model was trained to infer happy/sad emotional state using three publicly available emotional speech corpora in German and US English. We examined the model's predictive ability to classify the presence of depression on Danish speaking healthy controls (N = 42), patients with first-episode major depressive disorder (MDD) (N = 40), and the subset of the same patients who entered remission (N = 25) based on recorded clinical interviews. The model was evaluated on raw, de-noised, and speaker-diarized data. The model showed separation between healthy controls and depressed patients at the first visit, obtaining an AUC of 0.71. Further, speech from patients in remission was indistinguishable from that of the control group. Model predictions were stable throughout the interview, suggesting that 20-30 s of speech might be enough to accurately screen a patient. Background noise (but not speaker diarization) heavily impacted predictions. A generalizable speech emotion recognition model can effectively reveal changes in speaker depressive states before and after remission in patients with MDD. Data collection settings and data cleaning are crucial when considering automated voice analysis for clinical purposes.

Novel dual-channel long short-term memory compressed capsule networks for emotion recognition

Recent analysis on speech emotion recognition (SER) has made considerable advances with the use of MFCC’s spectrogram features and the implementation of neural network approaches such as convolutional neural networks (CNNs). The fundamental issue of CNNs is that the spatial information is not recorded in spectrograms. Capsule networks (CapsNet) have gained gratitude as alternatives to CNNs with their larger capacities for hierarchical representation. However, the concealed issue of CapsNet is the compression method that is employed in CNNs cannot be directly utilized in CapsNet. To address these issues, this research introduces a text-independent and speaker-independent SER novel architecture, where a dual-channel long short-term memory compressed-CapsNet (DC-LSTM COMP-CapsNet) algorithm is proposed based on the structural features of CapsNet. Our proposed novel classifier can ensure the energy efficiency of the model and adequate compression method in speech emotion recognition, which is not delivered through the original structure of a CapsNet. Moreover, the grid search (GS) approach is used to attain optimal solutions. Results witnessed an improved performance and reduction in the training and testing running time. The speech datasets used to evaluate our algorithm are: Arabic Emirati-accented corpus, English “speech under simulated and actual stress (SUSAS)” corpus, English Ryerson audio-visual database of emotional speech and song (RAVDESS) corpus, and crowd-sourced emotional multimodal actors dataset (CREMA-D). This work reveals that the optimum feature extraction method compared to other known methods is MFCCs delta-delta. Using the four datasets and the MFCCs delta-delta, DC-LSTM COMP-CapsNet surpasses all the state-of-the-art systems, classical classifiers, CNN, and the original CapsNet. Using the Arabic Emirati-accented corpus, our results demonstrate that the proposed work yields average emotion recognition accuracy of 89.3% compared to 84.7%, 82.2%, 69.8%, 69.2%, 53.8%, 42.6%, and 31.9% based on CapsNet, CNN, support vector machine (SVM), multi-layer perceptron (MLP), k-nearest neighbor (KNN), radial basis function (RBF), and naïve Bayes (NB), respectively.

A machine learning perspective on the emotional content of Parkinsonian speech

Patients with Parkinson's disease (PD) have distinctive voice patterns, often perceived as expressing sad emotion. While this characteristic of Parkinsonian speech has been supported through the perspective of listeners, where both PD and healthy control (HC) subjects repeat the same speaking tasks, it has never been explored through a machine learning modelling approach. Our work provides an objective evaluation of this characteristic of the PD speech, by building a transfer learning system to assess how the PD pathology affects the sadness perception. To do so we introduce a Mixture-of-Experts (MoE) architecture for speech emotion recognition designed to be transferable across datasets. Firstly, by relying on publicly available emotional speech corpora, we train the MoE model and then we use it to quantify perceived sadness in never seen before PD and matched HC speech recordings. To build our models (experts), we extracted spectral features of the voicing parts of speech and we trained a gradient boosting decision trees model in each corpus to predict happiness vs. sadness. MoE predictions are created by weighting each expert's prediction according to the distance between the new sample and the expert-specific training samples. The MoE approach systematically infers more negative emotional characteristics in PD speech than in HC. Crucially, these judgments are related to the disease severity and the severity of speech impairment in the PD patients: the more impairment, the more likely the speech is to be judged as sad. Our findings pave the way towards a better understanding of the characteristics of PD speech and show how publicly available datasets can be used to train models that provide interesting insights on clinical data.

King Saud University Emotions Corpus: Construction, Analysis, Evaluation, and Comparison

Emotional speech recognition for the Arabic language is insufficiently tackled in the literature compared to other languages. In this paper, we present the work of creating and verifying the King Saud University Emotions (KSUEmotions) corpus, which was released by the Linguistic Data Consortium (LDC) in 2017 as the first public Arabic emotional speech corpus. KSUEmotions contains an emotional speech of twenty-three speakers from Saudi Arabia, Syria, and Yemen, and includes the emotions: neutral, happiness, sadness, surprise, and anger. The corpus content is verified in two different ways: a human perceptual test by nine listeners who rate emotional performance in audio files, and automatic emotion recognition. Two automatic emotion recognition systems are experimented with: Residual Neural Network and Convolutional Neural Network. This work also experiments with emotion recognition for the English language using the Emotional Prosody Speech and Transcripts Corpus (EPST). The current experimental work is conducted in three tracks: (i) monolingual, where independent experiments for Arabic and English are carried out, (ii) multilingual, where the Arabic and English corpora are merged in as mixed corpus, and (iii) cross-lingual, where models are trained using one language and tested using the other. A challenge encountered in this work is that the two corpora do not contain the same emotions. That problem is tackled by mapping the emotions to the arousal-valance space.

Cross-Corpus Speech Emotion Recognition Based on Few-Shot Learning and Domain Adaptation

Within a single speech emotion corpus, deep neural networks have shown decent performance in speech emotion recognition. However, the performance of the emotion recognition based on data-driven learning methods degrades significantly for the cross-corpus scenario. To relieve this issue without any labeled samples from the target domain, we propose a cross-corpus speech emotion recognition based on few-shot learning and unsupervised domain adaptation, which is trained to learn the class (emotion) similarity from the source domain samples adapted to the target domain. In addition, we utilize multiple corpora in training to enhance the robustness of the emotion recognition to the unseen samples. Experiments on emotional speech corpora with three different languages showed that the proposed method outperformed other approaches.