Speech Emotion Features Research Articles

Application and Effectiveness Analysis of Multimodal Emotion Recognition Technology in Music Education

Abstract Emotions in music education affect learners’ cognitive activities, and failure to capture learners’ emotional changes in a timely manner can lead to music teachers’ inability to adjust their teaching strategies in a timely manner. In this paper, a convolutional neural network is utilized to extract speech and visual emotion features of students during the process of music education. The spatial plane fusion method is used to fuse the speech and visual emotion modalities. A cross-modal interactive attention mechanism is introduced to optimize the fusion effect of the multimodal emotion features. Then, a support vector machine is used to identify and classify the emotion features. The study shows that the multimodal emotion recognition model proposed in this paper can achieve an emotion recognition accuracy of 88.78%, can accurately recognize the emotional state of students, and can assist teachers in effectively intervening in the negative emotions of students. In the music classroom applying this technology, the average test score of the student’s music education program is 93.70, and their will to learn music education is 95.09% on average. This paper’s multimodal emotion recognition model helps teachers implement effective interventions in music education and establishes the foundation for improving students’ interest in music learning.

Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Goal</i> : As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Method:</i> In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Result and Conclusions:</i> Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

Spatio-temporal representation learning enhanced speech emotion recognition with multi-head attention mechanisms

Speech emotion recognition (SER) systems have become essential in various fields, including intelligent healthcare, customer service, call centers, automatic translation systems, and human–computer interaction. However, current approaches predominantly rely on single frame-level or utterance-level features, offering only shallow or deep characterization, and fail to fully exploit the diverse types, levels, and scales of emotion features. The limited ability of single features to capture speech emotion information, along with the ineffective combination of different features’ complementary advantages through simple fusion, pose significant challenges. To address these issues, this paper presents a novel spatio-temporal representation learning enhanced speech emotion recognition with multi-head attention mechanisms(STRL-SER). The proposed technique integrates fine-grained frame-level features and coarse-grained utterance-level emotion features, while employing separate modules to extract deep representations at different levels. In the frame-level module, we introduce parallel networks and utilize a bidirectional long short-term memory network (BiLSTM) and an attention-based multi-scale convolutional neural network (CNN) to capture the spatio-temporal representation details of diverse frame-level signals. Consequently, we extract deep representations of utterance-level features to effectively learn global speech emotion features. To leverage the advantages of different feature types, we introduce a multi-head attention mechanism that fuses the deep representations from various levels. This fusion approach retains the distinctive qualities of each feature type. Finally, we employ segment-level multiplexed decision making to generate the ultimate classification results. We evaluate the effectiveness of our proposed method on two widely recognized benchmark datasets: IEMOCAP and RAVDESS. The results demonstrate that our method achieves notable performance improvements compared to previous studies. On the IEMOCAP dataset, our method achieves a weighted accuracy (WA) of 81.60% and an unweighted accuracy (UA) of 79.32%. Similarly, on the RAVDESS dataset, we achieve a WA of 88.88% and a UA of 87.85%. These outcomes confirm the substantial advancements realized by our proposed method.

Speech emotion recognition based on Graph-LSTM neural network

Currently, Graph Neural Networks have been extended to the field of speech signal processing. It is the more compact and flexible way to represent speech sequences by graphs. However, the structures of the relationships in recent studies are tend to be relatively uncomplicated. Moreover, the graph convolution module exhibits limitations that impede its adaptability to intricate application scenarios. In this study, we establish the speech-graph using feature similarity and introduce a novel architecture for graph neural network that leverages an LSTM aggregator and weighted pooling. The unweighted accuracy of 65.39% and the weighted accuracy of 71.83% are obtained on the IEMOCAP dataset, achieving the performance comparable to or better than existing graph baselines. This method can improve the interpretability of the model to some extent, and identify speech emotion features effectively.

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

AbstractSpeech emotion recognition (SER) is an important research problem in human‐computer interaction systems. The representation and extraction of features are significant challenges in SER systems. Despite the promising results of recent studies, they generally do not leverage progressive fusion techniques for effective feature representation and increasing receptive fields. To mitigate this problem, this article proposes DeepCNN, which is a fusion of spectral and temporal features of emotional speech by parallelising convolutional neural networks (CNNs) and a convolution layer‐based transformer. Two parallel CNNs are applied to extract the spectral features (2D‐CNN) and temporal features (1D‐CNN) representations. A 2D‐convolution layer‐based transformer module extracts spectro‐temporal features and concatenates them with features from parallel CNNs. The learnt low‐level concatenated features are then applied to a deep framework of convolutional blocks, which retrieves high‐level feature representation and subsequently categorises the emotional states using an attention gated recurrent unit and classification layer. This fusion technique results in a deeper hierarchical feature representation at a lower computational cost while simultaneously expanding the filter depth and reducing the feature map. The Berlin Database of Emotional Speech (EMO‐BD) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) datasets are used in experiments to recognise distinct speech emotions. With efficient spectral and temporal feature representation, the proposed SER model achieves 94.2% accuracy for different emotions on the EMO‐BD and 81.1% accuracy on the IEMOCAP dataset respectively. The proposed SER system, DeepCNN, outperforms the baseline SER systems in terms of emotion recognition accuracy on the EMO‐BD and IEMOCAP datasets.

A Deep Learning Method Using Gender-Specific Features for Emotion Recognition.

Speech reflects people's mental state and using a microphone sensor is a potential method for human-computer interaction. Speech recognition using this sensor is conducive to the diagnosis of mental illnesses. The gender difference of speakers affects the process of speech emotion recognition based on specific acoustic features, resulting in the decline of emotion recognition accuracy. Therefore, we believe that the accuracy of speech emotion recognition can be effectively improved by selecting different features of speech for emotion recognition based on the speech representations of different genders. In this paper, we propose a speech emotion recognition method based on gender classification. First, we use MLP to classify the original speech by gender. Second, based on the different acoustic features of male and female speech, we analyze the influence weights of multiple speech emotion features in male and female speech, and establish the optimal feature sets for male and female emotion recognition, respectively. Finally, we train and test CNN and BiLSTM, respectively, by using the male and the female speech emotion feature sets. The results show that the proposed emotion recognition models have an advantage in terms of average recognition accuracy compared with gender-mixed recognition models.

An Analysis of Emotional Responses of Students in Bilingual Classes and Adjustment Strategies

Students’ willingness to participate in bilingual communication is greatly influenced by their positive emotions in the bilingual class. The automatic recognition of students’ emotional state in bilingual class can assist teachers to correctly master the laws of students’ emotional changes during the bilingual learning process as fast as possible. However, the speech emotion features extracted by existing speech emotion recognition models are not universal and not suitable for bilingual speech emotion recognition, and the accuracy needs to be improved. To cope with these issues, this paper aims to study the emotional responses of students in bilingual class based on emotional analysis and provide adjustment strategies for it. At first, the signals of students’ speech records in bilingual class were pre-processed, the one-dimensional features of the signals were converted into 2-dimensional speech spectrum features so as to attain more useful information to facilitate the emotional recognition of students’ speech records in the bilingual class. Then this paper combined the bi-linear Convolution Neural Network (CNN) with capsule network to construct a bilingual class student emotion recognition model, and experimental results verified the effectiveness of the constructed model.

Speech Emotion Recognition Application for Education

Based on convolutional neural networks, a speech recognition application capable of analyzing human emotions is designed. This speech emotion recognition can better assist teachers to understand students' emotional status in the learning process and enable them to improve their teaching methods with the help of the system, thus achieving the goal of improving students' learning efficiency. The application is based on PAD dimension, convolutional neural network to extract deep speech emotion features, and Least squares support vector machine for emotion recognition, thus improving the recognition accuracy of this application.

Research on Speech Emotion Recognition Based on the Fractional Fourier Transform

Speech emotion recognition is an important part of human–computer interaction, and the use of computers to analyze emotions and extract speech emotion features that can achieve high recognition rates is an important step. We applied the Fractional Fourier Transform (FrFT), and then constructed it to extract MFCC and combined it with a deep learning method for speech emotion recognition. Since the performance of FrFT depends on the transform order p, we utilized an ambiguity function to determine the optimal order for each frame of speech. The MFCC was extracted under the optimal order of FrFT for each frame of speech. Finally, combining the deep learning network LSTM for speech emotion recognition. Our experiment was conducted on the RAVDESS, and detailed confusion matrices and accuracy were given for analysis. The MFCC extracted using FrFT was shown to have better performance than ordinal FT, and the proposed model achieved a weighting accuracy of 79.86%.

Learning multi-scale features for speech emotion recognition with connection attention mechanism

Speech emotion recognition (SER) has become a crucial topic in the field of human–computer interactions. Feature representation plays an important role in SER, but there are still many challenges in feature representation such as the inability to predict which features are most effective for SER and the cultural differences in emotion expression. Most previous studies use a single type of feature for the recognition task or conduct early fusion of features. However, a single type of feature cannot well reflect the emotions of speech signals. Also, different features contain different information, direct fusion cannot integrate the advantages of different features. To overcome these challenges, this paper proposes a parallel network for multi-scale SER based on a connection attention mechanism (AMSNet). AMSNet fuses fine-grained frame-level manual features with coarse-grained utterance-level deep features. Meanwhile, it adopts different speech emotion feature extraction modules according to the temporal and spatial features of speech signals, which enriches features and improves feature characterization. The network consists of a frame-level representation learning module (FRLM) based on the time structure and an utterance-level representation learning module (URLM) based on the global structure. Besides, improved attention-based long short-term memory (LSTM) is introduced into FRLM to focus on the frames that contribute more to the final emotion recognition result. In URLM, a convolutional neural network with the squeeze-and-excitation block (SCNN) is introduced to extract deep features. In addition, the connection attention mechanism is proposed for feature fusion, which applies different weights to different features. Extensive experiments are conducted on the IEMOCAP and EmoDB datasets, and the results demonstrate the effectiveness and performance superiority of AMSNet. Our code will be publicly available at https://codeocean.com/capsule/8636967/tree/v1.

Speech Sentiment Analysis Using Hierarchical Conformer Networks

Multimodality has been widely used for sentiment analysis tasks, especially for speech sentiment analysis. Compared with the emotion expression of most text languages, speech is more intuitive for human emotion, as speech contains more and richer emotion features. Most of the current studies mainly involve the extraction of speech features, but the accuracy and prediction rate of the models still need to be improved. To improve the extraction and fusion of speech sentiment feature information, we present a new framework. The framework adopts a hierarchical conformer model and an attention-based GRU model to increase the accuracy of the model. The method has two main parts: a local feature learning group and a global feature learning group. The local feature learning group is mainly used to learn the spatio-temporal feature information of speech emotion features through the conformer model, and a combination of convolution and transformer is used to be able to enhance the extraction of long and short-term feature information. The global features are then extracted by the AUGRU model, and the fusion of features is performed by the attention mechanism to access the weights of feature information. Finally, the sentiment is identified by a fully connected network layer, and then classified by a central loss function and a softmax function. Compared with existing speech sentiment analysis models, we obtained better sentiment classification results on the IEMOCAP and RAVDESS benchmark datasets.

Progressively Discriminative Transfer Network for Cross-Corpus Speech Emotion Recognition

Cross-corpus speech emotion recognition (SER) is a challenging task, and its difficulty lies in the mismatch between the feature distributions of the training (source domain) and testing (target domain) data, leading to the performance degradation when the model deals with new domain data. Previous works explore utilizing domain adaptation (DA) to eliminate the domain shift between the source and target domains and have achieved the promising performance in SER. However, these methods mainly treat cross-corpus tasks simply as the DA problem, directly aligning the distributions across domains in a common feature space. In this case, excessively narrowing the domain distance will impair the emotion discrimination of speech features since it is difficult to maintain the completeness of the emotion space only by an emotion classifier. To overcome this issue, we propose a progressively discriminative transfer network (PDTN) for cross-corpus SER in this paper, which can enhance the emotion discrimination ability of speech features while eliminating the mismatch between the source and target corpora. In detail, we design two special losses in the feature layers of PDTN, i.e., emotion discriminant loss and distribution alignment loss . By incorporating prior knowledge of speech emotion into feature learning (i.e., high and low valence speech emotion features have their respective cluster centers), we integrate a valence-aware center loss and an emotion-aware center loss as the to guarantee the discriminative learning of speech emotions except an emotion classifier. Furthermore, a multi-layer distribution alignment loss is adopted to more precisely eliminate the discrepancy of feature distributions between the source and target domains. Finally, through the optimization of PDTN by combining three losses, i.e., cross-entropy loss , , and , we can gradually eliminate the domain mismatch between the source and target corpora while maintaining the emotion discrimination of speech features. Extensive experimental results of six cross-corpus tasks on three datasets, i.e., Emo-DB, eNTERFACE, and CASIA, reveal that our proposed PDTN outperforms the state-of-the-art methods.

Multi-Stream Convolution-Recurrent Neural Networks Based on Attention Mechanism Fusion for Speech Emotion Recognition

The quality of feature extraction plays a significant role in the performance of speech emotion recognition. In order to extract discriminative, affect-salient features from speech signals and then improve the performance of speech emotion recognition, in this paper, a multi-stream convolution-recurrent neural network based on attention mechanism (MSCRNN-A) is proposed. Firstly, a multi-stream sub-branches full convolution network (MSFCN) based on AlexNet is presented to limit the loss of emotional information. In MSFCN, sub-branches are added behind each pooling layer to retain the features of different resolutions, different features from which are fused by adding. Secondly, the MSFCN and Bi-LSTM network are combined to form a hybrid network to extract speech emotion features for the purpose of supplying the temporal structure information of emotional features. Finally, a feature fusion model based on a multi-head attention mechanism is developed to achieve the best fusion features. The proposed method uses an attention mechanism to calculate the contribution degree of different network features, and thereafter realizes the adaptive fusion of different network features by weighting different network features. Aiming to restrain the gradient divergence of the network, different network features and fusion features are connected through shortcut connection to obtain fusion features for recognition. The experimental results on three conventional SER corpora, CASIA, EMODB, and SAVEE, show that our proposed method significantly improves the network recognition performance, with a recognition rate superior to most of the existing state-of-the-art methods.

Construction of a Knowledge Map of Speech Emotion Features Based on Impulse-Coupled Neural Networks.

This paper constructs knowledge graphs of speech emotion feature to support the expression of rich semantic information due to their unique graphical structure and to provide new ideas for studying speech emotion recognition recommendations. Impulse-coupled neural networks, as a mathematical abstraction of the visual properties of the human eye, have been widely used in various fields of speech processing. In this paper, we take the classical impulse-coupled neural network model as the research object, aiming to explore, analyze, and study the method of constructing knowledge maps of speech emotion features based on the impulse-coupled neural network model, and propose several improved impulse-coupled neural network models, which are used in the fields of target detection, speech segmentation, and quantization compression. This paper further constructs various domain semantic concept maps and performs multidimensional semantic enhancement understanding of search text based on the concept maps and the constructed entity-relationship knowledge maps and proposes a knowledge-based interpretable recommendation method for cloud services and a generalized recommendation and sample enhancement method for cloud scenarios. The fusion algorithm based on multiscale analysis can decompose the source speech into subspeech at different scales and then fuse each subspeech separately, while typical objects in the real world are also composed of many components at different scales. Based on the factors of different degrees in the speech context, the imagery work ruler evaluation model is refined with the basic principles of risk perception and risk resolution in the context, and an exploratory design is carried out separately for the verbal description, graphics, text, color, animation, and sound signals in the audiovisual signal using the workshop format to obtain a large number of design samples and evaluate the important units in turn, finally integrating the context involved in the study. The perceptual design system is given as a specific contextual design method.

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

Articulation, emotion, and personality play strong roles in the orofacial movements. To improve the naturalness and expressiveness of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">virtual agents</i> (VAs), it is important that we carefully model the complex interplay between these factors. This paper proposes a conditional generative adversarial network, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">conditional sequential GAN</i> (CSG), which learns the relationship between emotion, lexical content and lip movements in a principled manner. This model uses a set of spectral and emotional speech features directly extracted from the speech signal as conditioning inputs, generating realistic movements. A key feature of the approach is that it is a speech-driven framework that does not require transcripts. Our experiments show the superiority of this model over three state-of-the-art baselines in terms of objective and subjective evaluations. When the target emotion is known, we propose to create emotionally dependent models by either adapting the base model with the target emotional data (CSG-Emo-Adapted), or adding emotional conditions as the input of the model (CSG-Emo-Aware). Objective evaluations of these models show improvements for the CSG-Emo-Adapted compared with the CSG model, as the trajectory sequences are closer to the original sequences. Subjective evaluations show significantly better results for this model compared with the CSG model when the target emotion is happiness.

Research Progress of Speech Emotion Recognition Based on Discrete Emotion Model

Speech emotion recognition is one of the important technologies of human-computer interaction, and neural networks have made great contributions in it. In this survey, the commonly used discrete emotion databases and speech emotion feature parameters are firstly introduced. Then the feature extraction methods and emotion recognition models in recent three years’ research in China are summarized, and the former one can be divided into two directions, selecting the feature subset from the existing features and using neural network to extract new features. Finally, the challenges encountered in speech emotion recognition are described, and the future research and development of speech emotion recognition based on discrete emotion models are prospected.

Algorithm for speech emotion recognition classification based on Mel-frequency Cepstral coefficients and broad learning system

Speech plays a major role in emotional transmitting information in humans, and speech emotion recognition has become an important part of the human–computer system, especially in specific systems with high requirements for real-time and accuracy. To improve the accuracy and real-time of speech emotion recognition, people have done a lot of work in speech emotion feature extraction and speech emotion recognition algorithms, but the recognition rate also needs improvement. In this paper, we propose a speech emotion recognition method based on Mel-frequency Cepstral coefficients (MFCC) and broad learning network. 39-dimensional MFCC features were extracted after preprocess of the speech signal. After labelling and standardizing the data, a data prediction model is built. Finally, the data set is split into training and test data onto a certain ratio (0.8). We experimented with broad learning network architecture. And then the data processing in the broad learning network is improved. The proposed algorithm is a neural network structure that does not rely on deep structure, which has a small amount of calculation, excellent calculation speed and simple structure. The experimental results show that the proposed network architecture achieves higher accuracy and it turned out to be the most accurate in recognizing emotions in CASIA Chinese emotion corpus. The recognition rate can reach 100%. Therefore, the proposed network architecture provides an effective method of speech emotion recognition.

On the Differences in Prosodic Features of Assamese Speech Based on Types of Emotions

Speech is a vocal communication through which we achieve the information about speaker, language, message etc. Speaking is the process of converting discrete phonemes into continuous acoustic signal. Speech carries linguistic information that is associated with emotions, along with vocal information, which can be extracted by speech processing methods. In this paper, we basically analyse on prosodic features of different emotional speech in Assamese language for identify the comparison between male and female speakers who linguistically react differently in the same situations. We also apply statistical paired t-Test on prosodic features for showing the significant difference between male and female speech in different emotions.

Speech Emotion Feature Analysis Based on Emotion Fingerprints

The speech recognition has caught the attention of more and more researchers as the important branch of artificial intelligence. However, speech recognition just can recognize the neutral meaning and ignore the important emotion information. So, this paper proposes an emotion fingerprint extraction algorithm. The speech signal is decomposed by lifting wavelet packets. And combining singular value entropy and sample entropy as the feature vector to characterize the speech time-frequency matrix coefficient. Then adopting the statistical values to extract emotion fingerprints. The experiments show that this algorithm can reflect the emotion information of the speech fully and distinguish several major emotions well.

Emotional speech analysis using harmonic plus noise model and Gaussian mixture model

Extracting the valuable information from the emotional speech is one of the major challenges in the areas of emotion recognition and human-machine interfaces. Most of the research in emotion recognition is based on the analysis of fundamental frequency, energy contour, duration of silence, formant, Mel-band energies, linear prediction cepstral coefficients, and Mel frequency cepstral coefficients. It was observed that emotion classification using sinusoidal features perform better as compared to the linear prediction and cepstral features. Harmonic models are considered as a variant of the sinusoidal model. In order to improve emotional speech classification rate and conversion of neutral speech to emotional speech, analysis using different harmonic features of emotional speech is a critical step. In this paper, investigations have been carried out using Berlin emotional speech database to analyze gender-based emotional speech using harmonic plus noise model (HNM) features and Gaussian mixture model (GMM). Analysis has been performed with the HNM features like pitch, harmonic amplitude, maximum voiced frequency and noise components. From the results, it can be observed that different emotional speech of male and female speakers can be represented with K components of GMM distribution. The optimal number of GMM components have been decided on the basis of Akaike information criterion (AIC) score.