Speech Emotion Recognition System Research Articles

Speech Emotion Recognition Using Feedforward Neural Network

Speech Emotion Recognition (SER) has gained increasing attention due to its application in fields such as human-computer interaction, healthcare, customer service automation, and affective computing. This review focuses on the application of Deep Neural Networks (DNNs), specifically Feedforward Neural Networks (FNNs), in detecting and classifying emotions from speech signals. Recent advancements in deep learning have significantly enhanced the ability of machines to interpret emotional cues from auditory data. This paper discusses the motivation, objectives, and scope of employing DNN FNN architectures for SER. We also address the technical feasibility and potential of this approach for real-time applications. By reviewing existing literature, this study aims to provide insights into the current progress, challenges, and future prospects of SER systems.

Emotion Classification from Speech Waveform Using Machine Learning and Deep Learning Techniques

Emotions play a key role in determining the human mental state and indirectly express an individual’s well- being. A speech emotion recognition system can extract a person’s emotions from his/her speech inputs. There are some universal emotions such as anger, disgust, fear, happiness, pleasantness, sadness and neutral. These emotions are of significance especially in a situation like the Covid pandemic, when the aged or sick are vulnerable to depression. In the current paper, we examined various classification models with finite computational strength and resources in order to determine the emotion of a person from his/her speech. Speech prosodic features like pitch, loudness, and tone of speech, and work spectral features such as Mel Frequency Capstral Coefficients (MFCCs) of the voice were used to analyze the emotions of a person. Although sequence to sequence state of the art models for speech detection that offer high levels of accuracy and precision are currently in use, the computational needs of such approaches are high and inefficient. Therefore, in this work, we emphasised analysis and comparison of different classification algorithms such as multi layer perceptron, decision tree, support vector machine, and deep neural networks such as convolutional neural network and long short term memory. Given an audio file, the emotions that were exhibited by the speaker were recognized using machine learning and deep learning techniques. A comparative study was performed to identify the most appropriate algorithms that could be used to recognize emotions. Based on the experiment results, the MLP classifier and convolutional neural network model offered better accuracy with smaller variations when compared with other models used for the study.

Cross-feature fusion speech emotion recognition based on attention mask residual network and Wav2vec 2.0

Speech Emotion Recognition (SER) has received widespread attention as a crucial way for understanding human emotional states. However, the impact of irrelevant information on speech signals and data sparsity limit the development of SER system. To address these issues, this paper proposes a framework that incorporates the Attentive Mask Residual Network (AM-ResNet) and the self-supervised learning model Wav2vec 2.0 to obtain AM-ResNet features and Wav2vec 2.0 features respectively, together with a cross-attention module to interact and fuse these two features. The AM-ResNet branch mainly consists of maximum amplitude difference detection, mask residual block, and an attention mechanism. Among them, the maximum amplitude difference detection and the mask residual block act on the pre-processing and the network, respectively, to reduce the impact of silent frames, and the attention mechanism assigns different weights to unvoiced and voiced speech to reduce redundant emotional information caused by unvoiced speech. In the Wav2vec 2.0 branch, this model is introduced as a feature extractor to obtain general speech features (Wav2vec 2.0 features) through pre-training with a large amount of unlabeled speech data, which can assist the SER task and cope with data sparsity problems. In the cross-attention module, AM-ResNet features and Wav2vec 2.0 features are interacted with and fused to obtain the cross-fused features, which are used to predict the final emotion. Furthermore, multi-label learning is also used to add ambiguous emotion utterances to deal with data limitations. Finally, experimental results illustrate the usefulness and superiority of our proposed framework over existing state-of-the-art approaches.

Extending speech emotion recognition systems to non-prototypical emotions using mixed-emotion model

In the conventional approach to speech emotion recognition (SER), the classifier is usually trained on acted emotional speech data to predict individual basic emotions. In this work, we extend the SER systems with the realistic assumption of the coexistence of multiple basic emotions in an utterance. We utilize the MSP-Podcast database for developing the SER system, which contains spontaneous speech utterances. From the primary and secondary emotion annotations of this database, we organize six prototypical and seven non-prototypical emotions. We then propose the mixed-emotion model and express the non-prototypical emotions as a linear combination of prototypical emotions. The combination weights of the mixed-emotion model are computed using a normalized dominance scale based algorithm inspired by the integration of basic emotion theory and dimensional emotion theory in human psychology. We first train a prototypical SER model using the ECAPA-TDNN architecture. The softmax predictions from this model serve as emotion profile inputs to the mixed-emotion model, which then predicts the non-prototypical emotions. Assuming the coexistence of multiple emotions, we only apply the utterances with uniform emotion profiles to the mixed-emotion model. The developed system continues with the conventional SER model if the emotion profile tends to delta function owing to the probable occurrence of a single prototypical emotion. We develop the proposed mixed-emotion model based SER framework using MFCC and wav2vec 2.0 extracted features. Further, we show that due to human perception variations, there exist prominent annotation variations in the non-prototypical emotion ground truths. To address that, we extend the supervised evaluation protocols in four different formulations that capture the subjective variability at different levels. The proposed system shows a best-case performance improvement of 7.10% and 8.39% over the conventional prototypical SER model for the MFCC and wav2vec 2.0 features, respectively.

Advanced differential evolution for gender-aware English speech emotion recognition

Speech emotion recognition (SER) technology involves feature extraction and prediction models. However, recognition efficiency tends to decrease because of gender differences and the large number of extracted features. Consequently, this paper introduces a SER system based on gender. First, gender and emotion features are extracted from speech signals to develop gender recognition and emotion classification models. Second, according to gender differences, distinct emotion recognition models are established for male and female speakers. The gender of speakers is determined before executing the corresponding emotion model. Third, the accuracy of these emotion models is enhanced by utilizing an advanced differential evolution algorithm (ADE) to select optimal features. ADE incorporates new difference vectors, mutation operators, and position learning, which effectively balance global and local searches. A new position repairing method is proposed to address gender differences. Finally, experiments on four English datasets demonstrate that ADE is superior to comparison algorithms in recognition accuracy, recall, precision, F1-score, the number of used features and execution time. The findings highlight the significance of gender in refining emotion models, while mel-frequency cepstral coefficients are important factors in gender differences.

Research on Deep Learning-based Speech Emotion Recognition System

Research on Deep Learning-based Speech Emotion Recognition System

An enhanced speech emotion recognition using vision transformer

In human–computer interaction systems, speech emotion recognition (SER) plays a crucial role because it enables computers to understand and react to users’ emotions. In the past, SER has significantly emphasised acoustic properties extracted from speech signals. The use of visual signals for enhancing SER performance, however, has been made possible by recent developments in deep learning and computer vision. This work utilizes a lightweight Vision Transformer (ViT) model to propose a novel method for improving speech emotion recognition. We leverage the ViT model’s capabilities to capture spatial dependencies and high-level features in images which are adequate indicators of emotional states from mel spectrogram input fed into the model. To determine the efficiency of our proposed approach, we conduct a comprehensive experiment on two benchmark speech emotion datasets, the Toronto English Speech Set (TESS) and the Berlin Emotional Database (EMODB). The results of our extensive experiment demonstrate a considerable improvement in speech emotion recognition accuracy attesting to its generalizability as it achieved 98%, 91%, and 93% (TESS-EMODB) accuracy respectively on the datasets. The outcomes of the comparative experiment show that the non-overlapping patch-based feature extraction method substantially improves the discipline of speech emotion recognition. Our research indicates the potential for integrating vision transformer models into SER systems, opening up fresh opportunities for real-world applications requiring accurate emotion recognition from speech compared with other state-of-the-art techniques.

Speech emotion recognition systems and their security aspects

Speech emotion recognition (SER) systems leverage information derived from sound waves produced by humans to identify the concealed emotions in utterances. Since 1996, researchers have placed effort on improving the accuracy of SER systems, their functionalities, and the diversity of emotions that can be identified by the system. Although SER systems have become very popular in a variety of domains in modern life and are highly connected to other systems and types of data, the security of SER systems has not been adequately explored. In this paper, we conduct a comprehensive analysis of potential cyber-attacks aimed at SER systems and the security mechanisms that may prevent such attacks. To do so, we first describe the core principles of SER systems and discuss prior work performed in this area, which was mainly aimed at expanding and improving the existing capabilities of SER systems. Then, we present the SER system ecosystem, describing the dataflow and interactions between each component and entity within SER systems and explore their vulnerabilities, which might be exploited by attackers. Based on the vulnerabilities we identified within the ecosystem, we then review existing cyber-attacks from different domains and discuss their relevance to SER systems. We also introduce potential cyber-attacks targeting SER systems that have not been proposed before. Our analysis showed that only 30% of the attacks can be addressed by existing security mechanisms, leaving SER systems unprotected in the face of the other 70% of potential attacks. Therefore, we also describe various concrete directions that could be explored in order to improve the security of SER systems.

Speech Emotion Recognition using Machine Learning

This research paper presents a Speech Emotion Recognition (SER) system utilizing a Multilayer Perceptron (MLP) classifier and real-time audio analysis. The system records audio samples, extracts relevant features, and employs machine learning techniques to predict emotions in spoken language. The study focuses on the development of an intuitive Graphical User Interface (GUI) using the Kivy framework, providing a user-friendly platform for real-time emotion analysis. In shortly In this project, we attempt to detect underlying emotions such as (sad, happy, neutral, angry, disgust, surprised, fearful and calm) in recorded speech by analysing the acoustic features of the audio data of recordings and Created an application to implement the same on user input.

Joint enhancement and classification constraints for noisy speech emotion recognition

In the natural environment, the received speech signal is often interfered by noise, which reduces the performance of speech emotion recognition (SER) system. To this end, a noisy SER method based on joint constraints, including enhancement constraint and arousal-valence classification constraint (EC-AVCC), is proposed. This method introduces speech enhancement features and constraints into emotion recognition for the first time, which helps to improve the performance of noisy SER. It does not require denoising pre-processing for noisy speech, thus saving testing time. Specifically, it uses multi-domain statistical features (MDSF) composed of emotional and enhanced features as the input of the SER model based on convolution neural network (CNN) and long short-term memory-attention (ALSTM). In addition, the model is constrained jointly by speech enhancement and arousal-valence classification to obtain the robust and discriminative deep emotion features. Furthermore, in the auxiliary speech enhancement task, a joint loss function that simultaneously constrains the error of the ideal ratio mask and the error of the corresponding MDSF is introduced to obtain more robust features. The experimental results on the CASIA Chinese emotional dataset and the EMO-DB German emotional database show that compared with the CNN baseline, the proposed method improves the accuracy of SER in white noise and babble noise by 4.7%-9.9%.

Squeeze-and-excitation 3D convolutional attention recurrent network for end-to-end speech emotion recognition

Speech emotion recognition (SER) is difficult since emotions are complex and dynamic processes involving multiple dimensions and sub-dimensions. Feature extraction is a challenging step in SER, where relevant features are extracted from the speech to identify emotional states accurately. Overcoming these challenges is essential to ensuring the effectiveness and robustness of the SER system. 3D-convolutional neural networks (3D-CNNs) are successfully used for feature extraction in SER. Speech signals can be converted into spectrogram-like representations where one axis represents time, another frequency, and the third can represent additional context or features. For modeling the dynamic nature of speech, a squeeze-and-excitation-based 3D-CNN model is employed for capturing temporal and spatial features of speech. Attention Gated Recurrent Units (AGRU) are applied to the extracted features for learning long-range temporal dependencies and selecting more informative features. The extraction and selection of the spatio-temporal feature representation lead to a hierarchical representation of the input speech. The fusion of squeeze-and-excitation 3D-CNN and AGRU (named SE3D-CARN) is evaluated on two datasets, EMO-BD and IEMOCAP, to identify various emotional states in speech. The proposed SER model reached an accuracy of 94.2% and 81.1% on the EMO-BD and IEMOCAP datasets, respectively.

Enhancements in Immediate Speech Emotion Detection: Harnessing Prosodic and Spectral Characteristics

Speech is essential to human communication for expressing and understanding feelings. Emotional speech processing has challenges with expert data sampling, dataset organization, and computational complexity in large-scale analysis. This study aims to reduce data redundancy and high dimensionality by introducing a new speech emotion recognition system. The system employs Diffusion Map to reduce dimensionality and includes Decision Trees and K-Nearest Neighbors(KNN)ensemble classifiers. These strategies are suggested to increase voice emotion recognition accuracy. Speech emotion recognition is gaining popularity in affective computing for usage in medical, industry, and academics. This project aims to provide an efficient and robust real-time emotion identification framework. In order to identify emotions using supervised machine learning models, this work makes use of paralinguistic factors such as intensity, pitch, and MFCC. In order to classify data, experimental analysis integrates prosodic and spectral information utilizing methods like Random Forest, Multilayer Perceptron, SVM, KNN, and Gaussian Naïve Bayes. Fast training times make these machine learning models excellent for real-time applications. SVM and MLP have the highest accuracy at 70.86% and 79.52%, respectively. Comparisons to benchmarks show significant improvements over earlier models.

Face and Speech Emotion Recognition System

Abstract: Emotions serve as the cornerstone of human communication, facilitating the expression of one's inner thoughts and feelings to others. Speech Emotion Recognition (SER) represents a pivotal endeavour aimed at deciphering the emotional nuances embedded within a speaker's voice signal. Universal emotions such as neutrality, anger, happiness, and sadness form the basis of this recognition process, allowing for the identification of fundamental emotional states. To achieve this, spectral and prosodic features are leveraged, each offering unique insights into the emotional content of speech. Spectral features, exemplified by the Mel Frequency Cepstral Coefficient (MFCC), provide a detailed analysis of the frequency distribution within speech signals, while prosodic features encompass elements like fundamental frequency, volume, pitch, speech intensity, and glottal parameters, capturing the rhythmic and tonal variations indicative of different emotional states. Through the integration of these features, SER systems can effectively simulate and classify a diverse range of emotional expressions, paving the way for enhanced human-computer interaction and communication technologies

Creation of a diverse mixed-lingual emotional speech corpus with a framework for enhanced emotion detection

Speech Emotion Recognition (SER) has advanced considerably during the past 20 years. Till date, various SER systems have been developed for monolingual, multilingual and cross corpus contexts. However, in a country like India where numerous languages are spoken and often humans converse in more than one language, a dedicated SER system for mixed-lingual scenario is more crucial to be established which is the focus of this work. A self-recorded database that includes speech emotion samples with 11 diverse Indian languages has been developed. In parallel, a mixed-lingual database is formed with three popular standard databases of Berlin, Baum and SAVEE to represent mixed-lingual environment for western background. A detailed investigation of GeMAPS (Geneva Minimalistic Acoustic Parameter Set) feature set for mixed-lingual SER is performed. A distinct set of MFCC (Mel Frequency Cepstral Coefficients) coefficients derived from sine and cosine-based filter banks enriches the GeMAPS feature set and are proven to be robust for mixed-lingual emotion recognition. Various Machine Learning (ML) and Deep Learning (DL) algorithms have been applied for emotion recognition. The experimental results demonstrate GeMAPS features classified from ML has been quite robust for recognizing all the emotions across the mixed-lingual database of the western languages. However, with diverse recording conditions and languages of the Indian self-recorded database the GeMAPS with enriched features and classified using DL are proven to be significant for mixed-lingual emotion recognition.

Research on Speech Emotion Recognition System Using Machine Learning

Speech Emotion Recognition (SER) remains a hot topic in the domain of affective computing, drawing considerable research interest. Its growing potential, advancements in algorithms, and real-world applications contribute to this interest. Human speech carries paralinguistic cues that can be quantitatively represented through features like pitch, intensity, and Mel-Frequency Cepstral Coefficients (MFCC). SER typically involves three main stages: data processing, feature extraction/selection, and classification based on emotional features present. These steps, tailored to the unique attributes of human speech, make machine learning methods a fitting choice for SER implementation. While recent studies in affective computing leverage various ML techniques for SER tasks, few delve into the techniques aiding these core steps. Moreover, the challenges within these stages and cutting-edge approaches addressing them often receive limited discussion or are overlooked in these works.This project introduces a pioneering Speech Emotion Recognition System leveraging BiLSTM Algorithm and Image Processing techniques, developed for implementation on the Raspberry Pi platform using Python. The system analyses recorded audio input, employing three LEDs for mood detection and interfacing with Raspberry Pi programming facilitated by an SD card. This abstract outlines an all-encompassing approach for real-time emotion recognition via audio analysis, catering to varied applications in emotional AI and human-computer interaction. KEYWORDS: Speech Emotion Recognition (SER), Machine Learning, Image Processing, Raspberry Pi.

A Survey on Speech Emotion Recognition System using CNN Algorithm

The speech emotion recognition system (SER) plays an important role in decoding and predicting the speaker's emotional state by analyzing audio signals. Emotions are often simplified and grouped into categories such as anger, happiness, sadness, and even neutral emotional states. These emotions serve to communicate one's thoughts and provide insight into one's physical and mental health. Speech is the most basic and natural form of communication when interacting with others. Voice signals enable rapid communication between individuals, making them a valuable and effective method of expression. Over the past decades, countless research initiatives have been devoted to the development of voice-based automatic emotion recognition systems, especially to improve human-machine communication. Voice is gradually taking center stage in the field of Human-Machine interfaces in the IT sector. This interdisciplinary field draws on a variety of fields, including computer science, signal processing, psychology, linguistics, and more. As technology advances, it allows for seamless communication between humans and machines. Speech recognition not only interprets speech but also captures nuances in a person's tone and expressions, similar to body language. Therefore, it becomes an essential element of the Human-Machine communication system.

Speech Emotion Recognition Model Using Deep Learning

Speech has long been recognized as the main form of communication between people and computers. Technology made it possible for humans and computers to interact through the development of human-computer interfaces. Although speech emotion recognition systems have advanced quickly in recent years, many difficulties have also arisen during this development, such as the inability to recognize emotions that lead to depression and mood swings, which can be used by therapists to track their patients' moods. It is necessary to create a model that detects the many emotions that contribute to depression to improve doctor-patient relationships and increase the effectiveness of spoken emotion recognition models. In this paper, over 2000 audio files were compiled. We curated a local dataset that accounts for 60% of the total dataset acquired, 40% of the dataset used was obtained from RAVDESS. To extract the proper vocal features, we employed the Mel-Frequency Cepstral Coefficients (MFCC), Zero Crossing Rate (ZCR), and Root Mean Square (RMS). The Tensorflow CONV1D with Relu activation function and several sequential layers was used to build the model. The batch size was 64 and the epoch size was 50. Seven emotional states, including anger, disgust, sadness, happiness, surprise, fear, and neutrality were extracted. The accuracy of the confusion matrix, which served as the performance metrics, is 96%.

Selective Acoustic Feature Enhancement for Speech Emotion Recognition With Noisy Speech.

A speech emotion recognition (SER) system deployed on a real-world application can encounter speech contaminated with unconstrained background noise. To deal with this issue, a speech enhancement (SE) module can be attached to the SER system to compensate for the environmental difference of an input. Although the SE module can improve the quality and intelligibility of a given speech, there is a risk of affecting discriminative acoustic features for SER that are resilient to environmental differences. Exploring this idea, we propose to enhance only weak features that degrade the emotion recognition performance. Our model first identifies weak feature sets by using multiple models trained with one acoustic feature at a time using clean speech. After training the single-feature models, we rank each speech feature by measuring three criteria: performance, robustness, and a joint rank ranking that combines performance and robustness. We group the weak features by cumulatively incrementing the features from the bottom to the top of each rank. Once the weak feature set is defined, we only enhance those weak features, keeping the resilient features unchanged. We implement these ideas with the low-level descriptors (LLDs). We show that directly enhancing the weak LLDs leads to better performance than extracting LLDs from an enhanced speech signal. Our experiment with clean and noisy versions of the MSP-Podcast corpus shows that the proposed approach yields a 17.7% (arousal), 21.2% (dominance), and 3.3% (valence) performance gains over a system that enhances all the LLDs for the 10dB signal-to-noise ratio (SNR) condition.

Triangular Region Cut-Mix Augmentation Algorithm-Based Speech Emotion Recognition System With Transfer Learning Approach

Triangular Region Cut-Mix Augmentation Algorithm-Based Speech Emotion Recognition System With Transfer Learning Approach

Speech Emotion Recognition using Deep Learning

It is frequently simple to identify emotions in human-to-human encounters since they may be seen in speech, body language, and facial expressions. Nevertheless, recognising human emotion in human-computer interactions (HCI) can be difficult. Speech emotion recognition (SER), with the aim of just identifying emotions through vocal intonation, has arisen as a way to enhance this connection. In this paper, a SER system based on deep learning methodologies is proposed.Here, RAVDESS dataset was utilised to assess the suggested system. To select the most appropriate vocal features that represent speech emotions for this MFCC is used. LSTM, CNN, and a hybrid model that combines CNN and LSTM are three different deep learning models we used to construct our SER system. By examining these various strategies, We were able to establish the most effective model for properly recognising emotional states from speech data in real-time situations. Overall, this work indicates the usefulness of the suggested deep learning approach.