Berlin Emotional Speech Database Research Articles

Combining Transformer, Convolutional Neural Network, and Long Short-Term Memory Architectures: A Novel Ensemble Learning Technique That Leverages Multi-Acoustic Features for Speech Emotion Recognition in Distance Education Classrooms

Speech emotion recognition (SER) is a technology that can be applied to distance education to analyze speech patterns and evaluate speakers’ emotional states in real time. It provides valuable insights and can be used to enhance students’ learning experiences by enabling the assessment of their instructors’ emotional stability, a factor that significantly impacts the effectiveness of information delivery. Students demonstrate different engagement levels during learning activities, and assessing this engagement is important for controlling the learning process and improving e-learning systems. An important aspect that may influence student engagement is their instructors’ emotional state. Accordingly, this study used deep learning techniques to create an automated system for recognizing instructors’ emotions in their speech when delivering distance learning. This methodology entailed integrating transformer, convolutional neural network, and long short-term memory architectures into an ensemble to enhance the SER. Feature extraction from audio data used Mel-frequency cepstral coefficients; chroma; a Mel spectrogram; the zero-crossing rate; spectral contrast, centroid, bandwidth, and roll-off; and the root-mean square, with subsequent optimization processes such as adding noise, conducting time stretching, and shifting the audio data. Several transformer blocks were incorporated, and a multi-head self-attention mechanism was employed to identify the relationships between the input sequence segments. The preprocessing and data augmentation methodologies significantly enhanced the precision of the results, with accuracy rates of 96.3%, 99.86%, 96.5%, and 85.3% for the Ryerson Audio–Visual Database of Emotional Speech and Song, Berlin Database of Emotional Speech, Surrey Audio–Visual Expressed Emotion, and Interactive Emotional Dyadic Motion Capture datasets, respectively. Furthermore, it achieved 83% accuracy on another dataset created for this study, the Saudi Higher-Education Instructor Emotions dataset. The results demonstrate the considerable accuracy of this model in detecting emotions in speech data across different languages and datasets.

Emotions recognition in audio signals using an extension of the latent block model

Emotion detection in human speech is a significant area of research, crucial for various applications such as affective computing and human–computer interaction. Despite advancements, accurately categorizing emotional states in speech remains challenging due to its subjective nature and the complexity of human emotions. To address this, we propose leveraging Mel frequency cepstral coefficients (MFCCS) and extend the latent block model (LBM) probabilistic clustering technique with a Gaussian multi-way latent block model (GMWLBM). Our objective is to categorize speech emotions into coherent groups based on the emotional states conveyed by speakers. We employ MFCCS from time-series audio data and utilize a variational Expectation Maximization method to estimate GMWLBM parameters. Additionally, we introduce an integrated Classification Likelihood (ICL) model selection criterion to determine the optimal number of clusters, enhancing robustness. Numerical experiments on real data from the Berlin Database of Emotional Speech (EMO-DB) demonstrate our method’s efficacy in accurately detecting and classifying emotional states in human speech, even in challenging real-world scenarios, thereby contributing significantly to affective computing and human–computer interaction applications.

Speech Emotion Recognition using Extreme Machine Learning

Detecting Emotion from Spoken Words (SER) is the task of detecting the underlying emotion in spoken language. It is a challenging task, as emotions are subjective and highly contextual. Machine learning algorithms have been widely used for SER, and one such algorithm is the Gaussian Mixture Model (GMM) algorithm. The GMM algorithm is a statistical model that represents the probability distribution of a random variable as a sum of Gaussian distributions. It has been widely used for speech recognition and classification tasks. In this article, we offer a method for SER using Extreme Machine Learning (EML) with the GMM algorithm. EML is a type of machine learning that uses randomization to achieve high accuracy at a low computational cost. It has been effectively utilised in various classification tasks. For the planned approach includes two steps: feature extraction and emotion classification. Cepstral Coefficients of Melody Frequency (MFCCs) are used in order to extract features. MFCCs are commonly used for speech processing and represent the spectral envelope of the speech signal. The GMM algorithm is used for emotion classification. The input features are modelled as a mixture of Gaussians, and the emotion is classified based on the likelihood of the input features belonging to each Gaussian. Measurements were taken of the suggested method on the The Berlin Database of Emotional Speech (EMO-DB) and achieved an accuracy of 74.33%. In conclusion, the proposed approach to SER using EML and the GMM algorithm shows promising results. It is a computationally efficient and effective approach to SER and can be used in various applications, such as speech-based emotion detection for virtual assistants, call centre analytics, and emotional analysis in psychotherapy.

Speech emotion recognition using multiple classification models based on MFCC feature values

In everyday life, people are often influenced by emotions in our behaviour and language. When people use different emotions to articulate the same text, it can have a completely different effect. With the increasing demand for speech emotion recognition (SER), more machine learning and deep learning methods are being used to perform SER. matlab was used as the experimental tool in this study. The Berlin Database of Emotional Speech was used as the database. Feature extraction was performed by Mel Frequency Cepstrum Coefficients (MFCC) and based on these feature values, Support Vector Machines (SVM), K-Nearest Neighbors Algorithm (KNN), Semi-supervised graph-based classifier, ECOC classification model, Naive Bayes model and long short-term memory for predictive classification. The results surface that among the six classifiers, the best sentiment recognition method is LSTM, with 93.2% and 73.03% accuracies in the training and test groups respectively.

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

AbstractSpeech emotion recognition (SER) is an important research problem in human‐computer interaction systems. The representation and extraction of features are significant challenges in SER systems. Despite the promising results of recent studies, they generally do not leverage progressive fusion techniques for effective feature representation and increasing receptive fields. To mitigate this problem, this article proposes DeepCNN, which is a fusion of spectral and temporal features of emotional speech by parallelising convolutional neural networks (CNNs) and a convolution layer‐based transformer. Two parallel CNNs are applied to extract the spectral features (2D‐CNN) and temporal features (1D‐CNN) representations. A 2D‐convolution layer‐based transformer module extracts spectro‐temporal features and concatenates them with features from parallel CNNs. The learnt low‐level concatenated features are then applied to a deep framework of convolutional blocks, which retrieves high‐level feature representation and subsequently categorises the emotional states using an attention gated recurrent unit and classification layer. This fusion technique results in a deeper hierarchical feature representation at a lower computational cost while simultaneously expanding the filter depth and reducing the feature map. The Berlin Database of Emotional Speech (EMO‐BD) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) datasets are used in experiments to recognise distinct speech emotions. With efficient spectral and temporal feature representation, the proposed SER model achieves 94.2% accuracy for different emotions on the EMO‐BD and 81.1% accuracy on the IEMOCAP dataset respectively. The proposed SER system, DeepCNN, outperforms the baseline SER systems in terms of emotion recognition accuracy on the EMO‐BD and IEMOCAP datasets.

Artificial Intelligent for Human Emotion Detection with the Mel-Frequency Cepstral Coefficient (MFCC)

Emotions are an important aspect of human communication. Expression of human emotions can be identified through sound. The development of voice detection or speech recognition is a technology that has developed rapidly to help improve human-machine interaction. This study aims to classify emotions through the detection of human voices. One of the most frequently used methods for sound detection is the Mel-Frequency Cepstrum Coefficient (MFCC) where sound waves are converted into several types of representation. Mel-frequency cepstral coefficients (MFCCs) are the coefficients that collectively represent the short-term power spectrum of a sound, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. The primary data used in this research is the data recorded by the author. The secondary data used is data from the "Berlin Database of Emotional Speech" in the amount of 500 voice recording data. The use of MFCC can extract implied information from the human voice, especially to recognize the feelings experienced by humans when pronouncing the sound. In this study, the highest accuracy was obtained when training with epochs of 10000 times, which was 85% accuracy.

A digital “flat affect”? Popular speech compression codecs and their effects on emotional prosody

IntroductionCalls via video apps, mobile phones and similar digital channels are a rapidly growing form of speech communication. Such calls are not only— and perhaps less and less— about exchanging content, but about creating, maintaining, and expanding social and business networks. In the phonetic code of speech, these social and emotional signals are considerably shaped by (or encoded in) prosody. However, according to previous studies, it is precisely this prosody that is significantly distorted by modern compression codecs. As a result, the identification of emotions becomes blurred and can even be lost to the extent that opposing emotions like joy and anger or disgust and sadness are no longer differentiated on the recipients' side. The present study searches for the acoustic origins of these perceptual findings.MethodA set of 108 sentences from the Berlin Database of Emotional Speech served as speech material in our study. The sentences were realized by professional actors (2m, 2f) with seven different emotions (neutral, fear, disgust, joy, boredom, anger, sadness) and acoustically analyzed in the original uncompressed (WAV) version and as well as in strongly compressed versions based on the four popular codecs AMR-WB, MP3, OPUS, and SPEEX. The analysis included 6 tonal (i.e. f0-related) and 7 non-tonal prosodic parameters (e.g., formants as well as acoustic-energy and spectral-slope estimates).ResultsResults show significant, codec-specific distortion effects on all 13 prosodic parameter measurements compared to the WAV reference condition. Means values of automatic measurement can, across sentences, deviate by up to 20% from the values of the WAV reference condition. Moreover, the effects go in opposite directions for tonal and non-tonal parameters. While tonal parameters are distorted by speech compression such that the acoustic differences between emotions are increased, compressing non-tonal parameters make the acoustic-prosodic profiles of emotions more similar to each other, particularly under MP3 and SPEEX compression.DiscussionThe term “flat affect” comes from the medical field and describes a person's inability to express or display emotions. So, does strong compression of emotional speech create a “digital flat affect”? The answer to this question is a conditional “yes”. We provided clear evidence for a “digital flat affect”. However, it seems less strongly pronounced in the present acoustic measurements than in previous perception data, and it manifests itself more strongly in non-tonal than in tonal parameters. We discuss the practical implications of our findings for the everyday use of digital communication devices and critically reflect on the generalizability of our findings, also with respect to their origins in the codecs' inner mechanics.

The design of intelligent fuzzy cognitive system of music emotion by product supply chain management

AbstractThe classification of musical emotion is a challenging problem due to the subjectivity of human perception. In order to analyse the emotion contained in music from the perspective of product supply chain management, the psychological model and characteristics of music emotion are introduced. Additionally, the current product supply chain and the composition of the two contracts are analysed based on the characteristics of online music products. Finally, a speech emotion recognition model based on a Fuzzy Cognitive Map (FCM) is proposed. Two corpora containing five emotions (sadness, anger, happiness, surprise, and boredom) are selected to test the performance and characteristics of the network. In order to evaluate the system performance, different acoustic features are extracted, including prosodic features, formant features, marginal spectral features, and Mel‐Frequency Cepstral Coefficients (MFCC). The results show that the average recognition rate of MFCC features on the Berlin emotional speech database is 70.36%, which is 0.75% higher than the second‐highest edge spectrum features and achieves better results. Meanwhile, on this database, the Musical Emotion Recognition System based on FCM (MERS‐FCM) is improved by 20.36%, 7.34%, and 4.12% compared with Back Propagation, K‐nearest neighbour, and support vector machine, respectively, which shows that MERS‐FCM has better performance. The research of the music emotion recognition system based on FCM can provide more reference paths for the use and development of music emotion information.

Improving CNN-based solutions for emotion recognition using evolutionary algorithms

AI-based approaches, especially deep learning have made remarkable achievements in Speech Emotion Recognition (SER). Needless to say, Convolutional Neural Networks (CNNs) have been the backbone of many of these solutions. Although the use of CNNs have resulted in high performing models, building them needs domain knowledge and direct human intervention. The same issue arises while improving a model. To solve this problem, we use techniques that were firstly introduced in Neural Architecture Search (NAS) and use a genetic process to search for models with improved accuracy. More specifically, we insert blocks with dynamic structures in between the layers of an already existing model and then use genetic operations (i.e. selection, mutation, and crossover) to find the best performing structures. To validate our method, we use this algorithm to improve architectures by searching on the Berlin Database of Emotional Speech (EMODB). The experimental results show at least 1.7% performance improvement in terms of Accuracy on EMODB test set.

Speech Emotion Recognition Based on Multiple Acoustic Features and Deep Convolutional Neural Network

Speech emotion recognition (SER) plays a vital role in human–machine interaction. A large number of SER schemes have been anticipated over the last decade. However, the performance of the SER systems is challenging due to the high complexity of the systems, poor feature distinctiveness, and noise. This paper presents the acoustic feature set based on Mel frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), wavelet packet transform (WPT), zero crossing rate (ZCR), spectrum centroid, spectral roll-off, spectral kurtosis, root mean square (RMS), pitch, jitter, and shimmer to improve the feature distinctiveness. Further, a lightweight compact one-dimensional deep convolutional neural network (1-D DCNN) is used to minimize the computational complexity and to represent the long-term dependencies of the speech emotion signal. The overall effectiveness of the proposed SER systems’ performance is evaluated on the Berlin Database of Emotional Speech (EMODB) and the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) datasets. The proposed system gives an overall accuracy of 93.31% and 94.18% for the EMODB and RAVDESS datasets, respectively. The proposed MFCC and 1-D DCNN provide greater accuracy and outpace the traditional SER techniques.

A Comparison of Machine Learning Algorithms and Feature Sets for Automatic Vocal Emotion Recognition in Speech.

Vocal emotion recognition (VER) in natural speech, often referred to as speech emotion recognition (SER), remains challenging for both humans and computers. Applied fields including clinical diagnosis and intervention, social interaction research or Human Computer Interaction (HCI) increasingly benefit from efficient VER algorithms. Several feature sets were used with machine-learning (ML) algorithms for discrete emotion classification. However, there is no consensus for which low-level-descriptors and classifiers are optimal. Therefore, we aimed to compare the performance of machine-learning algorithms with several different feature sets. Concretely, seven ML algorithms were compared on the Berlin Database of Emotional Speech: Multilayer Perceptron Neural Network (MLP), J48 Decision Tree (DT), Support Vector Machine with Sequential Minimal Optimization (SMO), Random Forest (RF), k-Nearest Neighbor (KNN), Simple Logistic Regression (LOG) and Multinomial Logistic Regression (MLR) with 10-fold cross validation using four openSMILE feature sets (i.e., IS-09, emobase, GeMAPS and eGeMAPS). Results indicated that SMO, MLP and LOG show better performance (reaching to 87.85%, 84.00% and 83.74% accuracies, respectively) compared to RF, DT, MLR and KNN (with minimum 73.46%, 53.08%, 70.65% and 58.69% accuracies, respectively). Overall, the emobase feature set performed best. We discuss the implications of these findings for applications in diagnosis, intervention or HCI.

Speech emotion recognition using recurrent neural networks with directional self-attention

As an important branch of affective computing, Speech Emotion Recognition (SER) plays a vital role in human–computer interaction. In order to mine the relevance of signals in audios an increase the diversity of information, Bi-directional Long-Short Term Memory with Directional Self-Attention (BLSTM-DSA) is proposed in this paper. Long Short-Term Memory (LSTM) can learn long-term dependencies from learned local features. Moreover, Bi-directional Long-Short Term Memory (BLSTM) can make the structure more robust by direction mechanism because that the directional analysis can better recognize the hidden emotions in sentence. At the same time, autocorrelation of speech frames can be used to deal with the lack of information, so that Self-Attention mechanism is introduced into SER. The attention weight of each frame is calculated with the output of the forward and backward LSTM respectively rather than calculated after adding them together. Thus, the algorithm can automatically annotate the weights of speech frames to correctly select frames with emotional information in temporal network. When evaluate it on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database and Berlin database of emotional speech (EMO-DB), the BLSTM-DSA demonstrates satisfactory performance on the task of speech emotion recognition. Especially in emotion recognizing of happiness and anger, BLSTM-DSA achieves the highest recognition accuracies.

Speech Emotion Recognition Using Clustering Based GA-Optimized Feature Set

Speech Emotion Recognition (SER) is a hot topic in academia and industry. Feature engineering plays a pivotal role in building an efficient SER. Although researchers have done a tremendous amount of work in this field, there are still the issues of speech feature choice and the correct application of feature engineering that remains to be solved in the domain of SER. In this research, a feature optimization approach that uses a clustering-based genetic algorithm is proposed. Instead of randomly selecting the new generation, clustering is applied at the fitness evaluation level to detect outliers for exclusion to be part of the next generation. The approach is compared with the standard Genetic Algorithm in the context of audio emotion recognition using Berlin Emotional Speech Database (EMO-DB), Ryerson Audio-Visual Database of Speech and Song (RAVDESS) and, Surrey Audio-Visual Expressed Emotion Dataset (SAVEE). Results signify that the proposed technique effectively improved the emotion classification in speech. The recognition rate of 89.6% for general speakers (both male and female), 86.2% for male speakers, and 88.3% for female speakers on EMO-DB, 82.5% for general speakers, 75.4% for male speakers, and 91.1% for female speaker on RAVDESS, and 77.7% for general speakers on SAVEE is obtained in speaker-dependent experiments. For speaker-independent experiments, we achieved the recognition rate of 77.5% on EMO-DB, 76.2% on RAVDESS and, 69.8 % on SAVEE. All the experiments were performed on MATLAB and the Support Vector Machine (SVM) was used for classification. Results confirm that the proposed method is capable of discriminating emotions effectively and performed better than the other approaches used for comparison in terms of performance measures

Robustness to noise for speech emotion classification using CNNs and attention mechanisms

Speech Emotion Recognition (SER) is an important task since emotion is a primary dimension in human communication and health. It has a wide variety of practical applications such as assessing the mood of callers to an emergency call center and as a diagnostic tool for therapists. Since most of the SER models in the literature are trained with clean noiseless data and non noise-robust input features, they are not very useful in real world conditions where noise is almost always present. Although there are methods to reduce these adverse effects of noise, a systematic analysis of these methods in the context of SER is lacking. In this paper several different methods to mitigate adverse effects of noise on CNN (Convolutional Neural Network) based SER are developed and analyzed. The SER models trained on the Berlin Database of Emotional Speech were tested with clean data and data mixed with 10 different noise types at different noise levels with signal to noise ratios of 10,15,20,25,30 and 35. We show that the noise robustness of SER models can be improved by combining the magnitude spectrogram with the modified group delay spectrogram, by including synthetic noise in the training data, and by using an attention mechanism. When trained with noisy data, the models trained with the combined input saw a 10% increase in average accuracy than the models using individual inputs and not trained with noise. Adding the attention mechanism to the previous model further improved the accuracy by 5%. Finally, by training and evaluating on the RAVDESS dataset, we demonstrated that the noise robust methods developed can be generalized into other datasets and emotions. We achieved an average accuracy of 81% on RAVDESS dataset under noisy conditions.

A modified feature selection method based on metaheuristic algorithms for speech emotion recognition

Feature selection plays an important role to build a successful speech emotion recognition system. In this paper, a feature selection approach which modifies the initial population generation stage of metaheuristic search algorithms, is proposed. The approach is evaluated on two metaheuristic search algorithms, a nondominated sorting genetic algorithm-II (NSGA-II) and Cuckoo Search in the context of speech emotion recognition using Berlin emotional speech database (EMO-DB) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) database. Results show that the presented feature selection algorithms reduce the number of features significantly and are still effective for emotion classification from speech. Specifically, in speaker-dependent experiments of the EMO-DB, recognition rates of 87.66% and 87.20% are obtained using selected features by modified Cuckoo Search and NSGA-II respectively, whereas, for the IEMOCAP database, the accuracies of 69.30% and 68.32% are obtained using SVM classifier. For the speaker-independent experiments, we achieved comparable results for both databases. Specifically, recognition rates of 76.80% and 76.82% for EMO-DB and 59.37% and 59.52% for IEMOCAP using modified NSGA-II and Cuckoo Search respectively.

Recognition of speech emotion using custom 2D-convolution neural network deep learning algorithm

Speech emotion recognition has become the heart of most human computer interaction applications in the modern world. The growing need to develop emotionally intelligent devices has opened up a lot of research opportunities. Most researchers in this field have applied the use of handcrafted features and machine learning techniques in recognising speech emotion. However, these techniques require extra processing steps and handcrafted features are usually not robust. They are computationally intensive because the curse of dimensionality results in low discriminating power. Research has shown that deep learning algorithms are effective for extracting robust and salient features in dataset. In this study, we have developed a custom 2D-convolution neural network that performs both feature extraction and classification of vocal utterances. The neural network has been evaluated against deep multilayer perceptron neural network and deep radial basis function neural network using the Berlin database of emotional speech, Ryerson audio-visual emotional speech database and Surrey audio-visual expressed emotion corpus. The described deep learning algorithm achieves the highest precision, recall and F1-scores when compared to other existing algorithms. It is observed that there may be need to develop customized solutions for different language settings depending on the area of applications.

Fusing Visual Attention CNN and Bag of Visual Words for Cross-Corpus Speech Emotion Recognition.

Speech emotion recognition (SER) classifies emotions using low-level features or a spectrogram of an utterance. When SER methods are trained and tested using different datasets, they have shown performance reduction. Cross-corpus SER research identifies speech emotion using different corpora and languages. Recent cross-corpus SER research has been conducted to improve generalization. To improve the cross-corpus SER performance, we pretrained the log-mel spectrograms of the source dataset using our designed visual attention convolutional neural network (VACNN), which has a 2D CNN base model with channel- and spatial-wise visual attention modules. To train the target dataset, we extracted the feature vector using a bag of visual words (BOVW) to assist the fine-tuned model. Because visual words represent local features in the image, the BOVW helps VACNN to learn global and local features in the log-mel spectrogram by constructing a frequency histogram of visual words. The proposed method shows an overall accuracy of 83.33%, 86.92%, and 75.00% in the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), the Berlin Database of Emotional Speech (EmoDB), and Surrey Audio-Visual Expressed Emotion (SAVEE), respectively. Experimental results on RAVDESS, EmoDB, SAVEE demonstrate improvements of 7.73%, 15.12%, and 2.34% compared to existing state-of-the-art cross-corpus SER approaches.

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features.

Artificial intelligence (AI) and machine learning (ML) are employed to make systems smarter. Today, the speech emotion recognition (SER) system evaluates the emotional state of the speaker by investigating his/her speech signal. Emotion recognition is a challenging task for a machine. In addition, making it smarter so that the emotions are efficiently recognized by AI is equally challenging. The speech signal is quite hard to examine using signal processing methods because it consists of different frequencies and features that vary according to emotions, such as anger, fear, sadness, happiness, boredom, disgust, and surprise. Even though different algorithms are being developed for the SER, the success rates are very low according to the languages, the emotions, and the databases. In this paper, we propose a new lightweight effective SER model that has a low computational complexity and a high recognition accuracy. The suggested method uses the convolutional neural network (CNN) approach to learn the deep frequency features by using a plain rectangular filter with a modified pooling strategy that have more discriminative power for the SER. The proposed CNN model was trained on the extracted frequency features from the speech data and was then tested to predict the emotions. The proposed SER model was evaluated over two benchmarks, which included the interactive emotional dyadic motion capture (IEMOCAP) and the berlin emotional speech database (EMO-DB) speech datasets, and it obtained 77.01% and 92.02% recognition results. The experimental results demonstrated that the proposed CNN-based SER system can achieve a better recognition performance than the state-of-the-art SER systems.

Speech emotion recognition using cepstral features extracted with novel triangular filter banks based on bark and ERB frequency scales

Speech emotion recognition (SER) refers to the process of recognizing the emotional state of the speaker from the speech utterance. In earlier studies, wide varieties of cepstral features have been proposed to develop SER systems. The mel-frequency cepstral coefficients (MFCC), and human-factor cepstral coefficients (HFCC) are two popularly used variants of cepstral features. MFCC and HFCC features are extracted from speech signals using mel and human-factor filter banks, respectively. The magnitude response of individual filters in these filter banks is triangular in shape. As a result, these filter banks are referred to as triangular filter banks (TFB) and the corresponding extracted cepstral coefficients can be denoted as TFBCC-M (in case of MFCC) and TFBCC-HF (in case of HFCC). The mel-filter bank (TFB-M) is constructed using mel-scale, while the human-factor filter bank (TFB-HF) is constructed using human factor scale, which is a combination of mel and equivalent rectangular bandwidth (ERB) scales. Similarly, different frequency scales can be used to realize different TFBs to extract different types of TFBCC features. In this direction, this paper proposes two new TFBs denoted as TFB-B and TFB-E, realized using bark and ERB scales to extract new cepstral features referred to as TFBCC-B and TFBCC-E, respectively. The mathematical background to construct the proposed TFB-B and TFB-E is presented. The proposed filter banks are used along with the conventional TFB-M and TFB-HF to extract four different types of TFBCC features. These features are extracted from the emotional speech signals of two databases, namely Berlin database of emotional speech (Emo-DB) and Surrey audio-visual expressed emotion speech database (SAVEE). The extracted features are used to develop speaker-dependent (SD) and speaker-independent (SI) based SER systems using support vector machines. The performance of the respective features is analyzed in terms of isolated and combined usage. The experimental results show that the cepstral features extracted using the proposed TFBs are effective in characterizing and recognizing emotions similar to conventional MFCC and HFCC features. Moreover, the combined use of different cepstral features have resulted to improve the overall recognition performance of SER systems. In case of Emo-DB database, isolated use of the proposed TFBCC-B and TFBCC-E features achieve recognition accuracies of 83.23% and 81.99% for SD scenario, and 75% and 60.94% for SI scenario, respectively. Similarly, for SAVEE database, the recognition accuracies of 75% and 66.67% for SD scenario, and 44.17% and 55% for SI scenario are achieved. In case of Emo-DB database, the maximum recognition accuracies of 86.96% (for different combinations of conventional and proposed features namely, TFBCC-{(M+E), (M+B+E), (HF+B+E), (M+HF+B+E)}) and 77.08% (for combination TFBCC-(M+B+E)) are achieved for SD and SI scenarios, respectively. Similarly, for SAVEE database, the maximum recognition accuracies of 77.08% (for combination TFBCC-(M+HF+E)), and 55.83% (for combination TFBCC-(B+E)) are achieved for SD and SI scenarios, respectively.

Speech emotion recognition using hybrid spectral-prosodic features of speech signal/glottal waveform, metaheuristic-based dimensionality reduction, and Gaussian elliptical basis function network classifier

In this paper, a hybrid system consisting of three stages of feature extraction, dimensionality reduction, and feature classification is proposed for speech emotion recognition (SER). At feature extraction stage, an informationally-rich spectral-prosodic hybrid feature vector comprised of perceptual-spectral features; that is, mel-frequency cepstral coefficient (MFCC), perceptual linear prediction coefficient (PLPC), and perceptual minimum variance distortionless response (PMVDR) coefficient along with the prosodic feature of pitch (i.e. F0) are extracted for each frame. This feature vector is extracted from both speech signal and its glottal-waveform. The first and the second-order derivatives are then added to the above-mentioned vector to form a high-dimensional hybrid feature vector characterized by a large number of dimensions. At the next stage, i.e. dimensionality reduction, the dimensionality of this feature vector is reduced using a new proposed quantum-behaved particle swarm optimization (QPSO)-based approach. In this paper, a new QPSO algorithm (so-called, pQPSO) is presented that makes use of a truncated Laplace distribution (TLD) to generate new particles and thus to produce solutions (i.e. particles) that are all within a valid range of a problem (contrary to the standard QPSO). The contraction-expansion (CE) factor of the proposed pQPSO is also selected adaptively. Using the proposed QPSO algorithm, an optimal discriminative dimensionality reduction matrix (i.e. projection matrix) is estimated with emotion classification accuracy as a class-discriminative criterion. At the subsequent stage, vectors with reduced feature dimensionality are fed into a Gaussian elliptical basis function (GEBF)-type neural network classifier to detect their speech emotion. To accelerate the training phase of the GEBF classifier, a fast-scaled conjugate gradient (SCG) algorithm is correspondingly employed that does not need to adjust the learning rate. Finally, the proposed method is evaluated on three standard emotional speech databases of Berlin Database of Emotional Speech (EMODB), Surrey Audio-Visual Expressed Emotion (SAVEE), and Interactive Emotional Dyadic Motion Capture (IEMOCAP). The experimental results showed that the proposed method was more accurate than state-of-the-art ones in terms of detecting speech emotions.