Modulation Spectral Features Research Articles

Advances in Speech Emotion Recognition and Analysis: A Review of Applied Machine Learning Methodologies

Abstract: The most important marketing tactics in the current environment is emotion detection based on individual’s interests, allowing extensive customization to cater to the varied needs of customers. For instance, contact centers may start playing music while a caller is on the line if they are agitated. A smart car that slows down when the driver is angry or scared is another illustration.The recognition of speech emotion has become a vital task in the field of computer science department . A very important aspect of human emotional state information is speech emotion that is the source of communication between human and computer interactions . The recognition of speech emotion is currently a very hot topic in the field of machine learning and deep learning and many systems have been build and are currently working on the recognition of these speech emotions using deep learning and neural networks algorithms. In this paper, various types of models and features like Mel-frequency Cepstral Coefficients (MFCC), Modulation spectral features (MSFs), Low-Level Descriptors(LLDs) for speech emotion recognition are categorized. Ultimately, the algorithms like recurrent neural network(RNN), support vector machine(SVM) used in different models are tabulated presenting their advantages and disadvantages along with their features and efficiency.

COVID-19 Detection via Fusion of Modulation Spectrum and Linear Prediction Speech Features

The coronavirus disease 2019 (COVID-19) pandemic has drastically impacted life around the globe. As life returns to pre-pandemic routines, COVID-19 testing has become a key component, assuring that travellers and citizens are free from the disease. Conventional tests can be expensive, time-consuming (results can take up to 48h), and require laboratory testing. Rapid antigen testing, in turn, can generate results within 15-30 minutes and can be done at home, but research shows they achieve very poor sensitivity rates. In this paper, we propose an alternative test based on speech signals recorded at home with a portable device. It has been well-documented that the virus affects many of the speech production systems (e.g., lungs, larynx, and articulators). As such, we propose the use of new modulation spectral features and linear prediction analysis to characterize these changes and design a two-stage COVID-19 prediction system by fusing the proposed features. Experiments with three COVID-19 speech datasets (CSS, DiCOVA2, and Cambridge subset) show that the two-stage feature fusion system outperforms the benchmark systems of CSS and Cambridge datasets while maintaining lower complexity compared to DL-based systems. Furthermore, the two-stage system demonstrates higher generalizability to unseen conditions in a cross-dataset testing evaluation scheme. The generalizability and interpretability of our proposed system demonstrate the potential for accessible, low-cost, at-home COVID-19 testing.

Modulation spectral features for speech emotion recognition using deep neural networks

This work explores the use of constant-Q transform based modulation spectral features (CQT-MSF) for speech emotion recognition (SER). The human perception and analysis of sound comprise of two important cognitive parts: early auditory analysis and cortex-based processing. The early auditory analysis considers spectrogram-based representation whereas cortex-based analysis includes extraction of temporal modulations from the spectrogram. This temporal modulation representation of spectrogram is called modulation spectral feature (MSF). As the constant-Q transform (CQT) provides higher resolution at emotion salient low-frequency regions of speech, we find that CQT-based spectrogram, together with its temporal modulations, provides a representation enriched with emotion-specific information. We argue that CQT-MSF when used with a 2-dimensional convolutional network can provide a time-shift invariant and deformation insensitive representation for SER. Our results show that CQT-MSF outperforms standard mel-scale based spectrogram and its modulation features on two popular SER databases, Berlin EmoDB and RAVDESS. We also show that our proposed feature outperforms the shift and deformation invariant scattering transform coefficients, hence, showing the importance of joint hand-crafted and self-learned feature extraction instead of reliance on complete hand-crafted features. Finally, we perform Grad-CAM analysis to visually inspect the contribution of constant-Q modulation features over SER.

Contribution of Common Modulation Spectral Features to Vocal-Emotion Recognition of Noise-Vocoded Speech in Noisy Reverberant Environments

In one study on vocal emotion recognition using noise-vocoded speech (NVS), the high similarities between modulation spectral features (MSFs) and the results of vocal-emotion-recognition experiments indicated that MSFs contribute to vocal emotion recognition in a clean environment (with no noise and no reverberation). Other studies also clarified that vocal emotion recognition using NVS is not affected by noisy reverberant environments (signal-to-noise ratio is greater than 10 dB and reverberation time is less than 1.0 s). However, the contribution of MSFs to vocal emotion recognition in noisy reverberant environments is still unclear. We aimed to clarify whether MSFs can be used to explain the vocal-emotion-recognition results in noisy reverberant environments. We analyzed the results of vocal-emotion-recognition experiments and used an auditory-based modulation filterbank to calculate the modulation spectrograms of NVS. We then extracted ten MSFs as higher-order statistics of modulation spectrograms. As shown from the relationship between MSFs and vocal-emotion-recognition results, except for extremely high noisy reverberant environments, there were high similarities between MSFs and the vocal emotion recognition results in noisy reverberant environments, which indicates that MSFs can be used to explain such results in noisy reverberant environments. We also found that there are two common MSFs (MSKTk (modulation spectral kurtosis) and MSTLk (modulation spectral tilt)) that contribute to vocal emotion recognition in all daily environments.

Novel feature representation using single frequency filtering and nonlinear energy operator for speech emotion recognition

In this paper, the intrinsic characteristics of speech modulations are estimated to propose the instant modulation spectral features for efficient emotion recognition. This feature representation is based on single frequency filtering (SFF) technique and higher order nonlinear energy operator. The speech signal is decomposed into frequency sub-bands using SFF, and associated nonlinear energies are estimated with higher order nonlinear energy operator. Then, the feature vector is realized using cepstral analysis. The high-resolution property of SFF technique is exploited to extract the amplitude envelope of the speech signal at a selected frequency with good time-frequency resolution. The fourth order nonlinear energy operator provides noise robustness in estimating the modulation components. The proposed feature set is tested for the emotion recognition task using the i-vector model with the probabilistic linear discriminant scoring scheme, support vector machine and random forest classifiers. The results demonstrate that the performance of this feature representation is better than the widely used spectral and prosody features, achieving detection accuracy of 85.75%, 59.88%, and 65.78% on three emotional databases, EMODB, FAU-AIBO, and IEMOCAP, respectively. Further, the proposed features are found to be robust in the presence of additive white Gaussian and vehicular noises.

Excitation Features of Speech for Emotion Recognition Using Neutral Speech as Reference

In generation of emotional speech, there are deviations in the speech production features when compared to neutral (non-emotional) speech. The objective of this study is to capture the deviations in features related to the excitation component of speech and to develop a system for automatic recognition of emotions based on these deviations. The emotions considered in this study are anger, happiness, sadness and neutral state. The study shows that there are useful features in the deviations of the excitation features, which can be exploited to develop an emotion recognition system. The excitation features used in this study are the instantaneous fundamental frequency (F_0), the strength of excitation, the energy of excitation and the ratio of the high-frequency to low-frequency band energy (beta ). A hierarchical binary decision tree approach is used to develop an emotion recognition system with neutral speech as reference. The recognition experiments showed that the excitation features are comparable or better than the existing prosody features and spectral features, such as mel-frequency cepstral coefficients, perceptual linear predictive coefficients and modulation spectral features.

Audio based Emotion Detection and Recognizing Tool Using Mel Frequency based Cepstral Coefficient

Human computer interaction (HCI) been more predominantly grown Speech Emotion appreciation in recent decade. Any interaction of mutual interest and benefits like service sectors including business and classrooms in turn could utilize this technology for ground analysis i.e., track the trace of emotional state of the receiver and so to adopt the appropriate strategy that helps the purpose of interaction. Interface with Robots upon analysis can provide details on the emotional state of the receiver of the service at the other end and paves way for application platforms such as Computer games, Customer relationship maintenance and management, e-learning, Banking and Classroom orchestration can be taken to the next level of sophisticated technology. The Work comprises of Emotional attribute extraction and Comparative Machine Classification. The comparative machine classification been done with Support vector machine (SVM). For final comparison purposes Mel-frequency Cepstrum Coefficients (MFCC) and with modulation spectral features (MSFs) are been explored and used whereas the final combinational comparisons are done between different databases for the specific features. The general experimental consequences display that the most amplitude of a voice sign varies from great sounds. For the sign it is taken on this paper the maximum amplitude is 5.5 V.

Automatic speech emotion recognition using an optimal combination of features based on EMD-TKEO

In this paper, we propose a global approach for speech emotion recognition (SER) system using empirical mode decomposition (EMD). Its use is motivated by the fact that the EMD combined with the Teager-Kaiser Energy Operator (TKEO) gives an efficient time-frequency analysis of the non-stationary signals. In this method, each signal is decomposed using EMD into oscillating components called intrinsic mode functions (IMFs). TKEO is used for estimating the time-varying amplitude envelope and instantaneous frequency of a signal that is supposed to be Amplitude Modulation-Frequency Modulation (AM-FM) signal. A subset of the IMFs was selected and used to extract features from speech signal to recognize different emotions. The main contribution of our work is to extract novel features named modulation spectral (MS) features and modulation frequency features (MFF) based on AM-FM modulation model and combined them with cepstral features. It is believed that the combination of all features will improve the performance of the emotion recognition system. Furthermore, we examine the effect of feature selection on SER system performance. For classification task, Support Vecto Machine (SVM) and Recurrent Neural Networks (RNN) are used to distinguish seven basic emotions. Two databases- the Berlin corpus, and the Spanish corpus- are used for the experiments. The results evaluated on the Spanish emotional database, using RNN classifier and a combination of all features extracted from the IMFs enhances the performance of the SER system and achieving 91.16% recognition rate. For the Berlin database, the combination of all features using SVM classifier has 86.22% recognition rate.

Contribution of modulation spectral features on the perception of vocal-emotion using noise-vocoded speech

Contribution of modulation spectral features on the perception of vocal-emotion using noise-vocoded speech

Study on the relationship between modulation spectral features and the perception of vocal emotion with noise-vocoded speech

Previous studies about vocal-emotion recognition with noise-vocoded speech showed that temporal modulation cues provided by the temporal envelope play an important role in the perception of vocal emotion. To clarify the exact feature of temporal envelope that contributes to the perception of vocal emotion, a method based on the mechanism of modulation frequency analysis in the auditory system is necessary. In this study, auditory-based modulation spectral features were used to account for the perceptual data collected from vocal-emotion recognition experiments using noise-vocoded speech. At first, the modulation spectrogram of the emotional noise-vocoded speech was calculated by using an auditory-based modulation filterbank. Then, ten types of modulation spectral features were extracted from the modulation spectrograms. Finally, modulation spectral features and the perceptual data were compared to investigate the contribution of temporal envelope to the perception of vocal emotion with noise-vocoded speech. The results showed that there were high correlations between modulation spectral features and the perceptual data. Therefore, the modulation spectral features should be useful for accounting for the perceptual processing of vocal emotion with noise-vocoded speech. [Work supported by JSPS KAKENHI Grant Number JP. 17J08312, and Grant in Aid for Scientific Research Innovative Areas (No. 18H05004) from MEXT, Japan.]

Fusion of bottleneck, spectral and modulation spectral features for improved speaker verification of neutral and whispered speech

Speech based biometrics is becoming a preferred method of identity management amongst users and companies. Current state-of-the-art speaker verification (SV) systems, however, are known to be strongly dependent on the condition of the speech material provided as input, and can be affected by unexpected variability presented during testing, such as with environmental noise or changes in vocal effort. In this paper, SV using whispered speech is explored, as whispered speech is known to be a natural speaking style with reduced perceptibility but containing relevant information regarding speaker identity and gender. We propose to fuse information from spectral, modulation spectral and so-called bottleneck features computed via deep neural networks at the feature- and score-levels. Bottleneck features have been recently shown to provide robustness against train/test mismatch conditions and have yet to be tested for whispered speech. Experimental results showed that relative improvements as high as 79% and 60% could be achieved for neutral and whispered speech, respectively, relative to a baseline system trained with i-vectors extracted from mel frequency cepstral coefficients. Results from our fusion experiments, show that the proposed strategies allow to efficiently use the limited resources available and to result in whispered speech performance inline with that obtained with normal speech.

Modulation Spectral Features: In Pursuit of Invariant Representations of Music with Application to Unsupervised Source Identification

Modulation frequency analysis has been studied predominantly in research areas such as communications, filtering and coding of digital signals, and representations of neurons in the biomedical field. In turn, modulation frequency features have surfaced in the area of music data mining, also known as music information retrieval (MIR). The term ‘modulation spectral features’ has been used rather loosely, but we refer to them as temporal patterns in a signal that may be revealed via the modulation spectrum. This paper consists of a literature survey of modulation features in which we review the historical use of general modulation features along with evolving interest in exploring modulation spectral features for data mining in music signals. We also discuss challenges encountered in employing modulation spectral features with music signals and suggest new directions in this area. Lastly, we further exploit modulation spectral features by applying these motivated concepts, properties, and parameters of modulation spectra and its features to our preliminary, unsupervised source identification method for sound sources of periodic, temporal patterns.

Residual Life Prediction of Rotating Machines Using Acoustic Noise Signals

While automated condition monitoring of rotating machines often use vibration signals for defect detection, diagnosis, and residual life predictions, in this paper, the acoustic noise signal (<; 25 kHz), acquired via non-contact microphone sensors, is used to predict the remaining useful life (RUL). Modulation spectral (MS) analysis of acoustic signals has the potential to provide additional long-term information over more conventional short-term signal spectral components. However, the high dimensionality of MS features has been cited as a limitation to their applicability in this area in the literature. Therefore, in this study, a novel approach is proposed which employs an information theoretic approach to feature subset selection of modulation spectra features. This approach does not require information regarding the spectral location of defect frequencies to be known or pre-estimated and leverages information regarding the chronological order of data samples for dimensionality reduction. The results of this study show significant improvements for this proposed approach over the other commonly used spectral-based approaches for the task of predicting RUL by up to 19% relative over the standard envelope analysis approach used in the literature. A further 16% improvement was achieved by applying a more rigorous approach to labeling of acoustic samples acquired over the lifetime of the machines over a fixed length class labeling approach. A detailed misclassification analysis is provided to interpret the relative cost of system errors for the task of residual life predictions of rotating machines used in industrial applications.

Voice Pathology Detection and Discrimination Based on Modulation Spectral Features

In this paper, we explore the information provided by a joint acoustic and modulation frequency representation, referred to as modulation spectrum, for detection and discrimination of voice disorders. The initial representation is first transformed to a lower dimensional domain using higher order singular value decomposition (HOSVD). From this dimension-reduced representation a feature selection process is suggested using an information-theoretic criterion based on the mutual information between voice classes (i.e., normophonic/dysphonic) and features. To evaluate the suggested approach and representation, we conducted cross-validation experiments on a database of sustained vowel recordings from healthy and pathological voices, using support vector machines (SVMs) for classification. For voice pathology detection, the suggested approach achieved a classification accuracy of 94.1±0.28% (95% confidence interval), which is comparable to the accuracy achieved using cepstral-based features. However, for voice pathology classification the suggested approach significantly outperformed the performance of cepstral-based features.

Msf-Based Speaker Automatic Emotional Recognition In Continuous Chinese Mandarin

In the paper, modulation spectral features (MSFs) are proposed for the automatic emotional recognition for speech signal. The features are extracted from an auditory-inspired long-term spectro-temporal(ST) representation. On an experiment assessing classification of 4 emotion categories, the MSFs show promising performance in comparison with features that are based on mel-frequency cepstral coefficients and perceptual linear prediction coefficients, two commonly used short-term spectral representations. The MSFs further express a substantial improvement in recognition performance when used to augment prosodic features, which have been extensively used for speech emotion recognition. Using both types of features, an overall recognition rate of 91.55% is obtained for classifying 4 emotion categories.

Discrimination of speech from nonspeeech in broadcast news based on modulation frequency features

In audio content analysis, the discrimination of speech and non-speech is the first processing step before speaker segmentation and recognition, or speech transcription. Speech/non-speech segmentation algorithms usually consist of a frame-based scoring phase using MFCC features, combined with a smoothing phase. In this paper, a content based speech discrimination algorithm is designed to exploit long-term information inherent in modulation spectrum. In order to address the varying degrees of redundancy and discriminative power of the acoustic and modulation frequency subspaces, we first employ a generalization of SVD to tensors (Higher Order SVD) to reduce dimensions. Projection of modulation spectral features on the principal axes with the higher energy in each subspace results in a compact set of features with minimum redundancy. We further estimate the relevance of these projections to speech discrimination based on mutual information to the target class. This system is built upon a segment-based SVM classifier in order to recognize the presence of voice activity in audio signal. Detection experiments using Greek and US English broadcast news data composed of many speakers in various acoustic conditions suggest that the system provides complementary information to state-of-the-art mel-cepstral features.

Automatic speech emotion recognition using modulation spectral features

In this study, modulation spectral features (MSFs) are proposed for the automatic recognition of human affective information from speech. The features are extracted from an auditory-inspired long-term spectro-temporal representation. Obtained using an auditory filterbank and a modulation filterbank for speech analysis, the representation captures both acoustic frequency and temporal modulation frequency components, thereby conveying information that is important for human speech perception but missing from conventional short-term spectral features. On an experiment assessing classification of discrete emotion categories, the MSFs show promising performance in comparison with features that are based on mel-frequency cepstral coefficients and perceptual linear prediction coefficients, two commonly used short-term spectral representations. The MSFs further render a substantial improvement in recognition performance when used to augment prosodic features, which have been extensively used for emotion recognition. Using both types of features, an overall recognition rate of 91.6% is obtained for classifying seven emotion categories. Moreover, in an experiment assessing recognition of continuous emotions, the proposed features in combination with prosodic features attain estimation performance comparable to human evaluation.

Robust speech detection in real acoustic backgrounds with perceptually motivated features

The current study presents an analysis of the robustness of a speech detector in real background sounds. One of the most important aspects of automatic speech/nonspeech classification is robustness in the presence of strongly varying external conditions. These include variations of the signal-to-noise ratio as well as fluctuations of the background noise. These variations are systematically evaluated by choosing different mismatched conditions between training and testing of the speech/nonspeech classifiers. The detection performance of the classifier with respect to these mismatched conditions is used as a measure of robustness and generalisation. The generalisation towards un-trained SNR conditions and unknown background noises is evaluated and compared to a matched baseline condition. The classifier consists of a feature front-end, which computes amplitude modulation spectral features (AMS), and a support vector machine (SVM) back-end. The AMS features are based on Fourier decomposition over time of short-term spectrograms. Mel-frequency cepstral coefficients (MFCC) as well as relative spectral features (RASTA) based on perceptual linear prediction (PLP) serve as baseline. The results show that RASTA-filtered PLP features perform best in the matched task. In the generalisation tasks however, the AMS features emerge as more robust in most cases, while MFCC features are outperformed by both other feature types. In a second set of experiments, a hierarchical approach is analysed which employs a background classification step prior to the speech/nonspeech classifier in order to improve the robustness of the detection scores in novel backgrounds. The background sounds used are recorded in typical everyday scenarios. The hierarchy provides a benefit in overall performance if the robust AMS features are employed. The generalisation capabilities of the hierarchy towards novel backgrounds and SNRs is found to be optimal when a limited number of training backgrounds is used (compared to the inclusion of all available background data). The best backgrounds in terms of generalisation capabilities are found to be backgrounds in which some component of speech (such as unintelligible background babble) is present, which corroborates the hypothesis that the AMS features provide a decomposition of signals which is by itself very suitable for training very general speech/nonspeech detectors. This is also supported by the finding that the SVMs combined with RASTA-PLPs require nonlinear kernels to reach a similar performance as the AMS patterns with linear kernels.

Modulation Spectral Features for Robust Far-Field Speaker Identification

In this paper, auditory inspired modulation spectral features are used to improve automatic speaker identification (ASI) performance in the presence of room reverberation. The modulation spectral signal representation is obtained by first filtering the speech signal with a 23-channel gammatone filterbank. An eight-channel modulation filterbank is then applied to the temporal envelope of each gammatone filter output. Features are extracted from modulation frequency bands ranging from 3-15 H z and are shown to be robust to mismatch between training and testing conditions and to increasing reverberation levels. To demonstrate the gains obtained with the proposed features, experiments are performed with clean speech, artificially generated reverberant speech, and reverberant speech recorded in a meeting room. Simulation results show that a Gaussian mixture model based ASI system, trained on the proposed features, consistently outperforms a baseline system trained on mel-frequency cepstral coefficients. For multimicrophone ASI applications, three multichannel score combination and adaptive channel selection techniques are investigated and shown to further improve ASI performance.