Raw Speech Signal Research Articles

Automatic speaker and age identification of children from raw speech using sincNet over ERB scale

This paper presents the newly developed non-native children’s English speech (NNCES) corpus to reveal the findings of automatic speaker and age recognition from raw speech. Convolutional neural networks (CNN), which have the ability to learn low-level speech representations, can be fed directly with raw speech signals instead of using traditional hand-crafted features. Moreover, the filters that were learned using standard CNNs appeared to be noisy because they consider all elements of each filter. In contrast, sincNet can be able to generate more meaningful filters simply by replacing the first convolutional layer by a sinc-layer in standard CNNs. The low and high cutoff frequencies of the rectangular band-pass filter are the only parameters that can be learned in sincNet, which has the potential to extract significant speech cues from the speaker, such as pitch and formants. In this work, the sincNet model is significantly changed by switching from baseline Mel scale initializations to equivalent rectangular bandwidth (ERB) initializations, which has the added benefit of allocating additional filters in the lower region of the spectrum. Additionally, it needs to be highlighted that the novel sincNet model is well suited to identify the age of the children. The investigations on both read and spontaneous speech tasks in speaker identification, gender independent & dependent age-group identification of children outperform the baseline models with varying relative improvements in terms of accuracy.

WavDepressionNet: Automatic Depression Level Prediction Via Raw Speech Signals

Physiological reports have confirmed that there are differences in speech signals between depressed and healthy individuals. Therefore, as an application in the field of affective computing, automatic depression level prediction through speech signals has received the attention of researchers, which often estimate the depression severity of individuals by the Fourier or Mel spectrograms of speech signals. However, some studies on speech emotion recognition suggest that directly modeling the raw speech signal is more helpful for extracting emotion-related information. Inspired by this fact, we develop a WavDepressionNet to model raw speech signals for the improvement of prediction accuracy. In our method, a representation block is proposed to find a set of basis vectors to construct the optimal transformation space and generate the transformation result (named Depression Feature Map, DFM) of speech signal for facilitating the perception of depression cues. We further propose an assessment block, which cannot only use the designed spatiotemporal self-calibration mechanism to calibrate the DFM and highlight the useful elements, but also aggregates the calibrated DFM across various temporal ranges with the dilated convolution. Experimental results on the AVEC 2013 and AVEC 2014 depression databases demonstrate the effectiveness of our approach over previous works.

MSER: Multimodal speech emotion recognition using cross-attention with deep fusion

In human–computer interaction (HCI) and especially speech signal processing, emotion recognition is one of the most important and challenging tasks due to multi-modality and limited data availability. Nowadays, an intelligent system is required for real-world applications to efficiently process and understand the speaker's emotional state and to enhance the analytical abilities to assist communication by a human-machine interface (HMI). Designing a reliable and robust Multimodal Speech Emotion Recognition (MSER) to efficiently recognize emotions through multi-modality such as speech and text is necessary. This paper propose a novel MSER model with a deep feature fusion technique using a multi-headed cross-attention mechanism. The proposed model utilize audio and text cues to predict the emotion label accordingly. Our proposed model process the raw speech signal and text by CNN and feeds to corresponding encoders for discriminative and semantic feature extractions. The cross-attention mechanism is applied to both features to enhance the interaction between text and audio cues by crossway to extract the most relevant information for emotion recognition. Finally, combining the region-wise weights from both encoders enables interaction among different layers and paths by the proposed deep feature fusion scheme. The authors evaluate the proposed system using the IEMOCAP and MELD datasets and conduct extensive experiments that obtain state-of-the-art (SOTA) results and show a 4.5% improved recognition rate, respectively. Our model secured a significant improvement over SOTA methods, which shows the robustness and effectiveness of the proposed MSER model.

Non-uniform Speaker Disentanglement For Depression Detection From Raw Speech Signals.

While speech-based depression detection methods that use speaker-identity features, such as speaker embeddings, are popular, they often compromise patient privacy. To address this issue, we propose a speaker disentanglement method that utilizes a non-uniform mechanism of adversarial SID loss maximization. This is achieved by varying the adversarial weight between different layers of a model during training. We find that a greater adversarial weight for the initial layers leads to performance improvement. Our approach using the ECAPA-TDNN model achieves an F1-score of 0.7349 (a 3.7% improvement over audio-only SOTA) on the DAIC-WoZ dataset, while simultaneously reducing the speaker-identification accuracy by 50%. Our findings suggest that identifying depression through speech signals can be accomplished without placing undue reliance on a speaker's identity, paving the way for privacy-preserving approaches of depression detection.

Improving Parkinson’s disease recognition through voice analysis using deep learning

Parkinson’s disease (PD) corresponds to one of the most common neurological diseases in the world, which is mainly manifested by motor, cognitive, and language disorders. The change in the patient’s voice is one of the most striking clinical signs and proves to be an element of interest to support the diagnosis and the assessment of PD. In this research paper, a new approach based on speech signal analysis is set forward to automatically detect Parkinson’s disease. The approach evaluates two learning techniques, namely Support Vector Machines (SVM) and Convolutional Neural Networks (CNN), to classify data obtained from speech tasks. Two input data,i.e., the raw speech signal’s values and the i-vector features of dimensions 100, 200 and 300 are extracted in this study. Eventually, an evaluation step is undertaken through the use of five evaluation metrics which are accuracy, precision, recall/sensitivity, specificity and f-score. The most pertinent obtained results for a test dataset composed of 28 participants are recorded as follows: an accuracy of 100%, precision of 0.99, recall/sensitivity of 0.98, specificity of 0.96 and f-score of 0.98. These findings corroborate that our approach can help in terms of PD detection.

A deep learning-based model for detecting depression in senior population.

With the attention paid to the early diagnosis of depression, this study tries to use the biological information of speech, combined with deep learning to build a rapid binary-classification model of depression in the elderly who use Mandarin and test its effectiveness. Demographic information and acoustic data of 56 Mandarin-speaking older adults with major depressive disorder (MDD), diagnosed with the Mini-International Neuropsychiatric Interview (MINI) and the fifth edition of Diagnostic and Statistical Manual of Mental Disorders (DSM-5), and 47 controls was collected. Acoustic data were recorded using different smart phones and analyzed by deep learning model which is developed and tested on independent validation set. The accuracy of the model is shown by the ROC curve. The quality of the collected speech affected the accuracy of the model. The initial sensitivity and specificity of the model were respectively 82.14% [95%CI, (70.16-90.00)] and 80.85% [95%CI, (67.64-89.58)]. This study provides a new method for rapid identification and diagnosis of depression utilizing deep learning technology. Vocal biomarkers extracted from raw speech signals have high potential for the early diagnosis of depression in older adults.

Whispered Speech Detection Using Glottal Flow-Based Features

Recent studies have reported that the performance of Automatic Speech Recognition (ASR) technologies designed for normal speech notably deteriorates when it is evaluated by whispered speech. Therefore, the detection of whispered speech is useful in order to attenuate the mismatch between training and testing situations. This paper proposes two new Glottal Flow (GF)-based features, namely, GF-based Mel-Frequency Cepstral Coefficient (GF-MFCC) as a magnitude-based feature and GF-based relative phase (GF-RP) as a phase-based feature for whispered speech detection. The main contribution of the proposed features is to extract magnitude and phase information obtained by the GF signal. In the GF-MFCC, Mel-frequency cepstral coefficient (MFCC) feature extraction is modified using the estimated GF signal derived from the iterative adaptive inverse filtering as the input to replace the raw speech signal. In a similar way, the GF-RP feature is the modification of the relative phase (RP) feature extraction by using the GF signal instead of the raw speech signal. The whispered speech production provides lower amplitude from the glottal source than normal speech production, thus, the whispered speech via Discrete Fourier Transformation (DFT) provides the lower magnitude and phase information, which make it different from a normal speech. Therefore, it is hypothesized that two types of our proposed features are useful for whispered speech detection. In addition, using the individual GF-MFCC/GF-RP feature, the feature-level and score-level combination are also proposed to further improve the detection performance. The performance of the proposed features and combinations in this study is investigated using the CHAIN corpus. The proposed GF-MFCC outperforms MFCC, while GF-RP has a higher performance than the RP. Further improved results are obtained via the feature-level combination of MFCC and GF-MFCC (MFCC&GF-MFCC)/RP and GF-RP(RP&GF-RP) compared with using either one alone. In addition, the combined score of MFCC&GF-MFCC and RP&GF-RP gives the best frame-level accuracy of 95.01% and the utterance-level accuracy of 100%.

CyTex: Transforming speech to textured images for speech emotion recognition

Speech emotion recognition is an important aspect of emotional state recognition in human–machine interaction. Approaches using speech-to-image transforms have become popular in recent years because they can utilise deep neural network models that have proven to be successful in the image processing domain. In this paper, we propose a new speech-to-image transform, CyTex, that maps the raw speech signal directly to a textured image by using calculations based on the fundamental frequency of each speech frame. The textured RGB images resulting from the CyTex transform can then be classified using standard deep neural network models for the recognition of different classes of emotion. Using this approach, we can report an improvement of classification accuracies over the previous state-of-the-art results by 0.81% for the Emo-DB database, and also by 0.5% for the IEMOCAP database.

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

The success of deep learning in speech emotion recognition has led to its application in resource-constrained devices. It has been applied in human-to-machine interaction applications like social living assistance, authentication, health monitoring and alertness systems. In order to ensure a good user experience, robust, accurate and computationally efficient deep learning models are necessary. Recurrent neural networks (RNN) like long short-term memory (LSTM), gated recurrent units (GRU) and their variants that operate sequentially are often used to learn time series sequences of the signal, analyze long-term dependencies and the contexts of the utterances in the speech signal. However, due to their sequential operation, they encounter problems in convergence and sluggish training that uses a lot of memory resources and encounters the vanishing gradient problem. In addition, they do not consider spatial cues that may exist in the speech signal. Therefore, we propose an attention-based multi-learning model (ABMD) that uses residual dilated causal convolution (RDCC) blocks and dilated convolution (DC) layers with multi-head attention. The proposed ABMD model achieves comparable performance while taking global contextualized long-term dependencies between features in a parallel manner using a large receptive field with less increase in the number of parameters compared to the number of layers and considers spatial cues among the speech features. Spectral and voice quality features extracted from the raw speech signals are used as inputs. The proposed ABMD model obtained a recognition accuracy and F1 score of 93.75% and 92.50% on the SAVEE datasets, 85.89% and 85.34% on the RAVDESS datasets and 95.93% and 95.83% on the EMODB datasets. The model’s robustness in terms of the confusion ratio of the individual discrete emotions especially happiness which is often confused with emotions that belong to the same dimensional plane with it also improved when validated on the same datasets.

End-to-End Pathological Speech Detection Using Wavelet Scattering Network

In recent years, developing robust systems for automatic detection of pathological speech has attracted increasing interest among researchers and clinicians. This study proposes an end-to-end approach based on wavelet scattering network (WSN) for detection of pathological speech. In the proposed approach, the WSN (which involves no learning) extracts suitable information from the input raw speech signal and this information is then passed through a multi-layer perceptron (MLP) in order to classify the speech signal as either healthy or pathological. The results show that the proposed approach outperformed a convolutional neural network (CNN) based end-to-end system in distinguishing pathological speech from healthy speech. Furthermore, the proposed system achieved comparable performance with a state-of-the-art traditional system based on hand-crafted features for uncompressed speech, but gave better performance than the traditional system for compressed speech of low bit rates.

Robust speech watermarking by a jointly trained embedder and detector using a DNN

This paper utilizes deep neural networks for robust digital watermarking and authentication of speech signals. We present two adversarial neural networks, called embedder and detector. The embedder network tends to achieve imperceptible watermark embedding by minimizing the differences between the original and watermarked signals. The detector strives towards errorless watermark detection. We have proposed joint optimization of these two networks to achieve a trade-off between these requirements. Two models are trained, one with raw speech signals and another with STFT signal representations. Proposed techniques are tested on attacked audio recordings with measures such as SNR, PESQ, BER, and bps. The proposed system achieves high robustness to common watermarking attacks with BERs well below 1%. SNR values greater than 38 dB and PESQ values of 4.33 demonstrate that the difference in original and watermarked signals is negligible. In comparison with other approaches, our scheme shows good overall performance in terms of imperceptibility, robustness, and capacity.

One-dimensional convolutional neural network and hybrid deep-learning paradigm for classification of specific language impaired children using their speech

Background and objectiveScreening children for communicational disorders such as specific language impairment (SLI) is always challenging as it requires clinicians to follow a series of steps to evaluate the subjects. Artificial intelligence and computer-aided diagnosis have supported health professionals in making swift and error-free decisions about the neurodevelopmental state of children vis-à-vis language comprehension and production. Past studies have claimed that typical developing (TD) and SLI children show distinct vocal characteristics that can serve as discriminating facets between them. The objective of this study is to group children in SLI or TD categories by processing their raw speech signals using two proposed approaches: a customized convolutional neural network (CNN) model and a hybrid deep-learning framework where CNN is combined with long-short-term-memory (LSTM). MethodWe considered a publicly available speech database of SLI and typical children of Czech accents for this study. The convolution filters in both the proposed CNN and hybrid models (CNN-LSTM) estimated fuzzy-automated features from the speech utterance. We performed the experiments in five separate sessions. Data augmentations were performed in each of those sessions to enhance the training strength. ResultsOur hybrid model exhibited a perfect 100% accuracy and F-measure for almost all the session-trials compared to CNN alone which achieved an average accuracy close to 90% and F-measure ≥ 92%. The models have further illustrated their robust classification essences by securing values of reliability indexes over 90%. ConclusionThe results confirm the effectiveness of proposed approaches for the detection of SLI in children using their raw speech signals. Both the models do not require any dedicated feature extraction unit for their operations. The models may also be suitable for screening SLI and other neurodevelopmental disorders in children of different linguistic accents.

A hybrid CNN-LiGRU acoustic modeling using raw waveform sincnet for Hindi ASR

Deep Neural Network (DNN) is currently playing the most vital role in Automatic Speech Recognition (ASR). Convolution Neural Network (CNN) and Recurrent Neural Network (RNN) are the advanced versions of DNN. CNN and RNN are right to deal with spatial and temporal properties of the speech signal, respectively, and both properties have a higher impact on accuracy. In today’s scenario, many acoustic modeling techniques often switches due to the battle of CNNs and RNNs. In the last few years, CNN, with raw speech signal, shows their superiority over precomputed acoustic features. Recently, a novel first convolution layer named SincNet was proposed to produce the interpretable filters with better accuracy. In this work, we proposed a hybrid SincNet-CNN-RNN architecture with low computation cost and high accuracy. Different configurations of the hybrid model were extensively examined to achieve this goal. All experiments were performed on the Hindi speech dataset.

LIS-Net: An end-to-end light interior search network for speech command recognition

With the rapid development of deep learning techniques, speech-based communication is getting more practically to be embedded into smart devices such as Alexa echo, TV, Fridge, etc. In this work, we have developed an efficient yet accurate Speech Command Recognition (SCR), that is particularly appropriate for low-resource devices. To this aim, a novel neural network, called Light Interior Search Network (LIS-Net), is presented that works with raw speech signal. LIS-Net is structurally composed of a sequence of parameterized LIS-Blocks, each of which is a stack of LIS-Cores, exploring the feature-map inheritance to learn highly distinctive and lightweight footprint of speech patterns. The proposed network is validated on Google Speech Commands benchmark speech datasets, demonstrating a significant improvement of accuracy and processing time in comparison with other state-of-the-art techniques.

Direct Speech-to-Image Translation

Direct speech-to-image translation without text is an interesting and useful topic due to the potential applications in human-computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In this paper, we attempt to translate the speech signals into the image signals without the transcription stage. Specifically, a speech encoder is designed to represent the input speech signals as an embedding feature, and it is trained with a pretrained image encoder using teacher-student learning to obtain better generalization ability on new classes. Subsequently, a stacked generative adversarial network is used to synthesize high-quality images conditioned on the embedding feature. Experimental results on both synthesized and real data show that our proposed method is effective to translate the raw speech signals into images without the middle text representation. Ablation study gives more insights about our method.

Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting

The learning of interpretable representations from raw data presents significant challenges for time series data like speech. In this work, we propose a relevance weighting scheme that allows the interpretation of the speech representations during the forward propagation of the model itself. The relevance weighting is achieved using a sub-network approach that performs the task of feature selection. A relevance sub-network, applied on the output of first layer of a convolutional neural network model operating on raw speech signals, acts as an acoustic filterbank (FB) layer with relevance weighting. A similar relevance sub-network applied on the second convolutional layer performs modulation filterbank learning with relevance weighting. The full acoustic model consisting of relevance sub-networks, convolutional layers and feed-forward layers is trained for a speech recognition task on noisy and reverberant speech in the Aurora-4, CHiME-3 and VOiCES datasets. The proposed representation learning framework is also applied for the task of sound classification in the UrbanSound8K dataset. A detailed analysis of the relevance weights learned by the model reveals that the relevance weights capture information regarding the underlying speech/audio content. In addition, speech recognition and sound classification experiments reveal that the incorporation of relevance weighting in the neural network architecture improves the performance significantly.

Glottal Source Information for Pathological Voice Detection

Automatic methods for the detection of pathological voice from healthy speech can be considered as potential clinical tools for medical treatment. This study investigates the effectiveness of glottal source information in the detection of pathological voice by comparing the classical pipeline approach to the end-to-end approach. The traditional pipeline approach consists of a feature extractor and a separate classifier. In the former, two sets of glottal features (computed using the quasi-closed phase glottal inverse filtering method) are used together with the widely used openSMILE features. Using both the glottal and openSMILE features extracted from voice utterances and the corresponding healthy/pathology labels, support vector machine (SVM) classifiers are trained. In building end-to-end systems, both raw speech signals and raw glottal flow waveforms are used to train two deep learning architectures: (1) a combination of convolutional neural network (CNN) and multilayer perceptron (MLP), and (2) a combination of CNN and long short-term memory (LSTM) network. Experiments were carried out using three publicly available databases, including dysarthric (the UA-Speech database and the TORGO database) and dysphonic voices (the UPM database). The performance analysis of the detection system based on the traditional pipeline approach showed best results when the glottal features were combined with the baseline openSMILE features. The results of the end-to-end approach indicated higher accuracies (about 2-3 % improvement in all three databases) when glottal flow was used as the raw time-domain input (87.93 % for UA-Speech, 81.12 % for TORGO and 76.66 % for UPM) compared to using raw speech waveform (85.12 % for UA-Speech, 78.83 % for TORGO and 73.71 % for UPM). The evaluation of both approaches demonstrate that automatic detection of pathological voice from healthy speech benefits from using glottal source information.

End-to-End Speech Emotion Recognition With Gender Information

Many works have focused on speech emotion recognition algorithms. However, most rely on the proper selection of speech acoustic features. In this paper, we propose a novel emotion recognition algorithm that does not rely on any speech acoustic features and combines speaker gender information. We aim to benefit from the rich information from speech raw data, without any artificial intervention. In general, speech emotion recognition systems require manual selection of appropriate traditional acoustic features as classifier input for emotion recognition. Utilizing deep learning algorithms, and the network automatically select important information from raw speech signal for the classification layer to accomplish emotion recognition. It can prevent the omission of emotion information that cannot be direct mathematically modeled as a speech acoustic characteristic. We also add speaker gender information to the proposed algorithm to further improve recognition accuracy. The proposed algorithm combines a Residual Convolutional Neural Network (R-CNN) and a gender information block. The raw speech data is sent to these two blocks simultaneously. The R-CNN network obtains the necessary emotional information from the speech data and classifies the emotional category. The proposed algorithm is evaluated on three public databases with different language systems. Experimental results show that the proposed algorithm has 5.6%, 7.3%, and 1.5%, respectively accuracy improvements in Mandarin, English, and German compared with existing highest-accuracy algorithms. In order to verify the generalization of the proposed algorithm, we use FAU and eNTERFACE databases, in these two independent databases, the proposed algorithm can also achieve 85.8% and 71.1% accuracy, respectively.

End-to-end acoustic modeling using convolutional neural networks for HMM-based automatic speech recognition

In hidden Markov model (HMM) based automatic speech recognition (ASR) system, modeling the statistical relationship between the acoustic speech signal and the HMM states that represent linguistically motivated subword units such as phonemes is a crucial step. This is typically achieved by first extracting acoustic features from the speech signal based on prior knowledge such as, speech perception or/and speech production knowledge, and, then training a classifier such as artificial neural networks (ANN), Gaussian mixture model that estimates the emission probabilities of the HMM states. This paper investigates an end-to-end acoustic modeling approach using convolutional neural networks (CNNs), where the CNN takes as input raw speech signal and estimates the HMM states class conditional probabilities at the output. Alternately, as opposed to a divide and conquer strategy (i.e., separating feature extraction and statistical modeling steps), in the proposed acoustic modeling approach the relevant features and the classifier are jointly learned from the raw speech signal. Through ASR studies and analyses on multiple languages and multiple tasks, we show that: (a) the proposed approach yields consistently a better system with fewer parameters when compared to the conventional approach of cepstral feature extraction followed by ANN training, (b) unlike conventional method of speech processing, in the proposed approach the relevant feature representations are learned by first processing the input raw speech at the sub-segmental level ( ≈ 2 ms). Specifically, through an analysis we show that the filters in the first convolution layer automatically learn “in-parts” formant-like information present in the sub-segmental speech, and (c) the intermediate feature representations obtained by subsequent filtering of the first convolution layer output are more discriminative compared to standard cepstral features and could be transferred across languages and domains.

Learning emotion-discriminative and domain-invariant features for domain adaptation in speech emotion recognition

Conventional approaches for Speech Emotion Recognition (SER) usually assume that the feature distributions between training and test set are identical. However, this assumption does not hold in many real scenarios. Although many Domain Adaptation (DA) methods have been proposed to solve this problem, the conventional emotion discriminative information is ignored. In this paper, we propose a DA based method called Emotion-discriminative and Domain-invariant Feature Learning Method (EDFLM) for SER, in which both the domain divergence and emotion discrimination are considered to learn emotion-discriminative and domain-invariant features by using emotion label constraint and domain label constraint. Furthermore, to disentangle the emotion-related factors from the emotion-unrelated factors, we introduce an orthogonal term to encourage the input to be disentangled into two blocks: emotion-related and emotion-unrelated features. Our method can learn emotion-discriminative and domain-invariant features through a back propagation network which uses the acoustic features of INTERSPEECH 2009 Emotion Challenge as the input rather than raw speech signals. Experiments on the INTERSPEECH 2009 Emotion Challenge two-class task show that the performance of our method is superior to other state-of-the-arts methods.