Representation Of Speech Signal Research Articles

Fine-Grained Recognition of Fidgety-Type Emotions Using Multi-Scale One-Dimensional Residual Siamese Network

Fidgety speech emotion has important research value, and many deep learning models have played a good role in feature modeling in recent years. In this paper, the problem of practical speech emotion is studied, and the improvement is made on fidgety-type emotion using a novel neural network model. First, we construct a large number of phonological features for modeling emotions. Second, the differences in fidgety speech between various groups of people were studied. Through the distribution of features, the individual features of fidgety emotion were studied. Third, we propose a fine-grained emotion classification method, which analyzes the subtle differences between emotional categories through Siamese neural networks. We propose to use multi-scale residual blocks within the network architecture, and alleviate the vanishing gradient problem. This allows the network to learn more meaningful representations of fidgety speech signal. Finally, the experimental results show that the proposed method can provide the versatility of modeling, and that fidgety emotion is well identified. It has great research value in practical applications.

Advantages and Pitfalls of Dataset Condensation: An Approach to Keyword Spotting with Time-Frequency Representations

With the exponential growth of data, the need for efficient techniques to extract relevant information from datasets becomes increasingly imperative. Reducing the training data can be useful for applications wherein storage space or computational resources are limited. In this work, we explore the concept of data condensation (DC) in the context of keyword spotting systems (KWS). Using deep learning architectures and time-frequency speech representations, we have obtained condensed speech signal representations using gradient matching with Efficient Synthetic-Data Parameterization. From a series of classification experiments, we analyze the models and condensed data performances in terms of accuracy and number of data per class. We also present results using cross-model techniques, wherein models are trained with condensed data obtained from a different architecture. Our findings demonstrate the potential of data condensation in the context of the speech domain for reducing the size of datasets while retaining their most important information and maintaining high accuracy for the model trained with the condensed dataset. We have obtained an accuracy of 80.75% with 30 condensed speech representations per class with ConvNet, representing an addition of 24.9% in absolute terms when compared to 30 random samples from the original training dataset. However, we demonstrate the limitations of this approach in the cross-model tests. We also highlight the challenges and opportunities for further improving the accuracy of condensed data obtained and trained with different neural network architectures.

“You don't sound well, you should take the day off”: Automatic detection of upper respiratory tract infections from speech using time-frequency domain deep convolutional neural network

The acoustic-prosodic qualities of a speech signal are influenced by various health-related factors, owing to their complex and intricate nature. In the speech and health domain, machine learning research is active and expanding, with a focus on devising paradigms to objectively extract and measure such effects. The field of biomedical engineering has great promise in the development of non-invasive diagnostic procedures utilizing voice as a means of assessment. The utilization of speech signals for the purpose of screening for the upper respiratory tract infections (URTI) such as common cold may offer potential advantages in terms of mitigating its transmission. In this study, we have proposed a novel time-frequency domain deep convolutional neural network for URTI detection from speech using the Chirplet transform. The time-frequency representation of speech signal is achieved using the Chirplet transform. Then, a deep convolutional neural network is used for the classification of the time-frequency representation of healthy and URTI speech signals. The effectiveness of the proposed approach is assessed through the utilization of the URTIC database. We have achieved the UAR of 68.97% and 67.34% on the develop and test set of the URTIC database, respectively.

Developing children's ASR system under low-resource conditions using end-to-end architecture

The work presented in this paper aims at enhancing the performance of end-to-end (E2E) speech recognition task for children's speech under low resource conditions. For majority of the languages, there is hardly any speech data from child speakers. Furthermore, even the available children's speech corpora are limited in terms of the number of hours of data. On the other hand, large amounts of adults' speech data are freely available for research as well as commercial purposes. As a consequence, developing an effective E2E automatic speech recognition (ASR) system for children becomes a very challenging task. One may develop an ASR system using adults' speech and then use it to transcribe children's data, but this leads to very poor recognition rates due to the stark differences in the acoustic attributes of adults' and children's speech. In order to overcome these hurdles and to develop a robust children's ASR system employing E2E architecture, we have resorted to several out-of-domain and in-domain data augmentation techniques. For out-of-domain data augmentation, we have explicitly modified adults' speech to render it acoustically similar to that of children's speech before pooling into training. On the other hand, in the case of in-domain data augmentation, we have slightly modified the pitch and duration of children's speech in order to create more data capturing greater diversity. Data augmentation approaches helps in mitigating the ill-effects resulting from the scarcity of data from child domain to a certain extent. This, in turn, reduces the error rates by a large margin. In addition to data augmentation, we have also studied the efficacy of Gamma-tone frequency cepstral coefficients (GFCC) and frequency domain linear prediction (FDLP) technique along with the most commonly used Mel-frequency cepstral coefficients (MFCC) for front-end speech parameterization. Both MFCC as well as GFCC capture and model the spectral envelope of speech. On the other hand, application of linear prediction on the frequency domain representation of speech signal helps to effectively capture the temporal envelope during front-end feature extraction. Employing FDLP features that model the temporal envelope provides important cues for the perception and understanding of stop bursts and, at times, complete phonemes. This motivated us to perform a comparative experimental study of the effectiveness of the three aforementioned front-end acoustic features. In our experimental explorations, the use of proposed data augmentation in combination of FDLP features has shown a relative improvement in character error rate by 67.6% over the baseline system. The combination of data augmentation with MFCC or GFCC features is observed to result in lower recognition performances.

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Abstract The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.

Deep Learning-Based End-to-End Speaker Identification Using Time–Frequency Representation of Speech Signal

Deep Learning-Based End-to-End Speaker Identification Using Time–Frequency Representation of Speech Signal

Speech feature extraction using linear Chirplet transform and its applications*

ABSTRACT Most speech processing models begin with feature extraction and then pass the feature vector to the primary processing model. The solution's performance mainly depends on the quality of the feature representation and the model architecture. Much research focuses on designing robust deep network architecture and ignoring feature representation's important role during the deep neural network era. This work aims to exploit a new approach to design a speech signal representation in the time-frequency domain via Linear Chirplet Transform (LCT). The proposed method provides a feature vector sensitive to the frequency change inside human speech with a solid mathematical foundation. This is a potential direction for many applications. The experimental results show the improvement of the feature based on LCT compared to MFCC or Fourier Transform. In both speaker gender recognition, dialect recognition, and speech recognition, LCT significantly improved compared with MFCC and other features. This result also implies that the feature based on LCT is independent of language, so it can be used in various applications.

Generative models for sequential dynamics in active inference

AbstractA central theme of theoretical neurobiology is that most of our cognitive operations require processing of discrete sequences of items. This processing in turn emerges from continuous neuronal dynamics. Notable examples are sequences of words during linguistic communication or sequences of locations during navigation. In this perspective, we address the problem of sequential brain processing from the perspective of active inference, which inherits from a Helmholtzian view of the predictive (Bayesian) brain. Underneath the active inference lies a generative model; namely, a probabilistic description of how (observable) consequences are generated by (unobservable) causes. We show that one can account for many aspects of sequential brain processing by assuming the brain entails a generative model of the sensed world that comprises central pattern generators, narratives, or well-defined sequences. We provide examples in the domains of motor control (e.g., handwriting), perception (e.g., birdsong recognition) through to planning and understanding (e.g., language). The solutions to these problems include the use of sequences of attracting points to direct complex movements—and the move from continuous representations of auditory speech signals to the discrete words that generate those signals.

Star DGT: a robust Gabor transform for speech denoising

In this paper, we address the speech denoising problem, where Gaussian and coloured additive noises are to be removed from a given speech signal. Our approach is based on a redundant, analysis-sparse representation of the original speech signal. We pick an eigenvector of the Zauner unitary matrix and—under certain assumptions on the ambient dimension—we use it as window vector to generate a spark deficient Gabor frame. The analysis operator associated with such a frame, is a (highly) redundant Gabor transform, which we use as a sparsifying transform in the denoising procedure. We conduct computational experiments on real-world speech data, using as baseline three Gabor transforms generated by state-of-the-art window vectors in time-frequency analysis and compare their performance to the proposed Gabor transform. The results show that the proposed redundant Gabor transform outperforms previous ones consistently for all types of examined signals of noise.

An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition

Precise recognition of emotion from speech signals aids in enhancing human–computer interaction (HCI). The performance of a speech emotion recognition (SER) system depends on the derived features from speech signals. However, selecting the optimal set of feature representations remains the most challenging task in SER because the effectiveness of features varies with emotions. Most studies extract hidden local speech features ignoring the global long-term contextual representations of speech signals. The existing SER system suffers from low recognition performance mainly due to the scarcity of available data and sub-optimal feature representations. Motivated by the efficient feature extraction of convolutional neural network (CNN), long short-term memory (LSTM), and gated recurrent unit (GRU), this article proposes an ensemble utilizing the combined predictive performance of three different architectures. The first architecture uses 1D CNN followed by Fully Connected Networks (FCN). In the other two architectures, LSTM-FCN and GRU-FCN layers follow the CNN layer respectively. All three individual models focus on extracting both local and long-term global contextual representations of speech signals. The ensemble uses a weighted average of the individual models. We evaluated the model's performance on five benchmark datasets: TESS, EMO-DB, RAVDESS, SAVEE, and CREMA-D. We have augmented the data by injecting additive white gaussian noise, pitch shifting, and stretching the signal level to obtain better model generalization. Five categories of features were extracted from the speech samples: mel-frequency cepstral coefficients, log mel-scaled spectrogram, zero-crossing rate, chromagram, and root mean square value from each audio file in those datasets. All four models perform exceptionally well in the SER task; notably, the ensemble model accomplishes the state-of-the-art (SOTA) weighted average accuracy of 99.46% for TESS, 95.42% for EMO-DB, 95.62% for RAVDESS, 93.22% for SAVEE, and 90.47% for CREMA-D datasets and thus significantly outperformed the SOTA models using the same datasets.

Detecting Lombard Speech Using Deep Learning Approach.

Robust Lombard speech-in-noise detecting is challenging. This study proposes a strategy to detect Lombard speech using a machine learning approach for applications such as public address systems that work in near real time. The paper starts with the background concerning the Lombard effect. Then, assumptions of the work performed for Lombard speech detection are outlined. The framework proposed combines convolutional neural networks (CNNs) and various two-dimensional (2D) speech signal representations. To reduce the computational cost and not resign from the 2D representation-based approach, a strategy for threshold-based averaging of the Lombard effect detection results is introduced. The pseudocode of the averaging process is also included. A series of experiments are performed to determine the most effective network structure and the 2D speech signal representation. Investigations are carried out on German and Polish recordings containing Lombard speech. All 2D signal speech representations are tested with and without augmentation. Augmentation means using the alpha channel to store additional data: gender of the speaker, F0 frequency, and first two MFCCs. The experimental results show that Lombard and neutral speech recordings can clearly be discerned, which is done with high detection accuracy. It is also demonstrated that the proposed speech detection process is capable of working in near real-time. These are the key contributions of this work.

A novel distortion-tolerant speech encryption scheme for secure voice communication

This paper details a novel distortion-tolerant speech encryption scheme for securing voice communications over digital voice-specific channels such as 3G/4G or VoIP voice calls. The scheme relies on the usual “loudness, pitch and timbre” parametric representation of speech signals but seen as points on various perceptually-continuous manifolds linked by quasi-isometric continuous mappings. The ciphering of the vocal parameters of the initial speech is done by provably-secure, pseudo-random continuous transforms on these manifolds and hence ensures resilience against moderate errors. The enciphered parameters are then re-encoded into a pseudo-speech signal and transmitted over lossy digital voice channels with compression. Upon reception of the distorted pseudo-speech signal, the legitimate receiver retrieves corrupted versions of the initial vocal parameters and reconstructs an intelligible speech signal, perceptually close to the original, using a machine-learning vocoder derived from the LPCNet.The important parts of our scheme are extensively tested through simulations and experiments with encrypted pseudo-speech signals between different mobile phones. Furthermore, we detail a thorough speech quality subjective assessment with a group of 44 participants that demonstrates how the proposed scheme maintains intelligibility with tolerable progressive quality degradation linked to channel distortion. Finally, a preliminary computational analysis is done to justify the feasibility of real-time implementation of our scheme on portable devices.

A Novel RBFNN-CNN Model for Speaker Identification in Stressful Talking Environments

Speaker identification systems perform almost ideally in neutral talking environments. However, these systems perform poorly in stressful talking environments. In this paper, we present an effective approach for enhancing the performance of speaker identification in stressful talking environments based on a novel radial basis function neural network-convolutional neural network (RBFNN-CNN) model. In this research, we applied our approach to two distinct speech databases: a local Arabic Emirati-accent dataset and a global English Speech Under Simulated and Actual Stress (SUSAS) corpus. To the best of our knowledge, this is the first work that addresses the use of an RBFNN-CNN model in speaker identification under stressful talking environments. Our speech identification models select the finest speech signal representation through the use of Mel-frequency cepstral coefficients (MFCCs) as a feature extraction method. A comparison among traditional classifiers such as support vector machine (SVM), multilayer perceptron (MLP), k-nearest neighbors algorithm (KNN) and deep learning models, such as convolutional neural network (CNN) and recurrent neural network (RNN), was conducted. The results of our experiments show that speaker identification performance in stressful environments based on the RBFNN-CNN model is higher than that with the classical and deep machine learning models.

Fusing traditionally extracted features with deep learned features from the speech spectrogram for anger and stress detection using convolution neural network.

Stress and anger are two negative emotions that affect individuals both mentally and physically; there is a need to tackle them as soon as possible. Automated systems are highly required to monitor mental states and to detect early signs of emotional health issues. In the present work convolutional neural network is proposed for anger and stress detection using handcrafted features and deep learned features from the spectrogram. The objective of using a combined feature set is gathering information from two different representations of speech signals to obtain more prominent features and to boost the accuracy of recognition. The proposed method of emotion assessment is more computationally efficient than similar approaches used for emotion assessment. The preliminary results obtained on experimental evaluation of the proposed approach on three datasets Toronto Emotional Speech Set (TESS), Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), and Berlin Emotional Database (EMO-DB) indicate that categorical accuracy is boosted and cross-entropy loss is reduced to a considerable extent. The proposed convolutional neural network (CNN) obtains training (T) and validation (V) categorical accuracy of T = 93.7%, V = 95.6% for TESS, T = 97.5%, V = 95.6% for EMO-DB and T = 96.7%, V = 96.7% for RAVDESS dataset.

A Method of Increasing the Accuracy of Segmentation of a Speech Signal Based on Its Fractal Characteristics

The paper considers the analysis of approaches to the segmentation of the speech signal into vocalized and not vocalized fragments. The necessity of improving the accuracy of segmentation due to a more accurate description of the process of speech signal representation is shown. A rational method for calculating the fractal dimension to increase the accuracy of the speech signal segmentation process has been determined.

КОДИРОВАНИЕ ИЗОБРАЖЕНИЙ СПЕКТРОГРАММ ДЛЯ ОБЕСПЕЧЕНИЯ ПЕРЕМЕННОЙ СКОРОСТИ ПЕРЕДАЧИ АУДИОДАННЫХ С СОХРАНЕНИЕМ КАЧЕСТВА ИХ ЗВУЧАНИЯ

In the applications of audio control and fixation in the conditions of information-technical counteraction, noise clearing, formation of digital watermarks, audio fingerprinting, protective text audio markers, etc., a compact representation of speech signals for subsequent transmission-storage is required while maximal preserving the similarity of the sound quality of restored speech with the original, elimination of accompanying interferences. Theproposed audio codec is based on the narrow-band sine Gaussian model of speech analysis/synthesis, where its representation as a superposition of harmonic components weighted by a Gaussian window applies to all types of speech frames, as well as on universal and special methods of construction and image processing of narrow-band dynamic spectrograms, in particular, by the application of compression-recovery algorithms to them, which will allow to regulate the speech stream speed within a wide range of 1.2–16Kbit/s with adaptation to changes of the audio data transmission-storage channel bandwidth, caused, in particular, by both objective factors and the actions of an intruder. This work aims to select the best parameters on the spectrogram images that reduce the overall bitrate, remove the influence of noise and interference and allow using of spectral inversion methods and algorithms to recover the speech signal with the same or better quality. The parameters are extracted from the spectrogram images obtained using of the short-time Fourier transform, using methods to extract the amplitudes, frequencies, phases and development tracks of selected local or global maxima (peaks) of the speech signal on the spectral slices. The communication channel can transmit either the parameters themselves, or the results of compression-encoding of the image to restore the image of the original spectrogram with the selection of peak parameters already on it with the subsequent synthesis of speech or for direct spectral inversion of the image into speech. It is possible to correct the reconstructed spectrogram by using a priori information about the speaker's speech from his pre-generated voice database.

Sparse representation and reproduction of speech signals in complex Fourier basis

Sparse representation and reproduction of speech signals in complex Fourier basis

A Clustering-Based Approach for Features Extraction in Spectro-Temporal Domain Using Artificial Neural Network

In this paper, a new feature extraction method is presented based on spectro-temporal representation of speech signal for phoneme classification. In the proposed method, an artificial neural network approach is used to cluster spectro-temporal domain. Self-organizing map artificial neural network (SOM) was applied to clustering of features space. Scale, rate and frequency were used as spatial information of each point and the magnitude component was used as similarity attribute in clustering algorithm. Three mechanisms were considered to select attributes in spectro-temporal features space. Spatial information of clusters, the magnitude component of samples in spectro-temporal domain and the average of the amplitude components of each cluster points were considered as secondary features. The proposed features vectors were used for phonemes classification. The results demonstrate that a significant improvement is obtained in classification rate of different sets of phonemes in comparison to previous clustering-based methods. The obtained results of new features indicate the system error is compensated in all vowels and consonants subsets in compare to weighted K-means clustering.

Speech signal processing on graphs: The graph frequency analysis and an improved graph Wiener filtering method

In the paper, we investigate a graph representation of speech signals and graph speech enhancement technology. Specifically, we first propose a new graph k-shift operator Ck to map speech signals into the graph domain and construct a novel graph Fourier basis by using its singular eigenvectors for speech graph signals (SGSs). On this basis, we propose an improved graph Wiener filtering method based on the minimum mean square error (MMSE) criterion to suppress the noise interference in noisy speech. Comparing with the traditional methods in DSP and the existed graph Wiener filtering methods by applying graph shift operators in GSP, our numerical simulation results show that the performance of the proposed method outperforms that of these methods in terms of both average SSNR and mean PESQ score. Moreover, the computational complexity of the proposed method is much lower than that of the existed graph Wiener filtering methods and a little higher than that of classical methods in DSP.

Analysis of Momentous Fragmentary Formants in Talaqi-like Neoteric Assessment of Quran Recitation using MFCC Miniature Features of Quranic Syllables

The use of technological speech recognition systems with a variety of approaches and techniques has grown rapidly in a variety of human-machine interaction applications. Further to this, a computerized assessment system to identify errors in reading the Qur'an can be developed to practice the advantages of technology that exist today. Based on Quranic syllable utterances, which contain Tajweed rules that generally consist of Makhraj (articulation process), Sifaat (letter features or pronunciation) and Harakat (pronunciation extension), this paper attempts to present the technological capabilities in realizing Quranic recitation assessment. The transformation of the digital signal of the Quranic voice with the identification of reading errors (based on the Law of Tajweed) is the main focus of this paper. This involves many stages in the process related to the representation of Quranic syllable-based Recitation Speech Signal (QRSS), feature extraction, non-phonetic transcription Quranic Recitation Acoustic Model (QRAM), and threshold classification processes. MFCC-Formants are used in a miniature state that are hybridized with three bands in representing QRSS combined vowels and consonants. A human-guided threshold classification approach is used to assess recitation based on Quranic syllables and threshold classification performance for the low, medium, and high band groups with performances of 87.27%, 86.86%and 86.33%, respectively.