Speech Waveform Research Articles

Multi-scale Information Aggregation for Spoofing Detection

Synthesis artifacts that span scales from small to large are important cues for spoofing detection. However, few spoofing detection models leverage artifacts across different scales together. In this paper, we propose a spoofing detection system built on SincNet and Deep Layer Aggregation (DLA), which leverages speech representations at different levels to distinguish synthetic speech. DLA is totally convolutional with an iterative tree-like structure. The unique topology of DLA makes possible compounding of speech features from convolution layers at different depths, and therefore the local and the global speech representations can be incorporated simultaneously. Moreover, SincNet is employed as the frontend feature extractor to circumvent manual feature extraction and selection. SincNet can learn fine-grained features directly from the input speech waveform, thus making the proposed spoofing detection system end-to-end. The proposed system outperforms the baselines when tested on ASVspoof LA and DF datasets. Notably, our single model surpasses all competing systems in ASVspoof DF competition with an equal error rate (EER) of 13.99%, which demonstrates the importance of multi-scale information aggregation for synthetic speech detection.

Emotion Classification from Speech Waveform Using Machine Learning and Deep Learning Techniques

Emotions play a key role in determining the human mental state and indirectly express an individual’s well- being. A speech emotion recognition system can extract a person’s emotions from his/her speech inputs. There are some universal emotions such as anger, disgust, fear, happiness, pleasantness, sadness and neutral. These emotions are of significance especially in a situation like the Covid pandemic, when the aged or sick are vulnerable to depression. In the current paper, we examined various classification models with finite computational strength and resources in order to determine the emotion of a person from his/her speech. Speech prosodic features like pitch, loudness, and tone of speech, and work spectral features such as Mel Frequency Capstral Coefficients (MFCCs) of the voice were used to analyze the emotions of a person. Although sequence to sequence state of the art models for speech detection that offer high levels of accuracy and precision are currently in use, the computational needs of such approaches are high and inefficient. Therefore, in this work, we emphasised analysis and comparison of different classification algorithms such as multi layer perceptron, decision tree, support vector machine, and deep neural networks such as convolutional neural network and long short term memory. Given an audio file, the emotions that were exhibited by the speaker were recognized using machine learning and deep learning techniques. A comparative study was performed to identify the most appropriate algorithms that could be used to recognize emotions. Based on the experiment results, the MLP classifier and convolutional neural network model offered better accuracy with smaller variations when compared with other models used for the study.

Anatomy of a Spectrogram

This video provides a brief explanation of how spectrograms are constructed as a collection of individual spectra from consecutive points in time across a speech waveform. The difference between “narrow-band” and “wide-band” spectrograms is also explained based on choice analysis window duration. Three-dimensional animations of waveforms, spectra, and spectrograms are used throughout the video to demonstrate spectrographic representations of sound.

Hybrid CNN-BiLSTM architecture with multiple attention mechanisms to enhance speech emotion recognition

During recent years, the concept of attention in deep learning has been increasingly used to boost the performance of Speech Emotion Recognition (SER) models. However, these models for SER exhibit shortcomings in jointly emphasizing the time-frequency and dynamic sequential variations, often under-utilizing the rich contextual emotion-related information. We propose a hybrid deep learning model for SER using Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory Networks (BiLSTM) with multiple attention mechanisms. Our model utilizes features from the speech waveform viz. Mel spectrograms and Mel Frequency Cepstral Coefficients (MFCC), along with their time derivatives as input to the CNN and BiLSTM modules, respectively. A Time–Frequency Attention (TFA) mechanism, optimally incorporated into CNN, helps to selectively focus on emotion-related energy–time–frequency variations in Mel spectrograms. Attention-based BiLSTM uses MFCC and its time derivatives to identify the positional information of emotion for addressing the dynamic sequential variations. Finally, we fuse the attention-learned features from the CNN and BiLSTM modules and feed them to a Deep Neural Network (DNN) for SER. The experiments were carried out using three different datasets: Emo-DB and IEMOCAP, which are public datasets, and Amritaemo_Arabic; a private dataset. The hybrid model exhibited superior performance on both the public and private datasets, generating an average SER accuracy of 94.62%, 67.85%, and 95.80% with Emo-DB, IEMOCAP, and Amritaemo_Arabic datasets, respectively, effectively outperforming several state-of-the-art models.

The extraction method used for English–Chinese machine translation corpus based on bilingual sentence pair coverage

Abstract To improve the effect of corpus extraction in bilingual English–Chinese machine translation (ECMT), this article combines a machine learning algorithm with the ECMT corpus extraction method based on bilingual sentence pair coverage. Aiming at the propagation characteristics of medium- and short-range speeches, and employing multi-band speech waveforms in the troposphere, the actual requirements of remote machine translation are combined with the free-space, 1546, and the dual-path models, respectively, to analyze three typical air-to-ground wave propagation models. A simulation is conducted for both predictions and comparisons. In addition, a radio wave propagation model within the line of sight in the troposphere is established. The results suggest that the improved dual-path model is consistent with the propagation characteristics of radio waves in the troposphere. Moreover, the experiments show that the proposed ECMT corpus extraction method based on bilingual sentence pair coverage can play an important role in more accurate English–Chinese translations.

Noise-Robust Automatic Speech Recognition: A Case Study for Communication Interference

An Automatic Speech Recognition (ASR) System is a software tool that converts a speech audio waveform into its corresponding text transcription. ASR systems are usually built using Artificial Intelligence techniques, particularly Machine Learning algorithms like Deep Learning, to address the multi-faceted complexity and variability of human speech. This allows these systems to learn from extensive speech datasets, adapt to several languages and accents, and continuously improve their performance over time, making them each time more versatile and effective in their purpose of transcribing spoken language to text. Much in the same way, we argue that the noises commonly present in the different environments also need to be explicitly dealt with, and, when possible, modeled within specific datasets with proper training. Our motivation comes from the observation that noise removal techniques (commonly called denoising), are not always fully (and generically) efficient. For instance, noise degeneration due to communication interference, which is almost always present in radio transmissions, has peculiarities that a simple mathematical formulation cannot model. This work presents a modeling technique composed of an augmented dataset-building approach and a profile identifier that can be used to build ASRs for noisy environments that perform similarly to those used in noise-free environments. As a case study, we developed a specific ASR for the interference noise in radio transmissions with its specific dataset, while comparing our results with other state-of-the-art work. As a result, we report a Character Error Rate value of 0.3163 for the developed ASR under several different noise conditions.

Speech Emotion Recognition under Noisy Environments with SNR Down to −6 dB Using Multi-Decoder Wave-U-Net

A speech emotion recognition (SER) model for noisy environments is proposed, by using four band-pass filtered speech waveforms as the model input instead of the simplified input features such as MFCC (Mel Frequency Cepstral Coefficients). The four waveforms retain the entire information of the original noisy speech while the simplified features keep only partial information of the noisy speech. The information reduction at the model input may cause the accuracy degradation under noisy environments. A normalized loss function is used for training to maintain the high-frequency details of the original noisy speech waveform. A multi-decoder Wave-U-Net model is used to perform the denoising operation and the Wave-U-Net output waveform is applied to an emotion classifier in this work. By this, the number of parameters is reduced to 2.8 M for inference from 4.2 M used for training. The Wave-U-Net model consists of an encoder, a 2-layer LSTM, six decoders, and skip-nets; out of the six decoders, four decoders are used for denoising four band-pass filtered waveforms, one decoder is used for denoising the pitch-related waveform, and one decoder is used to generate the emotion classifier input waveform. This work gives much less accuracy degradation than other SER works under noisy environments; compared to accuracy for the clean speech waveform, the accuracy degradation is 3.8% at 0 dB SNR in this work while it is larger than 15% in the other SER works. The accuracy degradation of this work at SNRs of 0 dB, −3 dB, and −6 dB is 3.8%, 5.2%, and 7.2%, respectively.

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods

Objective. Brain–computer interfaces (BCIs) are technologies that bypass damaged or disrupted neural pathways and directly decode brain signals to perform intended actions. BCIs for speech have the potential to restore communication by decoding the intended speech directly. Many studies have demonstrated promising results using invasive micro-electrode arrays and electrocorticography. However, the use of stereo-electroencephalography (sEEG) for speech decoding has not been fully recognized. Approach. In this research, recently released sEEG data were used to decode Dutch words spoken by epileptic participants. We decoded speech waveforms from sEEG data using advanced deep-learning methods. Three methods were implemented: a linear regression method, an recurrent neural network (RNN)-based sequence-to-sequence model (RNN), and a transformer model. Main results. Our RNN and transformer models outperformed the linear regression significantly, while no significant difference was found between the two deep-learning methods. Further investigation on individual electrodes showed that the same decoding result can be obtained using only a few of the electrodes. Significance. This study demonstrated that decoding speech from sEEG signals is possible, and the location of the electrodes is critical to the decoding performance.

Vocoder Detection of Spoofing Speech Based on GAN Fingerprints and Domain Generalization

As an important part of the text-to-speech (TTS) system, vocoders convert acoustic features into speech waveforms. The difference in vocoders is key to producing different types of forged speech in the TTS system. With the rapid development of general adversarial networks (GANs), an increasing number of GAN vocoders have been proposed. Detectors often encounter vocoders of unknown types, which leads to a decline in the generalization performance of models. However, existing studies lack research on detection generalization based on GAN vocoders. To solve this problem, this study proposes vocoder detection of spoofed speech based on GAN fingerprints and domain generalization. The framework can widen the distance between real speech and forged speech in feature space, improving the detection model’s performance. Specifically, we utilize a fingerprint extractor based on an autoencoder to extract GAN fingerprints from vocoders. We then weight them to the forged speech for subsequent classification to learn the forged speech features with high differentiation. Subsequently, domain generalization is used to further improve the generalization ability of the model for unseen forgery types. We achieve domain generalization using domain-adversarial learning and asymmetric triplet loss to learn a better generalized feature space in which real speech is compact and forged speech synthesized by different vocoders is dispersed. Finally, to optimize the training process, curriculum learning is used to dynamically adjust the contributions of the samples with different difficulties in the training process. Experimental results show that the proposed method achieves the most advanced detection results among four GAN vocoders. The code is available at https://github.com/multimedia-infomation-security/GAN-Vocoder-detection .

Speech-driven head motion generation from waveforms

Head motion generation task for speech-driven virtual agent animation is commonly explored with handcrafted audio features, such as MFCCs as input features, plus additional features, such as energy and F0 in the literature. In this paper, we study the direct use of speech waveform to generate head motion. We claim that creating a task-specific feature from waveform to generate head motion leads to better performance than using standard acoustic features to generate head motion overall. At the same time, we completely abandon the handcrafted feature extraction process, leading to more effectiveness. However, the difficulty of creating a task-specific feature from waveform is their staggering quantity of irrelevant information, implicating potential cumbrance for neural network training. Thus, we apply a canonical-correlation-constrained autoencoder (CCCAE), where we are able to compress the high-dimensional waveform into a low-dimensional embedded feature, with the minimal error in reconstruction, and sustain the relevant information with the maximal cannonical correlation to head motion. We extend our previous research by including more speakers in our dataset and also adapt with a recurrent neural network, to show the feasibility of our proposed feature. Through comparisons between different acoustic features, our proposed feature, WavCCCAE, shows at least a 20% improvement in the correlation from the waveform, and outperforms the popular acoustic feature, MFCC, by at least 5% respectively for all speakers. Through the comparison in the feedforward neural network regression (FNN-regression) system, the WavCCCAE-based system shows comparable performance in objective evaluation. In long short-term memory (LSTM) experiments, LSTM-models improve the overall performance in normalised mean square error (NMSE) and CCA metrics, and adapt the WavCCCAEfeature better, which makes the proposed LSTM-regression system outperform the MFCC-based system. We also re-design the subjective evaluation, and the subjective results show the animations generated by models where WavCCCAEwas chosen to be better than the other models by the participants of MUSHRA test.

Recalling-Enhanced Recurrent Neural Network optimized with Chimp Optimization Algorithm based speech enhancement for hearing aids

Background noise often distorts the speech signals obtained in a real-world environment. This deterioration occurs in certain applications, like speech recognition, hearing aids. The aim of Speech enhancement (SE) is to suppress the unnecessary background noise in the obtained speech signal. The existing approaches for speech enhancement (SE) face more challenges like low Source-distortion ratio and memory requirements. In this manuscript, Recalling-Enhanced Recurrent Neural Network (R-ERNN) optimized with Chimp Optimization Algorithm based speech enhancement is proposed for hearing aids (R-ERNN-COA-SE-HA). Initially, the clean speech and noisy speech are amassed from MS-SNSD dataset. The input speech signals are encoded using vocoder analysis, and then the Sample RNN decode the bit stream into samples. The input speech signals are extracted using Ternary pattern and discrete wavelet transforms (TP-DWT) in the training phase. In the enhancement stage, R-ERNN forecasts the associated clean speech spectra from noisy speech spectra, then reconstructs a clean speech waveform. Chimp Optimization Algorithm (COA) is considered for optimizing the R-ERNN which enhances speech. The proposed method is implemented in MATLAB, and its efficiency is evaluated under some metrics. The R-ERNN-COA-SE-HA method provides 23.74%, 24.81%, and 19.33% higher PESQ compared with existing methods, such as RGRNN-SE-HA, PACDNN-SE-HA, ARN-SE-HA respectively.

Fast Neural Speech Waveform Generative Models With Fully-Connected Layer-Based Upsampling

Fast Neural Speech Waveform Generative Models With Fully-Connected Layer-Based Upsampling

Feasibility Study of Parkinson's Speech Disorder Evaluation With Pre-Trained Deep Learning Model for Speech-to-Text Analysis.

This study investigates the feasibility of employing a pre-trained deep learning wave-to-vec model for speech-to-text analysis in individuals with speech disorders arising from Parkinson's disease (PD). A publicly available dataset containing speech recordings including the Hoehn and Yahr (H&Y) staging, Movement Disorder Society Unified Parkinson's Disease Rating Scale (UPDRS) Part I, UPDRS Part II scores, and gender information from both healthy controls (HC) and those diagnosed with PD was utilized. Employing the Wav2Vec model, a speech-to-text analysis method was implemented on PD patient data. Tasks conducted included word letter classification, word match probability assessment, and analysis of speech waveform characteristics as provided by the model's output. For the dataset comprising 20 cases, among individuals with PD, the H&Y score averaged 2.50±0.67, the UPDRS II-part 5 score averaged 0.70±1.00, and the UPDRS III-part 18 score averaged 0.80±0.98. Additionally, the number of words derived from decoded text subsequent to speech recognition was evaluated, resulting in mean values of 299.10±16.79 and 259.80±93.39 for the HC and PD groups, respectively. Furthermore, the calculated degree of agreement for all syllables was based on the speech process. The accuracy for the reading sentences was observed to be 0.31 and 0.10, respectively. This study aimed to demonstrate the effectiveness of wave-to-vec in enhancing speech-to-text analysis for patients with speech disorders. The findings could pave the way for the development of clinical tools for improved diagnosis, evaluation, and communication support for this population.

Experimental studies for improving the performance of children's speaker verification system using short utterances

The task of developing an automatic speaker verification (ASV) system for children's speech is a formidable one due to a number of reasons. The dearth of domain-specific data is one among them. The challenge further intensifies with the introduction of short utterances of speech, a relatively unexplored domain in the case of children's ASV. Voice-based biometric systems suffers miserably when speech data, inadequate both in volume as well as in duration, is used either for enrollment or verification. To circumvent the issue arising due to data scarcity, the work in this paper extensively explores in-domain as well as various out-of-domain data augmentation techniques. A data augmentation approach is proposed that encompasses both in-domain and out-of-domain data augmentation techniques. The in-domain data augmentation approach, incorporates speed perturbation of children's speech. The out-of-domain data used are from adult speakers which are known to have acoustic attributes in stark contrast to child speakers. The acoustic characteristics of the adult speech data in this study are altered on two fronts namely speech waveform modification and feature-level modification, in order to modify the adult acoustic features and render it acoustically similar to children's speech prior to augmentation. While the speech waveform modification involves various signal processing techniques like prosody modification, formant modification and voice-conversion. The feature-level modification on the other hand involves Vocal-tract length normalization technique (VTLN) which explicitly models and compensates for the ill-effects of variations in vocal tract length by linearly warping the frequency axis of speech signals. The proposed data augmentation approach helps not only in increasing the amount of training data but also in effectively capturing the missing target attributes which helps in boosting the verification performance. A relative improvement of 48.01% in equal error rate (EER) with respect to the baseline system is a testimony of it. Furthermore, the conventionally used Mel-frequency cepstral coefficients (MFCC) are known to average out the higher-frequency components. Prior literary works have shown that a significant amount of relevant acoustic information is available in the higher-frequency region of the children's speech. Therefore, effective preservation of higher-frequency contents in children's speech is of paramount importance which must be appropriately tackled for the development of a reliable and robust children's ASV system. In this regard, frame-level concatenation of the MFCC features with the Inverse-Mel-frequency cepstral coefficient (IMFCC) features is undertaken. The feature concatenation of MFCC and IMFCC is carried out with the sole intention of effectively preserving the higher-frequency contents in the children's speech data. The low canonical correlation existing between the MFCC and the IMFCC feature vectors provides the necessary impetus to go with their feature fusion. The feature concatenation approach, when combined with proposed data augmentation, helps in further improvement of the verification performance. The experimental results testify our claims, wherein we have achieved an overall relative reduction of 50.15% for equal error rate.

Effective Zero-Shot Multi-Speaker Text-to-Speech Technique Using Information Perturbation and a Speaker Encoder

Effective Zero-Shot Multi-Speaker Text-to-Speech Technique Using Information Perturbation and a Speaker Encoder

W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

Non-parallel data voice conversion (VC) has achieved considerable breakthroughs due to self-supervised pre-trained representation (SSPR) being used in recent years. Features extracted by the pre-trained model are expected to contain more content information. However, in common VC with SSPR, there is no special implementation to remove speaker information in the content representation extraction by SSPR, which prevents further purification of the speaker information from SSPR representation. Moreover, in conventional VC, Mel-spectrogram is often selected as the reconstructed acoustic feature, which is not consistent with the input of the content encoder and results in some information lost. Motivated by the above, we proposed W2VC to settle the issues. W2VC consists of three parts: (1) We reconstruct feature from WavLM representation (WLMR) that is more consistent with the input of content encoder; (2) Connectionist temporal classification (CTC) is used to align content representation and text context from phoneme level, content encoder plus gradient reversal layer (GRL) based speaker classifier are used to remove speaker information in the content representation extraction; (3) WLMR-based HiFi-GAN is trained to convert WLMR to waveform speech. VC experimental results show that GRL can purify well the content information of the self-supervised model. The GRL purification and CTC supervision on the content encoder are complementary in improving the VC performance. Moreover, the synthesized speech using the WLMR retrained vocoder achieves better results in both subjective and objective evaluation. The proposed method is evaluated on the VCTK and CMU databases. It is shown the method achieves 8.901 in objective MCD, 4.45 in speech naturalness, and 3.62 in speaker similarity of subjective MOS score, which is superior to the baseline.

MGlif: A software tool for manual glottal inverse filtering

Despite decades of advances in automated estimation of the voice source (glottal flow) from the speech signal, manual glottal inverse filtering remains the gold standard; yet, options are limited regarding available software tools. Here, a new software application for glottal-flow estimation is introduced, with a graphical user interface designed for manual specification and tuning of inverse-filter parameters while results are updated and displayed in real-time. The user can flexibly control the signal polarity and sampling rate, the analysis frame position and duration, and the type of window and preemphasis for vocal-tract modeling. The tunable inverse-filter parameters are the frequencies and bandwidths of vocal-tract poles and zeros, and the lip-radiation coefficient. A single window displays graphical information in several panels: the speech waveform (scrollable), the selected frame, its log-power spectrum before and after preemphasis, the model spectrum including vocal-tract and lip-radiation effects, the estimated glottal-flow waveform, its derivative, and their log-power spectra. All information pertaining to an analysis may be saved at any time, ready to be reloaded for review or analysis resumption. Key data may also be exported for further use outside the application. The software will be made freely available to the research community.

Retracted: Intelligent English Translation Based on Intelligent Speech Waveform Analysis

Retracted: Intelligent English Translation Based on Intelligent Speech Waveform Analysis

Emotion Recognition From Speech and Text using Long Short-Term Memory

Everyday interactions depend on more than just rational discourse; they also depend on emotional reactions. Having this information is crucial to making any kind of practical or even rational decision, as it can help to better understand one another by sharing our responses and providing recommendations on how they may feel. Several studies have recently begun to focus on emotion detection and labeling, proposing different methods for organizing feelings and detecting emotions in speech. Determining how emotions are conveyed through speech has been given major emphasis in social interactions during the last decade. However, the real efficiency of identification needs to be improved because of the severe lack of data on the primary temporal link of the speech waveform. Currently, a new approach to speech recognition is recommended, which couples structured audio information with long-term neural networks to fully take advantage of the shift in emotional content across phases. In addition to time series characteristics, structural speech features taken from the waveforms are now in charge of maintaining the underlying connection between layers of the actual speech. There are several Long-Short-Term Memory (LSTM) based algorithms for identifying emotional focus over numerous blocks. The proposed method (i) reduced overhead by optimizing the standard forgetting gate, reducing the amount of required processing time, (ii) applied an attention mechanism to both the time and feature dimension in the LSTM's final output to get task-related information, rather than using the output from the prior iteration of the standard technique, and (iii) employed a powerful strategy to locate the spatial characteristics in the final output of the LSTM to gain information, as opposed to using the findings from the prior phase of the regular method. The proposed method achieved an overall classification accuracy of 96.81%.

An Escalated Eavesdropping Attack on Mobile Devices via Low-Resolution Vibration Signals

With the global prevalence of mobile devices, concerns about mobile devices regarding privacy breaches and data leakage are rising. Although sensor permissions are required for mobile applications to access outputs of built-in sensors, motion sensors (e.g., accelerometer and gyroscope) can be visited directly without permission requirement. Extant studies have shown that motion sensors may cause breaches of confidential information, such as passwords, digits, and voice-based commands, but whether it is possible to synthesize intelligible speech waveforms from low-resolution motion sensors has been understudied. In this paper, we present an escalated side-channel attack of built-in speakers by synthesizing intelligible speech waveforms from low-resolution vibration signals. Opposite to traditional classification problems, we formulate this task as a generative problem and introduce an end-to-end synthesis framework dubbed as <i>AccMyrinx</i> to eavesdrop on the speaker via the low-resolution vibration signals. In <i>AccMyrinx</i>, we introduce the data alignment solution to provide the pair-wise voice-vibration sequences and present wavelet-based MelGAN (WMelGAN) with multi-scale time-frequency domain discriminators to generate intelligible acoustic waveforms. We conducted intensive experiments and demonstrated the feasibility of synthesizing the intelligible acoustic signals from low-resolution solid-borne vibration signals. Compared with existing synthesis solutions, our proposed solution outperforms the baselines in both subject and object metrics with the smoothed word error rate of 42.67&#x0025; and the Mel-Cepstral distortion of 0.298. In addition, the quality of synthetic speeches could be impacted by several factors, including gender, speech rate, volume, and sampling frequency.