Deep Speech Research Articles

Deep Speech Synthesis and Its Implications for News Verification: Lessons Learned in the RTVE-UGR Chair

This paper presents the multidisciplinary work carried out in the RTVE-UGR Chair within the IVERES project, whose main objective is the development of a tool for journalists to verify the veracity of the audios that reach the newsrooms. In the current context, voice synthesis has both beneficial and detrimental applications, with audio deepfakes being a significant concern in the world of journalism due to their ability to mislead and misinform. This is a multifaceted problem that can only be tackled adopting a multidisciplinary perspective. In this article, we describe the approach we adopted within the RTVE-UGR Chair to successfully address the challenges derived from audio deepfakes involving a team with different backgrounds and a specific methodology of iterative co-creation. As a result, we present several outcomes including the compilation and generation of audio datasets, the development and deployment of several audio fake detection models, and the development of a web audio verification tool addressed to journalists. As a conclusion, we highlight the importance of this systematic collaborative work in the fight against misinformation and the future potential of audio verification technologies in various applications.

Deep Speech Denoising with Minimal Dependence on Clean Speech Data

Deep Speech Denoising with Minimal Dependence on Clean Speech Data

Real-Time Multichannel Deep Speech Enhancement in Hearing Aids: Comparing Monaural and Binaural Processing in Complex Acoustic Scenarios

Real-Time Multichannel Deep Speech Enhancement in Hearing Aids: Comparing Monaural and Binaural Processing in Complex Acoustic Scenarios

Toward Effective Aircraft Call Sign Detection Using Fuzzy String-Matching between ASR and ADS-B Data

Recently, artificial intelligence and data science have witnessed dramatic progress and rapid growth, especially Automatic Speech Recognition (ASR) technology based on Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). Consequently, new end-to-end Recurrent Neural Network (RNN) toolkits were developed with higher speed and accuracy that can often achieve a Word Error Rate (WER) below 10%. These toolkits can nowadays be deployed, for instance, within aircraft cockpits and Air Traffic Control (ATC) systems in order to identify aircraft and display recognized voice messages related to flight data, especially for airports not equipped with radar. Hence, the performance of air traffic controllers and pilots can ultimately be improved by reducing workload and stress and enforcing safety standards. Our experiment conducted at Tangier’s International Airport ATC aimed to build an ASR model that is able to recognize aircraft call signs in a fast and accurate way. The acoustic and linguistic models were trained on the Ibn Battouta Speech Corpus (IBSC), resulting in an unprecedented speech dataset with approved transcription that includes real weather aerodrome observation data and flight information with a call sign captured by an ADS-B receiver. All of these data were synchronized with voice recordings in a structured format. We calculated the WER to evaluate the model’s accuracy and compared different methods of dataset training for model building and adaptation. Despite the high interference in the VHF radio communication channel and fast-speaking conditions that increased the WER level to 20%, our standalone and low-cost ASR system with a trained RNN model, supported by the Deep Speech toolkit, was able to achieve call sign detection rate scores up to 96% in air traffic controller messages and 90% in pilot messages while displaying related flight information from ADS-B data using the Fuzzy string-matching algorithm.

Leveraging Domain Features for Detecting Adversarial Attacks Against Deep Speech Recognition in Noise

In recent years, significant progress has been made in deep model-based automatic speech recognition (ASR), leading to its widespread deployment in the real world. At the same time, adversarial attacks against deep ASR systems are highly successful. Various methods have been proposed to defend ASR systems from these attacks. However, existing classification based methods focus on the design of deep learning models while lacking exploration of domain specific features. This work leverages filter bank-based features to better capture the characteristics of attacks for improved detection. Furthermore, the paper analyses the potentials of using speech and non-speech parts separately in detecting adversarial attacks. In the end, considering adverse environments where ASR systems may be deployed, we study the impact of acoustic noise of various types and signal-to-noise ratios. Extensive experiments show that the inverse filter bank features generally perform better in both clean and noisy environments, the detection is effective using either speech or non-speech part, and the acoustic noise can largely degrade the detection performance.

An Optimal Method for Speech Recognition Based on Neural Network

Natural language processing technologies have become more widely available in recent years, making them more useful in everyday situations. Machine learning systems that employ accessible datasets and corporate work to serve the whole spectrum of problems addressed in computational linguistics have lately yielded a number of promising breakthroughs. These methods were particularly advantageous for regional languages, as they were provided with cutting-edge language processing tools as soon as the requisite corporate information was generated. The bulk of modern people are unconcerned about the importance of reading. Reading aloud, on the other hand, is an effective technique for nourishing feelings as well as a necessary skill in the learning process. This paper proposed a novel approach for speech recognition based on neural networks. The attention mechanism is first utilized to determine the speech accuracy and fluency assessments, with the spectrum map as the feature extraction input. To increase phoneme identification accuracy, reading precision, for example, employs a new type of deep speech. It makes use of the exportchapter tool, which provides a corpus, as well as the TensorFlow framework in the experimental setting. The experimental findings reveal that the suggested model can more effectively assess spoken speech accuracy and reading fluency than the old model, and its evaluation model’s score outcomes are more accurate.

Self-optimizing multi-domain auxiliary fusion deep complex convolution recurrent network for speech enhancement

A self-optimizing multi-domain auxiliary fusion deep complex convolution recurrent network for speech enhancement (AMDCCRN) was developed to further improve the feature completeness, correlation expression, and global optimization efficiency of the deep speech enhancement model for speech signals. By constructing a multi-domain auxiliary fusion deep complex convolution recurrent network (MDCCRN), the correlation representation of speech features and features in different spatial domains was enriched. The sparrow search algorithm (PGSSA), which is based on a game theory parallel and global optimization strategy, was introduced to improve parallel search capability and to further augment the efficiency and performance of the model's hyperparameter adaptive search. At the same time, PGSSA was used for adaptive optimization of six key model parameters required for AMDCCRN construction. Finally, a speech enhancement model suitable for multi-domain auxiliary fusion with self-learning ability was constructed. Test results on two common speech corpus data sets, THCHS-30 and WSJ0, showed that the proposed method achieved a better speech enhancement effect than other existing methods and that the validity and generalization of the proposed method had been verified.

Speech Emotion Recognition Application for Education

Based on convolutional neural networks, a speech recognition application capable of analyzing human emotions is designed. This speech emotion recognition can better assist teachers to understand students' emotional status in the learning process and enable them to improve their teaching methods with the help of the system, thus achieving the goal of improving students' learning efficiency. The application is based on PAD dimension, convolutional neural network to extract deep speech emotion features, and Least squares support vector machine for emotion recognition, thus improving the recognition accuracy of this application.

A statistical feature extraction for deep speech emotion recognition in a bilingual scenario

A statistical feature extraction for deep speech emotion recognition in a bilingual scenario

Phase-aware deep speech enhancement: It's all about the frame length

Algorithmic latency in speech processing is dominated by the frame length used for Fourier analysis, which in turn limits the achievable performance of magnitude-centric approaches. As previous studies suggest the importance of phase grows with decreasing frame length, this work presents a systematic study on the contribution of phase and magnitude in modern deep neural network (DNN)-based speech enhancement at different frame lengths. Results indicate that DNNs can successfully estimate phase when using short frames, with similar or better overall performance compared to using longer frames. Thus, interestingly, modern phase-aware DNNs allow for low-latency speech enhancement at high quality.

Exploration of Spontaneous Speech Corpus Development in Urban Agriculture Instructional Videos

Video transcription can be obtained automatically based on the original language translation of the video maker's speech, but the quality of the transcription depends on the quality of the audio signal and the natural voice of the speaker. In this study, Deep Speech is used to predict letters based on acoustic recognition without understanding language rules. The Common Voice multilingual corpus helps Deep Seech to transcribe Indonesian. However, this corpus does not accommodate the special topic of urban agriculture, so an additional corpus is needed to build acoustic and language models with the urban agriculture domain. A total of 15 popular videos with closed captions and nine E-Books with the theme of Horticulture (fruit, vegetables and medicinal plants) were curated. The video data were extracted into audio and transcription according to specifications as training data, while the agricultural text data were transformed into language models, which were used to predict recognition results. The evaluation results show that the number of epochs has an effect on improving the transcription performance. The language model score used during prediction improved WER performance as it interpreted words with agricultural terms. Another finding was that the model was unable to predict short words with informal varieties and located at the end of the sentence.

Multilingual Speech to Text using Deep Learning based on MFCC Features

The proposed methodology presented in the paper deals with solving the problem of multilingual speech recognition. Current text and speech recognition and translation methods have a very low accuracy in translating sentences which contain a mixture of two or more different languages. The paper proposes a novel approach to tackling this problem and highlights some of the drawbacks of current recognition and translation methods. The proposed approach deals with recognition of audio queries which contain a mixture of words in two different languages - Kannada and English. The novelty in the approach presented, is the use of a next Word Prediction model in combination with a Deep Learning speech recognition model to accurately recognise and convert the input audio query to text. Another method proposed to solve the problem of multilingual speech recognition and translation is the use of cosine similarity between the audio features of words for fast and accurate recognition. The dataset used for training and testing the models was generated manually by the authors as there was no pre-existing audio and text dataset which contained sentences in a mixture of both Kannada and English. The DL speech recognition model in combination with the Word Prediction model gives an accuracy of 71% when tested on the in-house multilingual dataset. This method outperforms other existing translation and recognition solutions for the same test set. Multilingual translation and recognition is an important problem to tackle as there is a tendency for people to speak in a mixture of languages. By solving this problem, the barrier of language and communication can be lifted and thus can help people connect better and more comfortably with each other.

Research and DSP Implementation of Speech Enhancement Technology Based on Dynamic Mixed Features and Adaptive Mask

A deep learning speech enhancement algorithm based on dynamic hybrid feature and adaptive mask and DSP implementation is proposed in this paper, which solves the problem of feature loss and improves the performance of speech enhancement. The dynamic features incorporate the log Mel power spectrum, Mel cepstral coefficients, and Multiresolution Auditory Cepstral Coefficients (MRACC) and capture the speech transient information by deriving the derivatives to comprehensively represent the nonlinear structure of speech and reduce distortion. To make the system improve the speech quality while reducing the speech distortion as much as possible, a soft mask that can be adaptively adjusted considering the signal-to-noise ratio information is proposed, which can be automatically adjusted according to the different speech signal-to-noise ratio information to obtain the mask value under the corresponding signal-to-noise ratio conditions, and phase difference information that can improve the speech intelligibility is incorporated in it. Then, an improved deep neural network model is designed to effectively improve the speech enhancement performance. Finally, the hardware and algorithm software design of the DSP-based speech enhancement system is given. Experimental simulations are carried out for multiple voices in different noise backgrounds. The experimental results indicate that the performance indexes of the proposed method are significantly improved compared with the existing speech enhancement methods, which verifies the feasibility and superiority of the proposed method.

Removal of musical noise using deep speech prior

We propose a musical-noise-removal method using Deep speech Prior (DP). Musical noise is an artificial distortion caused by nonlinear processing applied to speech and music signals. Median filtering is one of the most widely used methods for removing musical noise from a signal. However, it also removes speech components. Deep Image Prior (DIP) is a natural image prior that DNNs generate unstructured noise reluctantly. DIP enables denoising without a pre-training by early stopping the training. A characteristic of musical noise is that it does not have a structure such as harmonics of speech. Consequently, there is the DP that DNNs generate speech components more easily than musical noise. On the basis of DP, the proposed method removes musical noise as well as DIP-based denoising. We also improved upon our method by focusing on the dependence of the training process on input values. Performance evaluations showed that the proposed method and its improved version can remove musical noise without degrading speech quality and improve subjective quality.

Unsupervised Adaptation of Deep Speech Activity Detection Models to Unseen Domains

Speech Activity Detection (SAD) aims to accurately classify audio fragments containing human speech. Current state-of-the-art systems for the SAD task are mainly based on deep learning solutions. These applications usually show a significant drop in performance when test data are different from training data due to the domain shift observed. Furthermore, machine learning algorithms require large amounts of labelled data, which may be hard to obtain in real applications. Considering both ideas, in this paper we evaluate three unsupervised domain adaptation techniques applied to the SAD task. A baseline system is trained on a combination of data from different domains and then adapted to a new unseen domain, namely, data from Apollo space missions coming from the Fearless Steps Challenge. Experimental results demonstrate that domain adaptation techniques seeking to minimise the statistical distribution shift provide the most promising results. In particular, Deep CORAL method reports a 13% relative improvement in the original evaluation metric when compared to the unadapted baseline model. Further experiments show that the cascaded application of Deep CORAL and pseudo-labelling techniques can improve even more the results, yielding a significant 24% relative improvement in the evaluation metric when compared to the baseline system.

Evaluation of the efficiency of state-of-the-art Speech Recognition engines

Speech Recognition is one of the several Artificial Intelligence applications. It helps us converting spoken words into text. It can be part of various daily use cases in order to deal with accessibility. Google Assistant and Amazon's Alexa are in the top of list of the well-known Speech Recognition tools. European companies cannot use these solutions as they should guarantee data sovereignty. Another important point is that these mentioned solutions are not customized. So that, it is not possible to deal with new accents or new vocabularies. To cope with these problems, one can either use European Automatic Speech Recognition (ASR) solutions or build his own personalized models using well-known open-source tools like Deep Speech or Kaldi. Choosing the best solution between both, Kaldi and DeepSpeech, is an important task. The criteria for judging the finest method are the Accuracy and the Inference Time. In this paper, we make theoretical and experimental study between DeepSpeech and Kaldi. Also, Vosk and LinTO, open-source solutions build in top of Kaldi, will be included in the comparison study.

Acoustic-to-Articulatory Mapping With Joint Optimization of Deep Speech Enhancement and Articulatory Inversion Models

Acoustic-to-Articulatory Mapping With Joint Optimization of Deep Speech Enhancement and Articulatory Inversion Models

Watermarking of Deep Recurrent Neural Network Using Adversarial Examples to Protect Intellectual Property

ABSTRACT In the present era, deep learning algorithms are the key elements of several state-of-the-art solutions. But developing these algorithms for production requires a huge volume of data, computational resources, and human expertise. Thus, illegal reproduction, distribution, and modification of these models can cause economic damage to developers and can lead to copyright infringement. We propose a novel watermarking algorithm for deep recurrent neural networks based on adversarial examples that can verify the ownership of the model in a black-box way. In this paper, a novel algorithm to watermark a popular pre-trained speech-to-text deep recurrent neural network model Deep Speech without affecting the accuracy of the model is demonstrated. Watermarking is done by generating a set of adversarial examples by adding noise to the input such that the DeepSpeech model predicts the given input as the target string. In the case of copyright infringement, these adversarial examples can be used to verify ownership of the model. If the alleged stolen model predicts the same target string for the adversarial examples, the ownership of the model is verified. This novel watermarking algorithm can minimize the economic damage to the owners of the deep learning models due to stealing and plagiarizing.

Machine learning of transcripts and audio recordings of spontaneous speech for diagnosis of Alzheimer’s disease

AbstractBackgroundLanguage deficits may occur in early stages of Alzheimer’s disease (AD). Analysis of individuals’ language changes has been shown to predict the diagnosis and severity of AD. This work explored machine learning methods to detect AD using voice recordings of the Cookie Theft picture description tasks.MethodAn open‐source dataset from Dementia Bank, which included manual transcripts and audio recordings of the Cookie Theft picture description task from 309 individuals with AD and 243 cognitively healthy older adults, was employed in the classification analysis. We analyzed transcripts generated using three different approaches (Figure 1), including a) automatically generated transcripts based on denoised voice recording; b) automatically generated transcripts without the denoising process; and c) manuscript transcripts. Denoising was implemented using noisereduce, a Python library to minimize background noise (CITATION). Automatic transcripts were produced using two toolkits: CMUSpinx pocketsphinx interpreter (PS) and Mozilla DeepSpeech (DS). The Small‐Bidirectional Encoder Representations from Transformers (BERT) model was implemented to extract high dimensional feature vectors to provide contextual representation of transcripts for classification analysis. The experiment was designed to compare the performance of transcript type (manual vs. automatic), recording quality (noise vs. denoise), as well as speech recognition methods (DS vs. PS). A neural network model was constructed to classify between those with AD and healthy controls. The classification analysis was conducted using 10‐fold cross validation. The classification was repeated 20 times and the averaged results were reported (see Table 1).ResultThe performance of each approach is shown as Table 1. Overall, the model that used the features extracted from DS transcripts with denoised speech recordings achieved the best performance which had 92.72% accuracy for AD diagnosis validation results. The comparison between raw and denoised speech recordings showed conflicting performances. Using features from PS transcripts with noisy speech recordings obtained better performance than the denoised one, which was different with features from DS transcripts.ConclusionThe analysis approach which combined transcription driven recognition and transfer learning indicated the potential of better performance in AD diagnosis. Further investigation is needed to understand the reason for its superior performance.

Clinical use of deep speech parameters derived from the semantic verbal fluency task

AbstractBackgroundPrevious research showed that semantic memory is a good indicator for cognitive decline in early phases of Alzheimer’s disease. Automatically derived deep speech parameters from the semantic verbal fluency (SVF) can potentially have an additional value to differentiate SCI, MCI and dementia, compared to the total fluency score. However, the added diagnostic value of (specific) deep speech parameters to the commonly clinically used SVF raw score remains unknown. In the Deepspa project, we investigated the (additional) value of automatically derived speech parameters in clinical practice. We also investigated the relation between automatically derived speech parameters of the SVF and other cognitive tasks, as well as disease severity and functioning in daily living.MethodIn the DeepSPA project, 140 participants were recruited from the memory clinic of the MUMC+ (SCI, MCI, ADD). All subjects underwent a cognitive assessment including the SVF. The SVF (animals, 60 seconds) was administered by use of the Delta application (ki elements). Disease severity and functioning in daily life were administered by the Clinical Dementia Rating Scale (CDR) and Disability Assessment for Dementia (DAD) respectively. The agreement between the automatic and clinical raw score of the SVF was assessed by the interclass correlation coefficient. The relation between the deep speech parameters, such as mean word frequency, temporal and semantic clusters, transition time between words etc., and syndrome diagnosis, DAD and CDR were investigated using stepwise regression analyses, corrected for age, education level and gender.ResultPreliminary results showed that deep speech parameters have an additional value in the classification of clinical diagnosis, and disease severity. More specifically, mean word frequency and the mean time between temporal clusters had an added value. The reliability between word count by the application and the clinician was good (ICC=.882, 95% CI = .820‐.923). Results about the relation between deep speech parameters and other cognitive performances will be available at the conference.ConclusionFirst results suggest that deep speech parameters have an additional value in the early diagnostics of cognitive impairments. More information about the value of these non‐invasive automatically derived deep speech parameters could improve diagnostic accuracy in clinical practice.