TORGO Database Research Articles

Accurate synthesis of dysarthric Speech for ASR data augmentation

Dysarthria is a motor speech disorder often characterized by reduced speech intelligibility through slow, uncoordinated control of speech production muscles. Automatic Speech Recognition (ASR) systems can help dysarthric talkers communicate more effectively. However, robust dysarthria-specific ASR requires a significant amount of training speech, which is not readily available for dysarthric talkers.This paper presents a new dysarthric speech synthesis method for the purpose of ASR training data augmentation. Differences in prosodic and acoustic characteristics of dysarthric spontaneous speech at varying severity levels are important components for dysarthric speech modeling, synthesis, and augmentation. For dysarthric speech synthesis, a modified neural multi-talker TTS is implemented by adding a dysarthria severity level coefficient and a pause insertion model to synthesize dysarthric speech for varying severity levels.To evaluate the effectiveness for synthesis of training data for ASR, dysarthria-specific speech recognition was used. Results show that a DNNHMM model trained on additional synthetic dysarthric speech achieves relative Word Error Rate (WER) improvement of 12.2 % compared to the baseline, and that the addition of the severity level and pause insertion controls decrease WER by 6.5 %, showing the effectiveness of adding these parameters. Overall results on the TORGO database demonstrate that using dysarthric synthetic speech to increase the amount of dysarthric-patterned speech for training has significant impact on the dysarthric ASR systems. In addition, we have conducted a subjective evaluation to evaluate the dysarthricness and similarity of synthesized speech. Our subjective evaluation shows that the perceived dysarthricness of synthesized speech is similar to that of true dysarthric speech, especially for higher levels of dysarthria. Audio samples are available at https://mohammadelc.github.io/SpeechGroupUKY/

Pre-trained models for detection and severity level classification of dysarthria from speech

Automatic detection and severity level classification of dysarthria from speech enables non-invasive and effective diagnosis that helps clinical decisions about medication and therapy of patients. In this work, three pre-trained models (wav2vec2-BASE, wav2vec2-LARGE, and HuBERT) are studied to extract features to build automatic detection and severity level classification systems for dysarthric speech. The experiments were conducted using two publicly available databases (UA-Speech and TORGO). One machine learning-based model (support vector machine, SVM) and one deep learning-based model (convolutional neural network, CNN) was used as the classifier. In order to compare the performance of the wav2vec2-BASE, wav2vec2-LARGE, and HuBERT features, three popular acoustic feature sets, namely, mel-frequency cepstral coefficients (MFCCs), openSMILE and extended Geneva minimalistic acoustic parameter set (eGeMAPS) were considered. Experimental results revealed that the features derived from the pre-trained models outperformed the three baseline features. It was also found that the HuBERT features performed better than the wav2vec2-BASE and wav2vec2-LARGE features. In particular, when compared to the best-performing baseline feature (openSMILE), the HuBERT features showed in the detection problem absolute accuracy improvements that varied between 1.33% (the SVM classifier, the TORGO database) and 2.86% (the SVM classifier, the UA-Speech database). In the severity level classification problem, the HuBERT features showed absolute accuracy improvements that varied between 6.54% (the SVM classifier, the TORGO database) and 10.46% (the SVM classifier, the UA-Speech database) compared to the best-performing baseline feature (eGeMAPS).

Disordered speech recognition considering low resources and abnormal articulation

The success of automatic speech recognition (ASR) benefits a great number of healthy people, but not people with disorders. The speech disordered may truly need support from technology, while they actually gain little. The difficulties of disordered ASR arise from the limited availability of data and the abnormal nature of speech, e.g, unclear, unstable, and incorrect pronunciations. To realize the ASR of disordered speech, this study addresses the problems of disordered speech in two respects, low resources, and articulatory abnormality. In order to solve the problem of low resources, this study proposes staged knowledge distillation (KD), which provides different references to the student models according to their mastery of knowledge, so as to avoid feature overfitting. To tackle the articulatory abnormalities in dysarthria, we propose an intended phonological perception method (IPPM) by applying the motor theory of speech perception to ASR, in which pieces of intended phonological features are estimated and provided to ASR. And further, we solve the challenges of disordered ASR by combining the staged KD and the IPPM. TORGO database and UASEECH corpus are two commonly used datasets of dysarthria which is the main cause of speech disorders. Experiments on the two datasets validated the effectiveness of the proposed methods. Compared with the baseline, the proposed method achieves 35.14%∼38.12% relative phoneme error rate reductions (PERRs) for speakers with varying degrees of dysarthria on the TORGO database and relative 8.17%∼13.00% PERRs on the UASPEECH corpus. The experiments demonstrated that addressing disordered speech from both low resources and speech abnormality is an effective way to solve the problems, and the proposed methods significantly improved the performance of ASR for disordered speech.

WNSA-Net: An Axial-Attention-Based Network for Schizophrenia Detection Using Wideband and Narrowband Spectrograms

Schizophrenia is a severe mental disease that affects patients' thoughts, feelings, and behaviors. Speech signal has proven to be a biomarker in the early diagnosis of schizophrenia. Previous studies on schizophrenic speech detection are mainly based on manual feature extraction engineering, which requires domain knowledge for researchers and has difficulties extracting effective features. This work proposes an end-to-end architecture, called Axial-attention-based Network using Wideband and Narrowband Spectrograms (WNSA-Net), to detect schizophrenia. Specifically, we adopt both wideband and narrowband spectrograms as inputs to represent speech signals using fine time and frequency structures. Then dilated convolution blocks are employed to capture detailed and long-range information in spectrograms. Axial-attention blocks are introduced to augment the information in feature maps along the time and frequency axes. In addition, we employ a gate mechanism to fuse the output feature maps from all channels. Experimental results on the Schizophrenia dataset and its subdatasets show that schizophrenic patients have difficulties in expressing emotions. To validate the performance of our WNSA-Net, experiments are conducted on Schizophrenia dataset and open-access TORGO database, achieving 97.37% and 98.16% accuracy in detecting schizophrenia and dysarthria, respectively. The results show promise for the proposed method in the diagnosis of disordered speech.

Speech intelligibility assessment of dysarthria using Fisher vector encoding

Neuromuscular disorders can lead to dysarthria. Dysarthria is a speech disorder that mainly affects the human motor speech system. It often results in reduced speech intelligibility. Speech intelligibility is one of the parameters to assess the severity of dysarthria. The intelligibility of speech decreases as the severity increases. Automatic speech intelligibility assessment could be reliable and cost-effective compared to the conventional methods which need experienced speech pathologists. Automatic intelligibility assessment includes feature extraction and classification. Features extracted from spectral and cepstral domains are investigated for use in intelligibility assessment. Frame level features are extracted from different signal representations in spectral and cepstral domains. Fisher vector encoding and temporal encoding that uses descriptive statistics are applied to convert frame level features into utterance level. These features are fed to PLDA and ANN classifiers for intelligibility assessment. Overall, the performance of Fisher vector encoding is found to be superior compared to temporal encoding. In all features, the performance of ANN is better in assessing the intelligibility levels compared to PLDA as expected. The STFT, as well as harmonics in spectral domain and CQCC in the cepstral domain, performed well for intelligibility assessment of unseen data of TORGO database.

RS-MSConvNet: A Novel End-to-End Pathological Voice Detection Model

Recent studies have reported the success of multi-scale convolution neural network (MSConvNet) model for many classification applications due to its powerful ability of exploring multi-scale convolution block to extract multi-scale representations to make a detection. However, a new design based on MSConvNet for pathological voice detection has not been explored. In this paper, we propose RS-MSConvNet, a novel end-to-end MSConvNet model using raw speech for pathological voice detection. The main contribution of the proposed RS-MSConvNet method is to exploit the multi-scale convolution block, followed by spatial-temporal feature block, and fully connected layer as classification. In addition, to further improve accuracy performance, we propose a novel hybrid detection model by integrating the feature extraction ability of the RS-MSConvNet model and the classifier of support vector machine (SVM) method, called RS-MSConvNet-SVM model. The effectiveness of our proposed models is investigated using the TORGO database. The experimental results reveal that the RS-MSConvNet model outperforms other baseline methods in the speaker-independent task. Moreover and as compared to the RS-MSConvNet-SVM model, a further improved accuracy is obtained using the RS-MSConvNet-SVM model. These outcomes exhibit that our proposed models are useful for pathological voice detection.

Consonantal Landmarks as Predictors of Dysarthria among English-Speaking Adults with Cerebral Palsy.

The current study explored the possibility that the consonantal landmarks served as predictors of dysarthric speech produced by English-speaking adults with cerebral palsy (CP). Additionally, the relationship between the perceptual severity of dysarthric speech and the consonantal landmarks was explored. The analyses included 210 sentences from the TORGO database produced by seven English-speaking CP speakers with dysarthria and seven typically developing controls matched in age and gender. The results indicated that the clinical group produced more total landmark features than did the control group. A binominal regression analysis revealed that the improper control of laryngeal vibration and the inability to tactically control the energy in a voiced segment would lead to the higher likelihood of dysarthric speech. A multinominal regression analysis revealed that producing too many +v and −v landmark features would lead to higher perceptual severity levels among the CP speakers. Together with literature, the current study proposed that the landmark-based acoustic analysis could quantify the differences in consonantal productions between dysarthric and non-dysarthric speech and reflect the underlying speech motor deficits of the population in concern.

Significant sensors and parameters in assessment of dysarthric speech

PurposeDysarthria is a neuromotor speech disorder caused by neuromuscular disturbances that affect one or more articulators resulting in unintelligible speech. Though inter-phoneme articulatory variations are well captured by formant frequency-based acoustic features, these variations are expected to be much higher for dysarthric speakers than normal. These substantial variations can be well captured by placing sensors in appropriate articulatory position. This study focuses to determine a set of articulatory sensors and parameters in order to assess articulatory dysfunctions in dysarthric speech.Design/methodology/approachThe current work aims to determine significant sensors and parameters associated using motion path and correlation analyzes on the TORGO database of dysarthric speech. Among eight informative sensor channels and six parameters per channel in positional data, the sensors such as tongue middle, back and tip, lower and upper lips and parameters (y, z, φ) are found to contribute significantly toward capturing the articulatory information. Acoustic and positional data analyzes are performed to validate these identified significant sensors. Furthermore, a convolutional neural network-based classifier is developed for both phone-and word-level classification of dysarthric speech using acoustic and positional data.FindingsThe average phone error rate is observed to be lower, up to 15.54% for positional data when compared with acoustic-only data. Further, word-level classification using a combination of both acoustic and positional information is performed to study that the positional data acquired using significant sensors will boost the performance of classification even for severe dysarthric speakers.Originality/valueThe proposed work shows that the significant sensors and parameters can be used to assess dysfunctions in dysarthric speech effectively. The articulatory sensor data helps in better assessment than the acoustic data even for severe dysarthric speakers.

Histogram of States Based Assistive System for Speech Impairment Due to Neurological Disorders.

Assistive speech technology is a challenging task because of the impaired nature of dysarthric speech, such as breathy voice, strained speech, distorted vowels, and consonants. Learning compact and discriminative embeddings for dysarthric speech utterances is essential for impaired speech recognition. We propose a Histogram of States (HoS)-based approach that uses Deep Neural Network-Hidden Markov Model (DNN-HMM) to learn word lattice-based compact and discriminative embeddings. Best state sequence chosen from word lattice is used to represent dysarthric speech utterance. A discriminative model-based classifier is then used to recognize these embeddings. The performance of the proposed approach is evaluated using three datasets, namely 15 acoustically similar words, 100-common words datasets of the UA-SPEECH database, and a 50-words dataset of the TORGO database. The proposed HoS-based approach performs significantly better than the traditional Hidden Markov Model and DNN-HMM-based approaches for all three datasets. The discriminative ability and the compactness of the proposed HoS-based embeddings lead to the best accuracy of impaired speech recognition.

Glottal Source Information for Pathological Voice Detection

Automatic methods for the detection of pathological voice from healthy speech can be considered as potential clinical tools for medical treatment. This study investigates the effectiveness of glottal source information in the detection of pathological voice by comparing the classical pipeline approach to the end-to-end approach. The traditional pipeline approach consists of a feature extractor and a separate classifier. In the former, two sets of glottal features (computed using the quasi-closed phase glottal inverse filtering method) are used together with the widely used openSMILE features. Using both the glottal and openSMILE features extracted from voice utterances and the corresponding healthy/pathology labels, support vector machine (SVM) classifiers are trained. In building end-to-end systems, both raw speech signals and raw glottal flow waveforms are used to train two deep learning architectures: (1) a combination of convolutional neural network (CNN) and multilayer perceptron (MLP), and (2) a combination of CNN and long short-term memory (LSTM) network. Experiments were carried out using three publicly available databases, including dysarthric (the UA-Speech database and the TORGO database) and dysphonic voices (the UPM database). The performance analysis of the detection system based on the traditional pipeline approach showed best results when the glottal features were combined with the baseline openSMILE features. The results of the end-to-end approach indicated higher accuracies (about 2-3 % improvement in all three databases) when glottal flow was used as the raw time-domain input (87.93 % for UA-Speech, 81.12 % for TORGO and 76.66 % for UPM) compared to using raw speech waveform (85.12 % for UA-Speech, 78.83 % for TORGO and 73.71 % for UPM). The evaluation of both approaches demonstrate that automatic detection of pathological voice from healthy speech benefits from using glottal source information.

Statistical Distribution Exploration of Tongue Movement for Pathological Articulation on Word/Sentence Level

Pathological articulation exploration, especially the study of the kinematic characteristics of motor organ, is helpful to further reveal the essence of motor dysarthria. Due to the scarcity of the available pathological pronunciation database, there has little research working on the statistic distribution analysis for patients and normal controlled people. This paper applied the distribution method on TORGO database to discover the cognitive and motor rules of dysarthria patients. Single phoneme analysis is effective for locating the specific tongue muscle but ignoring cognitive ability assessment, particularly for the content understanding and the fluency degree of expression by patients. The paper focused on the word/sentence level rather than single phoneme analysis. The reaction time was designed to reveal the relationship between the brain cognition and motor neuron activation. The statistic distribution tells that the cerebral palsy or amyotrophic lateral sclerosis does affect people's reflection and make the patients hard to control the tongue muscles effectively, resulting in unstable reaction time. The articulation velocity of patients appears 5mm/s faster than normal people, at 85mm/s, perhaps due to the factors of the word/sentence data and the big proportion of extra large displacement. It illustrates that the tongue moves relatively coherently and fluently once patients active the muscles, but hard to slow down as the muscle control ability decreases. The spatial occupancy was represented by the maximum articulation movement range (MAMR). We adapted the logarithmic normal distribution to find out the significant threshold for the diagnosis of dysarthria with the MAMR exceeding 7mm along left and right direction and the number of abnormal ranges surpassing 10% of total number. Primary test of MAMR, as an articulatory feature, for speech classification was carried out and it achieves 81% accuracy. These explorations convinced us to apply them to the pathological speech recognition task for improvement in future.

On the impact of dysarthric speech on contemporary ASR cloud platforms

The spread of voice-driven devices has a positive impact for people with disabilities in smart environments, since such devices allow them to perform a series of daily activities that were difficult or impossible before. As a result, their quality of life and autonomy increase. However, the speech recognition technology employed in such devices becomes limited with people having communication disorders, like dysarthria. People with dysarthria may be unable to control their smart environments, at least with the needed proficiency; this problem may negatively affect the perceived reliability of the entire environment. By exploiting the TORGO database of speech samples pronounced by people with dysarthria, this paper compares the accuracy of the dysarthric speech recognition as achieved by three speech recognition cloud platforms, namely IBM Watson Speech-to-Text, Google Cloud Speech, and Microsoft Azure Bing Speech. Such services, indeed, are used in many virtual assistants deployed in smart environments, such as Google Home. The goal is to investigate whether such cloud platforms are usable to recognize dysarthric speech, and to understand which of them is the most suitable for people with dysarthria. Results suggest that the three platforms have comparable performance in recognizing dysarthric speech and that the accuracy of the recognition is related to the speech intelligibility of the person. Overall, the platforms are limited when the dysarthric speech intelligibility is low (80–90% of word error rate), while they improve up to reach a word error rate of 15–25% for people without abnormality in their speech intelligibility.

Improving Acoustic Models in TORGO Dysarthric Speech Database.

Assistive speech-based technologies can improve the quality of life for people affected with dysarthria, a motor speech disorder. In this paper, we explore multiple ways to improve Gaussian mixture model and deep neural network (DNN) based hidden Markov model (HMM) automatic speech recognition systems for TORGO dysarthric speech database. This work shows significant improvements over the previous attempts in building such systems in TORGO. We trained speaker-specific acoustic models by tuning various acoustic model parameters, using speaker normalized cepstral features and building complex DNN-HMM models with dropout and sequence-discrimination strategies. The DNN-HMM models for severe and severe-moderate dysarthric speakers were further improved by leveraging specific information from dysarthric speech to DNN models trained on audio files from both dysarthric and normal speech, using generalized distillation framework. To the best of our knowledge, this paper presents the best recognition accuracies for TORGO database till date.

Perceptual Information Loss due to Impaired Speech Production

Phonological classes define articulatory-free and articulatory-bound phone attributes. Deep neural network is used to estimate the probability of phonological classes from the speech signal. In theory, a unique combination of phone attributes form a phoneme identity. Probabilistic inference of phonological classes thus enables estimation of their compositional phoneme probabilities. A novel information theoretic framework is devised to quantify the information conveyed by each phone attribute, and assess the speech production quality for perception of phonemes. As a use case, we hypothesize that disruption in speech production leads to information loss in phone attributes, and thus confusion in phoneme identification. We quantify the amount of information loss due to dysarthric articulation recorded in the TORGO database. A novel information measure is formulated to evaluate the deviation from an ideal phone attribute production leading us to distinguish healthy production from pathological speech.

Analysis of Speech Disabled People with Dysarthria using TORGO Database

This paper represents a method for assessment of speech skill of dysarthria disordered patients repeatedly, which is a motor speech disorder comes from neurological injury of the motor element of the motor-speech system. This structure comprises two major features: feature representation and prediction. In the feature representation, the speech sound is changed into a phone series with an regular speech recognition technique and joint with a canonical phone sequence from an intonation dictionary using a biased finite state transducer to obtain the pronunciation mappings such as match, substitution, and deletion. Next, in the prediction process, a structured sparse linear model exists with phonological information that concurrently addresses phonologically structured sparse feature selection and transparency prediction is identified. The TORGO database of dysarthric pronunciation consists of aligned acoustical and measured 3D articulatory features from speakers with either cerebral palsy (CP) or amyotrophic lateral sclerosis (ALS), which are two of the most common causes of speech disability. The major aim of this paper is to supply a flexible and suitable environment for the dysathria speech disordered peoples by recognizing their speech to make others understand them

Automatic intelligibility classification of sentence-level pathological speech

Pathological speech usually refers to the condition of speech distortion resulting from atypicalities in voice and/or in the articulatory mechanisms owing to disease, illness or other physical or biological insult to the production system. Although automatic evaluation of speech intelligibility and quality could come in handy in these scenarios to assist experts in diagnosis and treatment design, the many sources and types of variability often make it a very challenging computational processing problem. In this work we propose novel sentence-level features to capture abnormal variation in the prosodic, voice quality and pronunciation aspects in pathological speech. In addition, we propose a post-classification posterior smoothing scheme which refines the posterior of a test sample based on the posteriors of other test samples. Finally, we perform feature-level fusions and subsystem decision fusion for arriving at a final intelligibility decision. The performances are tested on two pathological speech datasets, the NKI CCRT Speech Corpus (advanced head and neck cancer) and the TORGO database (cerebral palsy or amyotrophic lateral sclerosis), by evaluating classification accuracy without overlapping subjects’ data among training and test partitions. Results show that the feature sets of each of the voice quality subsystem, prosodic subsystem, and pronunciation subsystem, offer significant discriminating power for binary intelligibility classification. We observe that the proposed posterior smoothing in the acoustic space can further reduce classification errors. The smoothed posterior score fusion of subsystems shows the best classification performance (73.5% for unweighted, and 72.8% for weighted, average recalls of the binary classes).

Vocal Tract Representation in the Recognition of Cerebral Palsied Speech

In this study, the authors explored articulatory information as a means of improving the recognition of dysarthric speech by machine. Data were derived chiefly from the TORGO database of dysarthric articulation (Rudzicz, Namasivayam, & Wolff, 2011) in which motions of various points in the vocal tract are measured during speech. In the 1st experiment, the authors provided a baseline model indicating a relatively low performance with traditional automatic speech recognition (ASR) using only acoustic data from dysarthric individuals. In the 2nd experiment, the authors used various measures of entropy (statistical disorder) to determine whether characteristics of dysarthric articulation can reduce uncertainty in features of dysarthric acoustics. These findings led to the 3rd experiment, in which recorded dysarthric articulation was directly encoded into the speech recognition process. The authors found that 18.3% of the statistical disorder in the acoustics of speakers with dysarthria can be removed if articulatory parameters are known. Using articulatory models reduces phoneme recognition errors relatively by up to 6% for speakers with dysarthria in speaker-dependent systems. Articulatory knowledge is useful in reducing rates of error in ASR for speakers with dysarthria and in reducing statistical uncertainty of their acoustic signals. These findings may help to guide clinical decisions related to the use of ASR in the future.

Using articulatory likelihoods in the recognition of dysarthric speech

Millions of individuals have congenital or acquired neuro-motor conditions that limit control of their muscles, including those that manipulate the vocal tract. These conditions, collectively called dysarthria, result in speech that is very difficult to understand both by human listeners and by traditional automatic speech recognition (ASR), which in some cases can be rendered completely unusable. In this work we first introduce a new method for acoustic-to-articulatory inversion which estimates positions of the vocal tract given acoustics using a nonlinear Hammerstein system. This is accomplished based on the theory of task-dynamics using the TORGO database of dysarthric articulation. Our approach uses adaptive kernel canonical correlation analysis and is found to be significantly more accurate than mixture density networks, at or above the 95% level of confidence for most vocal tract variables. Next, we introduce a new method for ASR in which acoustic-based hypotheses are re-evaluated according to the likelihoods of their articulatory realizations in task-dynamics. This approach incorporates high-level, long-term aspects of speech production and is found to be significantly more accurate than hidden Markov models, dynamic Bayesian networks, and switching Kalman filters.

The TORGO database of acoustic and articulatory speech from speakers with dysarthria

This paper describes the acquisition of a new database of dysarthric speech in terms of aligned acoustics and articulatory data. This database currently includes data from seven individuals with speech impediments caused by cerebral palsy or amyotrophic lateral sclerosis and age- and gender-matched control subjects. Each of the individuals with speech impediments are given standardized assessments of speech-motor function by a speech-language pathologist. Acoustic data is obtained by one head-mounted and one directional microphone. Articulatory data is obtained by electromagnetic articulography, which allows the measurement of the tongue and other articulators during speech, and by 3D reconstruction from binocular video sequences. The stimuli are obtained from a variety of sources including the TIMIT database, lists of identified phonetic contrasts, and assessments of speech intelligibility. This paper also includes some analysis as to how dysarthric speech differs from non-dysarthric speech according to features such as length of phonemes, and pronunciation errors.