Dysarthric Speech Recognition Research Articles

The role of rhythm perception in recognition and learning of disordered speech

The musical advantage describes the idea that musicians, relative to non-musicians, are more successful at deciphering speech in challenging listening conditions. One assumption is that fine-tuned rhythm perception skills in the musical domain translate to fine-tuned processing of rhythm cues in the speech domain. But what happens when the rhythm cues afforded by the speech signal are, themselves, disordered? This study investigated whether the ability to perceive musical rhythms provides a perceptual advantage for recognition (initial intelligibility) and learning (intelligibility improvement) of dysarthric speech—a neurologically degraded acoustic signal characterized by rhythm abnormalities. Fifty young, normal hearing adults participated in two key tests including a rhythm perception test and a perception and learning test with standard pretest, training, and post-test phases. Initial intelligibility scores for each participant reflected words correct on the pretest, and intelligibility improvement scores reflected post-test words correct minus pretest words correct. The results revealed that rhythm perception scores predicted intelligibility improvement scores but not initial intelligibility scores. Findings are discussed in relation to theoretical models regarding the link between music and speech processing, and offer direction for new models that consider the perceptual consequence of rhythm abnormalities in disordered speech.

Generative Model-Driven Feature Learning for dysarthric speech recognition

Recognition of speech uttered by severe dysarthric speakers needs a robust learning technique. One of the commonly used generative model-based classifiers for speech recognition is a hidden Markov model. Generative model-based classifiers do not do well for overlapping classes and due to insufficient training data. Dysarthric speech is normally partial or incomplete that leads to improper learning of temporal dynamics. To overcome these issues, we focus on learning features for dysarthric speech recognition that involves recognizing the sequential patterns of varying length utterances. We propose a Generative Model-Driven Feature Learning based discriminative framework that maps the sequence of feature vectors to fixed dimension vector spaces induced by the generative models. The discriminative classifier is built in that vector space. The proposed HMM-based fixed dimensional vector representation provides better discrimination for dysarthric speech than the conventional HMM. We examine the performance of the proposed method to recognize the isolated utterances from the UA-Speech database. The recognition accuracy of the proposed model is better than the conventional hidden Markov model-based approach.

Preliminary Studies towards Improving the Isolated Digit Recognition Performance of Dysarthric Speech by Prosodic Analysis

Abstract The objective of the present work is to improve the digit recognition performance of speech signals affected with dysarthria. The paper presents preliminary studies performed on universal access dysarthric speech recognition (UADSR) database. The works presented in the paper are organized into three stages. Firstly, the degradation in the digit recognition performance is demonstrated by testing the dysarthric digits with the acoustic models built using the digit samples spoken by controlled speakers. Secondly, the prosodic analysis is performed on the dysarthric isolated digits that are available in the database. Finally, the prosodic parameters of the dysarthric speech is manipulated to match with the normal speech which is used to build the acoustic models. Based on the experiments conducted, the manipulation of duration parameters using the state of the art time-domain pitch synchronous overlap add (TD-PSOLA) method observed to be significantly improving the recognition rates in contrast to other prosodic parameters. The improvement in the word recognition rates are also found to be in accordance with the intelligibility of the dysarthric speakers and hence proves the significance of using customized prosodic scaling factors according to the intelligibility levels of each of the subjects.

Convolutive Bottleneck Network with Dropout for Dysarthric Speech Recognition

In this paper, we investigate the recognition of speech produced by a person with an articulation disorder resulting from athetoid cerebral palsy. The articulation of the first spoken words tends to become unstable due to strain on speech muscles, and that causes degradation of speech recognition. Therefore, we propose a robust feature extraction method using a convolutive bottleneck network (CBN) instead of the well-known MFCC. The CBN stacks multiple various types of layers, such as a convolution layer, a subsampling layer, and a bottleneck layer, forming a deep network. Applying the CBN to feature extraction for dysarthric speech, we expect that the CBN will reduce the influence of the unstable speaking style caused by the athetoid symptoms. Furthermore, we also adopt dropout in the output layer since automatically assigned labels to the dysarthric speech are usually unreliable due to ambiguous phonemes uttered by the person with speech disorders. We confirmed its effectiveness through word recognition experiments, where the CNN based feature extraction method outperformed the conventional feature extraction method.

A Multi-Views Multi-Learners Approach Towards Dysarthric Speech Recognition Using Multi-Nets Artificial Neural Networks

Automatic speech recognition (ASR) can be very helpful for speakers who suffer from dysarthria, a neurological disability that damages the control of motor speech articulators. Although a few attempts have been made to apply ASR technologies to sufferers of dysarthria, previous studies show that such ASR systems have not attained an adequate level of performance. In this study, a dysarthric multi-networks speech recognizer (DM-NSR) model is provided using a realization of multi-views multi-learners approach called multi-nets artificial neural networks, which tolerates variability of dysarthric speech. In particular, the DM-NSR model employs several ANNs (as learners) to approximate the likelihood of ASR vocabulary words and to deal with the complexity of dysarthric speech. The proposed DM-NSR approach was presented as both speaker-dependent and speaker-independent paradigms. In order to highlight the performance of the proposed model over legacy models, multi-views single-learner models of the DM-NSRs were also provided and their efficiencies were compared in detail. Moreover, a comparison among the prominent dysarthric ASR methods and the proposed one is provided. The results show that the DM-NSR recorded improved recognition rate by up to 24.67% and the error rate was reduced by up to 8.63% over the reference model.

Weighted finite-state transducer-based dysarthric speech recognition error correction using context-dependent pronunciation variation modelling

In this paper, an error correction method is proposed to improve the performance of dysarthric automatic speech recognition (ASR). To this end, context-dependent pronunciation variations are modelled by using a weighted Kullback-Leibler (KL) distance between acoustic models of ASR. Then, the context-dependent pronunciation variation model is converted into a weighted finite-state transducer (WFST) and combined with a lexicon and a language model. It is shown from ASR experiments that average word error rate (WER) of a WFST-based ASR system with the proposed error correction method is relatively reduced by 19.73%, compared to an ASR system without error correction. Moreover, it is shown that the error correction method using a weighted KL distance relatively reduces average WER by 3.81%, compared to that using a KL distance.

Evolutionary approach for integration of multiple pronunciation patterns for enhancement of dysarthric speech recognition

Dysarthria is a motor speech disorder caused by neurological injury of the motor component of the motor-speech system. Because it affects respiration, phonation, and articulation, it leads to different types of impairments in intelligibility, audibility, and efficiency of vocal communication. Speech Assistive Technology (SAT) has been developed with different approaches for dysarthric speech and in this paper we focus on the approach that is based on modeling of pronunciation patterns. We present an approach that integrates multiple pronunciation patterns for enhancement of dysarthric speech recognition. This integration is performed by weighting the responses of an Automatic Speech Recognition (ASR) system when different language model restrictions are set. The weight for each response is estimated by a Genetic Algorithm (GA) that also optimizes the structure of the implementation technique (Metamodels) which is based on discrete Hidden Markov Models (HMMs). The GA makes use of dynamic uniform mutation/crossover to further diversify the candidate sets of weights and structures to improve the performance of the Metamodels. To test the approach with a larger vocabulary than in previous works, we orthographically and phonetically labeled extended acoustic resources from the Nemours database of dysarthric speech. ASR tests on these resources with the proposed approach showed recognition accuracies over those obtained with standard Metamodels and a well used speaker adaptation technique. These results were statistically significant.

Acoustic model adaptation using in-domain background models for dysarthric speech recognition

Speech production errors characteristic of dysarthria are chiefly responsible for the low accuracy of automatic speech recognition (ASR) when used by people diagnosed with it. A person with dysarthria produces speech in a rather reduced acoustic working space, causing typical measures of speech acoustics to have values in ranges very different from those characterizing unimpaired speech. It is unlikely then that models trained on unimpaired speech will be able to adjust to this mismatch when acted on by one of the currently well-studied adaptation algorithms (which make no attempt to address this extent of mismatch in population characteristics).In this work, we propose an interpolation-based technique for obtaining a prior acoustic model from one trained on unimpaired speech, before adapting it to the dysarthric talker. The method computes a ‘background’ model of the dysarthric talker's general speech characteristics and uses it to obtain a more suitable prior model for adaptation (compared to the speaker-independent model trained on unimpaired speech). The approach is tested with a corpus of dysarthric speech acquired by our research group, on speech of sixteen talkers with varying levels of dysarthria severity (as quantified by their intelligibility). This interpolation technique is tested in conjunction with the well-known maximum a posteriori (MAP) adaptation algorithm, and yields improvements of up to 8% absolute and up to 40% relative, over the standard MAP adapted baseline.

Familiarisation conditions and the mechanisms that underlie improved recognition of dysarthric speech

This investigation evaluated the familiarisation conditions required to promote subsequent and more long-term improvements in perceptual processing of dysarthric speech and examined the cognitive-perceptual processes that may underlie the experience-evoked learning response. Sixty listeners were randomly allocated to one of three experimental groups and were familiarised under the following conditions: (1) neurologically intact speech (control), (2) dysarthric speech (passive familiarisation), and (3) dysarthric speech coupled with written information (explicit familiarisation). All listeners completed an identical phrase transcription task immediately following familiarisation, and listeners familiarised with dysarthric speech also completed a follow-up phrase transcription task 7 days later. Listener transcripts were analysed for a measure of intelligibility (percent words correct), as well as error patterns at a segmental (percent syllable resemblance) and suprasegmental (lexical boundary errors) level of perceptual processing. The study found that intelligibility scores for listeners familiarised with dysarthric speech were significantly greater than those of the control group, with the greatest and most robust gains afforded by the explicit familiarisation condition. Relative perceptual gains in detecting phonetic and prosodic aspects of the signal varied dependent upon the familiarisation conditions, suggesting that passive familiarisation may recruit a different learning mechanism to that of a more explicit familiarisation experience involving supplementary written information. It appears that decisions regarding resource allocation during subsequent processing of dysarthric speech may be informed by the information afforded by the conditions of familiarisation.

A follow-up investigation into the mechanisms that underlie improved recognition of dysarthric speech

Differences in perceptual strategies for lexical segmentation of moderate hypokinetic dysarthric speech, apparently related to the conditions of the familiarization procedure, have been previously reported [Borrie et al., Language and Cognitive Processes (2012)]. The current follow-up investigation examined whether this difference was also observed when familiarization stimuli highlighted syllabic strength contrast cues. Forty listeners completed an identical transcription task following familiarization with dysarthric phrases presented under either passive or explicit learning conditions. Lexical boundary error patterns revealed that syllabic strength cues were exploited in both familiarization conditions. Comparisons with data previously reported afford further insight into perceptual learning of dysarthric speech.

Using speech rhythm knowledge to improve dysarthric speech recognition

We introduce a new framework to improve the dysarthric speech recognition by using the rhythm knowledge. This approach builds speaker-dependent (SD) recognizers with respect to the dysarthria sever...

Vocal Tract Representation in the Recognition of Cerebral Palsied Speech

In this study, the authors explored articulatory information as a means of improving the recognition of dysarthric speech by machine. Data were derived chiefly from the TORGO database of dysarthric articulation (Rudzicz, Namasivayam, & Wolff, 2011) in which motions of various points in the vocal tract are measured during speech. In the 1st experiment, the authors provided a baseline model indicating a relatively low performance with traditional automatic speech recognition (ASR) using only acoustic data from dysarthric individuals. In the 2nd experiment, the authors used various measures of entropy (statistical disorder) to determine whether characteristics of dysarthric articulation can reduce uncertainty in features of dysarthric acoustics. These findings led to the 3rd experiment, in which recorded dysarthric articulation was directly encoded into the speech recognition process. The authors found that 18.3% of the statistical disorder in the acoustics of speakers with dysarthria can be removed if articulatory parameters are known. Using articulatory models reduces phoneme recognition errors relatively by up to 6% for speakers with dysarthria in speaker-dependent systems. Articulatory knowledge is useful in reducing rates of error in ASR for speakers with dysarthria and in reducing statistical uncertainty of their acoustic signals. These findings may help to guide clinical decisions related to the use of ASR in the future.

Using articulatory likelihoods in the recognition of dysarthric speech

Millions of individuals have congenital or acquired neuro-motor conditions that limit control of their muscles, including those that manipulate the vocal tract. These conditions, collectively called dysarthria, result in speech that is very difficult to understand both by human listeners and by traditional automatic speech recognition (ASR), which in some cases can be rendered completely unusable. In this work we first introduce a new method for acoustic-to-articulatory inversion which estimates positions of the vocal tract given acoustics using a nonlinear Hammerstein system. This is accomplished based on the theory of task-dynamics using the TORGO database of dysarthric articulation. Our approach uses adaptive kernel canonical correlation analysis and is found to be significantly more accurate than mixture density networks, at or above the 95% level of confidence for most vocal tract variables. Next, we introduce a new method for ASR in which acoustic-based hypotheses are re-evaluated according to the likelihoods of their articulatory realizations in task-dynamics. This approach incorporates high-level, long-term aspects of speech production and is found to be significantly more accurate than hidden Markov models, dynamic Bayesian networks, and switching Kalman filters.

The effect of listener age on the recognition of dysarthric speech

A growing number of studies have attempted to address the cognitive–linguistic source of intelligibility deficits in dysarthria. These have focused predominantly on the ability of young adults with normal hearing to decipher dysarthric speech. However, dysarthria is commonly associated with aging; older listeners often form primary communication partners. It is also known that older adults exhibit difficulty understanding speech that has been temporally or spectrally degraded. It follows that the spectral and temporal degradations present in dysarthric speech may pose a greater perceptual challenge for older, as opposed to younger, listeners. Twenty younger listeners (mean age = 20 yr) and 15 older listeners with good hearing for their age (mean age = 65 yr) transcribed the speech of individuals with moderate hypokinetic dysarthria. Percent words correct (intelligibility) was calculated and underlying error patterns at the suprasegmental and segmental levels of processing were examined. While the younger and older listener groups achieved similar intelligibility scores, the younger group showed greater reliance on syllabic strength cues to inform word boundary decisions. Similar levels of attention to segmental cues were observed across both groups. These results suggest that the recognition of dysarthric speech may be comparable across younger and older listeners.

The role of linguistic and indexical information in improved recognition of dysarthric speech

This investigation examined perceptual learning of dysarthric speech. Forty listeners were randomly assigned to one of two identification training tasks, aimed at highlighting either the linguistic (word identification task) or indexical (speaker identification task) properties of the neurologically degraded signal. Immediately following familiarization, all listeners completed an identical phrase transcription task. Analysis of post-training listener transcripts revealed remarkably similar intelligibility improvements for listeners trained to attend either the linguistic or the indexical properties of the signal. Perceptual learning effects were also evaluated with regard to underlying error patterns indicative of segmental and suprasegmental processing. Comparisons revealed no significant difference at either level of perceptual processing for the two training groups. The findings of this study suggest that elements within both the linguistic and indexical properties of the dysarthric signal are learnable and interact to promote improved processing of this type and severity of speech degradation. Furthermore, error pattern analysis indicates that similar cognitive–perceptual mechanisms may underlie the processing of indexical and linguistic information. Thus, the current study extends support for the development of a model of perceptual processing in which the learning of indexical properties is encoded and retained alongside linguistic properties of the signal.

Using speech rhythm knowledge to improve dysarthric speech recognition

We introduce a new framework to improve the dysarthric speech recognition by using the rhythm knowledge. This approach builds speaker-dependent (SD) recognizers with respect to the dysarthria severity level of each speaker. This severity level is determined by a hybrid classifier combining class posterior distributions and a hierarchical structure of multilayer perceptrons. To perform this classification, rhythm-based features are used as input parameters since the preliminary evidence from perceptual experiments shows that rhythm troubles may be the common characteristic of various types of dysarthria. Then, a speaker-dependent dysarthric speech recognition is performed by using Hidden Markov Models (HMMs). The Nemours database of American dysarthric speakers is used throughout experiments. Results show the relevance of rhythm metrics and the effectiveness of the proposed framework to improve the performance of dysarthric speech recognition.

Articulatory Knowledge in the Recognition of Dysarthric Speech

Disabled speech is not compatible with modern generative and acoustic-only models of speech recognition (ASR). This work considers the use of theoretical and empirical knowledge of the vocal tract for atypical speech in labeling segmented and unsegmented sequences. These combined models are compared against discriminative models such as neural networks, support vector machines, and conditional random fields. Results show significant improvements in accuracy over the baseline through the use of production knowledge. Furthermore, although the statistics of vocal tract movement do not appear to be transferable between regular and disabled speakers, transforming the space of the former given knowledge of the latter before retraining gives high accuracy. This work may be applied within components of assistive software for speakers with dysarthria.

Integration of Metamodel and Acoustic Model for Dysarthric Speech Recognition

We investigated the speech recognition of a person with articulation disorders resulting from athetoid cerebral palsy. The articulation of the first words spoken tends to be unstable due to the strain placed on the speech-related muscles, and this causes degradation of speech recognition. Therefore, we proposed a robust feature extraction method based on PCA (Principal Component Analysis) instead of MFCC, where the main stable utterance element is projected onto low-order features and fluctuation elements of speech style are projected onto high-order features. Therefore, the PCA-based filter will be able to extract stable utterance features only. The fluctuation of speaking style may invoke phone fluctuations, such as substitutions, deletions and insertions. In this paper, we discuss our effort to integrate a Metamodel and an Acoustic model approach. Metamodels have a technique for incorporating a model of a speaker’s confusion matrix into the ASR process in such a way as to increase recognition accuracy. The integration of metamodels and acoustic models enables fluctuation suppression not only in feature extraction but also in recognition. The proposed method resulted in an improvement of 9.9% (from 79.1% to 89%) in the recognition rate compared to the conventional method.

Investigation of an HMM/ANN hybrid structure in pattern recognition application using cepstral analysis of dysarthric (distorted) speech signals

Computer speech recognition of individuals with dysarthria, such as cerebral palsy patients requires a robust technique that can handle conditions of very high variability and limited training data. In this study, application of a 10 state ergodic hidden Markov model (HMM)/artificial neural network (ANN) hybrid structure for a dysarthric speech (isolated word) recognition system, intended to act as an assistive tool, was investigated. A small size vocabulary spoken by three cerebral palsy subjects was chosen. The effect of such a structure on the recognition rate of the system was investigated by comparing it with an ergodic hidden Markov model as a control tool. This was done in order to determine if this modified technique contributed to enhanced recognition of dysarthric speech. The speech was sampled at 11 kHz. Mel frequency cepstral coefficients were extracted from them using 15 ms frames and served as training input to the hybrid model setup. The subsequent results demonstrated that the hybrid model structure was quite robust in its ability to handle the large variability and non-conformity of dysarthric speech. The level of variability in input dysarthric speech patterns sometimes limits the reliability of the system. However, its application as a rehabilitation/control tool to assist dysarthric motor impaired individuals holds sufficient promise.

Experiments with fast Fourier transform, linear predictive and cepstral coefficients in dysarthric speech recognition algorithms using hidden Markov model

In this study, a hidden Markov Model was constructed and conditions were investigated that would provide improved performance for a dysarthric speech (isolated word) recognition system. The speaker dependant system was intended to act as an assistive/control tool. A small size vocabulary spoken by three cerebral palsy subjects was chosen. Fast Fourier transform, linear predictive, and Mel frequency cepstral coefficients extracted from data provided training input to several whole-word hidden Markov model configurations. The effect of model structure, number of states, and frame rates were also investigated. It was noted that a 10-state ergodic model using 15 msec frames was better than other configurations. Furthermore, it was found that a Mel cepstrum based model outperformed a fast Fourier transform and linear prediction based model. The system offers effective and robust application as a rehabilitation and/or control tool to assist dysarthric motor impaired individuals.