Speech Acoustic Models Research Articles

Evaluation of Interpretable Speech Biomarkers for Monitoring Alzheimer’s Disease and Mild Cognitive Impairment Progression

AbstractBackgroundAlzheimer’s disease (AD) is a progressive neurodegenerative disease. Like for other dementias, biomarkers may help characterize distinct aspects of the underlying pathology, predict decline, and monitor disease progression. We extracted a composite array of 24 speech‐based biomarkers using machine learning and signal processing techniques. We then determined which of the given biomarkers was predictive of MCI and AD progression from baseline by conducting a correlation analysis.MethodSpoken responses to a Cookie Theft Picture description task were recorded from 2 subjects with AD, 4 subjects with Mild Cognitive Impairment (MCI) due to AD (biomarker confirmed AD), and 4 subjects with MCI. We automatically extracted acoustic, linguistic, and cognitive biomarkers using speech recognition, acoustic, and language modeling techniques. We later computed Kendall’s tau‐b (tb) correlation between the biomarker values and Montreal Cognitive Assessment (MoCA) scores. Recordings and MoCA scores were collected at baseline, at 6 months after baseline, and at 12 months for a subgroup of the subjects. To perform the correlation analysis, we used scipy.stats.kendalltau library in Python.ResultWith respect to the acoustic biomarkers, a strong negative correlation (tb = ‐0.50) was observed between the MoCA scores and features such as pause time, pause percentage, and pause speech ratio, while a moderate positive correlation (tb = 0.2) was found for rhythm standard deviation. With respect to the linguistic biomarkers a strong positive correlation(tb = 0.50) for the features of corrected type‐token ratio, average sentence length in words, and noun count was reported, while a moderate positive correlation (tb = 0.27) was found for the features of word type count, word token count, and moving average type‐token ratio. The other acoustic, linguistic, and cognitive biomarkers showed a very weak (tb = 0.00 ‐ 0.10) or weak correlation (tb = 0.10 ‐ 0.20).ConclusionAltogether, these results suggest that speech‐based interpretable biomarkers may help clinicians to diagnose AD at earlier stages and monitor disease progression. Our preliminary data suggest that AD patients encounter more problems delivering longer and linguistically elaborated narratives as the disease progresses.

Evaluation of Interpretable Speech Biomarkers for Monitoring Alzheimer’s Disease and Mild Cognitive Impairment Progression

AbstractBackgroundAlzheimer’s disease (AD) is a progressive neurodegenerative disease. Like for other dementias, biomarkers may help characterize distinct aspects of the underlying pathology, predict decline, and monitor disease progression. We extracted a composite array of 24 speech‐based biomarkers using machine learning and signal processing techniques. We then determined which of the given biomarkers was predictive of MCI and AD progression from baseline by conducting a correlation analysis.MethodSpoken responses to a Cookie Theft Picture description task were recorded from 2 subjects with AD, 4 subjects with Mild Cognitive Impairment (MCI) due to AD (biomarker confirmed AD), and 4 subjects with MCI. We automatically extracted acoustic, linguistic, and cognitive biomarkers using speech recognition, acoustic, and language modeling techniques. We later computed Kendall’s tau‐b (τb ) correlation between the biomarker values and Montreal Cognitive Assessment (MoCA) scores. Recordings and MoCA scores were collected at baseline, at 6 months after baseline, and at 12 months for a subgroup of the subjects. To perform the correlation analysis, we used scipy.stats.kendalltau library in Python.ResultWith respect to the acoustic biomarkers, a strong negative correlation (τb= ‐0.50) was observed between the MoCA scores and features such as pause time, pause percentage, and pause speech ratio, while a moderate positive correlation (τb= 0.2) was found for rhythm standard deviation. With respect to the linguistic biomarkers a strong positive correlation(τb= 0.50) for the features of corrected type‐token ratio, average sentence length in words, and noun count was reported, while a moderate positive correlation (τb= 0.27) was found for the features of word type count, word token count, and moving average type‐token ratio. The other acoustic, linguistic, and cognitive biomarkers showed a very weak (τb= 0.00 ‐ 0.10) or weak correlation (τb= 0.10 ‐ 0.20).ConclusionAltogether, these results suggest that speech‐based interpretable biomarkers may help clinicians to diagnose AD at earlier stages and monitor disease progression. Our preliminary data suggest that AD patients encounter more problems delivering longer and linguistically elaborated narratives as the disease progresses.

Speech Emotion Recognition Enhanced Traffic Efficiency Solution for Autonomous Vehicles in a 5G-Enabled Space–Air–Ground Integrated Intelligent Transportation System

Speech emotion recognition (SER) is becoming the main human–computer interaction logic for autonomous vehicles in the next generation of intelligent transportation systems (ITSs). It can improve not only the safety of autonomous vehicles but also the personalized in-vehicle experience. However, current vehicle-mounted SER systems still suffer from two major shortcomings. One is the insufficient service capacity of the vehicle communication network, which is unable to meet the SER needs of autonomous vehicles in next-generation ITSs in terms of the data transmission rate, power consumption, and latency. Second, the accuracy of SER is poor, and it cannot provide sufficient interactivity and personalization between users and vehicles. To address these issues, we propose an SER-enhanced traffic efficiency solution for autonomous vehicles in a 5G-enabled space–air–ground integrated network (SAGIN)-based ITS. First, we convert the vehicle speech information data into spectrograms and input them into an AlexNet network model to obtain the high-level features of the vehicle speech acoustic model. At the same time, we convert the vehicle speech information data into text information and input it into the Bidirectional Encoder Representations from Transformers (BERT) model to obtain the high-level features of the corresponding text model. Finally, these two sets of high-level features are cascaded together to obtain fused features, which are sent to a softmax classifier for emotion matching and classification. Experiments show that the proposed solution can improve not only the SAGIN’s service capabilities, resulting in a large capacity, high bandwidth, ultralow latency, and high reliability, but also the accuracy of vehicle SER as well as the performance, practicality, and user experience of the ITS

Deep Spoken Keyword Spotting: An Overview

Spoken keyword spotting (KWS) deals with the identification of keywords in audio streams and has become a fast-growing technology thanks to the paradigm shift introduced by deep learning a few years ago. This has allowed the rapid embedding of deep KWS in a myriad of small electronic devices with different purposes like the activation of voice assistants. Prospects suggest a sustained growth in terms of social use of this technology. Thus, it is not surprising that deep KWS has become a hot research topic among speech scientists, who constantly look for KWS performance improvement and computational complexity reduction. This context motivates this paper, in which we conduct a literature review into deep spoken KWS to assist practitioners and researchers who are interested in this technology. Specifically, this overview has a comprehensive nature by covering a thorough analysis of deep KWS systems (which includes speech features, acoustic modeling and posterior handling), robustness methods, applications, datasets, evaluation metrics, performance of deep KWS systems and audio-visual KWS. The analysis performed in this paper allows us to identify a number of directions for future research, including directions adopted from automatic speech recognition research and directions that are unique to the problem of spoken KWS.

Study about Chinese Speech Synthesis Algorithm and Acoustic Model Based on Wireless Communication Network

Chinese speech synthesis refers to the technology that machines transform human speech signals into corresponding texts or commands through recognition and understanding. This paper combines the classic VAD and GSM VAD1 algorithm simulations, improves on the above two algorithms to recognize and collect speech, and analyzes their Chinese proficiency by amplifying the signal through a filter, in order to study the adulthood of Zhengzhou University in Southeast Asian students (mother tongues are Indonesian and Thai) as the research objects, to explore the relationship between the Chinese phonetic proficiency and the acquisition motivation of Southeast Asian students. This article combines algorithm and language disciplines. According to the results of Praat and SPSS: 55‐80 points account for 70%, 55 points below 20% and 80 points above 10%, we find that intrinsic motivation plays a role in CSL acquisition, a vital role. Intrinsic motivation can help mature learners from Southeast Asia to acquire Chinese better and better. The earlier you learn Chinese, the higher your motivation, and the easier it is to set your Chinese learning goals. The greater the enthusiasm for learning Chinese, the better the Chinese scores (such as HSK test scores and Chinese phonetic test scores). Therefore, the Chinese proficiency of international students has a great relationship with their interest in Chinese language, that is, the greater the interest in Chinese, the stronger their motivation to learn, and the Chinese proficiency will be very good.

Attention-Based Convolution Skip Bidirectional Long Short-Term Memory Network for Speech Emotion Recognition

Speech emotion recognition is a challenging task in natural language processing. It relies heavily on the effectiveness of speech features and acoustic models. However, existing acoustic models may not handle speech emotion recognition efficiently for their built-in limitations. In this work, a novel deep-learning acoustic model called attention-based skip convolution bi-directional long short-term memory, abbreviated as SCBAMM, is proposed to recognize speech emotion. It has eight hidden layers, namely, two dense layers, convolutional layer, skip layer, mask layer, Bi-LSTM layer, attention layer, and pooling layer. SCBAMM makes better use of spatiotemporal information and captures emotion-related features more effectively. In addition, it solves the problems of gradient exploding and gradient vanishing in deep learning to some extent. On the databases EMO-DB and CASIA, the proposed model SCBAMM achieves an accuracy rate of 94.58% and 72.50%, respectively. As far as we know, compared with peer models, this is the best accuracy rate.

The Key Technology of Speech Interaction Based on Deep Learning

Language recognition is essentially a pattern recognition problem, and all kinds of pattern recognition methods are used for language recognition research. For language recognition studied in this project, the highly effective speech feature parameters, acoustic model and language recognition classifier, the Chinese, Tibetan, Mongolian, uygur and training of acoustic model are established, so as to identify the voice input languages. In recent years, the deep learning theory has been applied in the field of speech recognition, and the DNN-HMM model has become the mainstream of acoustic modeling. This project studies Tibetan, Mongolian, Uygur emotion modeling of acoustic model and language model, training methods, studies the decoding algorithm, for continuous speech recognition of input speech, converts it to ethnic Chinese words.

An End-to-End Deep Learning Approach to Simultaneous Speech Dereverberation and Acoustic Modeling for Robust Speech Recognition

We propose an integrated end-to-end automatic speech recognition (ASR) paradigm by joint learning of the front-end speech signal processing and back-end acoustic modeling. We believe that “only good signal processing can lead to top ASR performance” in challenging acoustic environments. This notion leads to a unified deep neural network (DNN) framework for distant speech processing that can achieve both high-quality enhanced speech and high-accuracy ASR simultaneously. Our goal is accomplished by two techniques, namely: (i) a reverberation-time-aware DNN based speech dereverberation architecture that can handle a wide range of reverberation times to enhance speech quality of reverberant and noisy speech, followed by (ii) DNN-based multicondition training that takes both clean-condition and multicondition speech into consideration, leveraging upon an exploitation of the data acquired and processed with multichannel microphone arrays, to improve ASR performance. The final end-to-end system is established by a joint optimization of the speech enhancement and recognition DNNs. The recent REverberant Voice Enhancement and Recognition Benchmark (REVERB) Challenge task is used as a test bed for evaluating our proposed framework. We first report on superior objective measures in enhanced speech to those listed in the 2014 REVERB Challenge Workshop on the simulated data test set. Moreover, we obtain the best single-system word error rate (WER) of 13.28% on the 1-channel REVERB simulated data with the proposed DNN-based pre-processing algorithm and clean-condition training. Leveraging upon joint training with more discriminative ASR features and improved neural network based language models, a low single-system WER of 4.46% is attained. Next, a new multi-channel-condition joint learning and testing scheme delivers a state-of-the-art WER of 3.76% on the 8-channel simulated data with a single ASR system. Finally, we also report on a preliminary yet promising experimentation with the REVERB real test data.

A Speaker-Dependent Approach to Single-Channel Joint Speech Separation and Acoustic Modeling Based on Deep Neural Networks for Robust Recognition of Multi-Talker Speech

We propose a novel speaker-dependent (SD) multi-condition (MC) training approach to joint learning of deep neural networks (DNNs) of acoustic models and an explicit speech separation structure for recognition of multi-talker mixed speech in a single-channel setting. First, an MC acoustic modeling framework is established to train a SD-DNN model in multi-talker scenarios. Such a recognizer significantly reduces the decoding complexity and improves the recognition accuracy over those using speaker-independent DNN models with a complicated joint decoding structure assuming the speaker identities in mixed speech are known. In addition, a SD regression DNN for mapping the acoustic features of mixed speech to the speech features of a target speaker is jointly trained with the SD-DNN based acoustic models. Experimental results on Speech Separation Challenge (SSC) small-vocabulary recognition show that the proposed approach under multi-condition training achieves an average word error rate (WER) of 3.8%, yielding a relative WER reduction of 65.1% from a top performance, DNN-based pre-processing only approach we proposed earlier under clean-condition training (Tu et al. 2016). Furthermore, the proposed joint training DNN framework generates a relative WER reduction of 13.2% from state-of-the-art systems under multi-condition training. Finally, the effectiveness of the proposed approach is also verified on the Wall Street Journal (WSJ0) task with medium-vocabulary continuous speech recognition in a simulated multi-talker setting.

Adaptation of Noisy Speech Acoustic Models Considering Variations in Training Noise Signals

잡음음성인식을 위한 여러 가지 기법들 중 HMM(Hidden Markov Model) 모델적응은 다른 방식에 비해서 음성인식기의 성능을 상당히 향상 시키는 것으로 알려져 있다. 또한 깨끗한 음성으로 훈련된 HMM을 베이스라인 인식기로 사용하는 대신 잡음음성으로 미리 훈련된 잡음음성 HMM을 사용하는 경우, 그 효과가 더욱 더 큰 것으로 알려져 있다. 잡음음성 HMM 모델적응을 위한 기존의 연구에서는 인식잡음음성에 대한 확률모델을 유도하는 과정에서 훈련시 사용된 잡음신호를 랜덤변수가 아닌 단순한 파라미터로 취급하는 약점이 있었다. 본 연구에서는 이러한 문제를 해결하기 위하여 훈련잡음신호의 변이를 고려하였으며, 평균벡터와 공분산 행렬을 미리 추정하고 이를 인식잡음음성에 대한 확률모델 계산시에 활용하였다. 잡음음성 HMM의 훈련을 위해서 MTR(Multi-condition TRaining) 방식을 사용하였고 음향모델의 복잡도에 따른 성능 변화를 관찰하기 위하여 HMM의 구조를 2가지로 나누어서 성능 실험을 실시하였다. Aurora 2 음성데이터베이스를 이용한 잡음음성인식 실험결과, 훈련잡음신호의 변이를 고려한 제안된 방식은 MTR 훈련된 잡음음성 HMM에 특징보상과 모델적응을 한 기존의 방식에 비해서 우수한 성능을 보임을 알 수 있었다.

Severity-Based Adaptation with Limited Data for ASR to Aid Dysarthric Speakers

Automatic speech recognition (ASR) is currently used in many assistive technologies, such as helping individuals with speech impairment in their communication ability. One challenge in ASR for speech-impaired individuals is the difficulty in obtaining a good speech database of impaired speakers for building an effective speech acoustic model. Because there are very few existing databases of impaired speech, which are also limited in size, the obvious solution to build a speech acoustic model of impaired speech is by employing adaptation techniques. However, issues that have not been addressed in existing studies in the area of adaptation for speech impairment are as follows: (1) identifying the most effective adaptation technique for impaired speech; and (2) the use of suitable source models to build an effective impaired-speech acoustic model. This research investigates the above-mentioned two issues on dysarthria, a type of speech impairment affecting millions of people. We applied both unimpaired and impaired speech as the source model with well-known adaptation techniques like the maximum likelihood linear regression (MLLR) and the constrained-MLLR(C-MLLR). The recognition accuracy of each impaired speech acoustic model is measured in terms of word error rate (WER), with further assessments, including phoneme insertion, substitution and deletion rates. Unimpaired speech when combined with limited high-quality speech-impaired data improves performance of ASR systems in recognising severely impaired dysarthric speech. The C-MLLR adaptation technique was also found to be better than MLLR in recognising mildly and moderately impaired speech based on the statistical analysis of the WER. It was found that phoneme substitution was the biggest contributing factor in WER in dysarthric speech for all levels of severity. The results show that the speech acoustic models derived from suitable adaptation techniques improve the performance of ASR systems in recognising impaired speech with limited adaptation data.

Emotional speech acoustic model for Malay: Iterative versus isolated unit training

The ability of speech synthesis system to synthesize emotional speech enhances the user's experience when using this kind of system and its related applications. However, the development of an emotional speech synthesis system is a daunting task in view of the complexity of human emotional speech. The more recent state-of-the-art speech synthesis systems, such as the one based on hidden Markov models, can synthesize emotional speech with acceptable naturalness with the use of a good emotional speech acoustic model. However, building an emotional speech acoustic model requires adequate resources including segment-phonetic labels of emotional speech, which is a problem for many under-resourced languages, including Malay. This research shows how it is possible to build an emotional speech acoustic model for Malay with minimal resources. To achieve this objective, two forms of initialization methods were considered: iterative training using the deterministic annealing expectation maximization algorithm and the isolated unit training. The seed model for the automatic segmentation is a neutral speech acoustic model, which was transformed to target emotion using two transformation techniques: model adaptation and context-dependent boundary refinement. Two forms of evaluation have been performed: an objective evaluation measuring the prosody error and a listening evaluation to measure the naturalness of the synthesized emotional speech.

A VTS Based Speech Feature Compensation Using Noisy Speech Acoustic Models

기존의 VTS(Vector Taylor Series) 기반의 잡음음성인식 방식에서는 깨끗한 음성데이터를 이용하여 음향모델을 훈련한 후 VTS를 사용하여 원래음성을 추정하거나 새로운 잡음음성에 대한 음향모델의 파라미터 값을 적응하는 방식을 사용하였다. 그러나 이러한 접근 방식은 잡음음성을 이용하여 미리 훈련된 음향모델을 사용하지 못하는 단점이 있다. 깨끗한 음성으로 훈련된 음향모델 대신에 MTR(Multi-condition TRaining)이나 MMSR(Multi-Model based Speech Recognition) 방식으로 훈련된 잡음음성 음향모델을 VTS 알고리듬에 직접 사용함으로서 보다 나은 잡음음성인식 성능이 가능하리라 생각된다. 본 논문에서는 이러한 점에 착안하여 채널잡음을 고려한 인식잡음음성과 훈련잡음음성간의 수학적 관계식을 로그스펙트럼 영역에서 유도하였으며 이를 VTS를 이용하여 근사화한 후 훈련잡음음성에 대한 MMSE(Minimum Mean Square Error)를 통해서 인식잡음음성과 훈련잡음음성간의 불일치를 줄이도록 하였다. 제안된 방식을 MMSR 및 MTR 방식으로 훈련된 잡음음성 음향모델에 적용하였으며, Aurora 2 데이터베이스를 이용한 잡음음성인식실험 결과 MMSR 및 MTR 방식에 비해서 각각 10.3%와 10.8%의 상대적인 단어 오인식 감축률을 얻을 수 있다.

Likelihood-Maximizing-Based Multiband Spectral Subtraction for Robust Speech Recognition

Automatic speech recognition performance degrades significantly when speech is affected by environmental noise. Nowadays, the major challenge is to achieve good robustness in adverse noisy conditions so that automatic speech recognizers can be used in real situations. Spectral subtraction (SS) is a well-known and effective approach; it was originally designed for improving the quality of speech signal judged by human listeners. SS techniques usually improve the quality and intelligibility of speech signal while speech recognition systems need compensation techniques to reduce mismatch between noisy speech features and clean trained acoustic model. Nevertheless, correlation can be expected between speech quality improvement and the increase in recognition accuracy. This paper proposes a novel approach for solving this problem by considering SS and the speech recognizer not as two independent entities cascaded together, but rather as two interconnected components of a single system, sharing the common goal of improved speech recognition accuracy. This will incorporate important information of the statistical models of the recognition engine as a feedback for tuning SS parameters. By using this architecture, we overcome the drawbacks of previously proposed methods and achieve better recognition accuracy. Experimental evaluations show that the proposed method can achieve significant improvement of recognition rates across a wide range of signal to noise ratios.

A method of joint compensation of additive and convolutive distortions for speaker-independent speech recognition

A speech recognizer operating in a mobile environment has to be robust to two distortion sources: ambient noise (additive distortion) and microphone changes (convolutive distortion). Explicitly and simultaneously modeling the two distortion sources has been a great challenge for speech recognition in adverse environments. In this paper, two log-spectral domain components are introduced in speech acoustic models to represent additive and convolutive distortions. A method, called JAC, jointly compensates both additive and convolutive distortions. For each utterance to be recognized, it adapts HMM mean vectors with a noise estimate and a channel estimate. The noise estimate is calculated from the pre-utterance pause and the channel estimate is calculated using an EM algorithm from speech utterances produced in the distortion environment. The algorithm is evaluated on a noisy speech database recorded in-vehicle with a hands-free distant microphone in several sessions, including parked, stop-and-go, and highway driving conditions. Experiments show that the method typically reduces recognition word error rate by an order of magnitude. The method makes it possible to obtain high performance for speaker-independent recognition in changing noisy environments without collecting any noisy speech for training.

Data-driven multimodal synthesis

This paper is a report on current efforts at the Department of Speech, Music and Hearing, KTH, on data-driven multimodal synthesis including both visual speech synthesis and acoustic modeling. In the research we try to combine both corpus based methods with knowledge based models and to explore the best of the two approaches. In the paper an attempt to build formant-synthesis systems based on both rule-generated and database driven methods is presented. A pilot experiment is also reported showing that this approach can be a very interesting path to explore further. Two studies on visual speech synthesis are reported, one on data acquisition using a combination of motion capture techniques and one concerned with coarticulation, comparing different models.

Method and system for adaptive speech recognition in a noisy environment

The system uses utterances recorded in low noise condition, such as a car engine off to optimally adapt speech acoustic models to transducer and speaker characteristics and uses speech pauses to adjust the adopted models to a changing background noise, such as when in a car with the engine running.

Children's speech training aid

A children's speech training aid compares a child's speech with models of speech, stored as sub-word acoustic models, and a general speech model to give an indication of whether or not the child has spoken correctly. An indication of how well the word has been pronounced may also be given. An adult operator enters the word to be tested into the training aid which then forms a model of that word from the stored sub-word speech models. The stored acoustic models are formed by first recording a plurality of words by a plurality of children from a given list of single words. These recordings are then processed off-line to give a basic acoustic model of an acceptable or correct sound for each phoneme in the context of the pre- and proceeding phonemes. The acoustic models are Hidden Markov Models. The limits of acceptable prononciation applied to different words and children may be adjusted by variable penalty values applied in association with the general speech acoustic model. The training aid generates accumulated word costs for each child's utterance and uses these costs to indicate correctness of pronunciation.