Models For Speech Recognition Research Articles

Evaluation of Interpretable Speech Biomarkers for Monitoring Alzheimer’s Disease and Mild Cognitive Impairment Progression

AbstractBackgroundAlzheimer’s disease (AD) is a progressive neurodegenerative disease. Like for other dementias, biomarkers may help characterize distinct aspects of the underlying pathology, predict decline, and monitor disease progression. We extracted a composite array of 24 speech‐based biomarkers using machine learning and signal processing techniques. We then determined which of the given biomarkers was predictive of MCI and AD progression from baseline by conducting a correlation analysis.MethodSpoken responses to a Cookie Theft Picture description task were recorded from 2 subjects with AD, 4 subjects with Mild Cognitive Impairment (MCI) due to AD (biomarker confirmed AD), and 4 subjects with MCI. We automatically extracted acoustic, linguistic, and cognitive biomarkers using speech recognition, acoustic, and language modeling techniques. We later computed Kendall’s tau‐b (τb ) correlation between the biomarker values and Montreal Cognitive Assessment (MoCA) scores. Recordings and MoCA scores were collected at baseline, at 6 months after baseline, and at 12 months for a subgroup of the subjects. To perform the correlation analysis, we used scipy.stats.kendalltau library in Python.ResultWith respect to the acoustic biomarkers, a strong negative correlation (τb= ‐0.50) was observed between the MoCA scores and features such as pause time, pause percentage, and pause speech ratio, while a moderate positive correlation (τb= 0.2) was found for rhythm standard deviation. With respect to the linguistic biomarkers a strong positive correlation(τb= 0.50) for the features of corrected type‐token ratio, average sentence length in words, and noun count was reported, while a moderate positive correlation (τb= 0.27) was found for the features of word type count, word token count, and moving average type‐token ratio. The other acoustic, linguistic, and cognitive biomarkers showed a very weak (τb= 0.00 ‐ 0.10) or weak correlation (τb= 0.10 ‐ 0.20).ConclusionAltogether, these results suggest that speech‐based interpretable biomarkers may help clinicians to diagnose AD at earlier stages and monitor disease progression. Our preliminary data suggest that AD patients encounter more problems delivering longer and linguistically elaborated narratives as the disease progresses.

Objective evaluation of voice quality by intermittent recording of sound on English voice

Continuous sound surveillance have the problem of unintentionally recording conversational content. On the other hand, if only coarse features are stored so that speech cannot be recovered, the use of the accumulated recorded information is very limited. In this study, we propose a method to record raw background sounds while making the conversational content inaudible by periodically repeating recording and stopping. In this presentation, we will show the results of an objective evaluation using a speech recognition model by varying the ratio of recording time and the recording stop frequency. The results showed that the recording time ratio can be maximized by selecting appropriate recording stop frequency even for the same objective voice quality.

Voice-Controlled Intelligent Personal Assistant for Call-Center Automation in the Uzbek Language

The demand for customer support call centers has surged across various sectors due to the pandemic. Yet, the constraints of round-the-clock human services and fluctuating wait times pose challenges in fully meeting customer needs. In response, there’s a growing need for automated customer service systems that can provide responses tailored to specific domains and in the native languages of customers, particularly in developing nations like Uzbekistan where call center usage is on the rise. Our system, “UzAssistant,” is designed to recognize user voices and accurately present customer issues in standardized Uzbek, as well as vocalize the responses to voice queries. It employs feature extraction and recurrent neural network (RNN)-based models for effective automatic speech recognition, achieving an impressive 96.4% accuracy in real-time tests with 56 participants. Additionally, the system incorporates a sentence similarity assessment method and a text-to-speech (TTS) synthesis feature specifically for the Uzbek language. The TTS component utilizes the WaveNet architecture to convert text into speech in Uzbek.

Intelligent speech recognition algorithm in multimedia visual interaction via BiLSTM and attention mechanism

With the rapid development of information technology in modern society, the application of multimedia integration platform is more and more extensive. Speech recognition has become an important subject in the process of multimedia visual interaction. The accuracy of speech recognition is dependent on a number of elements, two of which are the acoustic characteristics of speech and the speech recognition model. Speech data is complex and changeable. Most methods only extract a single type of feature of the signal to represent the speech signal. This single feature cannot express the hidden information. And, the excellent speech recognition model can also better learn the characteristic speech information to improve performance. This work proposes a new method for speech recognition in multimedia visual interaction. First of all, this work considers the problem that a single feature cannot fully represent complex speech information. This paper proposes three kinds of feature fusion structures to extract speech information from different angles. This extracts three different fusion features based on the low-level features and higher-level sparse representation. Secondly, this work relies on the strong learning ability of neural network and the weight distribution mechanism of attention model. In this paper, the fusion feature is combined with the bidirectional long and short memory network with attention. The extracted fusion features contain more speech information with strong discrimination. When the weight increases, it can further improve the influence of features on the predicted value and improve the performance. Finally, this paper has carried out systematic experiments on the proposed method, and the results verify the feasibility.

The construction of Bert fusion model of speech recognition and sensing for South China electricity charge service scenario

Electric charge service and management is an important part of electric power work. The effective recovery of the electric charge relates to the smooth development of daily work and continuous improvement of the operation and management of power supply enterprises. With the large-scale implementation of the card prepayment system, the problem of electricity customers defaulting on electricity charges has been solved to a large extent, but some large electricity users still fail to pay electricity charges on time. Therefore, under the current situation of power grid development, it is still necessary to strengthen the service and management of electricity charges to promote efficient recovery of electricity charges. Speech recognition technology has increasingly become the focus of research institutions at home and abroad. People are committed to enabling machines to understand human speech instructions and hope to control the machine through speech. The research and development of speech recognition will greatly facilitate people's lives shortly. The development of 5G technology and the proposal of 6G technology make the interconnection of all things not only a hope but also a reality. To realize the interconnection of all things, one of the key technical breakthroughs is the development of a new human–computer interaction sensing system. Under the guidance of relevant theories and methods, this paper systematically analyzes the user structure, electricity charge recovery management and service system, existing problems and causes in South China, and clarifies the necessity of design and application of electricity charge service system in South China power supply companies. The experimental data and empirical analysis results show that the optimized Bert fusion model can provide more digital support for the power supply companies in South China in terms of electricity charge recovery efficiency, management level system improvement, and electricity charge service.

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

We present an any-to-one voice conversion (VC) system, using an autoregressive model and LPCNet vocoder, aimed at enhancing the converted speech in terms of naturalness, intelligibility, and speaker similarity. As the name implies, non-parallel any-to-one voice conversion does not require paired source and target speeches and can be employed for arbitrary speech conversion tasks. Recent advancements in neural-based vocoders, such as WaveNet, have improved the efficiency of speech synthesis. However, in practice, we find that the trajectory of some generated waveforms is not consistently smooth, leading to occasional voice errors. To address this issue, we propose to use an autoregressive (AR) conversion model along with the high-fidelity LPCNet vocoder. This combination not only solves the problems of waveform fluidity but also produces more natural and clear speech, with the added capability of real-time speech generation. To precisely represent the linguistic content of a given utterance, we use speaker-independent PPG features (SI-PPG) computed from an automatic speech recognition (ASR) model trained on a multi-speaker corpus. Next, a conversion model maps the SI-PPG to the acoustic representations used as input features for the LPCNet. The proposed autoregressive structure enables our system to produce the following prediction step outputs from the acoustic features predicted in the previous step. We evaluate the effectiveness of our system by performing any-to-one conversion pairs between native English speakers. Experimental results show that the proposed method outperforms state-of-the-art systems, producing higher speech quality and greater speaker similarity.

AUTOMATIC SPEECH RECOGNITION USING DEEP NEURAL NETWORKS

This research paper revolves around the evolution and the present scenario of Automatic Speech Recognition systems using Deep Neural Networks. It includes the designs, techniques of training, the evaluations of the model performance, and the emerging trends that are specific to deep neural networks embedded in automatic speech recognition models. This research also incorporates the challenges faced while building and deploying the speech recognition model including limited data availability and adaptability. We have examined how deep neural networks have transformed automatic speech recognition and this research provides valuable insights to improve the speech recognition technology across a number of applications from healthcare to smart devices. INDEX TERMS - Automatic Speech Recognition, Deep Neural Networks, Language Modeling, Robustness to Noise, Speech Modelling

AUTOMATIC SPEECH RECOGNITION USING DEEP NEURAL NETWORKS

This research concentrates on the progression and present status of automatic speech recognition systems powered by deep neural networks. It discusses model architectures, training approaches, evaluating model efficacy, and recent advancements specific to deep neural networks applied in automatic speech recognition models. It considers the challenges faced in crafting these speech recognition models, such as data scarcity and the necessity for adaptability. Our exploration traces the evolution of automatic speech recognition through deep neural networks, presenting valuable insights aimed at propelling the domain of speech recognition for diverse applications, spanning from smart devices to healthcare. KEYWORDS - Automatic Speech Recognition, Deep Neural Networks, Language Modeling, Robustness to Noise, Speech Modelling

Automatic speech recognition of Portuguese phonemes using neural networks ensemble

The automatic speech recognition based on detection of phonemes provides advantages for online recognition of a speech represented by a sound signal. The development of a system for automatic speech recognition is multidisciplinary. It covers several areas of research, such as linguistics, signal processing and computational intelligence. In this work, the process starts with a speech signal pre-processing to extract the main features of the speech signal at a given instant of time. Inspired by the “divide and conquer” principle, we bridge the complexity gap of automatic speech recognition by devising models based on an ensemble of neural network experts, allowing to divide the huge decision space regarding speech recognition so that each expert takes care only of a delimited area of this decision space. This novel application of this strategy improves the precision, sensitivity and accuracy of the recognition process. Each included expert decides regarding each one of the pre-processed input samples. The decision set thus obtained is weighted. So, the expert with the highest weight for the output will determine the sample final classification. After that, a dynamic post-processing step, implemented as a recurrent neural network, is executed. It aims at mitigating the oscillatory effect that occurs during the recognition of classes with similar characteristics. In this work, two ensembles are investigated. The first is based on the clustering of similar phonetics classes while the second takes care of the imbalanced distribution of samples in the training set. The proposed model achieves 7.63% improvement in terms of accuracy with respect to the best so far related model for automatic speech recognition.

96 Proof of Principle: Can Paragraph Recall Pauses and Speech Frequencies Correctly Classify Cognitively Compromised Older Adults?

Objective:Recent research has found that machine learning based analysis of patient speech can be used to classify Alzheimer’s Disease. We know of no studies, however, which systematically explore the value of pausing events in speech for detecting cognitive limitations. Using retrospectively acquired voice data from paragraph memory tests, we created two types of pause features: a) the number and duration of pauses, and b) frequency components in speech immediately following pausing. Multiple machine learning models were used to assess how these features could effectively discriminate individuals classified into two groups: Cognitively Compromised versus Cognitively Well.Participants and Methods:Participants (age> 65 years, n= 67) completed the Newcomer paragraph memory test and a neuropsychological protocol as part of a federally funded prospective IRB approved investigation at the University of Florida. Participant vocal recordings were acquired for the immediate and delay conditions of the test. Speaker diarization was performed on the immediate free recall test condition to separate voices of patients from examiners. Features extracted from both test conditions included a) 3 pause characteristics (total number of pauses, total pause duration, and length of the longest pause), and b) 20 Mel Frequency Cepstral Coefficients (MFCC) pertaining to speech immediately (2.7 seconds) following pauses. These were combined with demographics (age, sex, race, education, and handedness) to create a total of 105 features that were used as inputs for multiple machine learning analytic models (random forest, logistic regression, naive Bayes, AdaBoost, Gradient Boost, and multi-layered perceptron). External neuropsychological metrics were used to initially classify Cognitively Compromised (i.e., < -1.0 standard deviation on > two of five test metrics: total immediate, delay, discrimination Hopkins Verbal Learning Test-Revised (HVLT-R),Controlled Oral Word Association (COWA) test, category fluency ('animals')). Pearson Product Moment Correlations were used to assess the linear relationships between pauses and speech frequency categories and neuropsychological metrics.Results:Neuropsychology metric classification using -1SD cut-off identified 27% (18/67 participants) as Cognitively Compromised. The Cognitively Compromised group and the Cognitively Well group did not show any difference in distributions of individual pause/frequency features (Mann Whitney U-test, p> 0.11). A negative correlation was found between total duration of short pauses and HVLT total immediate free recall, while a positive correlation was found between MFCC-10 and HVLT total immediate free recall. The best classification model was AdaBoost Classifier which predicted the Cognitively Compromised label with 0.91 area under receiver operating curve, 0.81 accuracy, 0.43 sensitivity, 1.0 specificity, 1.0 precision, 0.6 f1 score.Conclusions:Pause characteristics and frequency profiles of speech immediately following pauses from a paragraph memory test accurately identified older adults with compromised cognition, as measured by verbal learning and verbal fluency metrics. Furthermore, individuals with reduced HVLT immediate free recall generated more pauses, while individuals who recalled more words had higher power in mid-frequency bands (10th MFCC). Future research needs to replicate how paragraph recall pause characteristics and frequency the profile of speech immediately following pauses potentially provides a low resource alternative to automatic speech recognition models for detecting cognitive impairments.

Research on the application of machine learning models in speech recognition in noisy environments

In a world increasingly reliant on verbal communication with machines, this paper undertakes a rigorous exploration of the application of machine learning models in speech recognition within noisy environments. An important focus of this research lies in demonstrating the viability and robustness of these models when challenged by real-world noise interference. In particular, the utilization of deep learning models such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory Networks (LSTMs) is thoroughly examined. Using concrete data and case studies, this paper uncovers a promising trajectory in the development and implementation of noise-robust speech recognition systems. In so doing, it lays a foundation for future research to enhance the practicality and effectiveness of machine learning models in this area. The ultimate goal is to ensure more reliable and accurate speech recognition that can withstand various noise conditions, thus expanding the potential for speech recognition applications in diverse and adverse environments.

Optimizing medical personnel speech recognition models using speech synthesis and reinforcement learning

Text-to-Speech synthesis (TTS) can be used to generate training data for building Automatic Speech Recognition models (ASR). Access to medical speech data is because it is sensitive data that is difficult to obtain for privacy reasons. Speech can be synthesized by mimicking different accents, dialects, and speaking styles in a medical language. Reinforcement Learning (RL), in the context of ASR, can be used to optimize a model. A model can be trained to minimize errors in speech-to-text transcription, especially for technical medical terminology. In this case, the “reward” to the RL model can be negatively proportional to the number of transcription errors. The paper presents a method and experimental study from which it is concluded that the combination of TTS and RL can enable the creation of a speech recognition model suited to the specific needs of medical personnel, helping to expand the training data and optimize the model to minimize transcription errors. The learning process used reward functions based on Mean Opinion Score (MOS), a subjective metric for assessing speech quality, and Word Error Rate (WER), which evaluates the quality of speech-to-text transcription. [The Polish National Center for Research and Development (NCBR) supported the project: “ADMEDVOICE- Adaptive intelligent speech processing system of medical personnel with the structuring of test results and support of therapeutic process,” no. INFOSTRATEG4/0003/2022.]

A Language Model Optimization Method for Turkish Automatic Speech Recognition System

The current Automatic Speech Recognition (ASR) modeling strategy still suffers from huge performance degradation when faced with languages with limited resources such as Turkish. Especially when the Language Model (LM) does not support the Acoustic Model (AM) sufficiently, the Word Error Rate (WER) increases. Therefore, a robust LM makes a strong contribution to improving ASR performance by generating word relations from the existing corpus. However, developing a robust language model is a challenging task due to the agglutinative nature of Turkish. Therefore, within the scope of the study, a sentence-level LM optimization method is proposed to improve the WER performance of Turkish ASR. In the proposed method, instead of a fixed word sequence obtained from the Markov assumptions, the probability of the word sequence forming a sentence was calculated. A method with n-gram and skip-gram properties is presented to obtain the word sequence probability. The proposed method has been tested on both statistical and Artificial Neural Network (ANN) based LMs. In the experiments carried out using, not only words but also sub-word level, two Turkish corpora (METU and Bogazici) shared via Linguistic Data Consortium (LDC) and a separate corpus, which we separate corpus that we specially created as HS was used. According to the experimental results obtained from statistical-based LM, 0.5% WER increases for the METU corpus, 1.6% WER decreases for the Bogazici corpus, and a 2.5% WER decrease for the HS corpus were observed. In the Feedforward Neural Networks (FNN) based LM, WER decreases were observed 0.2% for the METU corpus, 0.8% for the Bogazici corpus, and 1.6% for the HS corpus. Also, in the Recurrent Neural Network (RNN)-Long Short Term Memory (LSTM) based LM, WER decreases were observed 0.6% for METU corpus, 1.1% for the Bogazici corpus and 1.5% for the HS corpus. As a result, when the proposed method was applied to the LMs required for ASR, WER decreased, and the total performance of ASR increased.

V-LSTM: An Efficient LSTM Accelerator Using Fixed Nonzero-Ratio Viterbi-Based Pruning

Long Short-Term Memory (LSTM) has been widely adopted in tasks with sequence data, such as speech recognition and language modeling. LSTM brought significant accuracy improvement by introducing additional parameters to Recurrent Neural Network (RNN). However, increasing number of parameters and computations also led to inefficiency in computing LSTM on edge devices with limited on-chip memory size and DRAM bandwidth. In order to reduce the latency and energy of LSTM computations, there has been a pressing need for model compression schemes and suitable hardware accelerators. In this paper, we first propose the Fixed Nonzero-ratio Viterbi-based Pruning, which can reduce the memory footprint of LSTM models by 96% with negligible accuracy loss. By applying additional constraints on the distribution of surviving weights in Viterbi-based Pruning, the proposed pruning scheme mitigates the load-imbalance problem and thereby increases the processing engine utilization rate. Then, we propose the V-LSTM, an efficient sparse LSTM accelerator based on the proposed pruning scheme. High compression ratio of the proposed pruning scheme allows the proposed accelerator to achieve 24.9% lower per-sample latency than that of state-of-the-art accelerators. The proposed accelerator is implemented on Xilinx VC-709 FPGA evaluation board running at 200MHz for evaluation.

Measurement of level of consciousness by AVPU scale assessment system based on automated video and speech recognition technology

ObjectiveTo develop an alert/verbal/painful/unresponsive (AVPU) scale assessment system based on automated video and speech recognition technology (AVPU-AVSR) that can automatically assess a patient's level of consciousness and evaluate its performance through clinical simulation. MethodsWe developed an AVPU-AVSR system with a whole-body camera, face camera, and microphone. The AVPU-AVSR system automatically extracted essential audiovisual features to assess the AVPU score from the recorded video files. Arm movement, pain stimulus, and eyes-open state were extracted using a rule-based approach using landmarks estimated from pre-trained pose and face estimation models. Verbal stimuli were extracted using a pre-trained speech-recognition model. Simulations of a physician examining the consciousness of 12 simulated patients for 16 simulation scenarios (4 for each of “Alert”, “Verbal”, “Painful”, and “Unresponsive”) were conducted under the AVPU-AVSR system. The accuracy, sensitivity, and specificity of the AVPU-AVSR system were assessed. ResultsA total of 192 cases with 12 simulated patients were assessed using the AVPU-AVSR system with a multi-class accuracy of 0.95 (95% confidence interval [CI] (0.92–0.98). The sensitivity and specificity (95% CIs) for detecting impaired consciousness were 1.00 (0.97–1.00) and 0.88 (0.75–0.95), respectively. The sensitivity and specificity of each extracted feature ranged from 0.88 to 1.00 and 0.98 to 1.00. ConclusionsThe AVPU-AVSR system showed good accuracy in assessing consciousness levels in a clinical simulation and has the potential to be implemented in clinical practice to automatically assess mental status.

Hybrid deep learning based automatic speech recognition model for recognizing non-Indian languages

Hybrid deep learning based automatic speech recognition model for recognizing non-Indian languages

Deep Models for Low-Resourced Speech Recognition: Livvi-Karelian Case

Recently, there has been a growth in the number of studies addressing the automatic processing of low-resource languages. The lack of speech and text data significantly hinders the development of speech technologies for such languages. This paper introduces an automatic speech recognition system for Livvi-Karelian. Acoustic models based on artificial neural networks with time delays and hidden Markov models were trained using a limited speech dataset of 3.5 h. To augment the data, pitch and speech rate perturbation, SpecAugment, and their combinations were employed. Language models based on 3-grams and neural networks were trained using written texts and transcripts. The achieved word error rate metric of 22.80% is comparable to other low-resource languages. To the best of our knowledge, this is the first speech recognition system for Livvi-Karelian. The results obtained can be of a certain significance for development of automatic speech recognition systems not only for Livvi-Karelian, but also for other low-resource languages, including the fields of speech recognition and machine translation systems. Future work includes experiments with Karelian data using techniques such as transfer learning and DNN language models.

Low-Complexity and Accurate Noise Suppression Based on an a Priori SNR Model for Robust Speech Recognition on Embedded Systems and Its Evaluation in a Car Environment

Low-Complexity and Accurate Noise Suppression Based on an a Priori SNR Model for Robust Speech Recognition on Embedded Systems and Its Evaluation in a Car Environment

Метод разработки моделей распознавания речи для использования в информационных системах энергетики

Currently, when developing automatic speech recognition models for specialized subject areas, in particular for energy facilities, deep neural network architectures are used, which require a large amount of training data. At the same time, models often turn out to be poorly suitable for use in specific information systems due to poor-quality recognition of highly specialized subject vocabulary. Additional training of models to improve their quality in a specific context of recognition encounters the difficulty to obtain a sufficient amount of data and the laboriousness of their markup. Thus, an urgent task is to create methods that allow reducing the complexity of developing applied speech recognition models and improving their quality when used in subject areas, in particular, in the field of energy. Methods of thematic text modeling based on language models for adapting open data are applied. A deep neural network is used as a pretrained speech recognition model. For training, open-source datasets are used. A method to develop automatic speech recognition models for specialized subject areas has been developed. It includes the stage of intermediate learning of subject area vocabulary based on open-source data selected using thematic sampling. Based on the method, the authors have developed and studied a model of automatic speech recognition for energy facilities. It has showed higher recognition results than models obtained by traditional methods. Approbation of the proposed method has confirmed its effectiveness. The applied neural network model developed on the method has demonstrated the possibility to work in the information systems of energy facilities in Russian and English without additional training on proprietary data.

Security and privacy problems in voice assistant applications: A survey

Voice assistant applications have become omniscient nowadays. Two models that provide the two most important functions for real-life applications (i.e., Google Home, Amazon Alexa, Siri, etc.) are Automatic Speech Recognition (ASR) models and Speaker Identification (SI) models. According to recent studies, security and privacy threats have also emerged with the rapid development of the Internet of Things (IoT). The security issues researched include attack techniques toward machine learning models and other hardware components widely used in voice assistant applications. The privacy issues include technical-wise information stealing and policy-wise privacy breaches. The voice assistant application takes a steadily growing market share every year, but their privacy and security issues never stopped causing huge economic losses and endangering users' personal sensitive information. Thus, it is important to have a comprehensive survey to outline the categorization of the current research regarding the security and privacy problems of voice assistant applications. This paper concludes and assesses five kinds of security attacks and three types of privacy threats in the papers published in the top-tier conferences of cyber security and voice domain.