Hidden Markov Model Neural Network Research Articles

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond n=3. Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse.

Advancements in end-to-end isolated Kannada ASR system by combining robust noise elimination technique and TDNN

The demonstration of the latest enhancements in the end-to-end (E2E) isolated Kannada automatic speech recognition (ASR) system, achieved by combining a robust background noise elimination technique and a time delay neural network (TDNN), is presented in this work. An E2E Kannada ASR system consists of an interactive voice response system (IVRS), ASR models, and databases containing weather and agricultural commodity prices information. In the earlier spoken query system (SQS), the presence of babble, street noise, and other background noises led to a decrease in both offline and online speech recognition accuracies. To properly train the models, we increase the size of the database by collecting the Kannada speech data from an additional 500 farmers under real-time conditions. Moreover, the proposed noise elimination technique is employed to enhance the degraded speech data. Additionally, the efficacy of the TDNN is explored to improve the recognition accuracy of ASR models and the SQS system.The outcomes of the proposed speech enhancement algorithm demonstrate the absence of audible musical noise and other types of background noises in enhanced NOIZEUS and isolated Kannada speech databases. Leveraging Kannada language resources and the amalgamation of the proposed noise reduction technique and TDNN, a significant 1.1% enhancement in speech recognition accuracy is achieved in comparison with the previously developed deep neural network-hidden Markov model (DNN-HMM) based SQS. The enhanced isolated E2E ASR SQS system undergoes testing by 500 farmers, enabling them to access real-time agricultural commodity prices and weather forecasting information in their native Kannada language/dialects. This practical validation highlights the applicability and effectiveness of our advancements in real-world scenarios. The source code of proposed noise elimination technique, experimental results of ASR models and demo conversation of SQS are made publicly available at: https://sites.google.com/view/thimmarajayadavag/downloads

JSUM: A Multitask Learning Speech Recognition Model for Jointly Supervised and Unsupervised Learning

In recent years, the end-to-end speech recognition model has emerged as a popular alternative to the traditional Deep Neural Network—Hidden Markov Model (DNN-HMM). This approach maps acoustic features directly onto text sequences via a single network architecture, significantly streamlining the model construction process. However, the training of end-to-end speech recognition models typically necessitates a significant quantity of supervised data to achieve good performance, which poses a challenge in low-resource conditions. The use of unsupervised representation significantly reduces this necessity. Recent research has focused on end-to-end techniques employing joint Connectionist Temporal Classification (CTC) and attention mechanisms, with some also concentrating on unsupervised presentation learning. This paper proposes a joint supervised and unsupervised multi-task learning model (JSUM). Our approach leverages the unsupervised pre-trained wav2vec 2.0 model as a shared encoder that integrates the joint CTC-Attention network and the generative adversarial network into a unified end-to-end architecture. Our method provides a new low-resource language speech recognition solution that optimally utilizes supervised and unsupervised datasets by combining CTC, attention, and generative adversarial losses. Furthermore, our proposed approach is suitable for both monolingual and cross-lingual scenarios.

A high-precision correction method in non-rigid 3D motion poses reconstruction

Occlusion, rotation and other factors affect human motion structure because of the incomplete acquired image sequence, resulting in poor performance of non-rigid three-dimensional (3D) motion pose reconstruction. A non-rigid 3D reconstruction and high-precision correction method for motion pose are studied in this paper. A non-rigid imaging model is designed to obtain 3D moving images. According to the frame difference and morphological processing, the background of image is separated and denoised. Combined with motion analysis, 3D motion pose features are extracted as identification of non-rigid 3D motion error actions in a hybrid Convolution Neural Network-Hidden Markov Model to train the correction coefficients, which are used to adjust the pose in 3D motion reconstruction and realise correction. Experimental results show that this method has high precision reconstruction and correction of non-rigid 3D motion pose.

Automatic Speech Recognition Method Based on Deep Learning Approaches for Uzbek Language.

Communication has been an important aspect of human life, civilization, and globalization for thousands of years. Biometric analysis, education, security, healthcare, and smart cities are only a few examples of speech recognition applications. Most studies have mainly concentrated on English, Spanish, Japanese, or Chinese, disregarding other low-resource languages, such as Uzbek, leaving their analysis open. In this paper, we propose an End-To-End Deep Neural Network-Hidden Markov Model speech recognition model and a hybrid Connectionist Temporal Classification (CTC)-attention network for the Uzbek language and its dialects. The proposed approach reduces training time and improves speech recognition accuracy by effectively using CTC objective function in attention model training. We evaluated the linguistic and lay-native speaker performances on the Uzbek language dataset, which was collected as a part of this study. Experimental results show that the proposed model achieved a word error rate of 14.3% using 207 h of recordings as an Uzbek language training dataset.

A deep learning model based anomalous behavior detection for supporting verifiable access control scheme in cloud servers

Due to the growth of Big Data (BD) storage and access in cloud computing infrastructure, the detection of anomalies for Cloud Servers (CSs) is essential to ensure data confidentiality. Over the past decades, different security systems have been designed based on various methods like encryption, Access Policy (AP) control schemes, signcryption and so on. Among many security systems, a new Improved NTRU (INTRU) decryption based on the AP strategy has been suggested to secure the BD processed by the CSs. Also, the shared secret data was authenticated to defend the clients from anomalies in the cloud. But, the AP upgrade must not degrade the confidentiality of storing information, reveal trust in the CS or cause any different security challenges. It is not considered that such security challenges occur when the data owner shares its data with many CSs. Hence in this article, an INTRU with Detecting Anomalous in CS (INTRU-DACS) system is proposed that employs a deep learning-based Anomaly Detection System (ADS) to handle and secure the BD stored in the CSs. The main goal of this method is to effectively identify the abnormalities in the real world by the conduct utilization, i.e., the System Call Identifier Sequences (SCISs) created from CSs in which these conducts are associated with BD. Initially, effective data summarization is constructed via different feature states to analyze the SCISs of specific durations. After that, an anomaly identification algorithm is proposed to train and test the streaming of raw SC sequences. This observable SCs execution task of CSs is gathered from log files. The variations of such SCISs having a specified duration are random for usual and unusual sequences. So, the fact of current normal and abnormal services is recognized regarding their SCISs. Such normal and abnormal behavioral states are learned from Convolutional Neural Network-Hidden Markov Model (CNNHMM) classifier to identify the anomalies in CSs. But, it is still a challenging process because of the patterns of usual and unusual events. The performance is not effective since it models only the conduct of a huge number of SCISs created from a single CS. As a result, a Secure Access Control Scheme with DACS (SACS-DACS) system is proposed in which a Multidimensional Feature Misbehavior Server Detection method (MFMSD) is introduced for detecting anomalies in multiple CSs. In this method, large-scale SCISs of multiple CSs are extracted, including different features such as network traffic sequence features, CPU energy usage and memory usage from host logs. These extracted multidimensional features are fed to the CNNHMM that identifies the anomalies and maximizes the detection accuracy. At last, the simulation results demonstrate the effectiveness of the SACS-DACS and INTRU-DACS as compared to the INTRU.

Using Automatic Speech Recognition to Assess Thai Speech Language Fluency in the Montreal Cognitive Assessment (MoCA).

The Montreal cognitive assessment (MoCA), a widely accepted screening tool for identifying patients with mild cognitive impairment (MCI), includes a language fluency test of verbal functioning; its scores are based on the number of unique correct words produced by the test taker. However, it is possible that unique words may be counted differently for various languages. This study focuses on Thai as a language that differs from English in terms of word combinations. We applied various automatic speech recognition (ASR) techniques to develop an assisted scoring system for the MoCA language fluency test with Thai language support. This was a challenge because Thai is a low-resource language for which domain-specific data are not publicly available, especially speech data from patients with MCIs. Furthermore, the great variety of pronunciation, intonation, tone, and accent of the patients, all of which might differ from healthy controls, bring more complexity to the model. We propose a hybrid time delay neural network hidden Markov model (TDNN-HMM) architecture for acoustic model training to create our ASR system that is robust to environmental noise and to the variation of voice quality impacted by MCI. The LOTUS Thai speech corpus was incorporated into the training set to improve the model’s generalization. A preprocessing algorithm was implemented to reduce the background noise and improve the overall data quality before feeding data into the TDNN-HMM system for automatic word detection and language fluency score calculation. The results show that the TDNN-HMM model in combination with data augmentation using lattice-free maximum mutual information (LF-MMI) objective function provides a word error rate (WER) of 30.77%. To our knowledge, this is the first study to develop an ASR with Thai language support to automate the scoring system of MoCA’s language fluency assessment.

Feature compensation based on the normalization of vocal tract length for the improvement of emotion-affected speech recognition

The performance of speech recognition systems trained with neutral utterances degrades significantly when these systems are tested with emotional speech. Since everybody can speak emotionally in the real-world environment, it is necessary to take account of the emotional states of speech in the performance of the automatic speech recognition system. Limited works have been performed in the field of emotion-affected speech recognition and so far, most of the researches have focused on the classification of speech emotions. In this paper, the vocal tract length normalization method is employed to enhance the robustness of the emotion-affected speech recognition system. For this purpose, two structures of the speech recognition system based on hybrids of hidden Markov model with Gaussian mixture model and deep neural network are used. To achieve this goal, frequency warping is applied to the filterbank and/or discrete-cosine transform domain(s) in the feature extraction process of the automatic speech recognition system. The warping process is conducted in a way to normalize the emotional feature components and make them close to their corresponding neutral feature components. The performance of the proposed system is evaluated in neutrally trained/emotionally tested conditions for different speech features and emotional states (i.e., Anger, Disgust, Fear, Happy, and Sad). In this system, frequency warping is employed for different acoustical features. The constructed emotion-affected speech recognition system is based on the Kaldi automatic speech recognition with the Persian emotional speech database and the crowd-sourced emotional multi-modal actors dataset as the input corpora. The experimental simulations reveal that, in general, the warped emotional features result in better performance of the emotion-affected speech recognition system as compared with their unwarped counterparts. Also, it can be seen that the performance of the speech recognition using the deep neural network-hidden Markov model outperforms the system employing the hybrid with the Gaussian mixture model.

Novel hybrid DNN approaches for speaker verification in emotional and stressful talking environments

In this work, we conducted an empirical comparative study of the performance of text-independent speaker verification in emotional and stressful environments. This work combined deep models with shallow architecture, which resulted in novel hybrid classifiers. Four distinct hybrid models were utilized: deep neural network-hidden Markov model (DNN-HMM), deep neural network-Gaussian mixture model (DNN-GMM), Gaussian mixture model-deep neural network (GMM-DNN), and hidden Markov model-deep neural network (HMM-DNN). All models were based on novel implemented architecture. The comparative study used three distinct speech datasets: a private Arabic dataset and two public English databases, namely Speech Under Simulated and Actual Stress (SUSAS) and Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). The test results of the aforementioned hybrid models demonstrated that the proposed HMM-DNN leveraged the verification performance in emotional and stressful environments. Results also showed that HMM-DNN outperformed all other hybrid models in terms of equal error rate (EER) and area under the curve (AUC) evaluation metrics. The average resulting verification system based on the three datasets yielded EERs of 7.19, 16.85, 11.51, and 11.90% based on HMM-DNN, DNN-HMM, DNN-GMM, and GMM-DNN, respectively. Furthermore, we found that the DNN-GMM model demonstrated the least computational complexity compared to all other hybrid models in both talking environments. Conversely, the HMM-DNN model required the greatest amount of training time. Findings also demonstrated that EER and AUC values depended on the database when comparing average emotional and stressful performances.

A Hybrid Hidden Markov Model for Pipeline Leakage Detection

In this paper, a deep neural network hidden Markov model (DNN-HMM) is proposed to detect pipeline leakage location. A long pipeline is divided into several sections and the leakage occurs in different section that is defined as different state of hidden Markov model (HMM). The hybrid HMM, i.e., DNN-HMM, consists of a deep neural network (DNN) with multiple layers to exploit the non-linear data. The DNN is initialized by using a deep belief network (DBN). The DBN is a pre-trained model built by stacking top-down restricted Boltzmann machines (RBM) that compute the emission probabilities for the HMM instead of Gaussian mixture model (GMM). Two comparative studies based on different numbers of states using Gaussian mixture model-hidden Markov model (GMM-HMM) and DNN-HMM are performed. The accuracy of the testing performance between detected state sequence and actual state sequence is measured by micro F1 score. The micro F1 score approaches 0.94 for GMM-HMM method and it is close to 0.95 for DNN-HMM method when the pipeline is divided into three sections. In the experiment that divides the pipeline as five sections, the micro F1 score for GMM-HMM is 0.69, while it approaches 0.96 with DNN-HMM method. The results demonstrate that the DNN-HMM can learn a better model of non-linear data and achieve better performance compared to GMM-HMM method.

Histogram of States Based Assistive System for Speech Impairment Due to Neurological Disorders.

Assistive speech technology is a challenging task because of the impaired nature of dysarthric speech, such as breathy voice, strained speech, distorted vowels, and consonants. Learning compact and discriminative embeddings for dysarthric speech utterances is essential for impaired speech recognition. We propose a Histogram of States (HoS)-based approach that uses Deep Neural Network-Hidden Markov Model (DNN-HMM) to learn word lattice-based compact and discriminative embeddings. Best state sequence chosen from word lattice is used to represent dysarthric speech utterance. A discriminative model-based classifier is then used to recognize these embeddings. The performance of the proposed approach is evaluated using three datasets, namely 15 acoustically similar words, 100-common words datasets of the UA-SPEECH database, and a 50-words dataset of the TORGO database. The proposed HoS-based approach performs significantly better than the traditional Hidden Markov Model and DNN-HMM-based approaches for all three datasets. The discriminative ability and the compactness of the proposed HoS-based embeddings lead to the best accuracy of impaired speech recognition.

Investigation of DNN-HMM and Lattice Free Maximum Mutual Information Approaches for Impaired Speech Recognition

Assistive tools that recognize impaired speech due to neurological disorders are emerging and its a fairly complex task. An Intelligent Impaired Speech Recognition system helps persons with speech impairment to improve their interactions with outside world. Impaired speakers have difficulty in pronouncing words which results in partial or incomplete speech contents. Existing Automatic Speech Recognition systems are not effective for Impaired Speech Recognition due to the speaker specific variations which depend on the severity of the neurological disorders. In this work, we have investigated two important approaches namely, Deep Neural Network-Hidden Markov Model and Lattice Free Maximum Mutual Information approach for effective recognition of impaired speech. The training and testing samples are collected from persons with different neurological disorders at varied intelligibility levels such as high, medium, low and very low. The recognition accuracy is evaluated and compared using two datasets namely 20 acoustically similar words and 50 words Impaired Speech Corpus in Tamil.

Neural candidate-aware language models for speech recognition

This paper presents novel neural network based language models that can correct automatic speech recognition (ASR) errors by using speech recognizer outputs as a context. Our proposed models, called neural candidate-aware language models (NCALMs), estimate the generative probability of a target sentence while considering ASR outputs including hypotheses and their posterior probabilities. Recently, neural network language models have achieved great success in ASR field because of their ability to learn long-range contexts and model the word representation in continuous space. However, they estimate a sentence probability without considering other candidates and their posterior probabilities, even though the competing hypotheses are available and include important information to increase the speech recognition accuracy. To overcome this limitation, our idea is to utilize ASR outputs in both the training phase and the inference phase. Our proposed models are conditional generative models consisting of a Transformer encoder and a Transformer decoder. The encoder embeds the candidates as context vectors and the decoder estimates a sentence probability given the context vectors. We evaluate the proposed models in Japanese lecture transcription and English conversational speech recognition tasks. Experimental results show that a NCALM has better ASR performance than a system including a deep neural network-hidden Markov model hybrid system. We further improve ASR performance by using a NCALM and a Transformer language model simultaneously.

Online Automatic Speech Recognition With Listen, Attend and Spell Model

The Listen, Attend and Spell (LAS) model and other attention-based automatic speech recognition (ASR) models have known limitations when operated in a fully online mode. In this letter, we analyze the online operation of LAS models to demonstrate that these limitations stem from the handling of silence regions and the reliability of online attention mechanism at the edge of input buffers. We propose a novel and simple technique that can achieve fully online recognition while meeting accuracy and latency targets. For the Mandarin dictation task, our proposed approach can achieve a character error rate in online operation that is within 4% relative to an offline LAS model. The proposed online LAS model operates at 12% lower latency relative to a conventional neural network hidden Markov model hybrid of comparable accuracy. We have validated the proposed method through a production scale deployment, which, to the best of our knowledge, is the first such deployment of a fully online LAS model.

Modular End-to-End Automatic Speech Recognition Framework for Acoustic-to-Word Model

End-to-end (E2E) systems have played a more and more important role in automatic speech recognition (ASR) and achieved great performance. However, E2E systems recognize output word sequences directly with the input acoustic feature, which can only be trained on limited acoustic data. The extra text data is widely used to improve the results of traditional artificial neural network-hidden Markov model (ANN-HMM) hybrid systems. The involving of extra text data to standard E2E ASR systems may break the E2E property during decoding. In this paper, a novel modular E2E ASR system is proposed. The modular E2E ASR system consists of two parts: an acoustic-to-phoneme (A2P) model and a phoneme-to-word (P2W) model. The A2P model is trained on acoustic data, while extra data including large scale text data can be used to train the P2W model. This additional data enables the modular E2E ASR system to model not only the acoustic part but also the language part. During the decoding phase, the two models will be integrated and act as a standard acoustic-to-word (A2W) model. In other words, the proposed modular E2E ASR system can be easily trained with extra text data and decoded in the same way as a standard E2E ASR system. Experimental results on the Switchboard corpus show that the modular E2E model achieves better word error rate (WER) than standard A2W models.

Developing in-vehicular noise robust children ASR system using Tandem-NN-based acoustic modelling

Processing of children's speech is always challenging due to data scarcity and inefficient modelling input feature vectors. Accuracy of the modelling phase is always dependent upon extracted input features. In this paper, posterior probabilities are estimated over a phone set using first discriminatively trained model through neural-net pre-processor. This Neural Network (NN) classifier is first trained on original speech and then context-independent phone posterior probabilities are estimated on Tandem-NN system. The output vectors are employed as default features which are processed on Deep Neural Network-Hidden Markov Model (DNN-HMM) models. The original data-based system performance is improved by extending it using data augmentation. To see the robustness of the augmented speech various in-vehicle data are investigated and found that it is superior to that of other systems. Finally, we combine all augmented data to overcome data scarcity challenges to enhance system performance. It gives a relative improvement of 23.77% over the baseline system.

Developing in-vehicular noise robust children ASR system using Tandem-NN-based acoustic modelling

Processing of children's speech is always challenging due to data scarcity and inefficient modelling input feature vectors. Accuracy of the modelling phase is always dependent upon extracted input features. In this paper, posterior probabilities are estimated over a phone set using first discriminatively trained model through neural-net pre-processor. This Neural Network (NN) classifier is first trained on original speech and then context-independent phone posterior probabilities are estimated on Tandem-NN system. The output vectors are employed as default features which are processed on Deep Neural Network-Hidden Markov Model (DNN-HMM) models. The original data-based system performance is improved by extending it using data augmentation. To see the robustness of the augmented speech various in-vehicle data are investigated and found that it is superior to that of other systems. Finally, we combine all augmented data to overcome data scarcity challenges to enhance system performance. It gives a relative improvement of 23.77% over the baseline system.

Writer-aware CNN for parsimonious HMM-based offline handwritten Chinese text recognition

Recently, the hybrid convolutional neural network hidden Markov model (CNN-HMM) has been introduced for offline handwritten Chinese text recognition (HCTR) and has achieved state-of-the-art performance. However, modeling each of the large vocabulary of Chinese characters with a uniform and fixed number of hidden states requires high memory and computational costs and makes the tens of thousands of HMM state classes confusing. Another key issue of CNN-HMM for HCTR is the diversified writing style, which leads to model strain and a significant performance decline for specific writers. To address these issues, we propose a writer-aware CNN based on parsimonious HMM (WCNN-PHMM). First, PHMM is designed using a data-driven state-tying algorithm to greatly reduce the total number of HMM states, which not only yields a compact CNN by state sharing of the same or similar radicals among different Chinese characters but also improves the recognition accuracy due to the more accurate modeling of tied states and the lower confusion among them. Second, WCNN integrates each convolutional layer with one adaptive layer fed by a writer-dependent vector, namely, the writer code, to extract the irrelevant variability in writer information to improve recognition performance. The parameters of writer-adaptive layers are jointly optimized with other network parameters in the training stage, while a multiple-pass decoding strategy is adopted to learn the writer code and generate recognition results. Validated on the ICDAR 2013 competition of CASIA-HWDB database, the more compact WCNN-PHMM of a 7360-class vocabulary can achieve a relative character error rate (CER) reduction of 16.6% over the conventional CNN-HMM without considering language modeling. By adopting a powerful hybrid language model (N-gram language model and recurrent neural network language model), the CER of WCNN-PHMM is reduced to 3.17%. Moreover, the state-tying results of PHMM explicitly show the information sharing among similar characters and the confusion reduction of tied state classes. Finally, we visualize the learned writer codes and demonstrate the strong relationship with the writing styles of different writers. To the best of our knowledge, WCNN-PHMM yields the best results on the ICDAR 2013 competition set, demonstrating its power when enlarging the size of the character vocabulary.

Automated and Interpretable Patient ECG Profiles for Disease Detection, Tracking, and Discovery.

The ECG remains the most widely used diagnostic test for characterization of cardiac structure and electrical activity. We hypothesized that parallel advances in computing power, machine learning algorithms, and availability of large-scale data could substantially expand the clinical inferences derived from the ECG while at the same time preserving interpretability for medical decision-making. We identified 36 186 ECGs from the University of California, San Francisco database that would enable training of models for estimation of cardiac structure or function or detection of disease. We segmented the ECG into standard component waveforms and intervals using a novel combination of convolutional neural networks and hidden Markov models and evaluated this segmentation by comparing resulting electrical intervals against 141 864 measurements produced during the clinical workflow. We then built a patient-level ECG profile, a 725-element feature vector and used this profile to train and interpret machine learning models for examples of cardiac structure (left ventricular mass, left atrial volume, and mitral annulus e-prime) and disease (pulmonary arterial hypertension, hypertrophic cardiomyopathy, cardiac amyloid, and mitral valve prolapse). ECG measurements derived from the convolutional neural network-hidden Markov model segmentation agreed with clinical estimates, with median absolute deviations as a fraction of observed value of 0.6% for heart rate and 4% for QT interval. Models trained using patient-level ECG profiles enabled surprising quantitative estimates of left ventricular mass and mitral annulus e' velocity (median absolute deviation of 16% and 19%, respectively) with good discrimination for left ventricular hypertrophy and diastolic dysfunction as binary traits. Model performance using our approach for disease detection demonstrated areas under the receiver operating characteristic curve of 0.94 for pulmonary arterial hypertension, 0.91 for hypertrophic cardiomyopathy, 0.86 for cardiac amyloid, and 0.77 for mitral valve prolapse. Modern machine learning methods can extend the 12-lead ECG to quantitative applications well beyond its current uses while preserving the transparency that is so fundamental to clinical care.

Recurrent Poisson Process Unit for Speech Recognition

Over the past few years, there has been a resurgence of interest in using recurrent neural network-hidden Markov model (RNN-HMM) for automatic speech recognition (ASR). Some modern recurrent network models, such as long shortterm memory (LSTM) and simple recurrent unit (SRU), have demonstrated promising results on this task. Recently, several scientific perspectives in the fields of neuroethology and speech production suggest that human speech signals may be represented in discrete point patterns involving acoustic events in the speech signal. Based on this hypothesis, it may pose some challenges for RNN-HMM acoustic modeling: firstly, it arbitrarily discretizes the continuous input into the interval features at a fixed frame rate, which may introduce discretization errors; secondly, the occurrences of such acoustic events are unknown. Furthermore, the training targets of RNN-HMM are obtained from other (inferior) models, giving rise to misalignments. In this paper, we propose a recurrent Poisson process (RPP) which can be seen as a collection of Poisson processes at a series of time intervals in which the intensity evolves according to the RNN hidden states that encode the history of the acoustic signal. It aims at allocating the latent acoustic events in continuous time. Such events are efficiently drawn from the RPP using a sampling-free solution in an analytic form. The speech signal containing latent acoustic events is reconstructed/sampled dynamically from the discretized acoustic features using linear interpolation, in which the weight parameters are estimated from the onset of these events. The above processes are further integrated into an SRU, forming our final model, called recurrent Poisson process unit (RPPU). Experimental evaluations on ASR tasks including ChiME-2, WSJ0 and WSJ0&1 demonstrate the effectiveness and benefits of the RPPU. For example, it achieves a relative WER reduction of 10.7% over state-of-the-art models on WSJ0.