Hours Of Training Data Research Articles

End-to-end automated speech recognition using a character based small scale transformer architecture

This study explores the feasibility of constructing a small-scale speech recognition system capable of competing with larger, modern automated speech recognition (ASR) systems in both performance and word error rate (WER). Our central hypothesis posits that a compact transformer-based ASR model can yield comparable results, specifically in terms of WER, to traditional ASR models while challenging contemporary ASR systems that boast significantly larger computational sizes. The aim is to extend ASR capabilities to under-resourced languages with limited corpora, catering to scenarios where practitioners face constraints in both data availability and computational resources. The model, comprising a compact convolutional neural network (CNN) and transformer architecture with 2.214 million parameters, challenges the conventional wisdom that large-scale transformer-based ASR systems are essential for achieving high accuracy. In comparison, contemporary ASR systems often deploy over 300 million parameters. Trained on a modest dataset of approximately 3000 h – significantly less than the 50,000 h used in larger systems – the proposed model leverages the Common Voice and LibriSpeech datasets. Evaluation on the LibriSpeech test-clean and test-other datasets produced character error rates (CERs) of 6.40% and 16.73% and WERs of 16.03% and 35.51% respectively. Comparisons with existing architectures showcase the efficiency of our model. A gated recurrent unit (GRU) architecture, albeit achieving lower error rates, incurred a computational cost 24 times larger than our proposed model. Large-scale transformer architectures, while achieving marginally lower WERs (2%–4% on LibriSpeech test-clean), require 200 times more parameters and 53,000 additional hours of training data. Modern large language models are used to improve the WERs, but require large computational resources. To further enhance performance, a small 4-g language model was integrated into our end-to-end ASR model, resulting in improved WERs. The overarching goal of this work is to provide a practical solution for practitioners dealing with limited datasets and computational resources, particularly in the context of under-resourced languages.

Detection of Behaviors of Risk exhibited by People with Dementia with Privacy Protecting Videos

AbstractBehavioural symptoms of dementia present a significant risk within long‐term care homes, resulting in difficulties supporting residents and monitoring their safety due to limited staffing resources. Existing video surveillance infrastructure can be used to monitor and automatically detect clinically important behaviours that can put the residents at risk. An anomaly detection approach can be used to detect behaviours of risk from videos due to their infrequent and diverse nature. However, most existing video anomaly detection approaches focus on appearance‐based features, which can put the privacy of a person at risk and is also susceptible to pixel‐based noise. In this study, we explored different privacy‐protecting approaches and present a conditional average neighbour score method to detect behaviours of risks in videos from a dementia care unit. The conditional average neighbour score method considers that behaviours of risk in people with dementia are sometimes momentary, but often they persist over time. In this study, we used semantic segmentation masks approach to protect participants’ privacy by obscuring them in the video. The privacy‐protecting inputs were used to train a customized spatio‐temporal convolutional autoencoder and identify behaviours of risk as anomalies. The video was collected on a dementia care unit and composed of approximately 21 hours of training data (normal activities) and 9 hours of test data (normal activities and behaviours of risk events). In comparison to RGB input (0.822), we obtained an equivalent area under the receiver operating characteristic curve (AUC ROC) performance of 0.823 for the segmentation mask‐based privacy‐protecting approach. Further, using the conditional average neighbour score method, we improved the AUCROC performance to 0.83 for the segmentation mask‐based privacy‐protecting approach. The results signify that it is possible to use surveillance videos from a dementia care unit to detect behaviours of risk as anomalies, and that privacy protecting methods work as well as methods using the raw videos. This research paves the way to improve the quality of life of residents and reduce injuries in residential care homes, while respecting their privacy.

Prosody-TTS: An End-to-End Speech Synthesis System with Prosody Control

End-to-end text-to-speech synthesis systems achieved immense success in recent times, with improved naturalness and intelligibility. However, the end-to-end models, which primarily depend on the attention-based alignment, do not offer an explicit provision to modify/incorporate the desired prosody while synthesizing the speech. Moreover, the state-of-the-art end-to-end systems use autoregressive models for synthesis, making the prediction sequential. Hence, the inference time and the computational complexity are quite high. This paper proposes Prosody-TTS, a data-efficient end-to-end speech synthesis model that combines the advantages of statistical parametric models and end-to-end neural network models. It also has a provision to modify or incorporate the desired prosody at the finer level by controlling the fundamental frequency (\(f_0\)) and the phone duration. Generating speech utterances with appropriate prosody and rhythm helps in improving the naturalness of the synthesized speech. We explicitly model the duration of the phoneme and the \(f_0\) to have a finer level control over them during the synthesis. The model is trained in an end-to-end fashion to directly generate the speech waveform from the input text, which in turn depends on the auxiliary subtasks of predicting the phoneme duration, \(f_0\), and Mel spectrogram. Experiments on the Telugu language data of the IndicTTS database show that the proposed Prosody-TTS model achieves state-of-the-art performance with a mean opinion score of 4.08, with a very low inference time using just 4 hours of training data.

A novel long-term water absorption and thickness swelling deep learning forecast method for corn husk fiber-polypropylene composite

Investigating long-term water absorption (WA) and thickness swelling (TS) behaviors of wood plastic composites demand long working hours and high laboratory costs. However, using artificial intelligence methods, these behaviors can be predicted in far less time and with a low degree of error. This paper aims to predict the long-term WA and TS behaviors of a cornhusk fiber (CHF) propylene (PP) composite using the deep learning field’s long short-term memory (LSTM) method. We assessed the network LSTM performance based on mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE). The experimental tests of WA and TS behaviors were performed on a CHF/PP composite using three different filler percentages over a period of 0–1500 h. The predictions were carried out for 200, 400, 600, 800, and 1000 h to construct a database to identify how many hours of training data are required to meet a MAPE criterion of 2% between the actual and predicted data. The results show that 200 h of training data is adequate for the LSTM method to achieve this MAPE metric. Furthermore, the metrics results validate the applicability of the proposed method. All the manufacturing metrics and codes are attached.

Addressing data inadequacy challenges in personal comfort models by combining pretrained comfort models

Occupant thermal-comfort complaints are the biggest operational headache of facilities managers. Many of the complaints can be attributed to the diverse nature of individuals’ thermal comfort needs which are not accounted for in the de facto standard for thermal comfort. This has motivated research on developing data-driven personal comfort models and incorporating them in control loops. But the progress on this front has been hampered by the lack of sufficient ground-truth thermal comfort data to train accurate thermal comfort models. To address this problem, in this paper we explore how artificial labels, indicating individuals’ true thermal preference, can be generated from their heating and cooling behaviour with a personal comfort system. Furthermore, we use clustering to identify individuals with similar comfort requirements in a rich dataset collected from 37 individuals in an office building, and develop a small number of group comfort models, each achieving a high accuracy in predicting the thermal comfort of individuals within the respective cluster. The pretrained group comfort models are then combined using an ensemble method to create a general thermal comfort model that can accurately predict the thermal comfort of any individual without knowing their thermal preferences or group membership a priori. We evaluate the efficacy of two ensemble methods as more training data becomes available and show that they outperform two conventional comfort models (PMV, Adaptive) and the personal comfort model that is developed from scratch for a particular individual. Specifically, the best ensemble comfort model yields on average 71% accuracy in predicting individuals’ thermal preference using only 6 hours of training data, excluding no occupancy periods.

LSTM‐based japanese speaker identification using an omnidirectional camera and voice information

Numerous companies in Japan currently desire to increase the efficiency of meetings, which is an important objective that should be addressed with computational methods. To this end, increasing the efficiency of minute‐taking tasks using voice‐recognition technology has been considered. The development of an automated system able to identify individual speakers by voice so as to add information encoding which speaker uttered which text would substantially improve the overall efficiency of the process. Several speaker‐identification methods have been proposed in prior works on the automatic generation of meeting minutes. However, such methods require the preparation of multiple microphones to register audio signals. In contrast, speakers can be identified with a smaller preparation cost by utilizing an omnidirectional camera; however, hundreds of hours of training data are required. In this study, we propose a speaker‐identification method that uses only tens of minutes of moving images as training data. Evaluation results demonstrate that the proposed method is able to identify speakers with maximum and average success rates of 93.0% and 87.2%, respectively. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

An Investigation of Multilingual TDNN-BLSTM Acoustic Modeling for Hindi Speech Recognition

Background: In India, thousands of languages or dialects are in use. Most Indian dialects are low asset dialects. A well-performing Automatic Speech Recognition (ASR) system for Indian languages is unavailable due to a lack of resources. Hindi is one of them as large vocabulary Hindi speech datasets are not freely available. We have only a few hours of transcribed Hindi speech dataset. There is a lot of time and money involved in creating a well-transcribed speech dataset. Thus, developing a real-time ASR system with a few hours of the training dataset is the most challenging task. The different techniques like data augmentation, semi-supervised training, multilingual architecture, and transfer learning, have been reported in the past to tackle the fewer speech data issues. In this paper, we examine the effect of multilingual acoustic modeling in ASR systems for the Hindi language. Objective: This article’s objective is to develop a high accuracy Hindi ASR system with a reasonable computational load and high accuracy using a few hours of training data. Method: To achieve this goal we used Multilingual training with Time Delay Neural Network- Bidirectional Long Short Term Memory (TDNN-BLSTM) acoustic modeling. Multilingual acoustic modeling has significantly improved the ASR system's performance for low and limited resource languages. The common practice is to train the acoustic model by merging data from similar languages. In this work, we use three Indian languages, namely Hindi, Marathi, and Bengali. Hindi with 2.5 hours of training data and Marathi with 5.5 hours of training data and Bengali with 28.5 hours of transcribed data, was used in this work to train the proposed model. Results: The Kaldi toolkit was used to perform all the experiments. The paper is investigated over three main points. First, we present the monolingual ASR system using various Neural Network (NN) based acoustic models. Second, we show that Recurrent Neural Network (RNN) language modeling helps to improve the ASR performance further. Finally, we show that a multilingual ASR system significantly reduces the Word Error Rate (WER) (absolute 2% WER reduction for Hindi and 3% for the Marathi language). In all the three languages, the proposed TDNN-BLSTM-A multilingual acoustic models help to get the lowest WER. Conclusion: The multilingual hybrid TDNN-BLSTM-A architecture shows a 13.67% relative improvement over the monolingual Hindi ASR system. The best WER of 8.65% was recorded for Hindi ASR. For Marathi and Bengali, the proposed TDNN-BLSTM-A acoustic model reports the best WER of 30.40% and 10.85%.

REFINING AUTOMATIC SPEECH RECOGNITION SYSTEM FOR OLDER ADULTS.

Building a high quality automatic speech recognition (ASR) system with limited training data has been a challenging task particularly for a narrow target population. Open-sourced ASR systems, trained on sufficient data from adults, are susceptible on seniors' speech due to acoustic mismatch between adults and seniors. With 12 hours of training data, we attempt to develop an ASR system for socially isolated seniors (80+ years old) with possible cognitive impairments. We experimentally identify that ASR for the adult population performs poorly on our target population and transfer learning (TL) can boost the system's performance. Standing on the fundamental idea of TL, tuning model parameters, we further improve the system by leveraging an attention mechanism to utilize the model's intermediate information. Our approach achieves 1.58% absolute improvements over the TL model.

End-to-End Classification of Reverberant Rooms Using DNNs

Reverberation is present in our workplaces, our homes, concert halls and theatres. This article investigates how deep learning can use the effect of reverberation on speech to classify a recording in terms of the room in which it was recorded. Existing approaches in the literature rely on domain expertise to manually select acoustic parameters as inputs to classifiers. Estimation of these parameters from reverberant speech is adversely affected by estimation errors, impacting the classification accuracy. In order to overcome the limitations of previously proposed methods, this paper shows how DNNs can perform the classification by operating directly on reverberant speech spectra and a CRNN with an attention-mechanism is proposed for the task. The relationship is investigated between the reverberant speech representations learned by the DNNs and acoustic parameters. For evaluation, AIRs are used from the ACE-challenge dataset that were measured in 7 real rooms. The classification accuracy of the CRNN classifier in the experiments is 78% when using 5 hours of training data and 90% when using 10 hours.

Cross-Lingual Phone Mapping for Large Vocabulary Speech Recognition of Under-Resourced Languages

SUMMARY This paper presents a novel acoustic modeling technique of large vocabulary automatic speech recognition for under-resourced languages by leveraging well-trained acoustic models of other languages (called source languages). The idea is to use source language acoustic model to score the acoustic features of the target language, and then map these scores to the posteriors of the target phones using a classifier. The target phone posteriors are then used for decoding in the usual way of hybrid acoustic modeling. The motivation of such a strategy is that human languages usually share similar phone sets and hence it may be easier to predict the target phone posteriors from the scores generated by source language acoustic models than to train from scratch an under-resourced language acoustic model. The proposed method is evaluated using on the Aurora-4 task with less than 1 hour of training data. Two types of source language acoustic models are considered, i.e. hybrid HMM/MLP and conventional HMM/GMM models. In addition, we also use triphone tied states in the mapping. Our experimental results show that by leveraging well trained Malay and Hungarian acoustic models, we achieved 9.0% word error rate (WER) given 55 minutes of English training data. This is close to the WER of 7.9% obtained by using the full 15 hours of training data and much better than the WER of 14.4% obtained by conventional acoustic modeling techniques with the same 55 minutes of training data.

Sparse Inverse Covariance Matrices for Low Resource Speech Recognition

We propose to use sparse inverse covariance matrices for acoustic model training when there is insufficient training data. Acoustic models trained with inadequate training data tend to over fit, generalizing poorly to unseen test data, especially when full covariance matrices are used. We address this problem by adding an L1 regularization term to the traditional objective function for maximum likelihood estimation, to penalize complex models. The structure of the inverse covariance matrices will be automatically sparsified using this new objective function. The Expectation Maximization algorithm is used to learn the parameters of the hidden Markov model using the new objective function. It is shown that the training procedures for all the hidden Markov model parameters are the same as that of maximum likelihood estimation except the inverse covariance matrices. The update equation for the inverse covariance matrices is concave and can be solved efficiently. Our experiments show that this proposed method can correctly learn the underlying correlations among the random variables of the speech feature vector. Experimental results on the Wall Street Journal data show that our proposed model significantly outperforms the diagonal covariance model and the full covariance model by 10.9% and 16.5% relative recognition accuracy, when only about 14 hours of training data are available. On our collected low resource language data-the Cantonese data set, the proposed model also significantly outperforms the diagonal covariance model and the full covariance model.

Large Scale Distributed Acoustic Modeling With Back-Off ${\rm N}$-Grams

The paper revives an older approach to acoustic modeling that borrows from n-gram language modeling in an attempt to scale up both the amount of training data and model size (as measured by the number of parameters in the model), to approximately 100 times larger than current sizes used in automatic speech recognition. In such a data-rich setting, we can expand the phonetic context significantly beyond triphones, as well as increase the number of Gaussian mixture components for the context-dependent states that allow it. We have experimented with contexts that span seven or more context-independent phones, and up to 620 mixture components per state. Dealing with unseen phonetic contexts is accomplished using the familiar back-off technique used in language modeling due to implementation simplicity. The back-off acoustic model is estimated, stored and served using MapReduce distributed computing infrastructure. Speech recognition experiments are carried out in an N-best list rescoring framework for Google Voice Search. Training big models on large amounts of data proves to be an effective way to increase the accuracy of a state-of-the-art automatic speech recognition system. We use 87,000 hours of training data (speech along with transcription) obtained by filtering utterances in Voice Search logs on automatic speech recognition confidence. Models ranging in size between 20--40 million Gaussians are estimated using maximum likelihood training. They achieve relative reductions in word-error-rate of 11% and 6% when combined with first-pass models trained using maximum likelihood, and boosted maximum mutual information, respectively. Increasing the context size beyond five phones (quinphones) does not help.

Large scale discriminative training of hidden Markov models for speech recognition

This paper describes, and evaluates on a large scale, the lattice based framework for discriminative training of large vocabulary speech recognition systems based on Gaussian mixture hidden Markov models (HMMs). This paper concentrates on the maximum mutual information estimation (MMIE) criterion which has been used to train HMM systems for conversational telephone speech transcription using up to 265 hours of training data. These experiments represent the largest-scale application of discriminative training techniques for speech recognition of which the authors are aware. Details are given of the MMIE lattice-based implementation used with the extended Baum-Welch algorithm, which makes training of such large systems computationally feasible. Techniques for improving generalization using acoustic scaling and weakened language models are discussed. The overall technique has allowed the estimation of triphone and quinphone HMM parameters which has led to significant reductions in word error rate for the transcription of conversational telephone speech relative to our best systems trained using maximum likelihood estimation (MLE). This is in contrast to some previous studies, which have concluded that there is little benefit in using discriminative training for the most difficult large vocabulary speech recognition tasks. The lattice MMIE-based discriminative training scheme is also shown to out-perform the frame discrimination technique. Various properties of the lattice-based MMIE training scheme are investigated including comparisons of different lattice processing strategies (full search and exact-match) and the effect of lattice size on performance. Furthermore a scheme based on the linear interpolation of the MMIE and MLE objective functions is shown to reduce the danger of over-training. It is shown that HMMs trained with MMIE benefit as much as MLE-trained HMMs from applying model adaptation using maximum likelihood linear regression (MLLR). This has allowed the straightforward integration of MMIE-trained HMMs into complex multi-pass systems for transcription of conversational telephone speech and has contributed to our MMIE-trained systems giving the lowest word error rates in both the 2000 and 2001 NIST Hub5 evaluations.