Automatic Speech Recognition Accuracy Research Articles

System dedicated to Polish Automatic Speech Recognition - overview of solutions

The paper presents the analysis of modern Artificial Intelligence algorithms for the automated system supporting human beings during their conversation in Polish language. Their task is to perform Automatic Speech Recognition (ASR) and process it further, for instance fill the computer-based form or perform the Natural Language Processing (NLP) to assign the conversation to one of predefined categories. The State-of-the-Art review is required to select the optimal set of tools to process speech in the difficult conditions, which degrade accuracy of ASR. The paper presents the top-level architecture of the system applicable for the task. Characteristics of Polish language are discussed. Next, existing ASR solutions and architectures with the End-To-End (E2E) deep neural network (DNN) based ASR models are presented in detail. Differences between Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN) and Transformers in the context of ASR technology are also discussed.

Effect of Face Masks on Automatic Speech Recognition Accuracy for Mandarin

Automatic speech recognition (ASR) has been widely used to realize daily human–machine interactions. Face masks have become everyday wear in our post-pandemic life, and speech through masks may have impaired the ASR. This study explored the effects of different kinds of face masks (e.g., surgical mask, KN95 mask, and cloth mask) on the Mandarin word accuracy of two ASR systems with or without noises. A mouth simulator was used to play speech audio with or without wearing a mask. Acoustic signals were recorded at distances of 0.2 m and 0.6 m. Recordings were mixed with two noises at a signal-to-noise ratio of +3 dB: restaurant noise and speech-shaped noise. Results showed that masks did not affect ASR accuracy without noise. Under noises, masks did not significantly influence ASR accuracy at 0.2 m but had significant effects at 0.6 m. The activated-carbon mask had the most significant impact on ASR accuracy at 0.6 m, reducing the accuracy by 18.5 percentage points compared to that without a mask, whereas the cloth mask had the least effect on ASR accuracy at 0.6 m, reducing the accuracy by 0.9 percentage points. The acoustic attenuation of masks on the high-frequency band at around 3.15 kHz of the speech signal attributed to the effects of masks on ASR accuracy. When training ASR models, it may be important to consider mask robustness.

The impact of non-native English speakers’ phonological and prosodic features on automatic speech recognition accuracy

The present study examines the impact of Arab speakers’ phonological and prosodic features on the accuracy of automatic speech recognition (ASR) of non-native English speech. The authors first investigated the perceptions of 30 Egyptian ESL teachers and 70 Egyptian university students towards the L1 (Arabic)-based errors affecting intelligibility and then carried out a data analysis of the ASR of the students’ English speech to find out whether the errors investigated resulted in intelligibility breakdowns in an ASR setting. In terms of the phonological features of non-native speech, the results showed that the teachers gave more weight to pronunciation features of accented speech that did not actually hinder recognition, that the students were mostly oblivious to the L2 errors they made and their impact on intelligibility, and that L2 errors which were not perceived as serious by both teachers and students had negative impacts on ASR accuracy levels. In regard to the prosodic features of non-native speech, it was found that lower speech rates resulted in more accurate speech recognition levels, higher speech intensity led to less deletion errors, and voice pitch did not seem to have any impact on ASR accuracy levels. The study, accordingly, recommends training ASR systems with more non-native data to increase their accuracy levels as well as paying more attention to remedying non-native speakers’ L1-based errors that are more likely to impact non-native automatic speech recognition.

Measuring the Accuracy of Automatic Speech Recognition Solutions

For d/Deaf and hard of hearing (DHH) people, captioning is an essential accessibility tool. Significant developments in artificial intelligence mean that automatic speech recognition (ASR) is now a part of many popular applications. This makes creating captions easy and broadly available—but transcription needs high levels of accuracy to be accessible. Scientific publications and industry report very low error rates, claiming that artificial intelligence has reached human parity or even outperforms manual transcription. At the same time, the DHH community reports serious issues with the accuracy and reliability of ASR. There seems to be a mismatch between technical innovations and the real-life experience for people who depend on transcription. Independent and comprehensive data is needed to capture the state of ASR. We measured the performance of 11 common ASR services with recordings of Higher Education lectures. We evaluated the influence of technical conditions like streaming, the use of vocabularies, and differences between languages. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events. Our study shows that despite the recent improvements of ASR, common services lack reliability in accuracy.

A novel semi-blind source separation framework towards maximum signal-to-interference ratio

For the independence-based blind source separation (BSS) methods, effective modeling of the source signals is important. When applying the auxiliary function techniques, different source models normally result in different forms of the weighted covariance matrices, which in turn are used to compute the demixing filters. This paper proposes a new algorithmic framework termed minimum variance independent component analysis (MVICA), which is rigorously derived from the conventional BSS methods by determining the weighted covariance matrices with the maximum signal-to-interference ratio (SIR) criterion.The maximum-SIR weighted covariance matrix is approximately proved to be the interference covariance matrix. A deep neural network-supported implementation of the proposed MVICA algorithm is subsequently presented. Furthermore, we give a revealing insight into the relationships between MVICA and the existing speech separation approaches including conventional BSS methods and several mainstream beamforming techniques. Experimental results show the superiority of MVICA over the state-of-the-art BSS techniques and beamformers under various conditions, in terms of not only SIR but also signal-to-distortion ratio, speech intelligibility and perceptual quality and automatic speech recognition accuracy.

Finetuning Pretrained Model with Embedding of Domain and Language Information for ASR of Very Low-Resource Settings

This study investigates the effective incorporation of meta-information such as domain and language in finetuning a pretrained model based on self-supervised learning (SSL) for automatic speech recognition (ASR) in very low-resource settings. SSL pretrained models have been shown to achieve comparable or even better performance to conventional end-to-end systems even when we finetune them with a small dataset. However, it still requires the specific target dataset with a considerable amount of labeled data, like 10 h, to achieve satisfactory performance. Thus, we propose to exploit heterogeneous datasets which are partially matched either in language or domain and apply multi-task learning (MTL) or adversarial learning (ADV) using the meta-information. The finetuning comprises (1) domain adaptation, which uses in-domain multi-lingual datasets, and (2) language adaptation, which uses datasets of the same language but different domains. The auxiliary task is domain identification for language adaptation and language identification for domain adaptation. We then embed the output of the auxiliary task into the encoder output of the ASR task. The target dataset is the Khmer corpus of the Extraordinary Chambers in the Courts of Cambodia (ECCC) in various sizes from one hour to 10 h. The experimental evaluations demonstrate that fusing the meta-information in MTL or ADV significantly improves ASR accuracy. Moreover, a two-step adaptation method which first conducts domain adaptation and then language adaptation is the most effective. We also show that the target labeled dataset of only 5 h gives an almost saturated performance.

Noise estimation based on optimal smoothing and minimum controlled through recursive averaging for speech enhancement

One of the most significant challenges in the real time speech processing applications is the elimination of noise in the corrupted speech data. This noise can significantly impact the efficacy/performance of the applications of speech processing. However, developing a robust noise reduction algorithm is crucial for improving the accuracy of automatic speech recognition (ASR) and other speech processing systems under uncontrolled conditions. In this paper, we propose an algorithm to reduce the background noise in the degraded speech data under highly non-stationary conditions. The proposed optimal smoothing and minima controlled (OSMC) technique uses recursive averaging to enhance degraded speech data. Initially, a smoothed periodogram and local minima of the degraded speech data are computed and determined the time-frequency dependent threshold factor. The ratio of smoothed periodogram to local minima is used to find the active regions of speech in the degraded speech data by adapting the Bayesian minimum cost decision rule. To calculate the estimated noise spectrum for each frequency bin, a time-frequency smoothing factors are used. The perceptual evaluation of speech quality (PESQ) and normalized covariance metric (NCM) are used to evaluate the speech quality and intelligibility of the proposed technique over competing algorithms after speech enhancement. The experimental results demonstrated that the proposed algorithm has given a significant improvement in terms of average values of PESQ by 15.03% and 16.71% and NCM by 3.45% and 5.73% for NOIZEUS and Kannada speech databases at 5 dB and 10 dB respectively, over unprocessed speech under highly non-stationary noisy environments.

Feedback From Automatic Speech Recognition to Elicit Clear Speech in Healthy Speakers.

This study assessed the effectiveness of feedback generated by automatic speech recognition (ASR) for eliciting clear speech from young, healthy individuals. As a preliminary step toward exploring a novel method for eliciting clear speech in patients with dysarthria, we investigated the effects of ASR feedback in healthy controls. If successful, ASR feedback has the potential to facilitate independent, at-home clear speech practice. Twenty-three healthy control speakers (ages 23-40 years) read sentences aloud in three speaking modes: Habitual, Clear (over-enunciated), and in response to ASR feedback (ASR). In the ASR condition, we used Mozilla DeepSpeech to transcribe speech samples and provide participants with a value indicating the accuracy of the ASR's transcription. For speakers who achieved sufficiently high ASR accuracy, noise was added to their speech at a participant-specific signal-to-noise ratio to ensure that each participant had to over-enunciate to achieve high ASR accuracy. Compared to habitual speech, speech produced in the ASR and Clear conditions was clearer, as rated by speech-language pathologists, and more intelligible, per speech-language pathologist transcriptions. Speech in the Clear and ASR conditions aligned on several acoustic measures, particularly those associated with increased vowel distinctiveness and decreased speaking rate. However, ASR accuracy, intelligibility, and clarity were each correlated with different speech features, which may have implications for how people change their speech for ASR feedback. ASR successfully elicited outcomes similar to clear speech in healthy speakers. Future work should investigate its efficacy in eliciting clear speech in people with dysarthria.

ENHANCING NON-NATIVE ACCENT RECOGNITION THROUGH A COMBINATION OF SPEAKER EMBEDDINGS, PROSODIC AND VOCAL SPEECH FEATURES

The transcription accuracy of automatic speech recognition (ASR) system may suffer when recognizing accented speech. The resulting bias in ASR system towards a specific accent due to under representation of that accent in the training dataset. Accent recognition of existing speech samples can help with the preparation of the training datasets, which is an important step toward closing the accent gap and eliminating biases in ASR system. For that we built a system to recognize accent from spoken speech data. In this study, we have explored some prosodic and vocal speech features as well as speaker embeddings for accent recognition on our custom English speech data that covers speakers from around the world with varying accents. We demonstrate that our selected speech features are more effective in recognizing nonnative accents. Additionally, we experimented with a hierarchical classification model for multi-level accent classification. To establish an accent hierarchy, we employed a bottom-up approach, combining regional accents and categorizing them as either native or non-native at the top level. Furthermore, we conducted a comparative study between flat classification and hierarchical classification using the accent hierarchy structure.

Automatic Fluency Assessment Method for Spontaneous Speech without Reference Text

The automatic fluency assessment of spontaneous speech without reference text is a challenging task that heavily depends on the accuracy of automatic speech recognition (ASR). Considering this scenario, it is necessary to explore an assessment method that combines ASR. This is mainly due to the fact that in addition to acoustic features being essential for assessment, the text features output by ASR may also contain potentially fluency information. However, most existing studies on automatic fluency assessment of spontaneous speech are based solely on audio features, without utilizing textual information, which may lead to a limited understanding of fluency features. To address this, we propose a multimodal automatic speech fluency assessment method that combines ASR output. Specifically, we first explore the relevance of the fluency assessment task to the ASR task and fine-tune the Wav2Vec2.0 model using multi-task learning to jointly optimize the ASR task and fluency assessment task, resulting in both the fluency assessment results and the ASR output. Then, the text features and audio features obtained from the fine-tuned model are fed into the multimodal fluency assessment model, using attention mechanisms to obtain more reliable assessment results. Finally, experiments on the PSCPSF and Speechocean762 dataset suggest that our proposed method performs well in different assessment scenarios.

Finnish parliament ASR corpus: Analysis, benchmarks and statistics.

Public sources like parliament meeting recordings and transcripts provide ever-growing material for the training and evaluation of automatic speech recognition (ASR) systems. In this paper, we publish and analyse the Finnish Parliament ASR Corpus, the most extensive publicly available collection of manually transcribed speech data for Finnish with over 3000h of speech and 449 speakers for which it provides rich demographic metadata. This corpus builds on earlier initial work, and as a result the corpus has a natural split into two training subsets from two periods of time. Similarly, there are two official, corrected test sets covering different times, setting an ASR task with longitudinal distribution-shift characteristics. An official development set is also provided. We developed a complete Kaldi-based data preparation pipeline and ASR recipes for hidden Markov models (HMM), hybrid deep neural networks (HMM-DNN), and attention-based encoder-decoders (AED). For HMM-DNN systems, we provide results with time-delay neural networks (TDNN) as well as state-of-the-art wav2vec 2.0 pretrained acoustic models. We set benchmarks on the official test sets and multiple other recently used test sets. Both temporal corpus subsets are already large, and we observe that beyond their scale, HMM-TDNN ASR performance on the official test sets has reached a plateau. In contrast, other domains and larger wav2vec 2.0 models benefit from added data. The HMM-DNN and AED approaches are compared in a carefully matched equal data setting, with the HMM-DNN system consistently performing better. Finally, the variation of the ASR accuracy is compared between the speaker categories available in the parliament metadata to detect potential biases based on factors such as gender, age, and education.

Estimate the noise effect on automatic speech recognition accuracy for mandarin by an approach associating articulation index

The automatic speech recognition (ASR) technology has become one of the fast-growing engineering technologies. The ASR accuracy is often reduced by environmental interference noise. Many studies attempted to improve the ASR word accuracy in noisy backgrounds, but few were concerned about estimating the influence of noise on the word accuracy of ASR systems. This study investigated the effect of stationary noise on the word accuracy for mandarin of two ASR systems (i.e., the Baidu Cloud and Microsoft Azure). We selected the speech material from an open Mandarin speech database. The interference noise contained pink noise, white noise, random noise with equivalent power spectrum to the average power spectrum of mandarin, band-stop filtered noise and several typical household appliances noise (i.e., hairdryer, range hood, blender and water purifier). The signal-to-noise ratios (SNRs) were set to the range −12–3 dB. The Azure ASR system was less affected by noise and tended to have a higher word accuracy under the same noise condition than the Baidu ASR system. We proposed an approach to associate the articulation index (AI) of interference noise with the speech recognition accuracy of ASR systems. A three-parameter logistic model could be established for a specific ASR system, and each parameter is linearly dependent on the AI of interference noise. The performance of ASR systems under different stationary noise conditions could be quantitatively assessed and compared through the approach. Different models should be established for different versions of the ASR systems through this AI-associating approach since the parameters of one model are only applicable to one particular version of an ASR system.

Gauging the accuracy of automatic speech data harvesting in five under-resourced languages

Recent research on deep-learning architectures has resulted in substantial improvements in automatic speech recognition accuracy. The leaps of progress made in well-resourced languages can be attributed to the fact that these architectures are able to effectively represent spoken language in all its diversity and complexity. However, developing advanced models of a language without appropriate corpora of speech and text data remains a challenge. For many under-resourced languages, including those spoken in South Africa, such resources simply do not exist. The aim of the work reported on in this paper is to address this situation by investigating the possibility to create diverse speech resources from unannotated broadcast data. The paper describes how existing speech and text resources were used to develop a semi-automatic data harvesting procedure for two genres of broadcast data, namely news bulletins and radio dramas. It was found that adapting acoustic models with less than 10 hours of manually annotated data from the same domain significantly reduced transcription error rates for speaking styles and acoustic conditions that are not represented in any of the existing speech corpora. Results also indicated that much more automatically transcribed adaptation data is required to achieve similar results.

Dysarthric Speech Enhancement Based on Convolution Neural Network.

Generally, those patients with dysarthria utter a distorted sound and the restrained intelligibility of a speech for both human and machine. To enhance the intelligibility of dysarthric speech, we applied a deep learning-based speech enhancement (SE) system in this task. Conventional SE approaches are used for shrinking noise components from the noise-corrupted input, and thus improve the sound quality and intelligibility simultaneously. In this study, we are focusing on reconstructing the severely distorted signal from the dysarthric speech for improving intelligibility. The proposed SE system prepares a convolutional neural network (CNN) model in the training phase, which is then used to process the dysarthric speech in the testing phase. During training, paired dysarthric-normal speech utterances are required. We adopt a dynamic time warping technique to align the dysarthric-normal utter-ances. The gained training data are used to train a CNN - based SE model. The proposed SE system is evaluated on the Google automatic speech recognition (ASR) system and a subjective listening test. The results showed that the proposed method could notably enhance the recognition performance for more than 10% in each of ASR and human recognitions from the unprocessed dysarthric speech. Clinical Relevance- This study enhances the intelligibility and ASR accuracy from a dysarthria speech to more than 10.

Code of Ethics for Automatic Speech Recognition

Automatic Speech Recognition (ASR), which turns human speech into text, has had a long history of development and continues to improve and expand. Both academic and industry discussions on the technology present ethical issues regarding the privacy, security, and use of user data, as well the accuracy of ASR for different demographics. This paper reviews the literature on such ethical issues and proposes a code of ethics that addresses them. A case study on Google ASR products is conducted, evaluating that the industry still has improvements to make, especially in user data collection and use, and in accuracy.

End-to-End Speech Recognition of Tamil Language

Research in speech recognition is progressing with numerous state-of-the-art results in recent times. However, relatively fewer research is being carried out in Automatic Speech Recognition (ASR) for languages with low resources. We present a method to develop speech recognition model with minimal resources using Mozilla DeepSpeech architecture. We have utilized freely available online computational resources for training, enabling similar approaches to be carried out for research in a low-resourced languages in a financially constrained environments. We also present novel ways to build an efficient language model from publicly available web resources to improve accuracy in ASR. The proposed ASR model gives the best result of 24.7% Word Error Rate (WER), compared to 55% WER by Google speech-to-text. We have also demonstrated a semi-supervised development of speech corpus using our trained ASR model, indicating a cost effective approach of building large vocabulary corpus for low resource language. The trained Tamil ASR model and the training sets are released in public domain and are available on GitHub.

Alleviating ASR Long-Tailed Problem by Decoupling the Learning of Representation and Classification

Recently, we have witnessed excellent improvement of end-to-end (E2E) automatic speech recognition (ASR). However, how to tackle the long-tailed data distribution problem while maintaining E2E ASR models' performance for high-frequency tokens is still challenging. To solve this challenge, we propose a novel decoupled ASR learning method for the sequence-to-sequence ASR architecture in this paper. Our method decouples the learning procedure of this model into two stages: representation learning and classification learning. In the representation learning stage, we use the encoder output of a pretrained language model as one of the ASR model’s learning targets, and propose threshold log cosine embedding loss (TLCE-loss) as the objective function. A frequency-mask cross-entropy loss (FMCE-loss) is also designed as an auxiliary loss. In the classification learning stage, we find that introducing a temperature into softmax function helps reduce the influence of negative samples on tail classes, thus mitigating the biased learning process for the classifier. Furthermore, we propose a weighted softmax (w-softmax) to adjust ASR posterior probabilities according to the token appearing frequency during inference. Additionally, we introduce tail word/character error rate (TWER / TCER) and head word/character error rate (HWER / HCER) that respectively evaluate the ASR accuracy for tail and head words/characters. Experimental results on the Switchboard and HKUST corpora show that our proposed method greatly outperforms the baseline, especially in TWER / TCER reduction. To the best of our knowledge, this is the first work to use a decoupled ASR learning method to alleviate the long-tailed problem in sequence-to-sequence ASR.

ETEH: Unified Attention-Based End-to-End ASR and KWS Architecture

Even though attention-based end-to-end (E2E) automatic speech recognition (ASR) models have been yielding state-of-the-art recognition accuracy, they still fall behind many of the ASR models deployed in the industry in some crucial functionalities such as online processing and precise timestamps generating. This weakness prevents attention-based E2E ASR models from being applied in several essential speech tasks, such as online speech recognition and keyword searching (KWS). In this paper, we describe our proposed unified attention-based E2E ASR and KWS architecture–ETEH, which supports, in one model, both online and offline ASR decoding modes, thus allowing for precise and reliable KWS. “ETE” stands for attention-based E2E modeling, whereas “H” represents the hybrid gaussian mixture model and hidden Markov model (GMM-HMM). As a combination of both, ETEH is an attention-based E2E ASR architecture which utilizes the frame-wise time alignment (FTA) generated by GMM-HMM ASR models. This FTA is used to better the model in two ways: first, it helps the monotonic attentions of ETEH models to capture more accurate word time stamps, thus resulting in lower latency for online decoding; second, it helps ETEH models to provide accurate and reliable KWS results. Furthermore, we are able to combine both offline and online modes in one ETEH model and establish a concise system by adopt the universal training strategy. ETEH is functional and unique, and to the best of our knowledge, we can hardly find a comparable single attention-based E2E ASR system as the baseline. To evaluate ASR accuracy and latency for ETEH, we use our previously proposed monotonic truncated attention (MTA) based online CTC/attention (OCA) ASR models as baselines. Experimental results show that ETEH ASR models gain significant improvement in ASR latency compared to the baseline. To evaluate KWS performance, we compare ETEH models with CTC-based KWS models. Results demonstrate that our ETEH models achieve significantly better KWS performance compared to the CTC baselines.

Recent Advances in End-to-End Automatic Speech Recognition

Recently, the speech community is seeing a significant trend of moving from deep neural network based hybrid modeling to end-to-end (E2E) modeling for automatic speech recognition (ASR). While E2E models achieve the state-of-the-art results in most benchmarks in terms of ASR accuracy, hybrid models are still used in a large proportion of commercial ASR systems at the current time. There are lots of practical factors that affect the production model deployment decision. Traditional hybrid models, being optimized for production for decades, are usually good at these factors. Without providing excellent solutions to all these factors, it is hard for E2E models to be widely commercialized. In this paper, we will overview the recent advances in E2E models, focusing on technologies addressing those challenges from the industry's perspective.

Robot Ego‐Noise Suppression with Labanotation‐Template Subtraction

AbstractIn this study, we aim to improve automatic‐speech‐recognition (ASR) accuracy in the presence of robot ego‐noise toward a better human‐robot interaction. Although several noise reduction methods have been proposed to increase ASR accuracy or signal‐to‐noise ratio (SNR) by predicting ego‐noises through a short‐time motion‐template subtraction or a neural network, these methods showed poor performance in some practical use cases, such as attenuating long‐term motion‐associated ego‐noise. Based on the motion‐template subtraction method, we address the problem of creating ego‐noise templates associated with a wide variety of robot motions. For representing robot motions, we employ a dance notation referred to as Labanotation. The rationales behind our approach are: (i) Labanotation allows quantizing infinite motion patterns using a finite number of Labanotation combinations; (ii) Labanotation‐based motion description is hardware‐independent; and (iii) long‐time noise templates facilitate the localization of noise templates in a speech‐with‐noise signal compared to short‐time templates. The effectiveness of the Labanotation‐template subtraction (LTS) method was tested for five commercial ASRs in terms of ASR accuracy, SNR, and source‐to‐distortion ratio. We show that LTS leads to a reasonable performance, comparable to the other methods. The contribution of this study is (i) to propose to use Labanotation to reasonably collect noise templates, (ii) to demonstrate the practical effectiveness of LTS as well as examples of Labanotations for household actions. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.