Performance Of Speech Recognition Systems Research Articles

Evaluating OpenAI's Whisper ASR: Performance analysis across diverse accents and speaker traits.

This study investigates Whisper's automatic speech recognition (ASR) system performance across diverse native and non-native English accents. Results reveal superior recognition in American compared to British and Australian English accents with similar performance in Canadian English. Overall, native English accents demonstrate higher accuracy than non-native accents. Exploring connections between speaker traits [sex, native language (L1) typology, and second language (L2) proficiency] and word error rate uncovers notable associations. Furthermore, Whisper exhibits enhanced performance in read speech over conversational speech with modifications based on speaker gender. The implications of these findings are discussed.

Speech enhancement augmentation for robust speech recognition in noisy environments

Abstract. The use of augmentations as a data enrichment method has become an important element in improving the performance of speech recognition systems. To work effectively in noisy conditions, augmentation is usually used to simulate the presence of background noise. However, the quality of speech recognition on samples pre-processed by noise reduction models does not increase. This paper proposes a new approach to speech data augmentation when training ASR systems, intended for their joint use with models for speech enhancement. It was based on the creation of several additional data samples containing speech samples processed by the enhancement model. The proposed approach was tested on the E-Branchformer neural network model using data from the Librispeech set. The quality of speech samples was assessed using the DNSMOS metric. By means of a 100-hour sample of clean speech samples it was shown that the proposed augmentation allows for an improvement in the WER metric of more than 4% in absolute value compared to the generally accepted approach based on adding noisy speech samples. Experiments on 960-hour data demonstrated the robustness of this approach as the training set size increased.

Crossing language identification: Multilingual ASR framework based on semantic dataset creation & Wav2Vec 2.0

This study proposes an innovative methodology to enhance the performance of multilingual Automatic Speech Recognition (ASR) systems by capitalizing on the high semantic similarity between sentences across different languages and eliminating the requirement for Language Identification (LID). To achieve this, special bilingual datasets were created from the Mozzila Common Voices datasets in Spanish, Russian, and Portuguese. The process involves computing sentence embeddings using Language-agnostic BERT and selecting sentence pairs based on high and low cosine similarity. Subsequently, we train the Wav2vec 2.0 XLSR53 model on these datasets and assess its performance utilizing Character Error Rate (CER) and Word Error Rate (WER) metrics. The experimental results indicate that models trained on high-similarity samples consistently surpass their low-similarity counterparts, emphasizing the significance of high semantic similarity data selection for precise and dependable ASR performance. Furthermore, the elimination of LID contributes to a simplified system with reduced computational costs and the capacity for real-time text output. The findings of this research offer valuable insights for the development of more efficient and accurate multilingual ASR systems, particularly in real-time and on-device applications.

Interactive System for Gender Classification

Abstract: A study on the detection of gender, based on voice using artificial neural networks. In today’s fast-moving world Gender classification through voice plays an important role to enhance performance in Speech recognition systems, Forensic investigation, and Marketing. The dataset has 3,168 recorded voice samples of male and female voices. The samples are produced by using acoustic analysis. The primary goal of the proposed model is to automate the system to identify gender based on the audio signals and to test the voice of a human on the spot. Multilayer Perceptron (MLP) with ReLU activation function as a model has been trained to predict gender. Nadam optimizer is used for the optimization of neural networks, K-Nearest Neighbor and Support Vector Machines are trained on the dataset of 3,168 records. The obtained best accuracy is 97% on the given dataset by MLP algorithm. An Interactive web page has been built to test the voice without interruption and to predict its gender. 125 real time samples are tested out of which model could classify every record into male/female. 19 male records are incorrectly classified.

Quantifying and Improving the Performance of Speech Recognition Systems on Dysphonic Speech.

This study seeks to quantify how current speech recognition systems perform on dysphonic input and if they can be improved. Experimental machine learning methods based on a retrospective database. Single academic voice center. A database of dysphonic speech recordings was created and tested against 3 speech recognition platforms. Platform performance on dysphonic voice input was compared to platform performance on normal voice input. A custom speech recognition model was trained on voice from patients with spasmodic dysphonia or vocal cord paralysis. Custom model performance was compared to base model performance. All platforms performed well on normal voice, and 2 platforms performed significantly worse on dysphonic speech. Accuracy metrics on dysphonic speech returned values of 84.55%, 88.57%, and 93.56% for International Business Machines (IBM) Watson, Amazon Transcribe, and Microsoft Azure, respectively. The secondary analysis demonstrated that the lower performance of IBM Watson and Amazon Transcribe was driven by performance on spasmodic dysphonia and vocal fold paralysis. Thus, a custom model was built to increase the accuracy of these pathologies on the Microsoft platform. Overall, the performance of the custom model on dysphonic voices was 96.43% and on normal voices was 97.62%. Current speech recognition systems generally perform worse on dysphonic speech than on normal speech. We theorize that poor performance is a consequence of a lack of dysphonic voices in each platform's original training dataset. We address this limitation with transfer learning used to increase the performance of these systems on all dysphonic speech.

Dysarthria severity assessment using squeeze-and-excitation networks

Automated dysarthria severity identification can aid clinicians in monitoring the patient’s progress, and can improve the performance of dysarthric speech recognition systems. In this paper, we explore the potency of squeeze-and-excitation (SE) networks for dysarthria severity level classification using mel spectrograms. Deep convolutional neural network (CNN) models built using SE components and residual blocks are analysed in this regard. A comparison of the system performance is done against a shallow CNN and a convolutional recurrent neural network built using a bidirectional long short-term memory network. The models are evaluated on the Universal Access dysarthric speech corpus under speaker-dependent (SD) and speaker-independent (SID) scenarios. The proposed model obtained appreciable results in the SD scenario, and its margin of about 10% over the baselines in the SID scenario promises its applicability in automated dysarthria severity level classification.

PITCH ESTIMATION FRAMEWORK FOR SPEECH SEGREGATION USING COCHLEAGRAM MORPHING

Computational auditory scene analysis (CASA) has significant role in speech segregation from monaural audio mixtures and generally a measure for performance of speech recognition systems. Pitch estimation has a substantial role in performance of CASA systems. This study presents a novel pitch estimation framework for speech segregation from monaural audio mixtures using cochleagram morphing. The proposed framework takes the rough estimation of target pitch from given audio mixtures containing speech and background interferences. Discrete set consisting morphed versions of cochleagram is obtained using k-Means clustering. The estimated pitch values are improved by validating and smoothing them to morphed cochleagram. Measure of refined estimated pitch contours along with harmonicity and temporal continuity are used to segregate target speech. The proposed framework produced 83.13% accuracy for MIR-1k dataset which is considerably higher than the existing methods.

Accurate Recognition of Natural language Using Machine Learning and Feature Fusion Processing

To enhance the performance of Chinese language pronunciation evaluation and speech recognition systems, researchers are focusing on developing intelligent techniques for multilevel fusion processing of data, features, and decisions using deep learning-based computer-aided systems. With a combination of score level, rank level, and hybrid level fusion, as well as fusion optimization and fusion score improvement, these systems can effectively combine multiple models and sensors to improve the accuracy of information fusion. Additionally, intelligent systems for information fusion, including those used in robotics and decision-making, can benefit from techniques such as multimedia data fusion and machine learning for data fusion. Furthermore, optimization algorithms and fuzzy approaches can be applied to data fusion applications in cloud environments and e-systems, while spatial data fusion can be used to enhance the quality of image and feature data In this paper, a new approach has been presented to identify the tonal language in continuous speech. This study proposes the Machine learning-assisted automatic speech recognition framework (ML-ASRF) for Chinese character and language prediction. Our focus is on extracting highly robust features and combining various speech signal sequences of deep models. The experimental results demonstrated that the machine learning neural network recognition rate is considerably higher than that of the conventional speech recognition algorithm, which performs more accurate human-computer interaction and increases the efficiency of determining Chinese language pronunciation accuracy.

Silent Speech Recognition Based on Surface Electromyography Using a Few Electrode Sites Under the Guidance From High-Density Electrode Arrays

Although surface electromyogram recorded from high-density electrode array is believed to carry sufficient spatial information that can benefit the decoding of motor intentions, the complexity of using the array hindered its widespread applications especially in wearable devices. This study is aimed to develop a non-acoustic modality of silent speech recognition that transfers knowledge learned from high-density array to a system using a few channels, with both high portability and performance. A convolutional neural network was established for recognizing a vocabulary of 33 Chinese words during subvocal speech production. The network was trained by the data recorded from face and neck muscles using two arrays with 64 channels in the source domain. Then it was calibrated through a transfer learning approach to grant its adaption to a new target domain with the data recorded by 8 separated electrodes, while its good capability of characterizing subvocal speech word patterns is expected to be maintained. The proposed method significantly outperformed three common classification approaches and the baseline approach without transfer learning (a network trained with data just from the target domain). Under conditions of electrode shift and cross-user variability, it still obtained performance improvements. The method is demonstrated to be viable for transfer learning across domains of electrode settings and it facilitates to improve the performance of silent speech recognition systems using separate electrode sites under the guidance from high-density of arrays.

Automatic Speech Recognition for Uyghur, Kazakh, and Kyrgyz: An Overview

With the emergence of deep learning, the performance of automatic speech recognition (ASR) systems has remarkably improved. Especially for resource-rich languages such as English and Chinese, commercial usage has been made feasible in a wide range of applications. However, most languages are low-resource languages, presenting three main difficulties for the development of ASR systems: (1) the scarcity of the data; (2) the uncertainty in the writing and pronunciation; (3) the individuality of each language. Uyghur, Kazakh, and Kyrgyz as examples are all low-resource languages, involving clear geographical variation in their pronunciation, and each language possesses its own unique acoustic properties and phonological rules. On the other hand, they all belong to the Altaic language family of the Altaic branch, so they share many commonalities. This paper presents an overview of speech recognition techniques developed for Uyghur, Kazakh, and Kyrgyz, with the purposes of (1) highlighting the techniques that are specifically effective for each language and generally effective for all of them and (2) discovering the important factors in promoting the speech recognition research of low-resource languages, by a comparative study of the development path of these three neighboring languages.

Ask2Mask: Guided Data Selection for Masked Speech Modeling

Masked speech modeling (MSM) methods such as wav2vec2 or w2v-BERT learn representations over speech frames which are randomly masked within an utterance. While these methods improve performance of Automatic Speech Recognition (ASR) systems, they have one major limitation. They treat all unsupervised speech samples with equal weight, which hinders learning as not all samples have relevant information to learn meaningful representations. In this work, we address this limitation. We propose ask2mask (ATM), a novel approach to focus on specific samples during MSM pre-training. ATM employs an external ASR model or <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scorer</i> to weight unsupervised input samples in two different ways: 1) A fine-grained data selection is performed by masking over the highly confident input frames as chosen by the scorer. This allows the model to learn meaningful representations. 2) ATM is further extended to focus at utterance-level by weighting the final MSM loss with the utterance-level confidence score. We conduct fine-tuning experiments on two well-benchmarked corpora: LibriSpeech (matching the pre-training data) and Commonvoice, TED-LIUM, AMI and CHiME-6 (not matching the pre-training data). The results substantiate the efficacy of ATM on significantly improving the recognition performance under mismatched conditions (up to 11.6% relative over published results and upto 4.46% relative over our internal baseline) while still yielding modest improvements under matched conditions.

Improving Hybrid CTC/Attention Architecture for Agglutinative Language Speech Recognition.

Unlike the traditional model, the end-to-end (E2E) ASR model does not require speech information such as a pronunciation dictionary, and its system is built through a single neural network and obtains performance comparable to that of traditional methods. However, the model requires massive amounts of training data. Recently, hybrid CTC/attention ASR systems have become more popular and have achieved good performance even under low-resource conditions, but they are rarely used in Central Asian languages such as Turkish and Uzbek. We extend the dataset by adding noise to the original audio and using speed perturbation. To develop the performance of an E2E agglutinative language speech recognition system, we propose a new feature extractor, MSPC, which uses different sizes of convolution kernels to extract and fuse features of different scales. The experimental results show that this structure is superior to VGGnet. In addition to this, the attention module is improved. By using the CTC objective function in training and the BERT model to initialize the language model in the decoding stage, the proposed method accelerates the convergence of the model and improves the accuracy of speech recognition. Compared with the baseline model, the character error rate (CER) and word error rate (WER) on the LibriSpeech test-other dataset increases by 2.42% and 2.96%, respectively. We apply the model structure to the Common Voice—Turkish (35 h) and Uzbek (78 h) datasets, and the WER is reduced by 7.07% and 7.08%, respectively. The results show that our method is close to the advanced E2E systems.

Towards improving speech recognition model with post-processing spell correction using BERT

Speech recognition has now become ubiquitous and plays an inevitable role in almost all sectors. Numerous works have been proposed on speech recognition; however, more accurate transcriptions are not possible. Exploration of various studies related to spell correction implies that several kinds of research have been carried out in this field but still it is a very challenging problem. This led to the need for a new spell corrector framework capable of leveraging the performance of the automatic speech recognition (ASR) system. The proposed work unveils state-of-the-art Bidirectional Encoder Representations from Transformers (BERT) based spell correction module developed on top of the deep recurrent neural network (RNN) based ASR system. The impact of BERT-based spell correction on the ASR system is evaluated on three different accent datasets in the perspective of word error rate (WER), character error rate (CER), and Bilingual evaluation understudy (BLEU) score. The experimental results inferred that the enhanced spell correction module is efficacious in detecting and correcting spell errors, by achieving the WER of 5.025% on librispeech corpus, 6.35% on voxforge, and 7.05% on NPTEL corpus.

Modeling and Simulation of English Speech Optimization Teaching Recognition Based on Intelligent Edge Detection Algorithm

English speech modeling is one of the key problems in the field of speech recognition. Its accuracy directly affects the performance of the English speech recognition system, and how to establish a more accurate acoustic model has always been the focus of researchers. This paper is based on the intelligent edge detection algorithm of the English speech optimization teaching recognition modeling simulation analysis to improve the accuracy of acoustic models such as parameters and the performance of continuous speech recognition system as the main purpose. In this paper, the accuracy of the neural network is improved on the premise of improving the training speed and decoding speed of the model. It proposes a new model and how to use the intelligent edge detection algorithm to build a complete English speech optimization teaching recognition system. The whole system includes the mobile terminal and the server, which realizes the most basic business logic of the speech recognition system. The experimental results of this paper show that from the point of view of the average recognition rate, the recognition effect of the optimized feature set has been further improved compared with the fusion feature. From the point of view of the recognition rate under different SNR environments, the recognition rate of the optimized feature set PCA-Features2 decreased by 0.47% compared with the FFPRLS_D + TEOCC feature set under 10dB10 words. Compared with the FFPLMS_D + TEOCC feature set, the recognition rate under 5dB10 words also drops by 0.47%.

A study of transformer-based end-to-end speech recognition system for Kazakh language

Today, the Transformer model, which allows parallelization and also has its own internal attention, has been widely used in the field of speech recognition. The great advantage of this architecture is the fast learning speed, and the lack of sequential operation, as with recurrent neural networks. In this work, Transformer models and an end-to-end model based on connectionist temporal classification were considered to build a system for automatic recognition of Kazakh speech. It is known that Kazakh is part of a number of agglutinative languages and has limited data for implementing speech recognition systems. Some studies have shown that the Transformer model improves system performance for low-resource languages. Based on our experiments, it was revealed that the joint use of Transformer and connectionist temporal classification models contributed to improving the performance of the Kazakh speech recognition system and with an integrated language model it showed the best character error rate 3.7% on a clean dataset.

Spectral warping and data augmentation for low resource language ASR system under mismatched conditions

The performance of an Automatic Speech Recognition System (ASR) system deteriorates while using it on children speech, due to large variations and mismatch of acoustic and linguistic variables between spoken utterances of adults and children. Another important reason for the low efficiency of ASR models is the data scarcity of children speech data for low resource-language like Punjabi. The proposed work in this paper tries to address the both challenges i.e. acoustic and linguistic variations challenge, and data scarcity problem, thereby improves performance of a children speech ASR system for Punjabi language. To handle the first issue of acoustic and linguistic variations, the proposed work uses formant modification as a spectral warping technique to reduce the variation between children speech and adult speech. Further, to address the second issue of data scarcity, this paper discusses training of ASR models on augmented children speech data. Also, the work combines well established Mel-Frequency Cepstral Coefficients (MFCC) features extraction technique with Frequency Domain Linear Prediction (FDLP) to propose MFCC-FDLP hybrid approach for front end feature extraction. For implementing the data augmentation, Tacotron 2, an end-to-end Text to Speech (TTS) generative model has been used. The proposed work uses MFCC, FDLP and hybrid MFCC + FDLP techniques for front end feature extraction, Time Delay Neural Network (TDNN) for backend acoustic modeling, and trigram language model to implement continuous Punjabi language ASR systems. To increase robustness of the proposed ASR system, we have included a batch of lexically diverse words in our pronunciation model which achieved a relative improvement of 29.44%.

An Innovative Approach Utilizing Binary-View Transformer for Speech Recognition Task

The deep learning advancements have greatly improved the performance of speech recognition systems, and most recent systems are based on the Recurrent Neural Network (RNN). Overall, the RNN works fine with the small sequence data, but suffers from the gradient vanishing problem in case of large sequence. The transformer networks have neutralized this issue and have shown state-of-the-art results on sequential or speech-related data. Generally, in speech recognition, the input audio is converted into an image using Mel-spectrogram to illustrate frequencies and intensities. The image is classified by the machine learning mechanism to generate a classification transcript. However, the audio frequency in the image has low resolution and causing inaccurate predictions. This paper presents a novel end-to-end binary view transformer-based architecture for speech recognition to cope with the frequency resolution problem. Firstly, the input audio signal is transformed into a 2D image using Mel-spectrogram. Secondly, the modified universal transformers utilize the multi-head attention to derive contextual information and derive different speech-related features. Moreover, a feed-forward neural network is also deployed for classification. The proposed system has generated robust results on Google's speech command dataset with an accuracy of 95.16% and with minimal loss. The binary-view transformer eradicates the eventuality of the over-fitting problem by deploying a multi-view mechanism to diversify the input data, and multi-head attention captures multiple contexts from the data's feature map.

A formant modification method for improved ASR of children’s speech

Differences in acoustic characteristics between children’s and adults’ speech degrade performance of automatic speech recognition systems when systems trained using adults’ speech are used to recognize children’s speech. This performance degradation is due to the acoustic mismatch between training and testing. One of the main sources of the acoustic mismatch is the difference in vocal tract resonances (formant frequencies) between adult and child speakers. The present study aims to reduce the mismatch in formant frequencies by modifying formants of children’s speech to better correspond to formants of adults’ speech. This is carried out by warping the linear prediction (LP) spectrum computed from children’s speech. The warped LP spectra computed in a frame-based manner from children’s speech are used with the corresponding LP residuals to synthesize speech whose formant structure is closer to that of adults’ speech. When used in testing of an ASR system trained using adults’ speech, the warping reduces the spectral mismatch in speech between training and testing and improves the system performance in recognition of children’s speech. Experiments were conducted using narrowband (8 kHz) and wideband (16 kHz) speech of adult and child speakers from the WSJCAM0 and PF_STAR databases, respectively, and by recognizing children’s speech using acoustic models trained with adults’ speech. The proposed method gave relative improvements of 24% and 11% for the DNN and TDNN acoustic models, respectively, for narrowband speech. For wideband speech, the technique gave relative improvements of 27% and 13% for the DNN and TDNN acoustic models, respectively. The performance of the proposed method was also compared to two speaker adaptation methods: vocal tract length normalization (VTLN) and speaking rate adaptation (SRA). This comparison showed the best recognition performance for the proposed method. We also combined the proposed method with VTLN and SRA, and found that the combined method gave a further reduction in WER. Moreover, our experiments carried out for noisy speech using various types of additive noise and signal-to-noise ratios showed that the proposed method performs well also for degraded speech.

Effect of Time-domain Windowing on Isolated Speech Recognition System Performance

Effect of Time-domain Windowing on Isolated Speech Recognition System Performance

Performance Evaluation of Offline Speech Recognition on Edge Devices

Deep learning–based speech recognition applications have made great strides in the past decade. Deep learning–based systems have evolved to achieve higher accuracy while using simpler end-to-end architectures, compared to their predecessor hybrid architectures. Most of these state-of-the-art systems run on backend servers with large amounts of memory and CPU/GPU resources. The major disadvantage of server-based speech recognition is the lack of privacy and security for user speech data. Additionally, because of network dependency, this server-based architecture cannot always be reliable, performant and available. Nevertheless, offline speech recognition on client devices overcomes these issues. However, resource constraints on smaller edge devices may pose challenges for achieving state-of-the-art speech recognition results. In this paper, we evaluate the performance and efficiency of transformer-based speech recognition systems on edge devices. We evaluate inference performance on two popular edge devices, Raspberry Pi and Nvidia Jetson Nano, running on CPU and GPU, respectively. We conclude that with PyTorch mobile optimization and quantization, the models can achieve real-time inference on the Raspberry Pi CPU with a small degradation to word error rate. On the Jetson Nano GPU, the inference latency is three to five times better, compared to Raspberry Pi. The word error rate on the edge is still higher, but it is not too far behind, compared to that on the server inference.