Speech Recognition Performance Research Articles

Two-stage deep spectrum fusion for noise-robust end-to-end speech recognition

Recently, speech enhancement (SE) methods have achieved quite good performances. However, because of the speech distortion problem, the enhanced speech may lose significant information, which degrades the performance of automatic speech recognition (ASR). To address this problem, this paper proposes a two-stage deep spectrum fusion with the joint training framework for noise-robust end-to-end (E2E) ASR. It consists of a masking and mapping fusion (MMF) and a gated recurrent fusion (GRF). The MMF is used as the first stage and focuses on SE, which explores the complementarity of the enhancement methods of masking-based and mapping-based to alleviate the problem of speech distortion. The GRF is used as the second stage and aims to further retrieve the lost information by fusing the enhanced speech of MMF and the original input. We conduct extensive experiments on an open Mandarin speech corpus AISHELL-1 with two noise datasets named 100 Nonspeech and NOISEX-92. Experimental results indicate that our proposed method significantly improves the performance and the character error rate (CER) is relatively reduced by 17.36% compared with the conventional joint training method.

Crossing language identification: Multilingual ASR framework based on semantic dataset creation & Wav2Vec 2.0

This study proposes an innovative methodology to enhance the performance of multilingual Automatic Speech Recognition (ASR) systems by capitalizing on the high semantic similarity between sentences across different languages and eliminating the requirement for Language Identification (LID). To achieve this, special bilingual datasets were created from the Mozzila Common Voices datasets in Spanish, Russian, and Portuguese. The process involves computing sentence embeddings using Language-agnostic BERT and selecting sentence pairs based on high and low cosine similarity. Subsequently, we train the Wav2vec 2.0 XLSR53 model on these datasets and assess its performance utilizing Character Error Rate (CER) and Word Error Rate (WER) metrics. The experimental results indicate that models trained on high-similarity samples consistently surpass their low-similarity counterparts, emphasizing the significance of high semantic similarity data selection for precise and dependable ASR performance. Furthermore, the elimination of LID contributes to a simplified system with reduced computational costs and the capacity for real-time text output. The findings of this research offer valuable insights for the development of more efficient and accurate multilingual ASR systems, particularly in real-time and on-device applications.

Early Datalogging Predicts Cochlear Implant Performance: Building a Recommendation for Daily Device Usage.

To quantify the effect of datalogging on speech recognition scores and time to achievement for a "benchmark" level of performance within the first year, and to provide a data-driven recommendation for minimum daily cochlear implant (CI) device usage to better guide patient counseling and future outcomes. Retrospective cohort. Tertiary referral center. Three hundred thirty-seven adult CI patients with data logging and speech recognition outcome data who were implanted between August 2015 and August 2020. Processor datalogging, speech recognition scores, achievement of "benchmark speech recognition performance" defined as 80% of the median score for speech recognition outcomes at our institution. The 1-month datalogging measure correlated positively with word and sentences scores at 1, 3, 6, and 12 months postactivation. Compared with age, sex, and preoperative performance, datalogging was the largest predictive factor of benchmark achievement on multivariate analysis. Each hour/day increase of device usage at 1 month resulted in a higher likelihood of achieving benchmark consonant-nucleus-consonant and AzBio scores within the first year (odds ratio = 1.21, p < 0.001) as well as earlier benchmark achievement. Receiver operating characteristic curve analysis identified the optimal data logging threshold at an average of 12 hours/day. Early CI device usage, as measured by 1-month datalogging, predicts benchmark speech recognition achievement in adults. Datalogging is an important predictor of CI performance within the first year postimplantation. These data support the recommended daily CI processor utilization of at least 12 hours/day to achieve optimal speech recognition performance for most patients.

Relationship between Effective Duration and Intelligibility of Japanese Monosyllables in Individuals with Sensorineural Hearing Loss

Among the temporal elements in the autocorrelation function, the effective duration (τe) is a useful indicator of speech recognition for patients with sensorineural hearing impairment. We assessed the influence of speech recognition performance on the relationship between the percentage of accurately perceived articulation and the median τe (τe-med) and the relationship between monosyllabic confusion and the τe-med. Significant correlations were observed between the articulation percentage and the average τe-med in groups with high, middle, and low speech recognition scores (SRSs). Two-factor mixed analysis of variance revealed significant main effects for the condition (presentation/response). There was no significant main effect for group (high-, middle-, or low-SRS) scores and no significant interaction between the groups. The average τe-med of the response was significantly longer than that of the presentation in all three groups. Monosyllables with short τe-med values tended to be misheard as monosyllables with a long τe-med when confusion occurred. The τe-med was useful for estimating monosyllables that patients with sensorineural hearing impairment find easy to listen to, independent of speech recognition performance.

The Effect of Background Noise and Music on Speech Recognition Performance of Individuals with Normal Hearing and Hearing Loss

This study explores speech recognition characteristics in background noise and music between normal hearing (NH) listeners, hearing aid (HA) users and cochlear implant (CI) users.Sixty individuals participated in the study: 20 with NH, 20 HA users, and 20 CI users. HA and CI users had a Categories of Auditory Performance score of 6 and open set sentence recognition of 85% or higher. They had been using the devices for at least one year. Babble noise (BN), piano solo (PS), piano + violin (PV), and piano + chorus (PC) were presented at +5- and +10 dB signal-to-noise ratio (SNR). The participants were asked to listen and repeat words and sentences from the Korean Standard Sentence List for Adults. At +5- and +10 dB SNRs, CI users performed worse than those with NH on word and sentence recognition in BN, PS, PV, and PC. HA users outperformed CI users in all conditions. Those with NH showed better sentence recognition than HA users across all conditions at +5 dB SNR and better word recognition in PV and BN at +5 dB SNR and in PC at +10 dB SNR. Correlational analysis revealed that the percentage of life with hearing loss before CI was not correlated with sentence and word recognition across all conditions in both SNRs. Statistically significant negative correlations were observed between the duration of deafness and sentence and word recognition in some conditions. Despite individuals with HL performing well on clinical tests, background music can still interfere with communication for those using hearing devices. To accurately evaluate individuals’ communication abilities clinical tools that include background music need to be developed. Studies using different types of music could help develop and standardize such tools for assessing speech and language abilities in individuals with hearing loss.

Audio-visual speech recognition based on joint training with audio-visual speech enhancement for robust speech recognition

Visual features are attractive cues that can be used for robust automatic speech recognition (ASR). In particular, speech recognition performance can be improved by combining audio with visual information obtained from the speaker’s face rather than using only audio in acoustically unfavorable environments. For this reason, recently, studies on various audio-visual speech recognition (AVSR) models have been actively conducted. However, from the experimental results of the AVSR models, important information for speech recognition is mainly concentrated on audio signals, and visual information plays a role in enhancing the robustness of recognition when the audio signal is corrupted in noisy environments. Therefore, there is a limit to the improvement of the recognition performance of conventional AVSR models in noisy environments. Unlike the conventional AVSR models that directly use input audio-visual information as it is, in this paper, we propose an AVSR model that first performs AVSE to enhance target speech based on audio-visual information and then uses both audio information enhanced by the AVSE and visual information such as the speaker’s lips or face. In particular, we propose a deep AVSR model that performs end-to-end training as one model by integrating an AVSR model based on the conformer with hybrid decoding and an AVSE model based on the U-net with recurrent neural network (RNN) attention (RA). Experimental results on the LRS2-BBC and LRS3-TED datasets demonstrate that the AVSE model effectively suppresses corrupting noise and the AVSR model successfully achieves noise robustness. Especially, the proposed jointly trained model integrating the AVSE and AVSR stages into one model showed better recognition performance than the other compared methods.

Effects of Cognitive Functions on Speech Recognition in Noise in Cochlear Implant Recipients

Introduction: There are substantial differences in speech recognition performance of adult cochlear implant (CI) recipients. This study investigated the effects of cognitive function on speech recognition in CI recipients. Methods: The verbal working memory of 36 adults with unilateral CIs was tested using digit span tests. Attention and inhibition abilities were assessed by using the Stroop test (both congruent and incongruent tasks). Speech recognition in noise was measured using the Turkish matrix test. Results: A moderate negative correlation was observed between the critical signal-to-noise ratio obtained via speech recognition in noise test and the digit span test scores (backward and digit span total scores). There was no correlation between Stroop test scores and speech recognition in noise in CI recipients. Conclusion: The findings indicated that verbal working memory correlated well with speech recognition outcomes in adult CI recipients and that higher working memory capacity led to better speech recognition performance in noise.

Bimodal electroencephalography-functional magnetic resonance imaging dataset for inner-speech recognition

The recognition of inner speech, which could give a ‘voice’ to patients that have no ability to speak or move, is a challenge for brain-computer interfaces (BCIs). A shortcoming of the available datasets is that they do not combine modalities to increase the performance of inner speech recognition. Multimodal datasets of brain data enable the fusion of neuroimaging modalities with complimentary properties, such as the high spatial resolution of functional magnetic resonance imaging (fMRI) and the temporal resolution of electroencephalography (EEG), and therefore are promising for decoding inner speech. This paper presents the first publicly available bimodal dataset containing EEG and fMRI data acquired nonsimultaneously during inner-speech production. Data were obtained from four healthy, right-handed participants during an inner-speech task with words in either a social or numerical category. Each of the 8-word stimuli were assessed with 40 trials, resulting in 320 trials in each modality for each participant. The aim of this work is to provide a publicly available bimodal dataset on inner speech, contributing towards speech prostheses.

Effects of the Number of Channels and Channel Stimulation Rate on Speech Recognition and Sound Quality Using Precurved Electrode Arrays.

This study investigated the relationship between the number of active electrodes, channel stimulation rate, and their interaction on speech recognition and sound quality measures while controlling for electrode placement. Cochlear implant (CI) recipients with precurved electrode arrays placed entirely within scala tympani and closer to the modiolus were hypothesized to be able to utilize more channels and possibly higher stimulation rates to achieve better speech recognition performance and sound quality ratings than recipients in previous studies. Participants included seven postlingually deafened adult CI recipients with Advanced Bionics Mid-Scala electrode arrays confirmed to be entirely within scala tympani using postoperative computerized tomography. Twelve conditions were tested using four, eight, 12, and 16 electrodes and channel stimulation rates of 600 pulse per second (pps), 1,200 pps, and each participant's maximum allowable rate (1,245-4,800 pps). Measures of speech recognition and sound quality were acutely assessed. For the effect of channels, results showed no significant improvements beyond eight channels for all measures. For the effect of channel stimulation rate, results showed no significant improvements with higher rates, suggesting that 600 pps was sufficient for maximum speech recognition performance and sound quality ratings. However, across all conditions, there was a significant relationship between mean electrode-to-modiolus distance and all measures, suggesting that a lower mean electrode-to-modiolus distance was correlated with higher speech recognition scores and sound quality ratings. These findings suggest that even well-placed precurved electrode array recipients may not be able to take advantage of more than eight channels or higher channel stimulation rates (> 600 pps), but that closer electrode array placement to the modiolus correlates with better outcomes for these recipients.

Wav2vec‐MoE: An unsupervised pre‐training and adaptation method for multi‐accent ASR

AbstractIn real life, either the subjective factors of speakers or the objective environment degrades the performance of automatic speech recognition (ASR). This study focuses on one of the subjective factors, accented speech, and attempts to build a multi‐accent ASR system to solve the degradation caused by different accents, one of whose characteristic is the low resource. To deal with the challenge of the low‐resource data and the different speech styles, a wav2vec‐MoE (mixture of experts) is proposed to adapt the wav2vec 2.0 for multi‐accent ASR. In the wav2vec‐MoE, a domain MoE is developed by introducing pseudo‐domain information in the pre‐training stage, where the domain denotes a collection of speech varied by the same influence factors. The MoE is trained with two strategies according to the proposed domain mismatch assessment between unlabeled speech and target speech, without requiring any explicit domain information. Experiments show that the wav2vec‐MoE achieves a 14.69% relative word error rate reduction (WERR) on the AESRC2020 accent dataset and an 8.79% relative WERR on the Common Voice English dataset.

Comparison of speech recognition performance with and without a face mask between a basic and a premium hearing aid in hearing-impaired listeners

BackgroundThe mask mandate during the COVID-19 pandemic leads to communication challenges as sound energy gets reduced and the visual cues are lost due to the face mask. This study examines the impact of a face mask on sound energy and compares speech recognition performance between a basic and a premium hearing aid. MethodsParticipants watched four video clips (a female and a male speaker with and without a face mask) and repeated the target sentences in various test conditions. Real-ear measurement was performed to investigate the changes in sound energy in no mask, surgical, and N95 mask conditions. ResultsWith the face mask on, sound energy significantly decreased for all types of masks. For speech recognition, the premium hearing aid showed significant improvement in the mask condition. ConclusionThe findings emphasize and encourage health care professionals to actively use communication strategies, such as speaking slowly and reducing background noise, when interacting with individuals with hearing loss.

Robust speech recognition based on deep learning for sports game review

Abstract To verify the feasibility of robust speech recognition based on deep learning in sports game review. In this paper, a robust speech recognition model is built based on the generative adversarial network GAN algorithm according to the deep learning model. And the loss function, optimization function and noise reduction front-end are introduced in the model to achieve the optimization of speech extraction features through denoising process to ensure that accurate speech review data can be derived even in the game scene under noisy environment. Finally, the experiments are conducted to verify the four directions of the model algorithm by comparing the speech features MFCC, FBANK and WAVE. The experimental results show that the speech recognition model trained by the GSDNet model algorithm can reach 89% accuracy, 56.24% reduction of auxiliary speech recognition word error rate, 92.61% accuracy of speech feature extraction, about 62.19% reduction of training sample data volume, and 94.75% improvement of speech recognition performance in the speech recognition task under noisy environment. It shows that the robust speech recognition based on deep learning can be applied to sports game reviews, and also can provide accurate voice review information from the noisy sports game scene, and also broaden the application area for deep learning models.

COVID-19 Diagnosis System Based on Chest X-ray Images Using Optimized Convolutional Neural Network

It is worth noting that this 21st century has experienced so many economic, social, cultural and political turbulences throughout the world. The 2019 novel coronavirus (COVID-19) outbreak has been regarded by the World Health Organization (WHO) as a public health crisis of global concern. Nowadays, the chest X-ray (CXR) and chest computed tomography (CT) are a more effective imaging technique for diagnosing lung related problems. Deep learning has been more mature in the field of supervised learning, but other areas of machine learning have just started, especially for the areas of unsupervised learning and reinforcement learning. Deep learning has very good performance in speech recognition and image recognition. Using deep learning approaches to diagnose COVID-19 can achieve better cures and treatments. This research presents the data augmentation and L2 regularization approach for transfer learning in several state-of-the-art deep learning models such as VGG16, VGG19, ResNet, and AlexNet, with Convolutional Block Attention Module (CBAM) to perform binary classification ( such as normal and COVID-19/pneumonia cases or COVID-19 and pneumonia cases) and also multi-class classification (such as COVID-19, pneumonia, and normal cases) of covid-chestxray-dataset and NIH datasets. In addition, the performance evaluation adopted the confused matrix to evaluate the results of these models. To sum up, the CBAM can improve the accuracy of all deep learning models to achieve better performance in contrast to that without this architecture. The findings can be a reference for the related COVID-19 diagnosis researches especially during the post-pandemic era.

Effect of cochlear implantation on speech recognition under different listening modes in children

Objective:To investigate speech recognition among children with a unilateral cochlear implant（CI）, bilateral CI, and bimodal hearing and identify the benefits of binaural hearing. Methods:A total of 67 children with severe sensorineural hearing loss who received bilateral CI, bimodal hearing, and unilateral CI from 2012 to 2021 were recruited, including 23 cases with unilateral CI, 25 cases of bimodal hearing, and 19 cases of bilateral CI. The aided hearing threshold at 250-4000 Hz and speech recognition performance of two-syllable words and short sentences in quiet and noisy environments were tested, respectively. Results:There was no significant difference in the hearing threshold results of the bilateral CI group, the bimodal hearing group, and the unilateral CI group by ANOVA（P>0.05）. According to the paired t-test within the group, the speech recognition of disyllabic words and short sentences in quiet and noisy environments were significantly different among the three groups（P<0.05）. There was no significant difference in the speech recognition rates of disyllabic words and short sentences among the three groups in a quiet environment（P>0.05）, but there was a statistically significant difference in a noisy environment（P<0.05）. Additional multiple linear regression analysis showed no significant difference in the recognition rates of disyllabic words and short sentences between the bimodal group and the bilateral CI group in quiet and noisy environments（β=-0.567, P>0.05; β=-0.643, P>0.05）. There was a statistically significant difference in speech recognition of disyllabic words and short sentences between the bimodal group and the unilateral CI group in the noise environment（β=-0.124, P<0.05）, but the difference was not statistically significant in a quiet environment（β=-0.671, P>0.05）; there was a statistically significant difference in the recognition rate of disyllabic words and short sentences between the bilateral CI group and the unilateral CI group in the noise environment（β=-0.226, P<0.05）, but the difference was not statistically significant in a quiet environment（β=-0.341, P>0.05）. Conclusion:Clinicians and guardians are encouraged to provide the most comprehensive audiological evaluation for children with hearing loss. Bilateral CI and bimodal hearing have demonstrated enormous advantages over unilateral CI regarding speech recognition in noise. Bimodal hearing deems a strong recommendation if the contralateral ear has sufficient speech recognition capability. Bilateral CI should be warranted if hearing aids deemed to be ineffective.

Medically related speech: Impacts of age, familiarity, and noise on word recognition

Verbal communication between patients and healthcare providers is an important piece of successful healthcare delivery. Less familiar medical terminology coupled with noisy hospital environments can lead to speech intelligibility challenges between healthcare providers and patients. Prior results showed lower word recognition accuracy for lower familiarity and frequency words and in the presence of noise in younger adults (Bent et al., 2022). The current study expands on the previous work with the inclusion of older adults and an additional noise masking condition. Online listening tests were conducted to study the impacts of word familiarity and background noise on speech perception in younger (age 18–35) and older adults (age 60–85). Participants were presented with a corpus of 160 medically related sentences with varying word familiarity and frequency characteristics, and word recognition accuracy was recorded. Sentences were presented in quiet, hospital noise, speech-shaped noise, and speech-shaped noise modulated by the hospital noise envelope, a novel condition from previous work. Results highlight a comparison of speech recognition performance between the two age groups. Effects of sentence familiarity/frequency rating and type of noise masker on speech intelligibility are discussed. [Work supported by the James S. McDonnell Foundation https://doi.org/10.37717/2021-3028.]

End-to-end emotional speech recognition using acoustic model adaptation based on knowledge distillation.

The end-to-end approach provides better performance in speech recognition compared to the traditional hidden Markov model-deep neural network (HMM-DNN)-based approach, but still shows poor performance in abnormal speech, especially emotional speech. The optimal solution is to build an acoustic model suitable for emotional speech recognition using only emotional speech data for each emotion, but it is impossible because it is difficult to collect sufficient amount of emotional speech data for each emotion. In this study, we propose a method to improve the emotional speech recognition performance by using the knowledge distillation technique that was originally introduced to decrease computational intensity of deep learning-based approaches by reducing the number of model parameters. In addition to its use as model compression, we employ this technique for model adaptation to emotional speech. The proposed method builds a basic model (referred to as a teacher model) with a number of model parameters using an amount of normal speech data, and then constructs a target model (referred to as a student model) with fewer model parameters using a small amount of emotional speech data (i.e., adaptation data). Since the student model is built with emotional speech data, it is expected to reflect the emotional characteristics of each emotion well. In the emotional speech recognition experiment, the student model maintained recognition performance regardless of the number of model parameters, whereas the teacher model degraded performance significantly as the number of parameters decreased, showing performance degradation of about 10% in word error rate. This result demonstrates that the student model serves as an acoustic model suitable for emotional speech recognition even though it does not require much emotional speech data.

Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy.

The current accuracy of speech recognition can reach over 97% on different datasets, but in noisy environments, it is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help to improve speech recognition performance. This is where the performance of lip recognition and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip possibly containing multiple meanings, we constructed a one-to-many mapping relationship model between lips and speech allowing for the lip reading model to consider which articulations are represented from the input lip movements. Audio representations are also preserved by modeling the inter-relationships between paired audiovisual representations. At the inference stage, the preserved audio representations could be extracted from memory by the learned inter-relationships using only video input. Second, a joint cross-fusion model using the attention mechanism could effectively exploit complementary intermodal relationships, and the model calculates cross-attention weights on the basis of the correlations between joint feature representations and individual modalities. Lastly, our proposed model achieved a 4.0% reduction in WER in a -15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method could achieve a significant improvement over speech recognition models in different noise environments.

A Study of Speech Recognition for Kazakh Based on Unsupervised Pre-Training

Building a good speech recognition system usually requires a lot of pairing data, which poses a big challenge for low-resource languages, such as Kazakh. In recent years, unsupervised pre-training has achieved good performance in low-resource speech recognition, but it is rarely used in Kazakh and other Central and West Asian languages. In this paper, wav2vec2.0 is improved by integrating a Factorized TDNN layer to better preserve the relationship between the voice and the time step before and after the quantization, which is called wav2vec-F. The unsupervised pre-training strategy was used to learn the potential speech representation from a large number of unlabeled audio data and was applied to the cross-language ASR task, which was optimized using the noise contrast binary classification task. At the same time, speech synthesis is used to promote the performance of speech recognition. The experiment shows that wav2vec-F can effectively utilize the unlabeled data from non-target languages, and the multi-language pre-training is obviously better than the single-language pre-training. The data enhancement method using speech synthesis can bring huge benefits. Compared with the baseline model, Librispeech's test-clean dataset has an average reduction of 1.9% in the word error rate. On the Kazakh KSC test set, the pre-training using only Kazakh reduced the word error rate by 3.8%. The pre-training of a small amount of Kazakh speech data synthesized by multi-language combined with TTS achieved a word error rate of 8.6% on the KSC test set when the labeled data were only 10 h, which was comparable to the results of the previous end-to-end model when the labeled data were 30 times less.

A Joint Speech Enhancement and Self-Supervised Representation Learning Framework for Noise-Robust Speech Recognition

Though speech enhancement (SE) can be used to improve speech quality in noisy environments, it may also cause distortions that degrade the performance of automatic speech recognition (ASR) models. Self-supervised pre-training, on the other hand, has been shown to improve the noise robustness of ASR models. However, the potential of the (optimal) integration of SE and self-supervised pre-training still remains unclear. In this paper, we propose a novel self-supervised pre-training framework that incorporates SE to improve ASR performance in noisy environments. First, in the pre-training phase the original noisy waveform or the waveform obtained by SE is fed into the self-supervised model to learn the contextual representation, where the quantized clean speech acts as the target. Second, we propose a dual-attention fusion method to fuse the features of noisy and enhanced speech, which can compensate for the information loss caused by separately using individual modules. Due to the flexible exploitation of clean/noisy/enhanced branches, the proposed method turns out to be a generalization of some existing noise-robust ASR models, e.g., enhanced wav2vec2.0. Finally, experimental results on both synthetic and real noisy datasets show that the proposed joint training approach can improve the ASR performance under various noisy settings, leading to a stronger noise robustness.

Speech Recognition Using Artificial Neural Network

Speech is a verbal communication used by humans through language. Likewise speech recognition is a process of converting speech to text. This paper provides a study of use of artificial neural networks(ANN) in speech recognition. Hidden Markov models (HMM) is a traditional statistical techniques for performing speech recognition. In speech detection software, Mel frequency cepstral coefficients (MFCCs) are frequently used. With different approaches evolving, we deal with the features used to recognize the speech pattern and implementation of speech recognition in the efficient types of artificial neural network (ANN).