Speech Recognition Performance Research Articles

Experienced Adult Cochlear Implant Users Show Improved Speech Recognition When Target Fitting Parameters Are Applied.

The aim of the present study was to investigate whether prediction models built by de Graaff et al. (2020) can be used to improve speech recognition in experienced adult postlingual implanted Cochlear CI users. de Graaff et al. (2020) found relationships between elevated aided thresholds and a not optimal electrical dynamic range (<50 CL or >60 CL), and poorer speech recognition in quiet and in noise. The primary hypothesis of the present study was that speech recognition improves both in quiet and in noise when the sound processor is refitted to match targets derived from the prediction models from de Graaff et al. (2020). A second hypothesis was that subjectively, most of the CI users would find the new setting too loud because of an increase in C levels, and therefore, prefer the old settings. A within-participant repeated measures design with 18 adult Cochlear CI users was used. T- and C-levels were changed to "optimized settings," as predicted by the model of de Graaff et al. (2020). Aided thresholds, speech recognition in quiet, and speech recognition in noise were measured with the old settings and after a 4-week acclimatization period with the optimized settings. Subjective benefit was measured using the Device Oriented Subjective Outcome Scale questionnaire. The mean electrical dynamic range changed from 41.1 (SD = 6.6) CL to 48.6 (SD = 3.0) CL. No significant change in aided thresholds was measured. Speech recognition improved for 16 out of 18 participants and remained stable for 2 participants. Average speech recognition scores in quiet significantly improved by 4.9% (SD = 3.8%). No significant change for speech recognition in noise was found. A significant improvement in subjective benefit was found for one of the Device Oriented Subjective Outcome subscales (speech cues) between the old and optimized settings. All participants chose to keep the optimized settings at the end of the study. We were able to improve speech recognition in quiet by optimizing the electrical dynamic range of experienced adult CI users, according to the prediction models built by de Graaff et al. (2020). There was no significant change in aided thresholds nor in speech recognition in noise. The findings of the present study suggest that improved performance for speech recognition in quiet in adult Cochlear CI users can be achieved by setting the dynamic range as close as possible to values between 50 and 60 CL when the volume level is at 10.

Multi-view Self-supervised Learning and Multi-scale Feature Fusion for Automatic Speech Recognition

To address the challenges of the poor representation capability and low data utilization rate of end-to-end speech recognition models in deep learning, this study proposes an end-to-end speech recognition model based on multi-scale feature fusion and multi-view self-supervised learning (MM-ASR). It adopts a multi-task learning paradigm for training. The proposed method emphasizes the importance of inter-layer information within shared encoders, aiming to enhance the model’s characterization capability via the multi-scale feature fusion module. Moreover, we apply multi-view self-supervised learning to effectively exploit data information. Our approach is rigorously evaluated on the Aishell-1 dataset and further validated its effectiveness on the English corpus WSJ. The experimental results demonstrate a noteworthy 4.6%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} reduction in character error rate, indicating significantly improved speech recognition performance . These findings showcase the effectiveness and potential of our proposed MM-ASR model for end-to-end speech recognition tasks.

The Enhanced Speech Recognition in Automated Home Lighting System using Adaptive Time-Frequency Domain Noise Removal Algorithm Filter

The Enhanced Speech Recognition in Automated Home Lighting System using Adaptive Time-Frequency Domain Noise Removal Algorithm Filter

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Self-supervised speech pre-training methods have developed rapidly in recent years, which show to be very effective for many near-field single-channel speech tasks. However, far-field multichannel speech processing is suffering from the scarcity of labeled multichannel data and complex ambient noises. The efficacy of self-supervised learning for far-field multichannel and multi-modal speech processing has not been well explored. Considering that visual information helps to improve speech recognition performance in noisy scenes, in this work we propose the multichannel multi-modal speech self-supervised learning framework AV-wav2vec2, which utilizes video and multichannel audio data as inputs. First, we propose a multi-path structure to process multi-channel audio streams and a visual stream in parallel, with intra-, and inter-channel contrastive as training targets to fully exploit the rich information in multi-channel speech data. Second, based on contrastive learning, we use additional single-channel audio data, which is trained jointly to improve the performance of multichannel multi-modal representation. Finally, we use a Chinese multichannel multi-modal dataset in real scenarios to validate the effectiveness of the proposed method on audio-visual speech recognition (AVSR), automatic speech recognition (ASR), visual speech recognition (VSR) and audio-visual speaker diarization (AVSD) tasks.

A closer look at reinforcement learning-based automatic speech recognition

Reinforcement learning (RL) has demonstrated effectiveness in improving model performance and robustness for automatic speech recognition (ASR) tasks. Researchers have employed RL-based training strategies to enhance performance beyond conventional supervised or semi-supervised learning. However, existing approaches treat RL as a supplementary tool, leaving the untapped potential of RL training largely unexplored. In this paper, we formulate a novel pure RL setting where an ASR model is trained exclusively through RL via human feedback metrics, e.g., word error rate (WER) or binary reward. This approach promises to significantly simplify the annotation process if we could replace the conventional onerous annotation with a single numeric value in a human–computer interaction (HCI) way. Our experiments demonstrate the feasibility of this new setting and also identify two main inherent issues in conventional RL-based ASR training that may lead to performance degradation: (1) the mismatch issue between the action and reward has been commonly overlooked in Connectionist Temporal Classification (CTC) based models, which is attributed to the inherent CTC alignment mapping issue; (2) the classic exploration–exploitation trade-off still exists in the sampling stage of RL-based ASR, and finding the balance between them becomes a challenge. To address these issues, we first propose a new RL-based approach named CTC-aligned Policy Gradient (CTC-PG), which provides a unified formulation for different sampling strategies and alleviates the mismatch issue of action and reward in CTC-based models. Moreover, we propose Focal Sampling to balance the trade-off between exploration and exploitation with a flexible temperature parameter. Experiment results on LibriSpeech dataset showcase the effectiveness and robustness of our methods by harnessing the full potential of RL in training ASR models.

Speech recognition based on mobile sensor networks application in English education intelligent assisted learning system

With the popularity of English learning and the development of mobile technology, the development of intelligent assisted learning system has become an effective learning method. The purpose of this study is to improve the speech recognition performance of English education intelligent learning assistance system by using mobile sensor networks, and improve the learning effect of learners. This paper uses a speech recognition algorithm based on mobile sensor network to acquire learners' speech signals with multiple sensor nodes. By preprocessing and feature extraction of acquired signals, machine learning algorithm is applied to recognize them, thus improving the recognition accuracy. The experimental results show that compared with traditional speech recognition methods, the speech recognition system based on mobile sensor network has better performance in the case of background noise and speaker change. When learners use the system to learn English, they can get more accurate and reliable speech recognition results, improve the learning effect and interactive experience, and provide a more effective learning tool for English learners.

Manual switching between programs intended for specific real-life listening environments by adult cochlear implant users: do they use the intended program?

Objective The aim of the current study was to investigate the use of manually and automatically switching programs in everyday day life by adult cochlear implant (CI) users Design Participants were fitted with an automatically switching sound processor setting and 2 manual programs for 3-week study periods. They received an extensive counselling session. Datalog information was used to analyse the listening environments identified by the sound processor, the program used and the number of program switches. Study samples Fifteen adult Cochlear CI users. Average age 69 years (range: 57–85 years). Results Speech recognition in noise was significantly better with the “noise” program than with the “quiet” program. On average, participants correctly classified 4 out of 5 listening environments in a laboratory setting. Participants switched, on average, less than once a day between the 2 manual programs and the sound processor was in the intended program 60% of the time. Conclusion Adult CI users switch rarely between two manual programs and leave the sound processor often in a program not intended for the specific listening environment. A program that switches automatically between settings, therefore, seems to be a more appropriate option to optimise speech recognition performance in daily listening environments.

Keep the noise down: On the performance of automatic speech recognition of voice-recordings in web surveys

Voice-recordings are increasingly implemented in web surveys, but the resulting audio data need to be transcribed before analysis. Since manual coding is too time- and work-intensive, researchers often rely on automatic speech recognition (ASR) systems for the transcription of the voice-recordings. However, ASR tools might create partly incorrect transcriptions and potentially change the content of responses. If the ASR performance (i.e., accuracy and validity) differs by subgroup and contextual factors, a bias is introduced in the analysis of open-ended questions. We assessed the impact of sociodemographic and contextual factors on the accuracy and validity of ASR transcriptions with data from the Longitudinal Internet Studies for the Social Sciences (LISS) panel collected in December 2020. We find that background noise reduces the accuracy and validity of ASR transcriptions. In addition, validity improved when the respondent was alone during the survey. Fortunately, we did not find any evidence of systematic differences across subgroups (age, sex, education), devices or respondent location.

Joint streaming model for backchannel prediction and automatic speech recognition

AbstractIn human conversations, listeners often utilize brief backchannels such as “uh‐huh” or “yeah.” Timely backchannels are crucial to understanding and increasing trust among conversational partners. In human–machine conversation systems, users can engage in natural conversations when a conversational agent generates backchannels like a human listener. We propose a method that simultaneously predicts backchannels and recognizes speech in real time. We use a streaming transformer and adopt multitask learning for concurrent backchannel prediction and speech recognition. The experimental results demonstrate the superior performance of our method compared with previous works while maintaining a similar single‐task speech recognition performance. Owing to the extremely imbalanced training data distribution, the single‐task backchannel prediction model fails to predict any of the backchannel categories, and the proposed multitask approach substantially enhances the backchannel prediction performance. Notably, in the streaming prediction scenario, the performance of backchannel prediction improves by up to 18.7% compared with existing methods.

The Effect of Hearing Loss and Working Memory Capacity on Context Use and Reliance on Context in Older Adults.

Older adults often complain of difficulty in communicating in noisy environments. Contextual information is considered an important cue for identifying everyday speech. To date, it has not been clear exactly how context use (CU) and reliance on context in older adults are affected by hearing status and cognitive function. The present study examined the effects of semantic context on the performance of speech recognition, recall, perceived listening effort (LE), and noise tolerance, and further explored the impacts of hearing loss and working memory capacity on CU and reliance on context among older adults. Fifty older adults with normal hearing and 56 older adults with mild-to-moderate hearing loss between the ages of 60 and 95 years participated in this study. A median split of the backward digit span further classified the participants into high working memory (HWM) and low working memory (LWM) capacity groups. Each participant performed high- and low-context Repeat and Recall tests, including a sentence repeat and delayed recall task, subjective assessments of LE, and tolerable time under seven signal to noise ratios (SNRs). CU was calculated as the difference between high- and low-context sentences for each outcome measure. The proportion of context use (PCU) in high-context performance was taken as the reliance on context to explain the degree to which participants relied on context when they repeated and recalled high-context sentences. Semantic context helps improve the performance of speech recognition and delayed recall, reduces perceived LE, and prolongs noise tolerance in older adults with and without hearing loss. In addition, the adverse effects of hearing loss on the performance of repeat tasks were more pronounced in low context than in high context, whereas the effects on recall tasks and noise tolerance time were more significant in high context than in low context. Compared with other tasks, the CU and PCU in repeat tasks were more affected by listening status and working memory capacity. In the repeat phase, hearing loss increased older adults' reliance on the context of a relatively challenging listening environment, as shown by the fact that when the SNR was 0 and -5 dB, the PCU (repeat) of the hearing loss group was significantly greater than that of the normal-hearing group, whereas there was no significant difference between the two hearing groups under the remaining SNRs. In addition, older adults with LWM had significantly greater CU and PCU in repeat tasks than those with HWM, especially at SNRs with moderate task demands. Taken together, semantic context not only improved speech perception intelligibility but also released cognitive resources for memory encoding in older adults. Mild-to-moderate hearing loss and LWM capacity in older adults significantly increased the use and reliance on semantic context, which was also modulated by the level of SNR.

Retracted: Use Brain-Like Audio Features to Improve Speech Recognition Performance

Retracted: Use Brain-Like Audio Features to Improve Speech Recognition Performance

CATNet: Cross-modal fusion for audio–visual speech recognition

Automatic speech recognition (ASR) is a typical pattern recognition technology that converts human speeches into texts. With the aid of advanced deep learning models, the performance of speech recognition is significantly improved. Especially, the emerging Audio–Visual Speech Recognition (AVSR) methods achieve satisfactory performance by combining audio-modal and visual-modal information. However, various complex environments, especially noises, limit the effectiveness of existing methods. In response to the noisy problem, in this paper, we propose a novel cross-modal audio–visual speech recognition model, named CATNet. First, we devise a cross-modal bidirectional fusion model to analyze the close relationship between audio and visual modalities. Second, we propose an audio–visual dual-modal network to preprocess audio and visual information, extract significant features and filter redundant noises. The experimental results demonstrate the effectiveness of CATNet, which achieves excellent WER, CER and converges speeds, outperforms other benchmark models and overcomes the challenge posed by noisy environments.

Boosting Speech Recognition Performance: A Robust and Accurate Ensemble Method based on HMMs

Boosting Speech Recognition Performance: A Robust and Accurate Ensemble Method based on HMMs

Spatial Release From Masking With Bilateral Bone Conduction Stimulation at Mastoid for Normal Hearing Subjects.

This study investigates the effect of spatial release from masking (SRM) in bilateral bone conduction (BC) stimulation at the mastoid. Nine adults with normal hearing were tested to determine SRM based on speech recognition thresholds (SRTs) in simulated spatial configurations ranging from 0 to 180 degrees. These configurations were based on nonindividualized head-related transfer functions. The participants were subjected to sound stimulation through either air conduction (AC) via headphones or BC. The results indicated that both the angular separation between the target and the masker, and the modality of sound stimulation, significantly influenced speech recognition performance. As the angular separation between the target and the masker increased up to 150°, both BC and AC SRTs decreased, indicating improved performance. However, performance slightly deteriorated when the angular separation exceeded 150°. For spatial separations less than 75°, BC stimulation provided greater spatial benefits than AC, although this difference was not statistically significant. For separations greater than 75°, AC stimulation offered significantly more spatial benefits than BC. When speech and noise originated from the same side of the head, the "better ear effect" did not significantly contribute to SRM. However, when speech and noise were located on opposite sides of the head, this effect became dominant in SRM.

Enhancing Armenian Automatic Speech Recognition Performance: A Comprehensive Strategy for Speed, Accuracy, and Linguistic Refinement

This research introduces a comprehensive strategy to enhance the performance of an existing automatic speech recognition (ASR) model, which has been previously documented in published articles. The study sets out to achieve several objectives. Firstly, it concentrates on updating the ASR model by retraining it with new datasets. This involves integrating samples from the latest Common Voice corpus release and data collected independently via the armspeech.com web application. Another key focus lies in optimizing the ASR model for near-real-time processing, intending to improve its speed and efficiency. The proposed adjustments to the model’s architecture aim to balance accuracy and processing speed, which is essential for applications requiring prompt speech recognition. Furthermore, the research explores the integration of Transformer models into the post-processing pipeline to introduce punctuation and capitalization into the ASR output. This step not only enhances the linguistic quality of transcriptions but also improves their readability and usability. In tandem with these advancements, the research presents a systematic approach to gathering, annotating, and storing datasets specifically tailored for punctuation and capitalization tasks. The methodology outlines the acquisition and organization of a dataset conducive to training Transformer models for these linguistic tasks. This comprehensive approach, which encompasses dataset enrichment, architectural modifications, and post-processing enhancements, aims to elevate the ASR model’s accuracy, speed, and linguistic refinement, with a particular focus on addressing the intricacies of the Armenian language. The research contributes valuable insights into the optimization of ASR systems, tackling both language-specific challenges and broader issues related to linguistic post-processing.

Boosting speech recognition performance: a robust and accurate ensemble method based on HMMs

Boosting speech recognition performance: a robust and accurate ensemble method based on HMMs

Enhancing Computer Digital Signal Processing through the Utilization of RNN Sequence Algorithms

With the increase in computing power and the availability of large amounts of data, deep learning techniques, especially convolutional neural networks (CNNS) and recurrent neural networks (RNNS), have become important tools for processing complex signals. These methods show excellent performance in speech recognition, image processing, natural language processing and so on. In this paper, we explore the application of recurrent neural network (RNN) sequence algorithms in the field of computer digital signal processing, highlighting current artificial intelligence techniques and their capabilities in solving complex signal processing problems. First, the paper reviews the basic principles and development of deep learning and RNN sequence algorithms, highlighting the advances these advanced technologies have made in simulating the way the human brain processes information. The practical application and effect of RNN sequence algorithm in computer digital signal processing are demonstrated through experimental data. By comparing with traditional algorithms, we demonstrate the efficiency and accuracy of RNN in processing complex signals, such as speech recognition in noisy environments and real-time video data processing. The experimental data not only demonstrate the effectiveness of RNNS in this field, but also highlight the unique advantages of deep learning methods when dealing with large and high-dimensional data. Through these empirical studies, this paper aims to provide researchers and engineers with an in-depth understanding of the potential of RNNS in digital signal processing applications, and looks forward to the future development direction of artificial intelligence technology in this field.

The Utilization of Psychometric Functions to Predict Speech Intelligibility in Vehicles

&lt;div&gt;In this study, a novel assessment approach of in-vehicle speech intelligibility is presented using psychometric curves. Speech recognition performance scores were modeled at an individual listener level for a set of speech recognition data previously collected under a variety of in-vehicle listening scenarios. The model coupled an objective metric of binaural speech intelligibility (i.e., the acoustic factors) with a psychometric curve indicating the listener’s speech recognition efficiency (i.e., the listener factors). In separate analyses, two objective metrics were used with one designed to capture spatial release from masking and the other designed to capture binaural loudness. The proposed approach is in contrast to the traditional approach of relying on the speech recognition threshold, the speech level at 50% recognition performance averaged across listeners, as the metric for in-vehicle speech intelligibility. Results from the presented analyses suggest the importance of considering speech recognition accuracy across a range of signal-to-noise ratios rather than the speech recognition threshold alone, and the importance of considering individual differences among listeners when evaluating in-vehicle speech intelligibility.&lt;/div&gt;

Development of a Smart Game Application for Auditory Training of Children with Spatial Processing Disorder in Iran: A Pilot Study

Background and Aim: Identifying sound localization defects in children and using appropriate rehabilitation methods lead to improve their problem to use binaural processing skills to successfully suppress background noise and selectively attend to a particular auditory source. This pilot study aims to design a game for auditory training of children with Spatial Processing Disorder (SPD) in Iran and compare their performance in engagement time and speech recognition in noise with those of normal peers.&#x0D; Methods: The game application was designed based on the spatial separation of target and competing stimuli by applying head-related transfer functions. The child’s task was to identify the picture of the target word from among the displayed options. Participants were 24 children with SPD and 27 normal peers aged 7–9 years, who performed the speech recognition tasks using the designed game.&#x0D; Results: The game was fascinating and easy to use for most participants. There was no significant difference in mean engagement time between the two groups. The mean engagement time of participants was significantly different between the two groups in terms of age. The children with SPD obtained lower scores in speech recognition in noise tasks. There was no significant difference in speech recognition scores between girls and boys.&#x0D; Conclusion: The designed game application has the potential for speech recognition in- noise training of children with SPD in Iran. It is a fascinating and user-friendly tool for simulating the real-life auditory situations for these children.&#x0D; Keywords: Smart game; speech recognition; application; auditory training; design; spatial auditory processing disorder.

Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition.

Automatic speech recognition (ASR) is the major human-machine interface in many intelligent systems, such as intelligent homes, autonomous driving, and servant robots. However, its performance usually significantly deteriorates in the presence of external noise, leading to limitations of its application scenes. The audio-visual speech recognition (AVSR) takes visual information as a complementary modality to enhance the performance of audio speech recognition effectively, particularly in noisy conditions. Recently, the transformer-based architectures have been used to model the audio and video sequences for the AVSR, which achieves a superior performance. However, its performance may be degraded in these architectures due to extracting irrelevant information while modeling long-term dependences. In addition, the motion feature is essential for capturing the spatio-temporal information within the lip region to best utilize visual sequences but has not been considered in the AVSR tasks. Therefore, we propose a multimodal sparse transformer network (MMST) in this article. The sparse self-attention mechanism can improve the concentration of attention on global information by selecting the most relevant parts wisely. Moreover, the motion features are seamlessly introduced into the MMST model. We subtly allow motion-modality information to flow into visual modality through the cross-modal attention module to enhance visual features, thereby further improving recognition performance. Extensive experiments conducted on different datasets validate that our proposed method outperforms several state-of-the-art methods in terms of the word error rate (WER).