Robust Speech Recognition Research Articles

Refining maritime Automatic Speech Recognition by leveraging synthetic speech

Maritime transport serves as a critical component of global trade and logistics, enabling the movement of goods and resources across oceans and waterways. Especially in busy waterways and ports, effective and accurate communication is essential, as it ensures the seamless exchange of information and the coordinated execution of port activities. However, comprehensibility is often hindered by factors such as poor audio quality, background noise, and diverse languages and accents. Automatic Speech Recognition (ASR) systems can mitigate these issues by providing real-time transcription and enabling the implementation of automated, value-adding services to enhance situational awareness. While pre-trained ASR models excel on general speech, maritime ASR faces unique challenges due to a lack of annotated data, diverse accents, and specialized terminology.To this end, we focus on improving the transcription quality of pre-trained ASR models for maritime communication with a particular focus on accurately recognizing maritime-specific terminology such as vessel and location names. Due to the scarcity of transcribed maritime communication, we create a synthetic training dataset tailored to regional maritime terminology. The synthetic audio is augmented with general human speech and used to fine-tune an end-to-end ASR model under various settings. The evaluation of the models employs a proprietary dataset of regional maritime radio communication from the port of Hamburg.The experimental results demonstrate a notable enhancement in ASR performance. Specifically, our approach yields an absolute improvement over the pre-trained baseline of 13.46% Word-Error-Rate and an increase of 41.57% recall for vessel names and 38.65% recall for locations. Our findings underscore the efficacy of integrating synthetic training data to address the challenges encountered in maritime ASR, paving the way for more robust and accurate speech recognition systems tailored to maritime applications.

Tensor-Based Least-Squares Solutions for Multirelational Signals and Applications.

The approach of least squares (LSs) has been quite popular and widely adopted for the common linear regression analysis, which can give rise to the solution to an arbitrary critically-, over-, or under-determined system. Such a linear regression analysis can be easily applied for linear estimation and equalization in signal processing for cybernetics. Nonetheless, the current LS approach for linear regression is unfortunately limited to the dimensionality of data, that is, the exact LS solution can involve only a data matrix. As the dimension of data increases and such data need to be represented by a tensor, the corresponding exact tensor-based LS (TLS) solution does not exist due to the lack of a pertinent mathematical framework. Lately, some alternatives such as tensor decomposition and tensor unfolding were proposed to approximate the TLS solutions to the linear regression problems involving tensor data, but these techniques cannot provide the exact or true TLS solution. In this work, we would like to make the first-ever attempt to present a new mathematical framework for facilitating the exact TLS solutions involving tensor data. To demonstrate the applicability of our proposed new scheme, numerical experiments regarding machine learning and robust speech recognition are illustrated and the associated memory and computational complexities are also studied.

Speech enhancement augmentation for robust speech recognition in noisy environments

Abstract. The use of augmentations as a data enrichment method has become an important element in improving the performance of speech recognition systems. To work effectively in noisy conditions, augmentation is usually used to simulate the presence of background noise. However, the quality of speech recognition on samples pre-processed by noise reduction models does not increase. This paper proposes a new approach to speech data augmentation when training ASR systems, intended for their joint use with models for speech enhancement. It was based on the creation of several additional data samples containing speech samples processed by the enhancement model. The proposed approach was tested on the E-Branchformer neural network model using data from the Librispeech set. The quality of speech samples was assessed using the DNSMOS metric. By means of a 100-hour sample of clean speech samples it was shown that the proposed augmentation allows for an improvement in the WER metric of more than 4% in absolute value compared to the generally accepted approach based on adding noisy speech samples. Experiments on 960-hour data demonstrated the robustness of this approach as the training set size increased.

Statistical Beamformer Exploiting Non-Stationarity and Sparsity With Spatially Constrained ICA for Robust Speech Recognition

Statistical Beamformer Exploiting Non-Stationarity and Sparsity With Spatially Constrained ICA for Robust Speech Recognition

A novel channel estimate for noise robust speech recognition

We propose a novel technique to estimate the channel characteristics for robust speech recognition. The method focuses on reliable time–frequency speech patches which are highly independent of the noise condition. Combined with a root-based approximation of the logarithm in the MFCC computation, this reduces the variance caused by the noise on the spectral features, and therefore also the constrain on the acoustic model in a multi-style training setup. We show that compared to the standard mean normalization, the proposed method estimates the channel equally well under clean conditions and better under noisy conditions. When integrated in the feature extraction pipeline, we show improvements in speech recognition accuracy on noisy speech and a status quo on clean speech. Our experiments reveal that this method helps the most for generative models that need to model the complex noise variability, and less so for discriminative models, which can learn to ignore noise instead of accurately modeling it. Our approach outperforms the state of the art on the noisy Aurora4 task.

Lessons Learned in Transcribing 5000 h of Air Traffic Control Communications for Robust Automatic Speech Understanding

Voice communication between air traffic controllers (ATCos) and pilots is critical for ensuring safe and efficient air traffic control (ATC). The handling of these voice communications requires high levels of awareness from ATCos and can be tedious and error-prone. Recent attempts aim at integrating artificial intelligence (AI) into ATC communications in order to lessen ATCos’s workload. However, the development of data-driven AI systems for understanding of spoken ATC communications demands large-scale annotated datasets, which are currently lacking in the field. This paper explores the lessons learned from the ATCO2 project, which aimed to develop an unique platform to collect, preprocess, and transcribe large amounts of ATC audio data from airspace in real time. This paper reviews (i) robust automatic speech recognition (ASR), (ii) natural language processing, (iii) English language identification, and (iv) contextual ASR biasing with surveillance data. The pipeline developed during the ATCO2 project, along with the open-sourcing of its data, encourages research in the ATC field, while the full corpus can be purchased through ELDA. ATCO2 corpora is suitable for developing ASR systems when little or near to no ATC audio transcribed data are available. For instance, the proposed ASR system trained with ATCO2 reaches as low as 17.9% WER on public ATC datasets which is 6.6% absolute WER better than with “out-of-domain” but gold transcriptions. Finally, the release of 5000 h of ASR transcribed speech—covering more than 10 airports worldwide—is a step forward towards more robust automatic speech understanding systems for ATC communications.

Boosting Low-Resource Speech Recognition in Air Traffic Communication via Pretrained Feature Aggregation and Multi-Task Learning

Developing a robust Automatic Speech Recognition (ASR) system usually requires a large amount of well-annotated samples which is extremely hard to build in the Air Traffic Control (ATC) due to domain-specific knowledge. In this work, we present a novel approach to improve ASR performance in the ATC domain by integrating Self-Supervised Learning and Multi-Task learning into a unified framework. Specifically, the proposed framework follows a two-stage training paradigm, i.e., (a) learning universal acoustic representations by employing the wav2vec 2.0 model and (b) jointly finetuning the model by the ASR, Speaker Role Identification, and Language Identification tasks. To capture the task-specific representations, an attention-guided feature aggregation module is dedicatedly designed to disentangle the discriminative representations from the pretrained features. In addition, the uncertainty-based loss combination strategy is employed to balance the loss weights for each task in a learnable manner. Finally, we conduct experiments to validate the technical improvements in a real-world ATC dataset. Experimental results demonstrated that the proposed framework outperforms competitive baselines among all tasks.

Low-Complexity and Accurate Noise Suppression Based on an a Priori SNR Model for Robust Speech Recognition on Embedded Systems and Its Evaluation in a Car Environment

Low-Complexity and Accurate Noise Suppression Based on an a Priori SNR Model for Robust Speech Recognition on Embedded Systems and Its Evaluation in a Car Environment

A Study on the Impact of Lombard Effect on Recognition of Hindi Syllabic Units Using CNN Based Multimodal ASR Systems

Research work on the design of robust multimodal speech recognition systems making use of acoustic and visual cues, extracted using the relatively noise robust alternate speech sensors is gaining interest in recent times among the speech processing research fraternity. The primary objective of this work is to study the exclusive influence of Lombard effect on the automatic recognition of the confusable syllabic consonant-vowel units of Hindi language, as a step towards building robust multimodal ASR systems in adverse environments in the context of Indian languages which are syllabic in nature. The dataset for this work comprises the confusable 145 consonant-vowel (CV) syllabic units of Hindi language recorded simultaneously using three modalities that capture the acoustic and visual speech cues, namely normal acoustic microphone (NM), throat microphone (TM) and a camera that captures the associated lip movements. The Lombard effect is induced by feeding crowd noise into the speaker’s headphone while recording. Convolutional Neural Network (CNN) models are built to categorise the CV units based on their place of articulation (POA), manner of articulation (MOA), and vowels (under clean and Lombard conditions). For validation purpose, corresponding Hidden Markov Models (HMM) are also built and tested. Unimodal Automatic Speech Recognition (ASR) systems built using each of the three speech cues from Lombard speech show a loss in recognition of MOA and vowels while POA gets a boost in all the systems due to Lombard effect. Combining the three complimentary speech cues to build bimodal and trimodal ASR systems shows that the recognition loss due to Lombard effect for MOA and vowels reduces compared to the unimodal systems, while the POA recognition is still better due to Lombard effect. A bimodal system is proposed using only alternate acoustic and visual cues which gives a better discrimination of the place and manner of articulation than even standard ASR system. Among the multimodal ASR systems studied, the proposed trimodal system based on Lombard speech gives the best recognition accuracy of 98%, 95%, and 76% for the vowels, MOA and POA, respectively, with an average improvement of 36% over the unimodal ASR systems and 9% improvement over the bimodal ASR systems.

Computational Approaches to Phonology: Advances in Speech Recognition and Synthesis

This article explores the field of computational approaches to phonology, with a particular focus on recent increases in popularity and speech synthesis. Phonology, as a department of linguistics, deals with the observation of sounds and their work in language. Recent improvements in this area have resulted in major improvements in accuracy and efficiency, towards the integration of deep recognition strategies together with recurrent neural networks (RNNs) and convolutional neural networks (CNNs). the models have tested the ability to capture complex phonetic patterns and decorate the overall performance of computerized speech reputation structures. These advances have seen packages in multiple domains, such as virtual assistants, text-to-speech structures, and language acquisition tools. This article looks at the main computational procedures used in phonology, along with characteristic-based representations, rule-based modes, and statistical methods. It explores demanding situations related to phonological evaluation, which include phoneme segmentation, pronunciation variants, and language-specific problems. In addition, this paper discusses mixing phonological knowledge with device learning strategies, which enables the development of robust and correct speech recognition and synthesis systems. As usual, this research represents remarkable advances from computational methods to phonology, particularly in the geographical area of speech recognition and synthesis. the combination of technology and linguistic insights has increased the accuracy, naturalness, and efficiency in processing spoken language. These enhancements open up new opportunities to enhance human-pc interaction, language recognition, and many other packages that rely on speech-based technologies.

Robust speech recognition in sports competition review based on natural language processing

Robust speech recognition in sports competition review based on natural language processing

Noise robust automatic speech recognition: review and analysis

Noise robust automatic speech recognition: review and analysis

Audio-visual speech recognition based on joint training with audio-visual speech enhancement for robust speech recognition

Visual features are attractive cues that can be used for robust automatic speech recognition (ASR). In particular, speech recognition performance can be improved by combining audio with visual information obtained from the speaker’s face rather than using only audio in acoustically unfavorable environments. For this reason, recently, studies on various audio-visual speech recognition (AVSR) models have been actively conducted. However, from the experimental results of the AVSR models, important information for speech recognition is mainly concentrated on audio signals, and visual information plays a role in enhancing the robustness of recognition when the audio signal is corrupted in noisy environments. Therefore, there is a limit to the improvement of the recognition performance of conventional AVSR models in noisy environments. Unlike the conventional AVSR models that directly use input audio-visual information as it is, in this paper, we propose an AVSR model that first performs AVSE to enhance target speech based on audio-visual information and then uses both audio information enhanced by the AVSE and visual information such as the speaker’s lips or face. In particular, we propose a deep AVSR model that performs end-to-end training as one model by integrating an AVSR model based on the conformer with hybrid decoding and an AVSE model based on the U-net with recurrent neural network (RNN) attention (RA). Experimental results on the LRS2-BBC and LRS3-TED datasets demonstrate that the AVSE model effectively suppresses corrupting noise and the AVSR model successfully achieves noise robustness. Especially, the proposed jointly trained model integrating the AVSE and AVSR stages into one model showed better recognition performance than the other compared methods.

Robust speech recognition based on deep learning for sports game review

Abstract To verify the feasibility of robust speech recognition based on deep learning in sports game review. In this paper, a robust speech recognition model is built based on the generative adversarial network GAN algorithm according to the deep learning model. And the loss function, optimization function and noise reduction front-end are introduced in the model to achieve the optimization of speech extraction features through denoising process to ensure that accurate speech review data can be derived even in the game scene under noisy environment. Finally, the experiments are conducted to verify the four directions of the model algorithm by comparing the speech features MFCC, FBANK and WAVE. The experimental results show that the speech recognition model trained by the GSDNet model algorithm can reach 89% accuracy, 56.24% reduction of auxiliary speech recognition word error rate, 92.61% accuracy of speech feature extraction, about 62.19% reduction of training sample data volume, and 94.75% improvement of speech recognition performance in the speech recognition task under noisy environment. It shows that the robust speech recognition based on deep learning can be applied to sports game reviews, and also can provide accurate voice review information from the noisy sports game scene, and also broaden the application area for deep learning models.

Robust Automatic Speech Recognition System for the Recognition of Continuous Kannada Speech Sentences in the Presence of Noise

Robust Automatic Speech Recognition System for the Recognition of Continuous Kannada Speech Sentences in the Presence of Noise

An Investigation into Audio–Visual Speech Recognition under a Realistic Home–TV Scenario

Robust speech recognition in real world situations is still an important problem, especially when it is affected by environmental interference factors and conversational multi-speaker interactions. Supplementing audio information with other modalities, such as audio–visual speech recognition (AVSR), is a promising direction for improving speech recognition. The end-to-end (E2E) framework can learn information between multiple modalities well; however, the model is not easy to train, especially when the amount of data is relatively small. In this paper, we focus on building an encoder–decoder-based end-to-end audio–visual speech recognition system for use under realistic scenarios. First, we discuss different pre-training methods which provide various kinds of initialization for the AVSR framework. Second, we explore different model architectures and audio–visual fusion methods. Finally, we evaluate the performance on the corpus from the first Multi-modal Information based Speech Processing (MISP) challenge, which is recorded in a real home television (TV) room. By system fusion, our final system achieves a 23.98% character error rate (CER), which is better than the champion system of the first MISP challenge (CER = 25.07%).

LimitAccess: on-device TinyML based robust speech recognition and age classification

Automakers from Honda to Lamborghini are incorporating voice interaction technology into their vehicles to improve the user experience and offer value-added services. Speech recognition systems are a key component of smart cars, enhancing convenience and safety for drivers and passengers. In the future, safety-critical features may rely on speech recognition, but this raises concerns about children accessing such services. To address this issue, the LimitAccess system is proposed, which uses TinyML for age classification and helps parents limit children’s access to critical speech recognition services. This study employs a lite convolutional neural network (CNN) model for two different reasons: First, CNN showed superior accuracy compared to other audio classification models for age classification problems. Second, the lite model will be integrated into a microcontroller to meet its limited resource requirements. To train and evaluate our model, we created a dataset that included child and adult voices of the keyword “open”. The system approach categorizes voices into age groups (child, adult) and then utilizes that categorization to grant access to a car. The robustness of the model was enhanced by adding a new class (recordings) to the dataset, which enabled our system to detect replay and synthetic voice attacks. If an adult voice is detected, access to start the car will be granted. However, if a child’s voice or a recording is detected, the system will display a warning message that educates the child about the dangers and consequences of the improper use of a car. Arduino Nano 33 BLE sensing was our embedded device of choice for integrating our trained, optimized model. Our system achieved an overall F1 score of 87.7% and 85.89% accuracy. LimitAccess detected replay and synthetic voice attacks with an 88% F1 score.

Noise-Separated Adaptive Feature Distillation for Robust Speech Recognition

This letter makes an improvement on feature-based knowledge distillation for robust speech recognition. The use of distillation techniques in speech recognition has been demonstrated to improve the robustness of the system. In this letter, we propose a noise-separated adaptive feature distillation method, including an adaptive distillation position selection strategy and a noise separation mechanism, assuming that there is a common network structure between the student and teacher. The proposed method has two improvements. First, distillation positions can be adaptively selected in each iteration by comparing loss values computed on intermediate representations of the student and the teacher, increasing the flexibility of knowledge transfer during distillation. Second, a noise separation module is proposed to constrain noise information elimination by explicitly separating the speech information and the noise information in noisy speech, which reduces the interference of noise information during distillation. Therefore, a better recognition performance is demonstrated with the proposed method compared to the standardized feature-based knowledge distillation method.

Robust automatic continuous speech recognition for 'Adi', a zero-resource indigenous language of Arunachal Pradesh

Robust automatic continuous speech recognition for 'Adi', a zero-resource indigenous language of Arunachal Pradesh

Robust Speech Recognition Using Teacher-Student Learning Domain Adaptation

Recently, robust speech recognition for real-world applications has attracted much attention. This paper proposes a robust speech recognition method based on the teacher-student learning framework for domain adaptation. In particular, the student network will be trained based on a novel optimization criterion defined by the encoder outputs of both teacher and student networks rather than the final output posterior probabilities, which aims to make the noisy audio map to the same embedding space as clean audio, so that the student network is adaptive in the noise domain. Comparative experiments demonstrate that the proposed method obtained good robustness against noise.