Robust Automatic Speech Recognition System Research Articles

Advancing Accessibility through Automatic Speech Recognition and NLP Integration

Abstract: The integration of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) technologies has the potential to revolutionize accessibility and inclusive communication. This article explores the fundamentals of ASR, including acoustic modeling, language modeling, and speech signal processing techniques, and discusses the challenges posed by language diversity in developing accurate ASR systems. The advent of ASR technology has opened up numerous possibilities for enhancing accessibility across various domains, such as assistive technology, education, healthcare, and media. The integration of ASR with NLP techniques enables the processing and analysis of spoken language data, leading to the development of voice-enabled virtual assistants, conversational AI systems, and cross-lingual communication tools. However, several challenges remain, including the need for robust and accurate ASR systems, privacy and security concerns, and ethical considerations in the development and deployment of these technologies. The article also presents future directions, such as the integration of ASR and NLP with emotion recognition and sentiment analysis, advances in deep learning techniques, and the application of these technologies in healthcare and accessibility. Overall, the integration of ASR and NLP holds immense promise for creating more natural, empathetic, and inclusive communication systems, but their development and deployment must be approached with care to ensure fairness, transparency, and user privacy.

Toward Robust ASR System against Audio Adversarial Examples using Agitated Logit

Automatic speech recognition (ASR) systems are vulnerable to audio adversarial examples, which aim at deceiving ASR systems by adding perturbations to benign speech signals. These audio adversarial examples appear indistinguishable from benign audio waves, but the ASR system decodes them as intentional malicious commands. Previous studies have demonstrated the feasibility of such attacks in simulated environments (over-line) and have further showcased the creation of robust physical audio adversarial examples (over-air). Various defense techniques have been proposed to counter these attacks. However, most of them have either failed to handle various types of attacks effectively or have resulted in significant time overhead. In this article, we propose a novel method for detecting audio adversarial examples. Our approach involves feeding both smoothed audio and original audio inputs into the ASR system. Subsequently, we introduce noise to the logits before providing them to the decoder of the ASR. We demonstrate that carefully selected noise can considerably influence the transcription results of audio adversarial examples while having minimal impact on the transcription of benign audio waves. Leveraging this characteristic, we detect audio adversarial examples by comparing the altered transcription, resulting from logit noising, with the original transcription. The proposed method can be easily applied to ASR systems without requiring any structural modifications or additional training. Experimental results indicate that the proposed method exhibits robustness against both over-line and over-air audio adversarial examples, outperforming state-of-the-art detection methods.

MDVC corpus: empowering Moroccan Darija speech recognition

Automatic speech recognition (ASR) technology has significantly transformed human-machine interactions, but it remains limited in its representation of diverse languages and dialects. Moroccan Darija, the lively Moroccan dialect, has long been underrepresented in the realm of language technology. To address this gap, we present a novel corpus of audio files accompanied by meticulously transcribed Moroccan Darija speech. The corpus comprises 1,000 hours of diverse content, featuring multiple Moroccan accents, extracted from 80 YouTube channels. To standardize the representation of Moroccan Darija in our corpus, we made efforts to establish consistent writing norms and conventions. In addition to the dataset creation, we applied fine-tuning using the Wav2Vec2 model on the Moroccan Darija voice corpus (MDVC) dataset achieving a remarkable word error rate (WER) of 9%. This article discusses the current state of Moroccan Darija research, highlighting the scarcity of resources and the need for robust ASR systems. Our contribution offers a valuable resource for researchers and developers, and by standardizing the Darija language, we strive to improve ASR system for this low resource language.

A Study on the Impact of Lombard Effect on Recognition of Hindi Syllabic Units Using CNN Based Multimodal ASR Systems

Research work on the design of robust multimodal speech recognition systems making use of acoustic and visual cues, extracted using the relatively noise robust alternate speech sensors is gaining interest in recent times among the speech processing research fraternity. The primary objective of this work is to study the exclusive influence of Lombard effect on the automatic recognition of the confusable syllabic consonant-vowel units of Hindi language, as a step towards building robust multimodal ASR systems in adverse environments in the context of Indian languages which are syllabic in nature. The dataset for this work comprises the confusable 145 consonant-vowel (CV) syllabic units of Hindi language recorded simultaneously using three modalities that capture the acoustic and visual speech cues, namely normal acoustic microphone (NM), throat microphone (TM) and a camera that captures the associated lip movements. The Lombard effect is induced by feeding crowd noise into the speaker’s headphone while recording. Convolutional Neural Network (CNN) models are built to categorise the CV units based on their place of articulation (POA), manner of articulation (MOA), and vowels (under clean and Lombard conditions). For validation purpose, corresponding Hidden Markov Models (HMM) are also built and tested. Unimodal Automatic Speech Recognition (ASR) systems built using each of the three speech cues from Lombard speech show a loss in recognition of MOA and vowels while POA gets a boost in all the systems due to Lombard effect. Combining the three complimentary speech cues to build bimodal and trimodal ASR systems shows that the recognition loss due to Lombard effect for MOA and vowels reduces compared to the unimodal systems, while the POA recognition is still better due to Lombard effect. A bimodal system is proposed using only alternate acoustic and visual cues which gives a better discrimination of the place and manner of articulation than even standard ASR system. Among the multimodal ASR systems studied, the proposed trimodal system based on Lombard speech gives the best recognition accuracy of 98%, 95%, and 76% for the vowels, MOA and POA, respectively, with an average improvement of 36% over the unimodal ASR systems and 9% improvement over the bimodal ASR systems.

Robust Automatic Speech Recognition System for the Recognition of Continuous Kannada Speech Sentences in the Presence of Noise

Robust Automatic Speech Recognition System for the Recognition of Continuous Kannada Speech Sentences in the Presence of Noise

End-to-End Multi-Modal Speech Recognition on an Air and Bone Conducted Speech Corpus

Automatic speech recognition (ASR) has been significantly improved in the past years. However, most robust ASR systems are based on air-conducted (AC) speech, and their performances in low signal-to-noise-ratio (SNR) conditions are not satisfactory. Bone-conducted (BC) speech is intrinsically insensitive to environmental noise, and therefore can be used as an auxiliary source for improving the performance of an ASR at low SNR. In this paper, we first develop a multi-modal Mandarin corpus, which contains air- and bone-conducted synchronized speech (ABCS). The multi-modal speeches are recorded with a headset equipped with both AC and BC microphones. To our knowledge, it is by far the largest corpus for conducting bone conduction ASR research. Then, we propose a multi-modal conformer ASR system based on a novel multi-modal transducer (MMT). The proposed system extracts semantic embeddings from the AC and BC speech signals by a conformer-based encoder and a transformer-based truncated decoder. The semantic embeddings of the two speech sources are fused dynamically with adaptive weights by the MMT module. Experimental results demonstrate the proposed multi-modal system outperforms single-modal systems with either AC or BC modality and multi-modal baseline system by a large margin at various SNR levels. It also shows the two modalities complement with each other, and our method can effectively utilize the complementary information of different sources.

HindiSpeech-Net: a deep learning based robust automatic speech recognition system for Hindi language

HindiSpeech-Net: a deep learning based robust automatic speech recognition system for Hindi language

Characterizing Dysarthria Diversity for Automatic Speech Recognition: A Tutorial from the Clinical Perspective.

Despite significant advancements in automatic speech recognition (ASR) technology, even the best performing ASR systems are inadequate for speakers with impaired speech. This inadequacy may be, in part, due to the challenges associated with acquiring a sufficiently diverse training sample of disordered speech. Speakers with dysarthria, which refers to a group of divergent speech disorders secondary to neurologic injury, exhibit highly variable speech patterns both within and across individuals. This diversity is currently poorly characterized and, consequently, difficult to adequately represent in disordered speech ASR corpora. In this paper, we consider the variable expressions of dysarthria within the context of established clinical taxonomies (e.g., Darley, Aronson, and Brown dysarthria subtypes). We also briefly consider past and recent efforts to capture this diversity quantitatively using speech analytics. Understanding dysarthria diversity from the clinical perspective and how this diversity may impact ASR performance could aid in (1) optimizing data collection strategies for minimizing bias; (2) ensuring representative ASR training sets; and (3) improving generalization of ASR across users and performance for difficult-to-recognize speakers. Our overarching goal is to facilitate the development of robust ASR systems for dysarthric speech using clinical knowledge.

A multimodal Lombard speech recognition system for the confusable Hindi syllabic units

Research work on the design of robust multimodal speech recognition systems making use of acoustic and visual cues, extracted using the relatively noise robust alternate speech sensors is gaining interest in recent times among the speech processing research fraternity. The primary objective of this work is to study the exclusive influence of Lombard effect on Automatic Speech Recognition (ASR) systems towards building robust multimodal ASR systems in adverse environments in the context of Indian languages which are syllabic in nature. The dataset for this work comprises the confusable 145 Consonant-Vowel (CV) syllabic units of Hindi language recorded simultaneously using three modalities that capture the acoustic and visual speech cues, namely Normal acoustic Microphone (NM), Throat Microphone (TM) and a camera that captures the associated lip movements. The Lombard effect is induced by feeding crowd noise into the speaker’s headphone while recording. HMM models are built to categorize the CV units based on their Place of Articulation (POA), Manner Of Articulation (MOA) and vowels (under clean and Lombard conditions). Unimodal ASR systems built using each speech cue show a recognition loss in all the systems due to Lombard effect. To overcome this loss, the complimentary speech cues taken from normal and throat microphone Lombard speech as well as from visual Lombard speech are used to build three bimodal and one trimodal ASR systems. Among the ASR systems studied, the trimodal system gives the best recognition accuracy of 98%, 95% and 76% for the vowels, MOA and POA, respectively, with an average improvement of 36% over the unimodal ASR systems and 9% improvement over the bimodal ASR systems.

Acoustic Analysis of the Speakers’ Variability for Regional Accent-Affected Pronunciation in Bangladeshi Bangla: A Study on Sylheti Accent

Accented pronunciation variability is one of the key elements that deteriorate the accuracy of the automatic speech recognition (ASR). This article reports the results of the acoustic analysis of the two groups of speakers’ variability caused by regional accent in Bangladeshi Bangla. The analysis considers the seven monophthongal and four diphthongal vowels of Bangla to investigate the acoustic characteristics of two groups of single-accent speakers and their correlation on the articulation of the Standard Colloquial Bangladeshi Bangla (SCBB). An accent is the speaker’s regional signature and shaped by his/her community and educational background. This study examines both male and female speakers from the Sylhet region, which has one of the extremely deviant dialects in Bangla, and comparatively less deviant speakers from different districts of North-West and Middle Part of Bangladesh. Accent-related acoustic features such as pitch slope, formant frequencies, and vowel duration have been considered to examine the prominent characteristics of the accents and to classify the accents from these features. Both gender groups are distinctly analyzed. It has been found that there are significant deviations in formant frequencies and various steepness of the rise/fall in pitch slope within accents of both gender groups. In this study, it has been observed that accent related changes in speech affect the ASR performance. This has emphasized the need for accent-specific acoustic models to handle the speakers from highly deviant dialects as well as considering the accent-affected speakers’ variability in the corpora development for robust ASR system in Bangladeshi Bangla.

Enhanced Automatic Speech Recognition System Based on Enhancing Power-Normalized Cepstral Coefficients

Many new consumer applications are based on the use of automatic speech recognition (ASR) systems, such as voice command interfaces, speech-to-text applications, and data entry processes. Although ASR systems have remarkably improved in recent decades, the speech recognition system performance still significantly degrades in the presence of noisy environments. Developing a robust ASR system that can work in real-world noise and other acoustic distorting conditions is an attractive research topic. Many advanced algorithms have been developed in the literature to deal with this problem; most of these algorithms are based on modeling the behavior of the human auditory system with perceived noisy speech. In this research, the power-normalized cepstral coefficient (PNCC) system is modified to increase robustness against the different types of environmental noises, where a new technique based on gammatone channel filtering combined with channel bias minimization is used to suppress the noise effects. The TIDIGITS database is utilized to evaluate the performance of the proposed system in comparison to the state-of-the-art techniques in the presence of additive white Gaussian noise (AWGN) and seven different types of environmental noises. In this research, one word is recognized from a set containing 11 possibilities only. The experimental results showed that the proposed method provides significant improvements in the recognition accuracy at low signal to noise ratios (SNR). In the case of subway noise at SNR = 5 dB, the proposed method outperforms the mel-frequency cepstral coefficient (MFCC) and relative spectral (RASTA)–perceptual linear predictive (PLP) methods by 55% and 47%, respectively. Moreover, the recognition rate of the proposed method is higher than the gammatone frequency cepstral coefficient (GFCC) and PNCC methods in the case of car noise. It is enhanced by 40% in comparison to the GFCC method at SNR 0dB, while it is improved by 20% in comparison to the PNCC method at SNR −5dB.

Performance evaluation of psycho-acoustically motivated front-end compensator for TIMIT phone recognition

Wavelet-based front-end processing technique has gained popularity for its noise removing capability. In this paper, a robust automatic speech recognition system is proposed by utilizing the advantages of psycho-acoustically motivated wavelet-based front-end compensator. In the front-end compensator block, voiced speech probability-based voice activity detector system is designed to separate voiced and unvoiced frames and to update noise statistics. The wavelet packet decomposition tree is designed according to equal rectangular bandwidth (ERB) scale. Wavelet decomposition based on ERB scale is utilized here as the central frequency of the ERB distribution resembles frequency response of human cochlea. Voiced and unvoiced frames are separately decomposed into 24 sub-bands to estimate average sub-band energy (ASE) of each frame. ASE is then used to calculate threshold value. Lastly, Wiener filtering is employed for reducing the residual noise before final reconstruction stage. The proposed system is evaluated on TIMIT database under various noise conditions. The phoneme recognition accuracy of the proposed system is compared with different baseline and robust features as well as with existing front-end compensation techniques. Additionally, the proposed front-end compensator is evaluated in terms of phoneme classification accuracy. Performance improvement is observed in all above experiments.

GFCC based discriminatively trained noise robust continuous ASR system for Hindi language

A statistically designed Automatic Speech Recognition (ASR) system extracts features from speech signals using feature extraction methods, links the extracted features with the expected phonetics of the hypothesis using acoustic models, uses language model to add prior information about the structure of the target language. For many years, Mel-frequency Cepstral coefficients (MFCC), n-gram, and Hidden Markov Model (HMM) approaches have been used predominantly for feature extraction, language modeling and acoustic modeling, respectively. However, performance degradation of MFCC in noisy conditions and inaccuracy of HMMs while handling large vocabularies have made researchers to propose more efficient methods. The proposed work uses noise robust method Gammatone Frequency Cepstral Coefficients(GFCC) for feature extraction, trigram language modeling, and HMM-Gaussian mixture model (GMM) based acoustic modeling to implement a continuous Hindi language ASR system. Also, it applies Differential Evolution (DE) technique to refine the GFCC features and discriminative techniques to enhance performance of the acoustic model. The performance of the implemented system has been evaluated by using different feature extraction methods, variants of n-gram language modeling techniques and different discriminative techniques in clean as well as noisy conditions. Initially, the results reveal that DE optimized GFCC with HMM-Gaussian Mixture Model (GMM) acoustic modeling performs better than MFCC, PLP and MF-PLP feature extraction methods. Secondly, the experimental results show that the Minimum Phone Error (MPE) outperforms Maximum Mutual Information (MMI) and Maximum Likelihood Estimation (MLE) and trigram based language modeling gives more accurate results than unigram and bigram language modeling. Finally, it has been concluded that the continuous Hindi language ASR system implemented using DE refined GFFC feature extraction method with MPE discriminative training technique and trigram based language modeling gives better accuracy in clean as-well-as noisy environments.

Speech enhancement by combining spectral subtraction and minimum mean square error-spectrum power estimator based on zero crossing

Speech data collected under uncontrolled environment need to be processed to build a robust automatic speech recognition system. In this paper, a method is proposed to process the degraded speech signal. Initially, the significance of the spectral subtraction with voice activity detection (SS-VAD) and magnitude squared spectrum estimators are studied for different types of noises. In SS-VAD method, the degraded speech data is sampled and windowed into 50% overlapping. The VAD is used to detect the voiced regions of speech signal. The minimum mean square error-short time power spectrum, minimum mean square error-spectrum power based on zero crossing (MMSE-SPZC) and maximum a posteriori estimators are studied individually. These MSS estimators are implemented on the assumption that the magnitude squared spectrum of the degraded speech signal is the sum of the clean (original) speech signal and noise model. The experimental results show that the MMSE-SPZC estimator gives better performance compared to the other two methods. This estimator is combined with SS-VAD method to improve the performance. In this paper, the combined SS-VAD and MMSE-SPZC method, yields better speech quality by reducing noise in degraded speech signal compared to the individual methods.

Local Feature based Gender Independent Bangla ASR

This paper presents an automatic speech recognition (ASR) for Bangla (widely used as Bengali) by suppressing the speaker gender types based on local features extracted from an input speech. Speaker-specific characteristics play an important role on the performance of Bangla automatic speech recognition (ASR). Gender factor shows adverse effect in the classifier while recognizing a speech by an opposite gender, such as, training a classifier by male but testing is done by female or vice-versa. To obtain a robust ASR system in practice it is necessary to invent a system that incorporates gender independent effect for particular gender. In this paper, we have proposed a Gender-Independent technique for ASR that focused on a gender factor. The proposed method trains the classifier with the both types of gender, male and female, and evaluates the classifier for the male and female. For the experiments, we have designed a medium size Bangla (widely known as Bengali) speech corpus for both the male and female.The proposed system has showed a significant improvement of word correct rates, word accuracies and sentence correct rates in comparison with the method that suffers from gender effects using. Moreover, it provides the highest level recognition performance by taking a fewer mixture component in hidden Markov model (HMMs).

A Novel Uncertainty Decoding Rule With Applications to Transmission Error Robust Speech Recognition

In this paper, we derive an uncertainty decoding rule for automatic speech recognition (ASR), which accounts for both corrupted observations and inter-frame correlation. The conditional independence assumption, prevalent in hidden Markov model-based ASR, is relaxed to obtain a clean speech posterior that is conditioned on the complete observed feature vector sequence. This is a more informative posterior than one conditioned only on the current observation. The novel decoding is used to obtain a transmission-error robust remote ASR system, where the speech capturing unit is connected to the decoder via an error-prone communication network. We show how the clean speech posterior can be computed for communication links being characterized by either bit errors or packet loss. Recognition results are presented for both distributed and network speech recognition, where in the latter case common voice-over-IP codecs are employed.

VTLN Through Frequency Warping Based on Pitch

This article describes a Vocal Tract Length Nor­ malization (VTLN) procedure through frequency warping based on pitch estimates. This procedure aims to reduce the inter-speaker variability of speech signals in order to obtain a robust automatic speech recognition system. Two additional methods are also described: one for reducing the environment variability and another for compensating the coarticulation effects on connected word pronunciation. En­ vironment variability is compensated by explicitly modeling some frequent noise phenomena. Coarticulation phenomena compensation reduces speech signal variability by modeling events that result from coarticulation between adjacent mod­ els. Inter-speaker variability removal is performed by a traditional speaker normalization method, which consists in expanding or compressing the Mel filterbank bandwidths, in order to normalize the Vocal Tract Length (VTL) of each speaker. Most of the existing methods for VTL estimation are based on formant estimation, but the difficulty of formant estimation is a known performance limitation. The proposed method over­ comes such a problem since it estimates the warping factor through pitch. The recognition results, obtained for a tele­ phone digit recognition task (with phones and sub words as units), prove that this procedure leads to similar improve­ ments to those obtained with traditional methods based on formant estimates, actually outperforming them in some sit­ uations.

Cross-language use of acoustic information for automatic speech recognition

Techniques are investigated that use acoustic information from existing source language databases to implement automatic speech recognition (ASR) systems for new target languages. The assumption is that the amount of target language data available is too little for the training of a robust ASR system. Strategies for cross-language use of acoustic information are evaluated which include (i) training on pooled source and target language data, (ii) adapting source language models using target language data, (iii) adapting models trained on pooled source and target language using target language data only and (iv) transforming source language data to augment target language data for model training. These strategies are allied with Bayesian and transformation-based techniques to present a framework for cross-language reuse of acoustic information. Experiments are performed for a large number of approaches from the framework, using relatively large amounts of English speech data from either a separate database or from the same-database as smaller amounts of Afrikaans speech data to improve the performance of an Afrikaans speech recogniser. Results indicate that a significant reduction in word error rate is achievable (between 14% and 48% for experiments), depending on the amount of target language data available.

ACCOR: instrumentation and database for cross-language study of coarticulation.

An unavoidable problem in speech technology, particularly in the development of robust automatic speech recognition systems, is the extreme variability in the acoustic attributes of segments. Segments are highly sensitive to context and bear little resemblance to their intrinsic characteristics manifested when they are uttered in isolation. However, the problem can become tractable if we model the linguistic and physiological aspects of coarticulatory processes, the main source of systemic variability at the segmental level.