Noise Robust Automatic Speech Recognition Research Articles

Knowledge Distillation-Based Training of Speech Enhancement for Noise-Robust Automatic Speech Recognition

Knowledge Distillation-Based Training of Speech Enhancement for Noise-Robust Automatic Speech Recognition

Noise robust automatic speech recognition: review and analysis

Noise robust automatic speech recognition: review and analysis

A Joint Speech Enhancement and Self-Supervised Representation Learning Framework for Noise-Robust Speech Recognition

Though speech enhancement (SE) can be used to improve speech quality in noisy environments, it may also cause distortions that degrade the performance of automatic speech recognition (ASR) models. Self-supervised pre-training, on the other hand, has been shown to improve the noise robustness of ASR models. However, the potential of the (optimal) integration of SE and self-supervised pre-training still remains unclear. In this paper, we propose a novel self-supervised pre-training framework that incorporates SE to improve ASR performance in noisy environments. First, in the pre-training phase the original noisy waveform or the waveform obtained by SE is fed into the self-supervised model to learn the contextual representation, where the quantized clean speech acts as the target. Second, we propose a dual-attention fusion method to fuse the features of noisy and enhanced speech, which can compensate for the information loss caused by separately using individual modules. Due to the flexible exploitation of clean/noisy/enhanced branches, the proposed method turns out to be a generalization of some existing noise-robust ASR models, e.g., enhanced wav2vec2.0. Finally, experimental results on both synthetic and real noisy datasets show that the proposed joint training approach can improve the ASR performance under various noisy settings, leading to a stronger noise robustness.

End-to-End Lip-Reading Open Cloud-Based Speech Architecture.

Deep learning technology has encouraged research on noise-robust automatic speech recognition (ASR). The combination of cloud computing technologies and artificial intelligence has significantly improved the performance of open cloud-based speech recognition application programming interfaces (OCSR APIs). Noise-robust ASRs for application in different environments are being developed. This study proposes noise-robust OCSR APIs based on an end-to-end lip-reading architecture for practical applications in various environments. Several OCSR APIs, including Google, Microsoft, Amazon, and Naver, were evaluated using the Google Voice Command Dataset v2 to obtain the optimum performance. Based on performance, the Microsoft API was integrated with Google’s trained word2vec model to enhance the keywords with more complete semantic information. The extracted word vector was integrated with the proposed lip-reading architecture for audio-visual speech recognition. Three forms of convolutional neural networks (3D CNN, 3D dense connection CNN, and multilayer 3D CNN) were used in the proposed lip-reading architecture. Vectors extracted from API and vision were classified after concatenation. The proposed architecture enhanced the OCSR API average accuracy rate by 14.42% using standard ASR evaluation measures along with the signal-to-noise ratio. The proposed model exhibits improved performance in various noise settings, increasing the dependability of OCSR APIs for practical applications.

Performance Analysis of various Front-end and Back End Amalgamations for Noise-robust DNN-based ASR

Introduction: An Automatic Speech Recognition (ASR) system enables to recognize the speech utterances and thus can be used to convert speech into text for various purposes. These systems are deployed in different environments such as clean or noisy and are used by all ages or types of people. These also present some of the major difficulties faced in the development of an ASR system. Thus, an ASR system need to be efficient, while also being accurate and robust. Our main goal is to minimize the error rate during training as well as testing phases, while implementing an ASR system. Performance of ASR depends upon different combinations of feature extraction techniques and back-end techniques. In this paper, using a continuous speech recognition system, the performance comparison of different combinations of feature extraction techniques and various types of back-end techniques has been presented Methods: Hidden Markov Models (HMMs), Subspace Gaussian Mixture Models (SGMMs) and Deep Neural Networks (DNNs) with DNN-HMM architecture, namely Karel’s, Dan’s and Hybrid DNN-SGMM architecture are used at the back-end of the implemented system. Mel frequency Cepstral Coefficient (MFCC), Perceptual Linear Prediction (PLP), and Gammatone Frequency Cepstral coefficients (GFCC) are used as feature extraction techniques at the front-end of the proposed system. Kaldi toolkit has been used for the implementation of the proposed work. The system is trained on the Texas Instruments-Massachusetts Institute of Technology (TIMIT) speech corpus for English language Results: The experimental results show that MFCC outperforms GFCC and PLP in noiseless conditions, while PLP tends to outperform MFCC and GFCC in noisy conditions. Furthermore, the hybrid of Dan’s DNN implementation along with SGMM performs the best for the back-end acoustic modeling. The proposed architecture with PLP feature extraction technique in the front end and hybrid of Dan’s DNN implementation along with SGMM at the back end outperforms the other combinations in a noisy environment. Conclusion: Automatic Speech recognition has numerous applications in our lives like Home automation, Personal assistant, Robotics etc. It is highly desirable to build an ASR system with good performance. The performance Automatic Speech Recognition is affected by various factors which include vocabulary size, whether system is speaker dependent or independent, whether speech is isolated, discontinuous or continuous, adverse conditions like noise. The paper presented an ensemble architecture that uses PLP for feature extraction at the front end and a hybrid of SGMM + Dan’s DNN in the backend to build a noise robust ASR system Discussion: The presented work in this paper discusses the performance comparison of continuous ASR systems developed using different combinations of front-end feature extraction (MFCC, PLP, and GFCC) and back-end acoustic modeling (mono-phone, tri-phone, SGMM, DNN and hybrid DNN-SGMM) techniques. Each type of front-end technique is tested in combination with each type of back-end technique. Finally, it compares the results of the combinations thus formed, to find out the best performing combination in noisy and clean conditions

Robust children’s speech recognition in zero resource condition

User applications such as voice-based web search, online learning, and video gaming require an effective speech recognition module to take user commands. Nowadays, even children are frequently using such tools, especially for online learning and gaming. This has increased the demand for developing a noise-robust automatic speech recognition (ASR) system that can effectively transcribe children’s data under varied ambient conditions. However, automatic recognition of children’s speech is extremely challenging due to the insufficiency of data from child speakers in the majority of the languages across the world. Consequently, in this zero-resource condition, we are forced to decode children’s speech on systems trained using adults’ data. However, the acoustic mismatch between adults’ and children’s speech, such as differences in pitch, formant frequencies, and speaking-rates, leads to highly degraded recognition performance. To enhance the recognition rate under zero-resource conditions, we have explored the role of formant and duration-modification-based out-of-domain data augmentation in this paper. For that purpose, the formant frequencies of the adults’ speech data are upscaled using warping of linear predictive coding coefficients. On pooling original and formant modified adults’ speech data into training, the mismatch in formant locations is reduced leading to better recognition performance. Further improvement in recognition rate can be achieved by simultaneously modifying the duration as well as the formant frequencies of the training data. This case of out-of-domain data augmentation has also been studied in this work and found to yield added gains. In addition to data augmentation, a noise- and pitch-robust front-end acoustic feature extraction approach exploiting higher-order spectral analysis (simple and cross-bispectrum) is also proposed in this paper. The proposed features are noise-robust due to the inherent immunity of the bispectrum towards additive noises. An added advantage of bispectrum is reduced pitch sensitivity as demonstrated in this work. This, in turn, helps alleviate the aforementioned pitch-induced acoustic mismatch. The experimental evaluations presented in this paper demonstrate that the use of proposed acoustic features, as well as the out-of-domain data augmentation techniques, are highly suited for zero-resource children’s speech recognition tasks under clean and noisy conditions.

GFCC based discriminatively trained noise robust continuous ASR system for Hindi language

A statistically designed Automatic Speech Recognition (ASR) system extracts features from speech signals using feature extraction methods, links the extracted features with the expected phonetics of the hypothesis using acoustic models, uses language model to add prior information about the structure of the target language. For many years, Mel-frequency Cepstral coefficients (MFCC), n-gram, and Hidden Markov Model (HMM) approaches have been used predominantly for feature extraction, language modeling and acoustic modeling, respectively. However, performance degradation of MFCC in noisy conditions and inaccuracy of HMMs while handling large vocabularies have made researchers to propose more efficient methods. The proposed work uses noise robust method Gammatone Frequency Cepstral Coefficients(GFCC) for feature extraction, trigram language modeling, and HMM-Gaussian mixture model (GMM) based acoustic modeling to implement a continuous Hindi language ASR system. Also, it applies Differential Evolution (DE) technique to refine the GFCC features and discriminative techniques to enhance performance of the acoustic model. The performance of the implemented system has been evaluated by using different feature extraction methods, variants of n-gram language modeling techniques and different discriminative techniques in clean as well as noisy conditions. Initially, the results reveal that DE optimized GFCC with HMM-Gaussian Mixture Model (GMM) acoustic modeling performs better than MFCC, PLP and MF-PLP feature extraction methods. Secondly, the experimental results show that the Minimum Phone Error (MPE) outperforms Maximum Mutual Information (MMI) and Maximum Likelihood Estimation (MLE) and trigram based language modeling gives more accurate results than unigram and bigram language modeling. Finally, it has been concluded that the continuous Hindi language ASR system implemented using DE refined GFFC feature extraction method with MPE discriminative training technique and trigram based language modeling gives better accuracy in clean as-well-as noisy environments.

Histogram equalization with Bayesian estimation for noise robust speech recognition.

The histogram equalization approach is an efficient feature normalization technique for noise robust automatic speech recognition. However, it suffers from performance degradation when some fundamental conditions are not satisfied in the test environment. To remedy these limitations of the original histogram equalization methods, class-based histogram equalization approach has been proposed. Although this approach showed substantial performance improvement under noise environments, it still suffers from performance degradation due to the overfitting problem when test data are insufficient. To address this issue, the proposed histogram equalization technique employs the Bayesian estimation method in the test cumulative distribution function estimation. It was reported in a previous study conducted on the Aurora-4 task that the proposed approach provided substantial performance gains in speech recognition systems based on the acoustic modeling of the Gaussian mixture model-hidden Markov model. In this work, the proposed approach was examined in speech recognition systems with deep neural network-hidden Markov model (DNN-HMM), the current mainstream speech recognition approach where it also showed meaningful performance improvement over the conventional maximum likelihood estimation-based method. The fusion of the proposed features with the mel-frequency cepstral coefficients provided additional performance gains in DNN-HMM systems, which otherwise suffer from performance degradation in the clean test condition.

Dual-channel spectral weighting for robust speech recognition in mobile devices

Many mobile devices now include an extra microphone, frequently placed at their rear, intended to obtain information about the environmental noise for speech de-noising purposes. Although this secondary sensor can be regarded as just another element in a microphone array when performing beamforming, in this paper we show that it can be considered differently in order to better exploit the information about the acoustic environment. In particular, we propose a novel spectral weighting based on Wiener filtering that takes benefit from this secondary microphone to perform noise-robust automatic speech recognition (ASR) in mobile devices. At first it is assumed that the secondary microphone only captures noise while a reference sensor in the array (primary microphone) observes the same noise spectrum (homogeneous noise field). Since both assumptions are not always accurate, the Wiener filter (WF) weighting is modified through 1) a bias correction term (to rectify the resulting spectral weights when a non-negligible speech component is present at the secondary channel) and 2) a novel noise equalization to be applied on the secondary channel before spectral weight computation. Speech recognition experiments are performed on a dual-microphone smartphone (AURORA2-2C-CT/FT corpora) and a tablet with six microphones (CHiME-3/4). Our results show the high performance of our approach as well as its great versatility regardless of the analyzed mobile device and usage scenario.

Online MVDR Beamformer Based on Complex Gaussian Mixture Model With Spatial Prior for Noise Robust ASR

This paper considers acoustic beamforming for noise robust automatic speech recognition. A beamformer attenuates background noise by enhancing sound components coming from a direction specified by a steering vector. Hence, accurate steering vector estimation is paramount for successful noise reduction. Recently, time-frequency masking has been proposed to estimate the steering vectors that are used for a beamformer. In particular, we have developed a new form of this approach, which uses a speech spectral model based on a complex Gaussian mixture model CGMM to estimate the time-frequency masks needed for steering vector estimation, and extended the CGMM-based beamformer to an online speech enhancement scenario. Our previous experiments showed that the proposed CGMM-based approach outperforms a recently proposed mask estimator based on a Watson mixture model and the baseline speech enhancement system of the CHiME-3 challenge. This paper provides additional experimental results for our online processing, which achieves performance comparable to that of batch processing with a suitable block-batch size. This online version reduces the CHiME-3 word error rate WER on the evaluation set from 8.37% to 8.06%. Moreover, in this paper, we introduce a probabilistic prior distribution for a spatial correlation matrix a CGMM parameter, which enables more stable steering vector estimation in the presence of interfering speakers. In practice, the performance of the proposed online beamformer degrades with observations that contain only noise or/and interference because of the failure of the CGMM parameter estimation. The introduced spatial prior enables the target speaker's parameter to avoid overfitting to noise or/and interference. Experimental results show that the spatial prior reduces the WER from 38.4% to 29.2% in a conversation recognition task compared with the CGMM-based approach without the prior, and outperforms a conventional online speech enhancement approach.

A Consolidated Perspective on Multimicrophone Speech Enhancement and Source Separation

Speech enhancement and separation are core problems in audio signal processing, with commercial applications in devices as diverse as mobile phones, conference call systems, hands-free systems, or hearing aids. In addition, they are crucial preprocessing steps for noise-robust automatic speech and speaker recognition. Many devices now have two to eight microphones. The enhancement and separation capabilities offered by these multichannel interfaces are usually greater than those of single-channel interfaces. Research in speech enhancement and separation has followed two convergent paths, starting with microphone array processing and blind source separation, respectively. These communities are now strongly interrelated and routinely borrow ideas from each other. Yet, a comprehensive overview of the common foundations and the differences between these approaches is lacking at present. In this paper, we propose to fill this gap by analyzing a large number of established and recent techniques according to four transverse axes: 1 the acoustic impulse response model, 2 the spatial filter design criterion, 3 the parameter estimation algorithm, and 4 optional postfiltering. We conclude this overview paper by providing a list of software and data resources and by discussing perspectives and future trends in the field.

Bayesian feature enhancement using independent vector analysis and reverberation parameter re-estimation for noisy reverberant speech recognition

Because speech recorded by distant microphones in real-world environments is contaminated by both additive noise and reverberation, the automatic speech recognition (ASR) performance is seriously degraded due to the mismatch between the training and testing environments. In the previous studies, some of the authors proposed a Bayesian feature enhancement (BFE) method with re-estimation of reverberation filter parameters for reverberant speech recognition and a BFE method employing independent vector analysis (IVA) to deal with speech corrupted by additive noise. Although both of them accomplish significant improvements in either reverberation- or noise-robust ASR, most of the real-world environments involve both additive noise and reverberation. For robust ASR in the noisy reverberant environments, in this paper, we present a hidden-Markov-model (HMM)-based BFE method using IVA and reverberation parameter re-estimation (RPR) to remove additive and reverberant distortion components in speech acquired by multi-microphones effectively by introducing Bayesian inference in the observation model of input speech features. Experimental results show that the presented method can further reduce the word error rates (WERs) compared with the BFE methods based on conventional noise and/or reverberation models and combinations of the BFE methods for reverberation- or noise-robust ASR.

Dual‐channel VTS feature compensation for noise‐robust speech recognition on mobile devices

One way to improve automatic speech recognition (ASR) performance on the latest mobile devices, which can be employed on a variety of noisy environments, consists of taking advantage of the small microphone arrays embedded in them. Since the performance of the classic beamforming techniques with small microphone arrays is rather limited, specific techniques are being developed to efficiently exploit this novel feature for noise-robust ASR purposes. In this study, a novel dual-channel minimum mean square error-based feature compensation method relying on a vector Taylor series (VTS) expansion of a dual-channel speech distortion model is proposed. In contrast to the single-channel VTS approach (which can be considered as the state-of-the-art for feature compensation), the authors’ technique particularly benefits from the spatial properties of speech and noise. Their proposal is assessed on a dual-microphone smartphone (a particular case of interest) by means of the AURORA2-2C synthetic corpus. Word recognition results, also validated with real noisy speech data, demonstrate the higher accuracy of their method by clearly outperforming minimum variance distortionless response beamforming and a single-channel VTS feature compensation approach, especially at low signal-to-noise ratios.

Spectral Reconstruction and Noise Model Estimation Based on a Masking Model for Noise Robust Speech Recognition

An effective way to increase noise robustness in automatic speech recognition (ASR) systems is feature enhancement based on an analytical distortion model that describes the effects of noise on the speech features. One of such distortion models that has been reported to achieve a good trade-off between accuracy and simplicity is the masking model. Under this model, speech distortion caused by environmental noise is seen as a spectral mask and, as a result, noisy speech features can be either reliable (speech is not masked by noise) or unreliable (speech is masked). In this paper, we present a detailed overview of this model and its applications to noise robust ASR. Firstly, using the masking model, we derive a spectral reconstruction technique aimed at enhancing the noisy speech features. Two problems must be solved in order to perform spectral reconstruction using the masking model: (1) mask estimation, i.e. determining the reliability of the noisy features, and (2) feature imputation, i.e. estimating speech for the unreliable features. Unlike missing data imputation techniques where the two problems are considered as independent, our technique jointly addresses them by exploiting a priori knowledge of the speech and noise sources in the form of a statistical model. Secondly, we propose an algorithm for estimating the noise model required by the feature enhancement technique. The proposed algorithm fits a Gaussian mixture model to the noise by iteratively maximising the likelihood of the noisy speech signal so that noise can be estimated even during speech-dominating frames. A comprehensive set of experiments carried out on the Aurora-2 and Aurora-4 databases shows that the proposed method achieves significant improvements over the baseline system and other similar missing data imputation techniques.

Modeling State-Conditional Observation Distribution Using Weighted Stereo Samples for Factorial Speech Processing Models

This paper investigates the effectiveness of factorial speech processing models in noise-robust automatic speech recognition tasks. For this purpose, the paper proposes an idealistic approach for modeling state-conditional observation distribution of factorial models based on weighted stereo samples. This approach is an extension to previous single pass retraining for ideal model compensation which is extended here to support multiple audio sources. Non-stationary noises can be considered as one of these audio sources with multiple states. Experiments of this paper over the set A of the Aurora 2 dataset show that recognition performance can be improved by this consideration. The improvement is significant in low signal to noise energy conditions, up to 4% absolute word recognition accuracy. In addition to the power of the proposed method in accurate representation of state-conditional observation distribution, it has an important advantage over previous methods by providing the opportunity to independently select feature spaces for both source and corrupted features. This opens a new window for seeking better feature spaces appropriate for noisy speech, independent from clean speech features.

Feature enhancement of reverberant speech by distribution matching and non-negative matrix factorization

This paper describes a novel two-stage dereverberation feature enhancement method for noise-robust automatic speech recognition. In the first stage, an estimate of the dereverberated speech is generated by matching the distribution of the observed reverberant speech to that of clean speech, in a decorrelated transformation domain that has a long temporal context in order to address the effects of reverberation. The second stage uses this dereverberated signal as an initial estimate within a non-negative matrix factorization framework, which jointly estimates a sparse representation of the clean speech signal and an estimate of the convolutional distortion. The proposed feature enhancement method, when used in conjunction with automatic speech recognizer back-end processing, is shown to improve the recognition performance compared to three other state-of-the-art techniques.

Kernel Power Flow Orientation Coefficients for Noise-Robust Speech Recognition

Noise-robustness has become a crucial parameter in Automatic Speech Recognition (ASR) systems today with their increased use in noise-filled real-world environments. One way to address this issue is to develop features that are innately noise-robust. The Kernel Power flow Orientation Coefficients (KPOCs) are a novel feature set based on spectro-temporal analysis that uses a bank of 2D kernels to extract the dominant orientation of the power flow at each point in the auditory spectrogram of the speech signal. The collection of dominant power flow orientation angles forms a novel representation of the speech signal named the Power flow Orientation Spectrogram (POS), which is innately resistant to the spectral masking introduced by the presence of noise and reverberation. This approach not only grants KPOC its noise robustness, but also keeps the number of output coefficients inherently small, thus eliminating the need of the feature dimensionality reduction otherwise necessary in the conventional the spectro-temporal approach. KPOCs performance has been evaluated on three experimental frameworks, and the results have shown that they outperform a number of well-known noise-robust features for average and low SNRs. The relative improvement in Word Recognition Accuracy (WRA) to the classic Mel Frequency Cepstral Coefficients (MFCCs) for the Aurora 2 task goes from 32% up to 190% for SNRs in the range from 10 down to $ - {\hbox {5}}~\hbox{dB}$ . The experimental results also show that in clean training the performance of KPOC approaches that of the state-of-the-art noise-robust ASR frontends in all noise scenarios for small vocabulary ASR tasks.

Minimum Mean-Square Error Estimation of Mel-Frequency Cepstral Features–A Theoretically Consistent Approach

In this work, we consider the problem of feature enhancement for noise-robust automatic speech recognition (ASR). We propose a method for minimum mean-square error (MMSE) estimation of mel-frequency cepstral features, which is based on a minimum number of well-established, theoretically consistent statistical assumptions. More specifically, the method belongs to the class of methods relying on the statistical framework proposed in Ephraim and Malah's original work (“Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator,” IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-32, no. 6, 1984). The method is general in that it allows MMSE estimation of mel-frequency cepstral coefficients (MFCC's), cepstral-mean subtracted (CMS-) MFCC's, autoregressive-moving-average (ARMA)-filtered CMS-MFCC's, velocity, and acceleration coefficients. In addition, the method is easily modified to take into account other compressive non-linearities than the logarithm traditionally used for MFCC computation. In terms of MFCC estimation performance, as measured by MFCC mean-square error, the proposed method shows performance which is identical to or better than other state-of-the-art methods. In terms of ASR performance, no statistical difference could be found between the proposed method and the state-of-the-art methods. We conclude that existing state-of-the-art MFCC feature enhancement algorithms within this class of algorithms, while theoretically suboptimal or based on theoretically inconsistent assumptions, perform close to optimally in the MMSE sense.

Noise Robust Automatic Speech Recognition Scheme with Histogram of Oriented Gradient Features

In this paper, we propose a novel technique for noise robust automatic speech recognition (ASR). The development of ASR techniques has made it possible to recognize isolated words with a near perfect word recognition rate. However, in a highly noisy environment, a distinct mismatch between the trained speech and the test data results in a significantly degraded word recognition rate (WRA). Unlike conventional ASR systems employing Mel-frequency cepstral coefficients (MFCCs) and a hidden Markov model (HMM), this study employ histogram of oriented gradient (HOG) features and a Support Vector Machine (SVM) to ASR tasks to overcome this problem. Our proposed ASR system is less vulnerable to external interference noise, and achieves a higher WRA compared to a conventional ASR system equipped with MFCCs and an HMM. The performance of our proposed ASR system was evaluated using a phonetically balanced word (PBW) set mixed with artificially added noise.

Sparse coding of the modulation spectrum for noise-robust automatic speech recognition

AbstractThe full modulation spectrum is a high-dimensional representation of one-dimensional audio signals. Most previous research in automatic speech recognition converted this very rich representation into the equivalent of a sequence of short-time power spectra, mainly to simplify the computation of the posterior probability that a frame of an unknown speech signal is related to a specific state. In this paper we use the raw output of a modulation spectrum analyser in combination with sparse coding as a means for obtaining state posterior probabilities. The modulation spectrum analyser uses 15 gammatone filters. The Hilbert envelope of the output of these filters is then processed by nine modulation frequency filters, with bandwidths up to 16 Hz. Experiments using the AURORA-2 task show that the novel approach is promising. We found that the representation of medium-term dynamics in the modulation spectrum analyser must be improved. We also found that we should move towards sparse classification, by modifying the cost function in sparse coding such that the class(es) represented by the exemplars weigh in, in addition to the accuracy with which unknown observations are reconstructed. This creates two challenges: (1) developing a method for dictionary learning that takes the class occupancy of exemplars into account and (2) developing a method for learning a mapping from exemplar activations to state posterior probabilities that keeps the generalization to unseen conditions that is one of the strongest advantages of sparse coding.