Noisy Speech Recognition Research Articles

Task-Oriented Document-Grounded Dialog Systems by HLTPR@RWTH for DSTC9 and DSTC10

This paper summarizes our contributions to the document-grounded dialog tasks at the 9th and 10th Dialog System Technology Challenges (DSTC9 and DSTC10). In both iterations the task consists of three subtasks: first detect whether the current turn is knowledge seeking, second select a relevant knowledge document, and third generate a response grounded on the selected document. For DSTC9 we proposed different approaches to make the selection task more efficient. The best method, Hierarchical Selection, actually improves the results compared to the original baseline and gives a speedup of 24x. In the DSTC10 iteration of the task, the challenge was to adapt systems trained on written dialogs to perform well on noisy automatic speech recognition transcripts. Therefore, we proposed data augmentation techniques to increase the robustness of the models as well as methods to adapt the style of generated responses to fit well into the proceeding dialog. Additionally, we proposed a noisy channel model that allows for increasing the factuality of the generated responses. In addition to summarizing our previous contributions, in this work, we also report on a few small improvements and reconsider the automatic evaluation metrics for the generation task which have shown a low correlation to human judgments.

End-to-End Noisy Speech Recognition Using Fourier and Hilbert Spectrum Features

Despite the progress of deep neural networks over the last decade, the state-of-the-art speech recognizers in noisy environment conditions are still far from reaching satisfactory performance. Methods to improve noise robustness usually include adding components to the recognition system that often need optimization. For this reason, data augmentation of the input features derived from the Short-Time Fourier Transform (STFT) has become a popular approach. However, for many speech processing tasks, there is an evidence that the combination of STFT-based and Hilbert–Huang transform (HHT)-based features improves the overall performance. The Hilbert spectrum can be obtained using adaptive mode decomposition (AMD) techniques, which are noise-robust and suitable for non-linear and non-stationary signal analysis. In this study, we developed a DeepSpeech2-based recognition system by adding a combination of STFT and HHT spectrum-based features. We propose several ways to combine those features at different levels of the neural network. All evaluations were performed using the WSJ and CHiME-4 databases. Experimental results show that combining STFT and HHT spectra leads to a 5–7% relative improvement in noisy speech recognition.

A Multi-Target SNR-Progressive Learning Approach to Regression Based Speech Enhancement

We propose a multi-target, signal-to-noise-ratio (SNR)-progressive learning (SNR-PL) framework for regression based speech enhancement (SE). At low SNR levels, it is often not easy to directly learn the complicated regression required in SE. We therefore decompose the original SE problem of mapping noisy to clean speech features, with a large SNR gap, into a series of sub-problems, each with a small SNR increment and presumably easier to learn. In our configurations, each hidden layer of the proposed regression neural network is guided to explicitly learn an intermediate target with a specified but small SNR gain. Tested on both deep neural network (DNN) and long short-term memory (LSTM) architectures, SNR-PL consistently outperforms the conventional “black box” DNN framework in terms of both objective measure superiority and network model compactness. Furthermore, with the best configured LSTM-based SNR-PL model, we often observe that the performance is easily saturated or even degraded when increasing the number of intermediate targets, due to the fact that useful information is lost in dimension reduction when involving more target layers. Accordingly, to address this information loss issue, we explore densely connected networks on top of the LSTM structure where the input and the preceding intermediate targets are concatenated together to learn the next target. Finally, to fully utilize the rich and complementary information of intermediate targets, a simple post-processing strategy is adopted to further improve the performance. Evaluated on the simulation speech data, experimental results in unseen noises cases demonstrate that the proposed approach consistently performs better than the conventional LSTM approach in terms of objective speech enhancement measures for speech intelligibility and quality. Furthermore, when evaluated on real data provided by the CHiME-4 Challenge for automatic speech recognition (ASR) of noisy microphone array speech, we show that the proposed approach with intermediate outputs can directly improve the ASR performance, while the conventional LSTM approach increases the word error rate.

Feature Level Solution to Noise Robust Speech Recognition in the context of Tonal Languages

Performance of a speech recognition system is highly dependent on the operational environments. The mismatched ambient conditions have adverse impact on the performance of an Automatic Speech Recognition (ASR) system. The speech parameterization techniques for tonal speech recognition are different from those used for non-tonal speech recognition. It is due to the fact that tonal speech has two components – basic linguistic unit and tone. The basic linguistic unit with different tones convey different meanings. Therefore, the feature set used for tonal speech recognition must have the capability to representing both of them. Tone is determined by the fundamental frequency of the speech signal which is highly sensitive to noise. Since at the time of parameterization of the non-tonal speech recognition systems, these highly noise-sensitive tone related information are discarded, the traditional noise elimination methods used for non-tonal speech recognition fail to deliver robust performance in tonal speech recognition. In the present study, we have analyze the performance of different commonly used feature sets for noisy tonal speech recognition. Hidden Markov Model (HMM) based speech recognizer has been used for performance evaluation. Noise elimination techniques sub-band spectral subtraction and Wiener filter have been used for noise reduction and their relative performance have been evaluated.

Japanese speech intelligibility estimation and prediction using objective intelligibility indices under noisy and reverberant conditions

Objective measures of intelligibility are preferable to subjective ones in the evaluation of speech systems used in real environments. In this study, subjective evaluations of eight types of indoor noise environments were used to compare four intelligibility indices to objectively evaluate Japanese speech intelligibility. These indices were as follows: short-time objective intelligibility (STOI), which has been widely used in recent years; speech intelligibility prediction based on mutual information (SIMI), which is derived from STOI; extended STOI (ESTOI), which is an improved version of STOI; and frequency weighted segmental signal to noise ratio (fwSNRseg), which incorporates both time and frequency components. These indices were subjectively evaluated in the eight noisy environments included in the corpus and environments for noisy speech recognition 4 (CENSREC-4) dataset using the familiarity-controlled word lists 2007 (FW07) as the speech data for the intelligibility evaluations. The results of the subjective evaluation of the four indices were then used to train predictive intelligibility estimation models. We evaluated the model performance using cross validation, which involved repeated training of seven of the eight environments and predicting the speech intelligibility under the remaining one environment. In the simulation results, the prediction accuracy of the SIMI index was significantly higher than that of the other indices, with a root mean squared error of 0.160 and a correlation coefficient of 0.934.

Wiener Filter in Wavelet Domain for Mel-LPC based Noisy Speech Recognition

Wiener Filter in Wavelet Domain for Mel-LPC based Noisy Speech Recognition

Noisy Speech Recognition by Mel-LPC based AR-HMM with Power and Time Derivative Parameters

Noisy Speech Recognition by Mel-LPC based AR-HMM with Power and Time Derivative Parameters

Factored front-end CMLLR for joint speaker and environment normalization under DNN-HMM

In this paper Factored front-end CMLLR (F-FE-CMLLR) is investigated for the task of joint speaker and environment normalization in the frame-work of DNN-HMM acoustic modeling. It is a feature-space transform comprising of the composition of front-end CMLLR for environment and global CMLLR for speaker normalizations. The transform is applied to the input noisy, speaker-independent features and the resulting canonical features are passed on to the DNN-HMM for training and decoding. Two estimation procedures for F-FE-CMLLR are investigated, namely, sequential and iterative training. One of the key attributes of F-FE-CMLLR is that in the iterative training paradigm it is likely to foster acoustic factorization, which enables more effective transfer of the environment transform from one condition to another. Moreover, being a feature space transform, it becomes straightforward to use it in the context of DNN-HMM acoustic modeling. The performance of the proposed scheme is evaluated on the Aurora-4 noisy speech recognition task. The dominant acoustic factors in the task are the microphone variability, additive noise with varying SNRs and speakers. It is shown that F-FE-CMLLR yields a large improvement in performance compared to the baseline features, which are processed with CMLLR for speaker adaptive training (SAT). The improvement is observed in all acoustic conditions existing in the test sets. Moreover, the iterative training of F-FE-CMLLR outperforms sequential training under all test conditions. Specifically, when all three type of acoustic conditions co-exist, the sequential training yields a 13% relative improvement over SAT features. The iterative training provides an additional improvement on the top, amounting to an 18% relative gain over-all. It is argued that the improvement over sequential training is observed due to acoustic factorization that holds in an implicit sense.

Phoneme class based feature adaptation for mismatch acoustic modeling and recognition of distant noisy speech

Distant speech capture in lecture halls and auditoriums offers unique challenges in algorithm development for automatic speech recognition. In this study, a new adaptation strategy for distant nois...

Autocorrelation-based noise subtraction method with smoothing, overestimation, energy, and cepstral mean and variance normalization for noisy speech recognition

Autocorrelation domain is a proper domain for clean speech signal and noise separation. In this paper, a method is proposed to decrease effects of noise on the clean speech signal, autocorrelation-based noise subtraction (ANS). Then to deal with the error introduced by assumption that noise and clean speech signal are uncorrelated, two methods are proposed. Also to improve recognition rate of speech recognition system, overestimation parameter is used. Finally, with the addition of energy and cepstral mean and variance normalization to features of speech, recognition rate has improved considerably in comparison to standard features and other correlation-based methods. The proposed methods are tested on the Aurora 2 database. Between different proposed methods, autocorrelation-based noise subtraction method with smoothing, overestimation, energy, and cepstral mean and variance normalization (ANSSOEMV) method has a best recognition rate improvement in average than MFCC features which is 64.91% on the Aurora 2 database.

Bayesian feature enhancement using independent vector analysis and reverberation parameter re-estimation for noisy reverberant speech recognition

Because speech recorded by distant microphones in real-world environments is contaminated by both additive noise and reverberation, the automatic speech recognition (ASR) performance is seriously degraded due to the mismatch between the training and testing environments. In the previous studies, some of the authors proposed a Bayesian feature enhancement (BFE) method with re-estimation of reverberation filter parameters for reverberant speech recognition and a BFE method employing independent vector analysis (IVA) to deal with speech corrupted by additive noise. Although both of them accomplish significant improvements in either reverberation- or noise-robust ASR, most of the real-world environments involve both additive noise and reverberation. For robust ASR in the noisy reverberant environments, in this paper, we present a hidden-Markov-model (HMM)-based BFE method using IVA and reverberation parameter re-estimation (RPR) to remove additive and reverberant distortion components in speech acquired by multi-microphones effectively by introducing Bayesian inference in the observation model of input speech features. Experimental results show that the presented method can further reduce the word error rates (WERs) compared with the BFE methods based on conventional noise and/or reverberation models and combinations of the BFE methods for reverberation- or noise-robust ASR.

Prior-based Binary Masking and Discriminative Methods for Reverberant and Noisy Speech Recognition Using Distant Stereo Microphones

Prior-based Binary Masking and Discriminative Methods for Reverberant and Noisy Speech Recognition Using Distant Stereo Microphones

Auditory driven subband speech enhancement for automatic recognition of noisy speech

Speech recognizers achieve high recognition accuracy under quiet acoustic environments, but their performance degrades drastically when they are deployed in real environments, where the speech is degraded by additive ambient noise. This paper advocates a two phase approach for robust speech recognition in such environment. Firstly, a front end subband speech enhancement with adaptive noise estimation (ANE) approach is used to filter the noisy speech. The whole noisy speech spectrum is portioned into eighteen dissimilar subbands based on Bark scale and noise power from each subband is estimated by the ANE approach, which does not require the speech pause detection. Secondly, the filtered speech spectrum is processed by the non parametric frequency domain algorithm based on human perception along with the back end building a robust classifier to recognize the utterance. A suite of experiments is conducted to evaluate the performance of the speech recognizer in a variety of real environments, with and without the use of a front end speech enhancement stage. Recognition accuracy is evaluated at the word level, and at a wide range of signal to noise ratios for real world noises. Experimental evaluations show that the proposed algorithm attains good recognition performance when signal to noise ratio is lower than 5 dB.

Improvement of the noisy speech recognition using method of thresholds and emphasizing wavelet coefficients for clean speech

In recent years, speech recognition under the noisy environment is one of the very important technologies. This study improves a speech recognition method for the signals under the noisy environment using method of thresholds and emphasizing wavelet coefficients for clean signal data. Noisy speech recognitions are extremely difficult problem. To analyze noise problem, in general, the majority of people have used the Fourier analysis. But the Fourier transform reveals only the frequency information. The general noise filters reduce specific frequency band contained both signal and noise. It is difficult to eliminate only noise component from a signal containing noise components. To overcome this difficulty, we applied the wavelet analysis. In this study, we improve the speech recognition under the noisy environment by bringing it close to reference data and noisy input data by modifying the spectrum by the wavelet transform using thresholds and emphasizing methods for reference clean speech data. We apply ...

Nanophotonic reservoir computing for noisy speech recognition

An optical architecture for reservoir computing based on nanophotonic crystal cavities is introduced in this paper. To evaluate the effectiveness of the proposed structure, its performance has been studied in the context of noisy speech recognition. The accuracy of noise-free speech recognition using nanophotonic reservoir computing is 100 % and for noisy speech signal with signal-to-noise ratios of 0–10 dB is from 92.4 to 98.3 %. It shows an improvement over the classic reservoir from 9.3 to 18.4 %.

Feature Extraction Method for Improving Speech Recognition in Noisy Environments

The paper presents a feature extraction method, named as Normalized Gammachirp Cepstral Coefficients (NGCC) that incorporates the properties of the peripheral auditory system to improve robustness in noisy speech recognition. The proposed method is based on a second order low-pass filter and normalized gammachirp filterbank to emulate the mechanisms performed in the outer/middle ear and cochlea. The speech recognition performance of this method is conducted on the speech signals in real-world noisy environments. Experimental results demonstrate that method outperformed the classical feature extraction methods in terms of speech recognition rate. The used Hidden Markov Models based speech recognition system is employed on the HTK 3.4.1 platform (Hidden Markov Model Toolkit).

Performance estimation of noisy speech recognition using spectral distortion and recognition task complexity

Performance estimation of noisy speech recognition using spectral distortion and recognition task complexity

Speech Recognition using ERB-like Admissible Wavelet Packet Decomposition based on Perceptual sub-band Weighting

ABSTRACTIn the recent years, wavelet packet (WP) transform has been used as an important speech representation tool. WP-based acoustic features have found to be more efficient than the short-time Fourier transform (STFT)-based features to capture the information of unvoiced phoneme from continuous speech. In this paper, a new 24 sub-band equivalent rectangular bandwidth (ERB)-like wavelet filter is proposed by employing perceptual Wiener filter on each sub-band of decomposed noisy speech. Wiener filtered output is then proceeded according to the Johnston model to calculate auditory masking threshold for each wavelet decomposed sub-band. This threshold is used to design the perceptual sub-band weighting (PSW) filter. The output from each perceptually weighted sub-band is processed further to calculate acoustic front end features. This technique aims to enhance the noisy speech signal by using standard Wiener filter on psychoacoustically motivated decomposed wavelet sub-band by controlling the sub-band weighting factor. Hindi continuous digit database and TIMIT database is used to evaluate the performance of the proposed feature. Obtained results show that proposed feature is effective for noisy speech recognition compared to some recently proposed feature extraction techniques.

Noise Robust Speech Parameterization using Relative Spectra and Auditory Filterbank

In the present study, a new feature extraction method based on relative spectra and gammachirp auditory filterbank is proposed for robust noisy speech recognition. The relative spectra filtering are applied to the log of the output of the gammachirp filterbank which incorporates the properties of the cochlear filter in order to remove uncorrelated additive noise components. The performances of this method have been evaluated on the isolated speech word corrupted by real-world noisy environments using the continuous Gausian-Mixture density Hidden Markov Model. The evaluation of the experimental results shows that the proposed method achieves best recognition rates compared to the conventional techniques like Perceptual Linear Prediction (PLP), Linear Predictive Cepstral Coefficients (LPCC) and Mel-Frequency Cepstral Coefficients (MFCC).

Laplace Group Sensing for Acoustic Models

This paper presents the group sparse learning for acoustic models where a sequence of acoustic features is driven by Markov chain and each feature vector is represented by groups of basis vectors. The group of common bases represents the features across Markov states within a regression class. The group of individual basis compensates the intra-state residual information. Laplace distribution is used as the sparse prior of sensing weights for group basis representation. Laplace parameter serves as regularization parameter or automatic relevance determination which controls the selection of relevant bases for acoustic modeling. The groups of regularization parameters and basis vectors are estimated from training data by maximizing the marginal likelihood over sensing weights which is implemented by Laplace approximation using the Hessian matrix and the maximum a posteriori parameters. Model uncertainty is compensated through full Bayesian treatment. The connection of Laplace group sensing to lasso regularization is illustrated. Experiments on noisy speech recognition show the robustness of group sparse acoustic models in presence of different noise types and SNRs.