Noisy Speech Samples Research Articles

Speech enhancement augmentation for robust speech recognition in noisy environments

Abstract. The use of augmentations as a data enrichment method has become an important element in improving the performance of speech recognition systems. To work effectively in noisy conditions, augmentation is usually used to simulate the presence of background noise. However, the quality of speech recognition on samples pre-processed by noise reduction models does not increase. This paper proposes a new approach to speech data augmentation when training ASR systems, intended for their joint use with models for speech enhancement. It was based on the creation of several additional data samples containing speech samples processed by the enhancement model. The proposed approach was tested on the E-Branchformer neural network model using data from the Librispeech set. The quality of speech samples was assessed using the DNSMOS metric. By means of a 100-hour sample of clean speech samples it was shown that the proposed augmentation allows for an improvement in the WER metric of more than 4% in absolute value compared to the generally accepted approach based on adding noisy speech samples. Experiments on 960-hour data demonstrated the robustness of this approach as the training set size increased.

Non-Parallel Whisper-to-Normal Speaking Style Conversion Using Auxiliary Classifier Variational Autoencoder

This paper is concerned with non-parallel whisper-to-normal speaking-style conversion (W2N-SC), which converts whispered speech into normal speech without using parallel training data. The most relevant to this task is voice conversion (VC), which converts one speaker’s voice to another. However, it differs in three main respects. First, unlike normal speech, whispered speech contains no or little pitch information. Second, whispered speech usually has significantly less energy than normal speech and is therefore more susceptible to external noise. Third, in the actual usage scenario of W2N-SC, users may suddenly switch voice modes from whispered to normal speech, or vice versa, meaning that the speaking-style of input speech cannot be assumed in advance. To clarify whether existing VC techniques can successfully handle these task-specific concerns and how they should be modified to better address them, we consider a variational autoencoder (VAE)-based VC method as a baseline and examine what modifications to this method would be effective for the current task. Specifically, we study the effects of (1) a self-supervised training scheme called fill-in-frames (FIF), (2) data augmentation (DA) using noisy speech samples, and (3) an architecture that allows for any-to-many conversions. Through experimental evaluation of the W2N-SC and speaker conversion tasks, we confirmed that, especially in the W2N-SC task, the version incorporating the above modifications works better than the baseline VC model applied as is.

Unsupervised Speech Enhancement Using Dynamical Variational Autoencoders

Dynamical variational autoencoders (DVAEs) are a class of deep generative models with latent variables, dedicated to model time series of high-dimensional data. DVAEs can be considered as extensions of the variational autoencoder (VAE) that include temporal dependencies between successive observed and/or latent vectors. Previous work has shown the interest of using DVAEs over the VAE for speech spectrograms modeling. Independently, the VAE has been successfully applied to speech enhancement in noise, in an unsupervised noise-agnostic set-up that requires neither noise samples nor noisy speech samples at training time, but only requires clean speech signals. In this paper, we extend these works to DVAE-based single-channel unsupervised speech enhancement, hence exploiting both speech signals unsupervised representation learning and dynamics modeling. We propose an unsupervised speech enhancement algorithm that combines a DVAE speech prior pre-trained on clean speech signals with a noise model based on nonnegative matrix factorization, and we derive a variational expectation-maximization (VEM) algorithm to perform speech enhancement. The algorithm is presented with the most general DVAE formulation and is then applied with three specific DVAE models to illustrate the versatility of the framework. Experimental results show that the proposed DVAE-based approach outperforms its VAE-based counterpart, as well as several supervised and unsupervised noise-dependent baselines, especially when the noise type is unseen during training.

Noise Reduction Using Neural Lateral Inhibition for Speech Enhancement

Recurrent spiking neurons with lateral inhibition connection play a vital role in human’s brain functional abilities. In this paper, we propose a novel noise reduction method that is based on neuron rate coding and bio-inspired spiking neural network architecture. The excitatory-inhibitory topology in the network acts as the temporal characteristic synchrony and coincidence detector that removes uncorrelated noisy spikes. A LIF source encoder is introduced along with the network. The network uses generated binary Short-Time Fourier Transform (STFT) masks according to the rate of processed spike train, which is used to reconstruct the denoised speech signal. The technique is evaluated on noisy speech samples with 5 types of real-world additive noise with different noise strength.

An Improved Fully Convolutional Network Based on Post-Processing with Global Variance Equalization and Noise-Aware Training for Speech Enhancement

An improved fully convolutional network based on post-processing with global variance (GV) equalization and noise-aware training (PN-FCN) for speech enhancement model is proposed. It aims at reducing the complexity of the speech improvement system, and it solves overly smooth speech signal spectrogram problem and poor generalization capability. The PN-FCN is fed with the noisy speech samples augmented with an estimate of the noise. In this way, the PN-FCN uses additional online noise information to better predict the clean speech. Besides, PN-FCN uses the global variance information, which improve the subjective score in a voice conversion task. Finally, the proposed framework adopts FCN, and the number of parameters is one-seventh of deep neural network (DNN). Results of experiments on the Valentini-Botinhaos dataset demonstrate that the proposed framework achieves improvements in both denoising effect and model training speed.

Examination of Energy Based Voice Activity Detection Algorithms for Noisy Speech Signals

This paper examines the behavior of two different energy-based voice activity detector (VAD) algorithms for noisy input signals. The examined detectors use time-domain methods to find speech boundaries. Time-domain short time energy features and/or zero-crossing rate of speech signals are used to evaluate the performance of the methods. In the first stage of both algorithms, time-domain short-time energy (STE) features are calculated for each speech segment. Then energy ratios and threshold values are used to detect any voicing activity of speech signals. The decision threshold value is calculated by evaluating the average STE of an initial silence period. The effectiveness of the selected methods is tested for clean and noisy speech samples. The methods are tested using the noisy speech signals under different SNR levels. The results indicated that both methods achieve a reasonable accuracy as low as an SNR value nearly 0dB with a slowly decreasing performance. But, under 0dB SNR, both methods lose their effectiveness against noisy conditions.

A Signal Subspace Speech Enhancement Approach Based on Joint Low-Rank and Sparse Matrix Decomposition

Abstract Subspace-based methods have been effectively used to estimate enhanced speech from noisy speech samples. In the traditional subspace approaches, a critical step is splitting of two invariant subspaces associated with signal and noise via subspace decomposition, which is often performed by singular-value decomposition or eigenvalue decomposition. However, these decomposition algorithms are highly sensitive to the presence of large corruptions, resulting in a large amount of residual noise within enhanced speech in low signal-to-noise ratio (SNR) situations. In this paper, a joint low-rank and sparse matrix decomposition (JLSMD) based subspace method is proposed for speech enhancement. In the proposed method, we firstly structure the corrupted data as a Toeplitz matrix and estimate its effective rank value for the underlying clean speech matrix. Then the subspace decomposition is performed by means of JLSMD, where the decomposed low-rank part corresponds to enhanced speech and the sparse part corresponds to noise signal, respectively. An extensive set of experiments have been carried out for both of white Gaussian noise and real-world noise. Experimental results show that the proposed method performs better than conventional methods in many types of strong noise conditions, in terms of yielding less residual noise and lower speech distortion.

English

In this work, an automatic gender and age recognizer from speech is investigated. The relevant features to gender recognition are selected from the first four formant frequencies and twelve MFCCs and feed the SVM classifier. While the relevant features to age has been used with k-NN classifier for the age recognizer model, using MATLAB as a simulation tool. A special selection of robust features is used in this work to improve the results of the gender and age classifiers based on the frequency range that the feature represents. The gender and age classification algorithms are evaluated using 114 (clean and noisy) speech samples uttered in Kurdish language. The model of two classes (adult males and adult females) gender recognition, reached 96% recognition accuracy. While for three categories classification (adult males, adult females, and children), the model achieved 94% recognition accuracy. For the age recognition model, seven groups according to their ages are categorized. The model performance after selecting the relevant features to age achieved 75.3%. For further improvement a de-noising technique is used with the noisy speech signals, followed by selecting the proper features that are affected by the de-noising process and result in 81.44% recognition accuracy.

A speech enhancement approach based on noise classification

For speech enhancement, most existing approaches do not consider the differences, between various types of noise, which significantly affect the performance of speech enhancement. In this paper, we propose a novel speech enhancement approach by taking into account the different characteristic statistical properties of various noise on the basis of noise classification. To classify noise, an effective noise classification method is firstly developed by exploiting the features of noise energy distribution in the Bark domain. Then, based on the noise types, the speech enhancement approach is obtained by forming the optimal parameter combinations for the optimally modified log-spectral amplitude (OM-LSA) speech estimator with the improved minima controlled recursive averaging (IMCRA) noise estimator, where the parameter combinations consisting of the smoothing parameters for smoothing the noisy power spectrum and the recursive averaging in the noise spectrum estimation as well as the weighting factor for the a priori SNR estimation, are built through the enhancement of noisy speech samples. Finally, extensive experiments are carried out in terms of objective evaluation under various noise conditions, and the experimental results show that the proposed approach yields better performance compared with the conventional OM-LSA with IMCRA in speech enhancement.

Source and system features for phone recognition

In this work, we have explored excitation source features in addition to vocal tract system features to improve the performance of phone recognition systems (PRSs). The excitation source information is derived by processing linear prediction residual of the speech signal. The vocal tract information is captured using Mel-frequency cepstral coefficient features. The PRSs are developed using hidden Markov models. The robustness of proposed excitation source features is demonstrated using white and babble noisy speech samples. In this work, TIMIT and Bengali speech databases are used for developing PRSs. The tandem PRSs are developed using the phone posteriors obtained from feedforward neural networks. From the results, it is observed that the tandem PRSs developed using the combination of excitation source and vocal tract system features, outperform the conventional tandem systems developed using system features alone. It is also observed that the PRSs developed using the combination of excitation source and vocal tract features, are more robust to noise than the PRSs developed using vocal tract features alone.

Bayesian on-line spectral change point detection: a soft computing approach for on-line ASR

Current automatic speech recognition (ASR) works in off-line mode and needs prior knowledge of the stationary or quasi-stationary test conditions for expected word recognition accuracy. These requirements limit the application of ASR for real-world applications where test conditions are highly non-stationary and are not known a priori. This paper presents an innovative frame dynamic rapid adaptation and noise compensation technique for tracking highly non-stationary noises and its application for on-line ASR. The proposed algorithm is based on a soft computing model using Bayesian on-line inference for spectral change point detection (BOSCPD) in unknown non-stationary noises. BOSCPD is tested with the MCRA noise tracking technique for on-line rapid environmental change learning in different non-stationary noise scenarios. The test results show that the proposed BOSCPD technique reduces the delay in spectral change point detection significantly compared to the baseline MCRA and its derivatives. The proposed BOSCPD soft computing model is tested for joint additive and channel distortions compensation (JAC)-based on-line ASR in unknown test conditions using non-stationary noisy speech samples from the Aurora 2 speech database. The simulation results for the on-line AR show significant improvement in recognition accuracy compared to the baseline Aurora 2 distributed speech recognition (DSR) in batch-mode.

Noisy Speech Recognition Based on RBF Neural Network

A noisy speech recognition method based on improved RBF neural network is presented, which the parameters of hidden layer are trained dynamically, and Akaike’s final prediction error standard (FPE) is employed to simplify the network. Comparing with two other training methods of RBF network, experimental results based on noisy speech samples show that this method achieves excellent performance in terms of recognition rate and recognition speed.

Study on the Selection of Specific Filters for Enhancement of Recorded Speech for Speaker Identification

Understanding the noise characteristics for finding appropriate filtering technique/s so as to obtain sufficiently clear speech samples for Speaker Identification, is one of the challenging tasks in Forensic Acoustics. Speaker's idiosyncratic speech should not be affected when the noise reduction is carried out; otherwise, Speaker Identification becomes highly erroneous. We have collected fifty noisy speech samples reported to be recorded in different modes from actual crime cases received in the laboratory. The samples are analyzed after subjecting to various filtering techniques and compared with the clear speech mixed with the noise collected from non-speech portion. Distortion levels on the speech are studied at various stages of application of filters in terms of SNR and Speaker Specific Information. Retaining the Speaker Specific Information as primary concern of our study, the limitation of filtering techniques depending on the characteristic and intensity level of noise is worked out for noisy speech samples. Subsequently a statistical study is also conducted. Listening tests were conducted to ensure that the perceptual features of the original noisy speech are preserved while applying filters. This work demonstrates the efficiency of Noise reduction filters in improving SNR and their controlled applications for preserving Speaker dependent features depending on the various noise characteristics embedded on speech samples. Audio Forensics has a challenging history of enhancement problems of speech samples received for examination. It is observed that out of the total speech samples received for Speaker Identification in the Laboratory, a large number of recordings requires enhancement. Speech is a non-linear time series represented in terms of complex number. Hence separating noise from noisy speech in spectral domain results into countless solutions. The main objective of a Noise Cancellation system is to obtain a clear signal with higher quality of speech signal. The presence of noise in speech signals can create higher degree of mismatch in performance of speech processing systems used for Speaker Identification as well as Speech Recognition. Inappropriate filtering of noise corresponds to extracting features of noise together with the actual speech signal during the feature extraction process. However, the desired parametric representation carries a high amount of error rate. The presence of broadband noise and a very low SNR deteriorate the intelligibility of most of the recorded speech samples. Speaker's idiosyncratic speech is affected when the noise reduction is carried out. Thus the Speaker Identification

Speech enhancement employing Laplacian-Gaussian mixture

A new, efficient speech enhancement algorithm (SEA) is developed in this paper. In this low-complexity SEA, a noisy speech signal is first decorrelated and then the clean speech components are estimated from the decorrelated noisy speech samples. The distributions of clean speech and noise signals are assumed to be Laplacian and Gaussian, respectively. The clean speech components are estimated either by maximum likelihood (ML) or minimum-mean-square-error (MMSE) estimators. These estimators require some statistical parameters derived from speech and noise. These parameters are adaptively extracted by the ML approach during the active speech or silence intervals, respectively. In addition, a voice activity detector (VAD) that uses the same statistical model is employed to detect whether the speech is active or not. The simulation results show that our SEA approach performs as well as a recent high efficiency SEA that employs the Wiener filter. The computational complexity of this algorithm is very low compared with existing SEAs with low computational complexity.

Hearing impaired speech in noisy classrooms

Noisy classrooms have been shown to induce among students patterns of interaction similar to those used by hearing impaired people [W. H. McKellin et al., GURT (2003)]. In this research, the speech of children in a noisy classroom setting was investigated to determine if noisy classrooms have an effect on students’ speech. Audio recordings were made of the speech of students during group work in their regular classrooms (grades 1–7), and of the speech of the same students in a sound booth. Noise level readings in the classrooms were also recorded. Each student’s noisy and quiet environment speech samples were acoustically analyzed for prosodic and segmental properties (f0, pitch range, pitch variation, phoneme duration, vowel formants), and compared. The analysis showed that the students’ speech in the noisy classrooms had characteristics of the speech of hearing-impaired persons [e.g., R. O’Halpin, Clin. Ling. and Phon. 15, 529–550 (2001)]. Some educational implications of our findings were identified. [Work supported by the Peter Wall Institute for Advanced Studies, University of British Columbia.]