Noisy Speech Research Articles

Masked multi-head self-attention for causal speech enhancement

Accurately modelling the long-term dependencies of noisy speech is critical to the performance of a speech enhancement system. Current deep learning approaches to speech enhancement employ either a recurrent neural network (RNN) or a temporal convolutional network (TCN). However, RNNs and TCNs both demonstrate deficiencies when modelling long-term dependencies. Enter multi-head attention (MHA) — a mechanism that has outperformed both RNNs and TCNs in tasks such as machine translation. By using sequence similarity, MHA possesses the ability to more efficiently model long-term dependencies. Moreover, masking can be employed to ensure that the MHA mechanism remains causal — an attribute critical for real-time processing. Motivated by these points, we investigate a deep neural network (DNN) that utilises masked MHA for causal speech enhancement. The conditions used to evaluate the proposed DNN include real-world non-stationary and coloured noise sources at multiple SNR levels. Our extensive experimental investigation demonstrates that the proposed DNN can produce enhanced speech at a higher quality and intelligibility than both RNNs and TCNs. We conclude that deep learning approaches employing masked MHA are more suited for causal speech enhancement than RNNs and TCNs. Availability—MHANet is available at https://github.com/anicolson/DeepXi

Real-time single-channel deep neural network-based speech enhancement on edge devices.

In this paper, we present a deep neural network architecture comprising of both convolutional neural network (CNN) and recurrent neural network (RNN) layers for real-time single-channel speech enhancement (SE). The proposed neural network model focuses on enhancing the noisy speech magnitude spectrum on a frame-by-frame process. The developed model is implemented on the smartphone (edge device), to demonstrate the real-time usability of the proposed method. Perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) test results are used to compare the proposed algorithm to previously published conventional and deep learning-based SE methods. Subjective ratings show the performance improvement of the proposed model over the other baseline SE methods.

Weak observer–level correlation and strong stimulus-level correlation between the McGurk effect and audiovisual speech-in-noise: A causal inference explanation

The McGurk effect is a widely used measure of multisensory integration during speech perception. Two observations have raised questions about the validity of the effect as a tool for understanding speech perception. First, there is high variability in perception of the McGurk effect across different stimuli and observers. Second, across observers there is low correlation between McGurk susceptibility and recognition of visual speech paired with auditory speech-in-noise, another common measure of multisensory integration. Using the framework of the causal inference of multisensory speech (CIMS) model, we explored the relationship between the McGurk effect, syllable perception, and sentence perception in seven experiments with a total of 296 different participants. Perceptual reports revealed a relationship between the efficacy of different McGurk stimuli created from the same talker and perception of the auditory component of the McGurk stimuli presented in isolation, both with and without added noise. The CIMS model explained this strong stimulus-level correlation using the principles of noisy sensory encoding followed by optimal cue combination within a common representational space across speech types. Because the McGurk effect (but not speech-in-noise) requires the resolution of conflicting cues between modalities, there is an additional source of individual variability that can explain the weak observer–level correlation between McGurk and noisy speech. Power calculations show that detecting this weak correlation requires studies with many more participants than those conducted to-date. Perception of the McGurk effect and other types of speech can be explained by a common theoretical framework that includes causal inference, suggesting that the McGurk effect is a valid and useful experimental tool.

Combining adaptive sparse NMF feature extraction and soft mask to optimize DNN for speech enhancement

In masking-based deep neural network (DNN) speech enhancement, the time–frequency masking value cannot be estimated accurately because the potential structure information of speech is ignored. In this paper, a speech enhancement method is proposed by combining adaptive sparse non-negative matrix factorization (NMF) feature extraction and soft mask to optimize DNN, using the advantages of the sparse matrix in catching the protruding structure of speech and combining with optimized masking-based prediction. First, considering the dominance of speech and noise interference in different noisy speech signals, this paper proposes a new method for estimating soft mask value, and the initial soft mask value is estimated by using speech cochleagram and noise cochleagram. Then, speech cochleagram and noise cochleagram are learned separately by the sparse NMF (SNMF) to obtain a joint dictionary. The noisy speech is sparsely represented on the joint dictionary, and the adaptive adjustment factor related to the changes of speech and noise dictionary is added to obtain the sparse coefficient. The sparse coefficient is used as the input of the DNN model, and the initial soft mask value is used as the learning label to estimate the final soft mask value. Finally, the estimated soft mask value is combined with the noisy speech cochleagram to obtain enhanced speech. Compared with other methods, the results show that 1.6039 dB increases the average signal-to-noise ratio (SNR) of the proposed method, the average perceptual evaluation of speech quality (PESQ) is increased by 0.1994, and the average short-time objective intelligibility (STOI) is improved by 0.0271, which fully illustrate the superiority of the proposed algorithm.

Investigation of a temporal modulation based method on the intelligibility of speech in speech-spectrum shaped noise

Temporal modulation is frequently considered for improving the intelligibility of noisy speech owing to the importance of the envelope in conveying speech information. In this study, we evaluated the performance of temporal modulation on word scores obtained with speech-in-noise. We firstly divided the noisy speech into 16 contiguous subbands from 200 Hz to 6 kHz with bandwidth approximately 1.5xERB of an auditory filter. The temporal envelope was constructed for each subband from the absolute value. It was then low-pass filtered at 16 Hz and used as the instantaneous gain for the subband's noisy speech. We established a psychophysical test (Modified Rhyme Test) with different speech-in-noise SNR values of 0, −3, −6, and −9 dB. Eleven native speakers (age 29 ± 8) with normal hearing were recruited for the study. The signals were processed by MATLAB and presented over earphones to participants seated in an audiometric room. Comparing the processed and unprocessed noisy speech, the mean differences in word scores for SNRs of 0, −3, −6, and −9 dB were −0.4%, 1.5%, −10.5%, and 1.4%, respectively. Using ANOVA, we concluded that the temporal modulation-based algorithm does not produce a statistically significant improvement in speech intelligibility. [Work supported by NIOSH.]

Effect of envelope signal-to-noise ratio on the intelligibility of speech in speech-spectrum shaped noise

In attempts to improve speech intelligibility, the envelope of speech in noise is commonly processed in the time domain and used to amplitude modulate the corrupted speech. In this study the envelope was constructed from the noisy speech, but separately and independently of the SNR of the noisy speech. Envelopes corresponding to 0, −3, −6, and −9 dB SNR were applied to the noisy speech when the latter had an SNR of either −9, −6, −3, or 0 dB, to evaluate the benefit to intelligibility of improving the SNR in the modulation domain. Signals from 200 Hz to 6 kHz were processed by MATLAB into sixteen contiguous subbands with bandwidth approximately 1.5×ERB of an auditory filter. The subband envelopes were formed from the absolute value of the signals and low-pass filtered. Eleven subjects aged 29 ± 8 years (mean and range) with normal hearing underwent the Modified Rhyme Test to assess speech intelligibility. The stimuli were presented diotically over earphones with the subject seated in an audiometric room. Statistically significant increases in mean word score of up to 35% could be obtained by improving the envelope SNR, suggesting this processing may benefit speech intelligibility. [Work supported by NIOSH.]

Reconstructing fundamental frequency from noisy speech using initialized autoencoders

In this paper, we present a new approach for fundamental frequency (f0) detection in noisy speech, based on Long Short-term Memory Neural Networks (LSTM). f0 is one of the most important parameters of human speech. Its detection is relevant in many speech signal processing areas and remains an important challenge for severely degraded signals. In previous references for f0 detection in speech enhancement and noise reduction tasks, LSTM has been initialized with random weights, following a back-propagation through time algorithm to adjust them. Our proposal is an alternative for a more efficient initialization, based on the weights of an Autoassociative network. This initialization is a better starting point for the f0 detection in noisy speech. We show the advantages of pre-training using objective measures for the parameter and the training process, with artificial and natural noise added at different signal-to-noise levels. Results show the performance of the LSTM increases in comparison to the random initialization, and represents a significant improvement in comparison with traditional initialization of neural networks for f0 detection in noisy conditions.

Unsupervised feature selection and NMF de-noising for robust Speech Emotion Recognition

Speech feature fusion is the most commonly used phenomenon for improving the accuracy in Speech Emotion Recognition (SER). But in this, there is a disadvantage of increasing the complexity in SER system in terms of processing time. Besides this, some of the features could be redundant and do not contribute for SER and lead to incorrect emotion prediction and reduction in SER accuracy. To overcome this problem, in this paper, unsupervised feature selection is applied to the feature set with the combination of INTERSPEECH 2010 paralinguistic features, Gammatone Cepstral Coefficients (GTCC) and Power Normalized Cepstral Coefficients (PNCC). The Feature Selection with Adaptive Structure Learning (FSASL), Unsupervised Feature Selection with Ordinal Locality (UFSOL) and a novel Subset Feature Selection (SuFS) algorithm are the feature dimension reduction techniques used to acquire better SER performance in this work. The proposed SER system is analyzed in both clean and noisy environments. The EMO-DB and IEMOCAP emotion databases are considered for evaluating the proposed SER performance. For noise analysis, the clean speech is corrupted with different noises of Aurora noise database and white Gaussian noise at different Signal to Noise Ratio (SNR) levels from −5dB to 20 dB. Support Vector Machine (SVM) classifier with linear and Radial Basis Function (RBF) kernels using 10-fold cross-validation and hold-out validation is used in this analysis with classification accuracy and computation time as the performance metrics. The results show that the proposed SER system outperforms the baseline SER system as well as many of the existing literature works both in clean and noisy conditions. For SNR levels >15 dB, the proposed SER system in presence of different noises performs same as the SER in clean environments. Whereas, for lower SNRs <15 dB the performance is likely to be reduced. Therefore, to overcome this drawback and improve the SER performance in noisy conditions, a dense Non-Negative Matrix Factorization (denseNMF) method is adopted for de-noising the noisy speech signal prior to SER achieving noise robustness.

Real-Time Estimation of Direction of Arrival of Speech Source Using Three Microphones.

In this paper, we present a real-time noise-robust direction of arrival (DOA) estimation technique using only the three built-in microphones of the modern Android-based smartphone. The proposed method eliminates the 'front-back' ambiguity caused by the symmetry of the two microphones reported previously and improves the performance of DOA estimation in noisy speech environments. Our method enhances the spatial awareness of hearing-impaired users by displaying the precise DOA angle of speech source on their smartphone screen. For increased efficiency, noise-robustness, and accuracy of the proposed DOA estimation method, a spectral pre-filtering technique and a Voice Activity Detector (VAD) based post-filtering are used along with a modified generalized cross-correlation (GCC) technique. Real recorded and simulated data under realistic noisy conditions are used in the evaluations of the proposed algorithm. Real-time implementation of the proposed system is carried out on an Android-based smartphone without any additional hardware or external microphone attachments. Experimental results show the performance of the proposed method versus those without pre or post-filtering under three different noisy conditions with 0dB to 10dB signal to noise ratios (SNRs).

A time–frequency smoothing neural network for speech enhancement

In the existing speech enhancement methods based on deep neural network (DNN), the network architectures are not designed for speech enhancement specially, which extract local features of noisy speech in a non-causal way. In this paper, inspired by the feature calculation method based on the time–frequency correlation in the improved minima controlled recursive averaging (IMCRA), by using the long short-term memory (LSTM) and convolutional neural network (CNN) to model the correlation in the time and frequency dimensions respectively, a time–frequency smoothing neural network is proposed for speech enhancement. In order to verify the effectiveness of the proposed network in speech enhancement, various causal speech enhancement systems are established based on different networks, and extensive experiments are carried out in terms of speech quality and intelligibility. The experimental results show that the proposed network yields better speech enhancement performance compared with the other networks.

Musical noise suppression using a low-rank and sparse matrix decomposition approach

We address the problem of suppressing musical noise from speech enhanced using a short-time processing algorithm. Enhancement algorithms rely on noise statistics and errors in estimating the statistics lead to residual noise in the enhanced signal. A frequently encountered residual noise type is the so-called musical noise, which is a consequence of spurious peaks occurring at random locations in the time-frequency (t-f) plane. Typically, speech enhancement algorithms operate on a short-time basis and perform attenuation of noisy speech spectral coefficients, effectively leading to a spectrotemporal gain function. We show that in case of speech distorted by musical noise, the spectrotemporal gain function has a distinct signature: the musical noise components are sparse in the t-f domain, whereas the spectrotemporal gain corresponding to the speech region exhibits a low-rank structure. Based on this observation, we propose a low-rank and sparse matrix decomposition of the spectrotemporal gain function. We show that musical noise can be effectively suppressed by reconstructing the speech signal using only the low-rank component. Performance comparison in terms of subjective scores and spectrographic analysis shows that the proposed technique is superior compared with two benchmark techniques. The proposed technique could be used in tandem with any speech enhancement algorithm that gives rise to musical noise.

Real‐time speech enhancement using optimised empirical mode decomposition and non‐local means estimation

In this study, the authors present a novel speech enhancement method by exploring the benefits of non-local means (NLM) estimation and optimised empirical mode decomposition (OEMD) adopting cubic-spline interpolation. The optimal parameters responsible for improving the performance are estimated using the path-finder algorithm. At first, the noisy speech signal is decomposed into many scaled signals called intrinsic-mode functions (IMFs) through the use of a temporary decomposition method is called sifting process in OEMD approach. The obtained IMFs are processed by NLM estimation technique in terms of non-local similarities present in each IMF, to reduce the ill-effects caused by interfering noise. The proposed NLM-based method is effective to eliminate the noise of less-frequency. Each IMF contains essential information about the signals, on some scale or frequency band. Field programmable gate array architecture is implemented on a Xilinx ISE 14.5 and the result of the proposed method offers good performance with a high signal-to-noise ratio (SNR) and low mean-square error compared to other approaches. The performance evolution is carried out for different speech signals taken from the TIMIT database and noises taken from the NOISEX-92 database in different SNR stages of 0, 5 and 10 dB, respectively.

Pitch Extraction Using Fourth-Root Spectrum in Noisy Speech

Pitch Extraction Using Fourth-Root Spectrum in Noisy Speech

Analysis of algorithms to estimate glottal closure instants from speech signals

Estimation of glottal closure instants (GCIs) plays a vital role in pitch-synchronous speech processing. The current work performs a qualitative and quantitative review of six existing GCI estimation algorithms, namely, group delay (GD)-based algorithm, DYPSA, YAGA, ZFF, SEDREAMS and DPI algorithm. This paper differs from existing review papers in that, a detailed analysis on the parameters affecting each algorithm is presented. The optimized set of parameters, derived from this analysis, is then used to perform a comparative analysis of the algorithms. Further, in addition to evaluating the performance of the algorithms on clean and noisy speech, performance on telephone speech is analyzed as well. The algorithms are also evaluated on pathological speech, to analyze their performance in the presence of pitch jitter. In terms of the identification rate, the DPI algorithm outperforms the other algorithms on clean speech, while SEDREAMS and ZFF are observed to be highly robust to noise. On telephone speech, however, DYPSA and GD-based algorithm exhibit superior performance. The GD algorithm also performs better than the other algorithms in the presence of pitch jitter. The algorithms are also evaluated in terms of the computation time, and ZFF is observed to be faster than the rest.

Binary Time-Frequency Mask for Improved Malay Speech Intelligibility at Low SNR Condition

A binary masking is often seen as a medium to enhance speech signal especially in hearing aid and speech recognition applications due to its simplicity and efficient approach for supervised source separation. High intelligibility could be obtained by applying the binary time-frequency masking to noisy speech signal. Since the issue of linear filtering algorithms might affect the intelligibility of noisy speech signal, so, this paper presents the binary time-frequency mask for improved Malay speech intelligibility at low SNR condition. The clean Malay speech signals that were contaminated by car and train noise with different signal to noise ratio (SNR) and local criterion (LC) level in forming the binary mask are analysed. The performance of intelligibility improvement was evaluated using a normalized sub band enveloped correlation (nSec). Overall results showed that the proposed approach produces slightly improved speech intelligibility at low SNR value.

Multi-scale decomposition based supervised single channel deep speech enhancement

Speech signals reaching our ears are in general contaminated by the background noise distortion which is detrimental to both speech quality and intelligibility. In this paper, we propose a nonlinear multi-scale decomposition-based deep speech enhancement method to improve the quality and intelligibility of the contaminated speech. In the proposed method, we have applied Hurst exponent-based Empirical Mode Decomposition (HEMD) to the noisy speech and obtained a set of intrinsic mode functions (IMFs) and a residual. The Deep Neural Networks (DNNs) are trained for each of the extracted IMF and residual to learn a non-linear mapping with a deep hidden structure to construct a time–frequency mask. We have formulated three deep speech enhancement structures, established on three time–frequency​ masks comprised of Ideal Ratio Mask (IRM), Ideal Binary Mask (IBM), and Phase Sensitive Mask (PSM). Background noise also degrades the original phase of the clean speech; therefore, introduces perceptual disturbance which leads to negative impacts on the speech quality and intelligibility. To avoid speech quality and intelligibility degradations, an iterative procedure is adopted to compensate the phase during noisy backgrounds. Nonlinear Mel-scale weighted MSE (LMW−MSE) is used as a loss function during network training, and computed the gradients which are based on the perceptually motivated nonlinear frequency scale. Usually, the output features of the conventional deep neural networks are over-smoothed which deteriorates the quality of the speech. To alleviate over-smoothness; frequency-independent spectral variance equalization is applied as a post-filtering method. The performance of the proposed deep enhancement methods is extensively evaluated and compared to the DNNs established on same time–frequency mask in various adverse noisy environments. The results have demonstrated that the proposed deep speech enhancement performed better in terms of the perceived speech quality and intelligibility.

A hybrid speech enhancement system with DNN based speech reconstruction and Kalman filtering

In this paper, we propose a hybrid speech enhancement system that exploits deep neural network (DNN) for speech reconstruction and Kalman filtering for further denoising, with the aim to improve performance under unseen noise conditions. Firstly, two separate DNNs are trained to learn the mapping from noisy acoustic features to the clean speech magnitudes and line spectrum frequencies (LSFs), respectively. Then the estimated clean magnitudes are combined with the phase of the noisy speech to reconstruct the estimated clean speech, while the LSFs are converted to linear prediction coefficients (LPCs) to implement Kalman filtering. Finally, the reconstructed speech is Kalman-filtered for further removing the residual noises. The proposed hybrid system takes advantage of both the DNN based reconstruction and traditional Kalman filtering, and can work reliably in either matched or unmatched acoustic environments. Computer based experiments are conducted to evaluate the proposed hybrid system with comparison to traditional iterative Kalman filtering and several state-of-the-art DNN based methods under both seen and unseen noises. It is shown that compared to the DNN based methods, the hybrid system achieves similar performance under seen noise, but notably better performance under unseen noise, in terms of both speech quality and intelligibility.

Speech Enhancement using a Variable Level Decomposition DWT

The wavelet transform shows a promising solution for a non-stationary signal. The denoising of a noisy speech signal carried with a wavelet thresholding technique. The noisy signal decomposed into different frequency bands, and this decomposition level (DL) decided independent of non-stationary noise. In this letter, a new DL detection procedure presented, and it decides the decomposition level based on signal energy and speech dominance. The proposed DL applies to the speech denoising model, and the obtained results compared with the unversal thresholding technique and the minimum mean square error algorithm. The performance of the enhanced speech signal measures with speech intelligibility measure (STOI), and speech quality measure (PESQ). The experimental results revealed that the proposed scheme outperforms the conventional methods in all SNR and non-stationary environments.

Improving speech enhancement by focusing on smaller values using relative loss

The task of single-channel speech enhancement is to restore clean speech from noisy speech. Recently, speech enhancement has been greatly improved with the introduction of deep learning. Previous work proved that using ideal ratio mask or phase-sensitive mask as intermediation to recover clean speech can yield better performance. In this case, the mean square error is usually selected as the loss function. However, after conducting experiments, the authors find that the mean square error has a problem. It considers absolute error values, meaning that the gradients of the network depend on absolute differences between estimated values and true values, so the points in magnitude spectra with smaller values contribute little to the gradients. To solve this problem, they propose relative loss, which pays more attention to relative differences between magnitude spectra, rather than the absolute differences, and is more in accordance with human sensory characteristics. The perceptual evaluation of speech quality, the short-time objective intelligibility, the signal-to-distortion ratio, and the segmental signal-to-noise ratio are used to evaluate the performance of the relative loss. Experimental results show that it can greatly improve speech enhancement by focusing on smaller values.

Linear versus deep learning methods for noisy speech separation for EEG-informed attention decoding

Objective. A hearing aid’s noise reduction algorithm cannot infer to which speaker the user intends to listen to. Auditory attention decoding (AAD) algorithms allow to infer this information from neural signals, which leads to the concept of neuro-steered hearing aids. We aim to evaluate and demonstrate the feasibility of AAD-supported speech enhancement in challenging noisy conditions based on electroencephalography recordings. Approach. The AAD performance with a linear versus a deep neural network (DNN) based speaker separation was evaluated for same-gender speaker mixtures using three different speaker positions and three different noise conditions. Main results. AAD results based on the linear approach were found to be at least on par and sometimes even better than pure DNN-based approaches in terms of AAD accuracy in all tested conditions. However, when using the DNN to support a linear data-driven beamformer, a performance improvement over the purely linear approach was obtained in the most challenging scenarios. The use of multiple microphones was also found to improve speaker separation and AAD performance over single-microphone systems. Significance. Recent proof-of-concept studies in this context each focus on a different method in a different experimental setting, which makes it hard to compare them. Furthermore, they are tested in highly idealized experimental conditions, which are still far from a realistic hearing aid setting. This work provides a systematic comparison of a linear and non-linear neuro-steered speech enhancement model, as well as a more realistic validation in challenging conditions.