Monaural Speech Enhancement Research Articles

A Joint Framework of Denoising Autoencoder and Generative Vocoder for Monaural Speech Enhancement

Conventional monaural speech enhancement methods usually enhance the magnitude spectrum of noisy speech and leave the phase unchanged. Recent studies suggest that phase is also important for both speech intelligibility and perceptual quality. Although deep learning exhibits great potential on enhancing the magnitude and phase spectra in complex spectrogram domain and waveform domain, complex spectrogram and waveform are always more difficult to predict than the magnitude spectrum due to lack of clear structure in them. In this study, a Mel-domain denoising autoencoder and a deep generative vocoder are stacked to form a joint framework for monaural speech enhancement, in which the clean speech waveform is reconstructed without using the phase. Specifically, a convolutional recurrent network (CRN) is employed as the denoising autoencoder to enhance the Mel power spectrum of noisy speech. Then, the enhanced Mel power spectrum is fed to a deep generative vocoder to synthesize the speech waveform. Furthermore, the denoising autoencoder and generative vocoder are jointly fine-tuned. Experimental results show that the proposed method significantly improves speech intelligibility and perceptual quality. More importantly, our method achieves much better generalization ability for untrained noises than previous methods.

A Model Compression Method With Matrix Product Operators for Speech Enhancement

The deep neural network (DNN) based speech enhancement approaches have achieved promising performance. However, the number of parameters involved in these methods is usually enormous for the real applications of speech enhancement on the device with the limited resources. This seriously restricts the applications. To deal with this issue, model compression techniques are being widely studied. In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement. In this method, the weight matrices in the linear transformations of neural network model are replaced by the MPO decomposition format before training. In experiment, this process is applied to the causal neural network models, such as the feedforward multilayer perceptron (MLP) and long short-term memory (LSTM) models. Both MLP and LSTM models with/without compression are then utilized to estimate the ideal ratio mask for monaural speech enhancement. The experimental results show that our proposed MPO-based method outperforms the widely-used pruning method for speech enhancement under various compression rates, and further improvement can be achieved with respect to low compression rates. Our proposal provides an effective model compression method for speech enhancement, especially in cloud-free application.

Combining Multi-Perspective Attention Mechanism With Convolutional Networks for Monaural Speech Enhancement

The redundant convolutional encoder-decoder network has been proven useful in speech enhancement tasks. This network can capture the localized time-frequency details of speech signals through the fully convolutional network structure and the feature selection capability that results from the encoder-decoder mechanism. However, extracting informative features, which we regard as important for the representational capability of speech enhancement models, is not considered explicitly. To solve this problem, we introduce the attention mechanism into the convolutional encoder-decoder model to explicitly emphasize useful information from three aspects, namely, channel, space, and concurrent space-and-channel. Furthermore, the attention operation is specifically achieved through the squeeze-and-excitation mechanism and its variants. The model can adaptively emphasize valuable information and suppress useless ones by assigning weights from different perspectives according to global information, thereby improving its representational capability. Experimental results show that the proposed attention mechanisms can employ a small fraction of parameters to effectively improve the performance of CNN-based models compared with their normal versions, and generalize well to unseen noises, signal-to-noise ratios (SNR) and speakers. Among these mechanisms, the concurrent space-channel-wise attention exhibits the most significant improvement. And when comparing with the state-of-the-art, they can produce comparable or better results. We also integrate the proposed attention mechanisms with other convolutional neural network (CNN)-based models and gain performance. Moreover, we visualize the enhancement results to show the effect of the attention mechanisms more clearly.

Learning Complex Spectral Mapping with Gated Convolutional Recurrent Networks for Monaural Speech Enhancement.

Phase is important for perceptual quality of speech. However, it seems intractable to directly estimate phase spectra through supervised learning due to their lack of spectrotemporal structure in it. Complex spectral mapping aims to estimate the real and imaginary spectrograms of clean speech from those of noisy speech, which simultaneously enhances magnitude and phase responses of speech. Inspired by multi-task learning, we propose a gated convolutional recurrent network (GCRN) for complex spectral mapping, which amounts to a causal system for monaural speech enhancement. Our experimental results suggest that the proposed GCRN substantially outperforms an existing convolutional neural network (CNN) for complex spectral mapping in terms of both objective speech intelligibility and quality. Moreover, the proposed approach yields significantly higher STOI and PESQ than magnitude spectral mapping and complex ratio masking. We also find that complex spectral mapping with the proposed GCRN provides an effective phase estimate.

Bridging the Gap Between Monaural Speech Enhancement and Recognition With Distortion-Independent Acoustic Modeling

Monaural speech enhancement has made dramatic advances since the introduction of deep learning a few years ago. Although enhanced speech has been demonstrated to have better intelligibility and quality for human listeners, feeding it directly to automatic speech recognition (ASR) systems trained with noisy speech has not produced expected improvements in ASR performance. The lack of an enhancement benefit on recognition, or the gap between monaural speech enhancement and recognition, is often attributed to speech distortions introduced in the enhancement process. In this article, we analyze the distortion problem, compare different acoustic models, and investigate a distortion-independent training scheme for monaural speech recognition. Experimental results suggest that distortion-independent acoustic modeling is able to overcome the distortion problem. Such an acoustic model can also work with speech enhancement models different from the one used during training. Moreover, the models investigated in this paper outperform the previous best system on the CHiME-2 corpus.

Modulation-Domain Kalman Filtering for Monaural Blind Speech Denoising and Dereverberation

We describe a monaural speech enhancement algorithm based on modulation-domain Kalman filtering to blindly track the time–frequency log-magnitude spectra of speech and reverberation. We propose an adaptive algorithm that performs blind joint denoising and dereverberation, while accounting for the inter-frame speech dynamics, by estimating the posterior distribution of the speech log-magnitude spectrum given the log-magnitude spectrum of the noisy reverberant speech. The Kalman filter update step models the non-linear relations between the speech, noise, and reverberation log spectra. The Kalman filtering algorithm uses a signal model that takes into account the reverberation parameters of the reverberation time $T_{60}$ and the direct-to-reverberant energy ratio (DRR) and also estimates and tracks $T_{60}$ and the DRR in every frequency bin to improve the estimation of the speech log spectrum. The proposed algorithm is evaluated in terms of speech quality, speech intelligibility, and dereverberation performance for a range of reverberation parameters and reverberant speech to noise ratios, in different noises, and is also compared to competing denoising and dereverberation techniques. Experimental results using noisy reverberant speech demonstrate the effectiveness of the enhancement algorithm.

Monaural Speech Enhancement for a Hose-Shaped Rescue Robot with Air-Jet Noise

Monaural Speech Enhancement for a Hose-Shaped Rescue Robot with Air-Jet Noise

Deep Neural Network Based Monaural Speech Enhancement with Sparse Non-Negative Matrix Factorization

Deep Neural Network Based Monaural Speech Enhancement with Sparse Non-Negative Matrix Factorization

Gated Residual Networks with Dilated Convolutions for Monaural Speech Enhancement.

For supervised speech enhancement, contextual information is important for accurate mask estimation or spectral mapping. However, commonly used deep neural networks (DNNs) are limited in capturing temporal contexts. To leverage long-term contexts for tracking a target speaker, we treat speech enhancement as a sequence-to-sequence mapping, and present a novel convolutional neural network (CNN) architecture for monaural speech enhancement. The key idea is to systematically aggregate contexts through dilated convolutions, which significantly expand receptive fields. The CNN model additionally incorporates gating mechanisms and residual learning. Our experimental results suggest that the proposed model generalizes well to untrained noises and untrained speakers. It consistently outperforms a DNN, a unidirectional long short-term memory (LSTM) model and a bidirectional LSTM model in terms of objective speech intelligibility and quality metrics. Moreover, the proposed model has far fewer parameters than DNN and LSTM models.

Deep Neural Network Based Monaural Speech Enhancement with Low-Rank Analysis and Speech Present Probability

Deep Neural Network Based Monaural Speech Enhancement with Low-Rank Analysis and Speech Present Probability

DNN-based monaural speech enhancement with temporal and spectral variations equalization

Recently, deep neural networks (DNNs) were successfully introduced to the speech enhancement area. Conventional DNN-based algorithms generally produce over-smoothed output features which deteriorate the quality of the enhanced speech. In addition, their performance measures calculated in the linear frequency scale do not match the human auditory perception where the sensitivity follows the Mel-frequency scale. In this paper, we propose a novel objective function for DNN-based speech enhancement algorithm. In the proposed technique, a new objective function which consists of the Mel-scale weighted mean square error, and temporal and spectral variations similarities between the enhanced and clean speech is employed in the DNN training stage. The proposed objective function helps to compute the gradients based on a perceptually motivated non-linear frequency scale and alleviates the over-smoothness of the estimated speech. In the experiments, the performance of the proposed algorithm was compared to the conventional DNN-based speech enhancement algorithm in matched and mismatched noise conditions. From the experimental results, we can see that the proposed algorithm performs better than the conventional algorithm in terms of both the objective and subjective measures.

Supervised monaural speech enhancement using two-level complementary joint sparse representations

Recently, a new complementary joint sparse representations (CJSR) method was proposed for monaural speech enhancement, which utilizes the relationships among speech, noise and mixture. One of the joint sparse representations (JSRs) uses the mapping relationship between mixture and speech while the other uses the mapping relationship between mixture and noise. However, since they only use the joint information and overcomplete dictionaries, there may be some confusion components between the estimated speech and noise. In this paper, a novel model with fusion process is proposed, which further using an additional composite dictionary composed of clean speech dictionary and noise dictionary as prior knowledge. When the estimated speech and noise are obtained by JSRs, they are further sparsely represented on the composite dictionary, respectively. Because that the composite dictionary has the discriminative property, the source confusion problem of previous methods is coped well. In order to take advantage of the complementary knowledge of the two estimated signals, we propose a weighting parameter based on the residuals of sparse representation on composite dictionary. If one of the residuals is smaller, the corresponding weighting is greater, and this weighting parameter varies with noise type and speaker flexibly and effectively. Experimental results show that the proposed algorithm has superior performance compared with other tested approaches.

Supervised Monaural Speech Enhancement Using Complementary Joint Sparse Representations

Sparse representation is one of the most well-known methods that are applied to monaural speech enhancement. In order to make full use of the relationships among speech, noise, and mixture in sparse representation for speech enhancement, this letter proposes a novel sparsity model that consists of a couple of joint sparse representations (JSRs). One JSR uses the mapping relationship between mixture and speech while the other uses that between mixture and noise. Both relationships are used to constrain the joint dictionary learning, which effectively solves the source confusion problem of traditional methods. Moreover, the latter JSR can be complementary to the former JSR, depending on the level of structure of the noise. Thus, we propose a Gini index based weighting parameter to take their complementary advantages. The experimental results show that the proposed method outperforms state-of-the-art methods using various objective measures.

Speech Enhancement Using Generative Dictionary Learning

The enhancement of speech degraded by real-world interferers is a highly relevant and difficult task. Its importance arises from the multitude of practical applications, whereas the difficulty is due to the fact that interferers are often nonstationary and potentially similar to speech. The goal of monaural speech enhancement is to separate a single mixture into its underlying clean speech and interferer components. This under-determined problem is solved by incorporating prior knowledge in the form of learned speech and interferer dictionaries. The clean speech is recovered from the degraded speech by sparse coding of the mixture in a composite dictionary consisting of the concatenation of a speech and an interferer dictionary. Enhancement performance is measured using objective measures and is limited by two effects. A too sparse coding of the mixture causes the speech component to be explained with too few speech dictionary atoms, which induces an approximation error we denote source distortion. However, a too dense coding of the mixture results in source confusion, where parts of the speech component are explained by interferer dictionary atoms and vice-versa. Our method enables the control of the source distortion and source confusion trade-off, and therefore achieves superior performance compared to powerful approaches like geometric spectral subtraction and codebook-based filtering, for a number of challenging interferer classes such as speech babble and wind noise.

A Speech Enhancement Method Based on Kalman Filtering

The enhancement of speech degraded by non-stationary interferers is a highly relevant and difficult task for many signal processing applications. In this study, we present a monaural speech enhancement method based on spectral subtraction and Kalman filtering (KF) by extracting the Liljencrants–Fant (LF) excitation during voiced speech, in which the nature of glottal flow can be maintained. Therefore, the approach could preserve the glottal pulse’s nature characteristic in Kalman filtering and thus achieve significant improvements on objective quality. The quality of the enhanced speech has been evaluated by perceptual evaluation of speech quality (PESQ) score. The results indicate that the proposed algorithm could improve the output speech quality compared with the conventional KF algorithm and sub-band spectral subtraction.