Speech Enhancement Performance Research Articles

An Iterative Graph Spectral Subtraction Method for Speech Enhancement

In this paper, we investigate the application of graph signal processing (GSP) theory in speech enhancement. We first propose a set of shift operators to construct graph speech signals, and then analyze their spectrum in the graph Fourier domain. By leveraging the differences between the spectrum of graph speech and graph noise signals, we further propose the graph spectral subtraction (GSS) method to suppress the noise interference in noisy speech. Moreover, based on GSS, we propose the iterative graph spectral subtraction (IGSS) method to further improve the speech enhancement performance. Our experimental results show that the proposed operators are suitable for graph speech signals, and the proposed methods outperform the traditional basic spectral subtraction (BSS) method and iterative basic spectral subtraction (IBSS) method in terms of both signal-to-noise ratios (SNR) and mean Perceptual Evaluation of Speech Quality (PESQ).

Multiresolution Speech Enhancement Based on Proposed Circular Nested Microphone Array in Combination with Sub-Band Affine Projection Algorithm

Speech enhancement is one of the most important fields in audio and speech signal processing. The speech enhancement methods are divided into the single and multi-channel algorithms. The multi-channel methods increase the speech enhancement performance by providing more information with the use of more microphones. In addition, spatial aliasing is one of the destructive factors in speech enhancement strategies. In this article, we first propose a uniform circular nested microphone array (CNMA) for data recording. The microphone array increases the accuracy of the speech processing methods by increasing the information. Moreover, the proposed nested structure eliminates the spatial aliasing between microphone signals. The circular shape in the proposed nested microphone array implements the speech enhancement algorithm with the same probability for the speakers in all directions. In addition, the speech signal information is different in frequency bands, where the sub-band processing is proposed by the use of the analysis filter bank. The frequency resolution is increased in low frequency components by implementing the proposed filter bank. Then, the affine projection algorithm (APA) is implemented as an adaptive filter on sub-bands that were obtained by the proposed nested microphone array and analysis filter bank. This algorithm adaptively enhances the noisy speech signal. Next, the synthesis filters are implemented for reconstructing the enhanced speech signal. The proposed circular nested microphone array in combination with the sub-band affine projection algorithm (CNMA-SBAPA) is compared with the least mean square (LMS), recursive least square (RLS), traditional APA, distributed multichannel Wiener filter (DB-MWF), and multichannel nonnegative matrix factorization-minimum variance distortionless response (MNMF-MVDR) in terms of the segmental signal-to-noise ratio (SegSNR), perceptual evaluation of speech quality (PESQ), mean opinion score (MOS), short-time objective intelligibility (STOI), and speed of convergence on real and simulated data for white and colored noises. In all scenarios, the proposed method has high accuracy at different levels and noise types by the lower distortion in comparison with other works and, furthermore, the speed of convergence is higher than the compared researches.

Deep Residual-Dense Lattice Network for Speech Enhancement

Convolutional neural networks (CNNs) with residual links (ResNets) and causal dilated convolutional units have been the network of choice for deep learning approaches to speech enhancement. While residual links improve gradient flow during training, feature diminution of shallow layer outputs can occur due to repetitive summations with deeper layer outputs. One strategy to improve feature re-usage is to fuse both ResNets and densely connected CNNs (DenseNets). DenseNets, however, over-allocate parameters for feature re-usage. Motivated by this, we propose the residual-dense lattice network (RDL-Net), which is a new CNN for speech enhancement that employs both residual and dense aggregations without over-allocating parameters for feature re-usage. This is managed through the topology of the RDL blocks, which limit the number of outputs used for dense aggregations. Our extensive experimental investigation shows that RDL-Nets are able to achieve a higher speech enhancement performance than CNNs that employ residual and/or dense aggregations. RDL-Nets also use substantially fewer parameters and have a lower computational requirement. Furthermore, we demonstrate that RDL-Nets outperform many state-of-the-art deep learning approaches to speech enhancement. Availability: https://github.com/nick-nikzad/RDL-SE.

A multi-objective learning speech enhancement algorithm based on IRM post-processing with joint estimation of SCNN and TCNN

In this study, a novel multi-objective speech enhancement algorithm is proposed. First, we construct a deep learning architecture based on a stacked and temporal convolutional neural network (STCNN). Second, the main log-power spectra (LPS) features are input into a stacked convolutional neural network (SCNN) to extract advanced abstract features. Third, an improved power function compression Mel-frequency cepstral coefficient (PC-MFCC) feature—more consistent with human hearing characteristics than a Mel-frequency cepstral coefficient (MFCC)—is proposed. Then, a temporal convolutional neural network (TCNN) uses PC-MFCC and learned features from SCNN as input, and separately predicts a clean LPS, PC-MFCC and Ideal Ratio Mask (IRM). In this training phase, PC-MFCC constrains the LPS and IRM through a loss function to obtain the optimal network structure. Finally, IRM-based post-processing is used on the estimated clean LPS and IRM, which adjusts the weight between the above LPS and IRM to synthesise enhanced speech based on voice presence information. A series of experiments show that PC-MFCC is effective and shows complementarity with LPS in speech enhancement tasks. The proposed STCNN architecture has a higher speech enhancement performance than the comparative neural network models with good feature extraction and sequence modelling capabilities. Additionally, IRM-based post-processing further enhances the listening quality of reconstructed speech. Compared with the contrasting algorithm, the speech quality and intelligibility of enhanced speech based on the proposed multi-objective speech enhancement algorithm are further improved.

A Human Auditory Perception Loss Function Using Modified Bark Spectral Distortion for Speech Enhancement

Human listeners often have difficulties understanding speech in the presence of background noise in daily speech communication environments. Recently, deep neural network (DNN)-based techniques have been successfully applied to speech enhancement and achieved significant improvements over the conventional approaches. However, existing DNN-based methods usually minimize the log-power spectral-based or the masking-based mean squared error (MSE) between the enhanced output and the training target (e.g., the ideal ratio mask (IRM) of the clean speech), which is not closely related to human auditory perception. In this letter, a modified bark spectral distortion loss function, which can be considered as an auditory perception-based MSE, is proposed to replace the conventional MSE in DNN-based speech enhancement approaches to further improve the objective perceptual quality. Experimental results reveal that the proposed method can obtain improved speech enhancement performance, especially in terms of objective perceptual quality in all experimental settings when compared with the DNN-based methods using the conventional MSE criterion.

A Modified Tunable – Q Wavelet Transform Approach for Tamil Speech Enhancement

ABSTRACT In this paper, a new speech enhancement approach using the modified Tunable-Q Wavelet Transform (TQWT) for Tamil speaker recognition systems is proposed. Different forms of the existing wavelet transforms like continuous wavelet transform and discrete wavelet transform provide little ability to tune the Q factor and redundancy of the wavelet and have less reconstruction property. TQWT is fully discrete, has perfect reconstruction property and is modestly overcomplete. The Q-factor and redundancy can be easily tuned. Speech signals vary in both time and space. The entire speech signal is divided into different parts. Depending on the oscillatory contents of the speech signal, different Q factors are applied for each part instead of applying a single Q factor for the entire data. This gives more accuracy in decomposition. The proposed method is evaluated on several speakers and under various noise conditions including babble, street and exhibition noises. In order to tune the quality factor (Q) of the TQWT and to obtain good reconstruction property, speech enhancement has been conducted with varying values of Q. The reconstruction error, cost function, Signal to Noise Ratio(SNR) and Mean Opinion Score(MOS) show that the new approach highly improves the performance of speech enhancement based on the modified TQWT. Hardware implementation is also done using TMS320C6713 DSK and the results are similar to simulation.

Development and use of a new Speech Quality Evaluation Parameter ESNR using ANN and Grey Wolf Optimizer

The performance of Speech Enhancement (SE) Algorithms is evaluated using various objective and subjective evaluation parameters. Recently, few objective evaluation parameters are developed for the measurement of speech quality and intelligibility. But still, there are ample scopes determining statistical parameters to predict the SNR of a noisy speech signal without using any reference of clean signal and noise. In this paper, this problem has been addressed and three types of Artificial Neural Networks (ANN) are developed for efficient prediction of the estimated SNR (E-SNR) of a given noisy speech signal. To further improve the accuracy of prediction of the SNR of the ANN, the coefficients of ANN are tuned using the bio-inspired optimization technique. In this paper, a popular and efficient Grey wolf Optimization is chosen for the purpose. Several audio features are studied and appropriate features are chosen as the inputs to the ANN. Finally, a comparative performance analysis is carried out using two standard speech databases and the best performing ANN and audio features are identified to provide the best ESNR.

Performance Analysis of LMS, NLMS Adaptive Algorithms for Speech Enhancement in Noisy Environment

The speech enhancement is an important technique to remove noise from corrupted speech signal. Here several Adaptive Algorithms were proposed to improve quality of speech signal. In this paper to estimate the speech enhancement performance with variety of noise reduction algorithms using adaptive filters like LMS, NLMS. Simulations were performed on noisy data which was prepared by adding machinegun, Factory, vehicle and Traffic noise at 0dB, 5dB and 10dB SNR levels to clean speech samples The performance comparison of adaptive noise cancellation (ANC) system using LMS and NLMS algorithms was carried by means of signal to noise ratio (SNR), mean square error (MSE) and root mean square error (RMSE). Based on performance analysis, the NLMS algorithm was found to be a better optimal adaptive noise canceller for speech signal

Upgraded NLMS algorithm for speech enhancement with sparse and dispersive impulse responses

To have the effective speech communication, the information should be clearly passed in a noise-free environment. However, in real-world environment, the existence of background noise degrades the performance of the system. Based on the blind source separation strategy, various adaptive algorithms are designed and implemented using both dispersive impulse response and sparse impulse response. Even though the existing dual fast normalized least mean square algorithm works well under different noisy situations and gives a good performance, the problem is that it involves large number of processing steps. To overcome the complexity in finding the signal prediction parameter and to improve the performance of speech enhancement, we propose three adaptive filtering algorithms namely revised twofold rapid normalized least mean square algorithm, diminished twofold normalized least mean square algorithm and upgraded balanced twofold normalized least mean square algorithm (UBTNLMS). Taking the performance objective criteria into account, these algorithms have been tested for segmental signal-to-noise ratio, segmental mean square error, signal-to-noise ratio, mean square error and cepstral distance. On comparing the performance of the existing and proposed algorithms, UBTNLMS performs better than the other algorithms.

Phase Differences Based Coherence Filter for Dual-Microphone System

This paper proposes a dual-microphone coherence filter, which based on phase difference of two noisy signals. This method passes, without dirtortion target speaker. In real noisy environment, it’s difficult to get exactly information of condition noise field, especially in complex environment. Therefore a post-filtering, which is a function depends on speech presence probability, is suitable for compact dual-microphone and is adapted in the author's approach. The performance of suggested algorithm proved the stability and efficiency of the algorithm. This approach allows increasing performance of speech enhancement in term of signal-to-noise ratio and the amount of noise reduction. The author intends to continue using phase differences as conditions for different algorithms speech quality improvement under various noise conditions.

Cognitive-Driven Binaural Beamforming Using EEG-Based Auditory Attention Decoding

Identifying the target speaker in hearing aid applications is an essential ingredient to improve speech intelligibility. Recently, a least-squares-based auditory attention decoding (AAD) method has been proposed to identify the target speaker from single-trial EEG recordings in an acoustic scenario with two competing speakers. Aiming at enhancing the target speaker and suppressing the interfering speaker and ambient noise, in this article, we propose a cognitive-driven speech enhancement system, consisting of a binaural beamformer which is steered based on AAD and estimated relative transfer function (RTF) vectors, which require estimates of the direction-of-arrivals (DOAs) of both speakers. For binaural beamforming and to generate reference signals for AAD, we consider either minimum-variance-distortionless-response (MVDR) beamformers or linearly-constrained-minimum-variance (LCMV) beamformers. Contrary to the binaural MVDR beamformer, the binaural LCMV beamformer allows to preserve the spatial impression of the acoustic scene and to control the suppression of the interfering speaker, which is important when intending to switch attention between speakers. The speech enhancement performance of the proposed system is evaluated in terms of the binaural signal-to-interference-plus-noise ratio ( $\text {SINR}$ ) improvement in anechoic and reverberant conditions. Furthermore, we investigate the impact of RTF and DOA estimation errors and AAD errors on the speech enhancement performance. The experimental results show that the proposed system using LCMV beamformers yields a larger decoding performance and binaural $\text {SINR}$ improvement compared to using MVDR beamformers.

Improved Relativistic Cycle-Consistent GAN With Dilated Residual Network and Multi-Attention for Speech Enhancement

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

A Parallel-Data-Free Speech Enhancement Method Using Multi-Objective Learning Cycle-Consistent Generative Adversarial Network

Recently, deep neural networks (DNNs) have become the mainstream strategy for speech enhancement task because it can achieve the higher speech quality and intelligibility than the traditional methods. However, these DNN-based methods always need a large number of parallel corpus consisting of clean speech and noise to produce noisy data for the training of the DNN in order to improve the generalization of the network. As a result, this implies that many noisy speech signals that are collected in real environment cannot be used to train the DNN because of the lack of corresponding clean speech and noise. Additionally, as we know, noise varies with the time and scenario, so we cannot obtain parallel speech and noise due to infinite noise data and some limited speech data. Thus, the network training with unparallel speech and noise data is essential for the generalization of the network. To address this problem, we propose a novel parallel-data-free speech enhancement method, in which the cycle-consistent generative adversarial network (CycleGAN) and multi-objective learning are employed. Our method is also able to make best use of the benefits of multi-objective learning. On the training stage, we utilize two different encoders to encode the features of clean speech and noisy speech, respectively. Then, two forward generators are immediately used to predict the ideal time-frequency (T-F) mask and log-power spectrum (LPS) of clean speech. Two inverse generators are applied to map the magnitude spectrum (MS) and LPS of noisy speech, respectively. In addition, four discriminators are used to distinguish the real speech features from the generated features. Two encoders, four generators and four discriminators are simultaneously trained by using adversarial, identity-mapping, latent similarity and cycle-consistent loss. On the test stage, we directly utilize the forward generators and encoders to acquire the enhanced speech. The experimental results indicate that the proposed approach is able to achieve the better speech enhancement performance than the reference methods. Moreover, the proposed method is also effective to improve speech quality and intelligibility when the networks are trained under the parallel data.

Speech Enhancement Based on Denoising Autoencoder With Multi-Branched Encoders

Deep learning-based models have greatly advanced the performance of speech enhancement (SE) systems. However, two problems remain unsolved, which are closely related to model generalizability to noisy conditions: (1) mismatched noisy condition during testing, i.e., the performance is generally sub-optimal when models are tested with unseen noise types that are not involved in the training data; (2) local focus on specific noisy conditions, i.e., models trained using multiple types of noises cannot optimally remove a specific noise type even though the noise type has been involved in the training data. These problems are common in real applications. In this article, we propose a novel denoising autoencoder with a multi-branched encoder (termed DAEME) model to deal with these two problems. In the DAEME model, two stages are involved: training and testing. In the training stage, we build multiple component models to form a multi-branched encoder based on a decision tree (DSDT). The DSDT is built based on prior knowledge of speech and noisy conditions (the speaker, environment, and signal factors are considered in this paper), where each component of the multi-branched encoder performs a particular mapping from noisy to clean speech along the branch in the DSDT. Finally, a decoder is trained on top of the multi-branched encoder. In the testing stage, noisy speech is first processed by each component model. The multiple outputs from these models are then integrated into the decoder to determine the final enhanced speech. Experimental results show that DAEME is superior to several baseline models in terms of objective evaluation metrics, automatic speech recognition results, and quality in subjective human listening tests.

Time-Domain Multi-Modal Bone/Air Conducted Speech Enhancement

Previous studies have proven that integrating video signals, as a complementary modality, can facilitate improved performance for speech enhancement (SE). However, video clips usually contain large amounts of data and pose a high cost in terms of computational resources and thus may complicate the SE system. As an alternative source, a bone-conducted speech signal has a moderate data size while manifesting speech-phoneme structures, and thus complements its air-conducted counterpart. In this study, we propose a novel multi-modal SE structure in the time domain that leverages bone- and air-conducted signals. In addition, we examine two ensemble-learning-based strategies, early fusion (EF) and late fusion (LF), to integrate the two types of speech signals, and adopt a deep learning-based fully convolutional network to conduct the enhancement. The experiment results on the Mandarin corpus indicate that this newly presented multi-modal (integrating bone- and air-conducted signals) SE structure significantly outperforms the single-source SE counterparts (with a bone- or air-conducted signal only) in various speech evaluation metrics. In addition, the adoption of an LF strategy other than an EF in this novel SE multi-modal structure achieves better results.

Audio-Visual Speech Enhancement Using Conditional Variational Auto-Encoders

Variational auto-encoders (VAEs) are deep generative latent variable models that can be used for learning the distribution of complex data. VAEs have been successfully used to learn a probabilistic prior over speech signals, which is then used to perform speech enhancement. One advantage of this generative approach is that it does not require pairs of clean and noisy speech signals at training. In this paper, we propose audio-visual variants of VAEs for single-channel and speaker-independent speech enhancement. We develop a conditional VAE (CVAE) where the audio speech generative process is conditioned on visual information of the lip region. At test time, the audio-visual speech generative model is combined with a noise model based on nonnegative matrix factorization, and speech enhancement relies on a Monte Carlo expectation-maximization algorithm. Experiments are conducted with the recently published NTCD-TIMIT dataset as well as the GRID corpus. The results confirm that the proposed audio-visual CVAE effectively fuses audio and visual information, and it improves the speech enhancement performance compared with the audio-only VAE model, especially when the speech signal is highly corrupted by noise. We also show that the proposed unsupervised audio-visual speech enhancement approach outperforms a state-of-the-art supervised deep learning method.

Embedding Encoder-Decoder With Attention Mechanism for Monaural Speech Enhancement

The auditory selection framework with attention and memory (ASAM), which has an attention mechanism, embedding generator, generated embedding array, and life-long memory, is used to deal with mixed speech. When ASAM is applied to speech enhancement, the discrepancy between the voice and noise feature memories is huge and the separability of noise and voice is increased. However, ASAM cannot achieve desirable performance in terms of speech enhancement because it fails to utilize the time-frequency dependence of the embedding vectors to generate a corresponding mask unit. This work proposes a novel embedding encoder-decoder (EED), and a convolutional neural network (CNN) is used as decoder. The CNN structure is good at detecting local patterns, which can be exploited to extract correlation embedding data from the embedding array to generate the target spectrogram. This work evaluates a similar ASAM, EED with an LSTM encoder and a CNN decoder (RC-EED), RC-EED with an attention mechanism (RC-AEED), other similar EED structures and baseline models. Experiment results show that RC-EED and RC-AEED networks have good performance on speech enhancement task at low signal-to-noise ratio conditions. In addition, RC-AEED exhibits superior speech enhancement performance over ASAM and achieves better speech quality than do deep recurrent network and convolutional recurrent network.

Speech enhancement based on noise classification and deep neural network

Deep neural network (DNN) has recently been successfully adopted as a regression model in speech enhancement. Nonetheless, training machines to adapt different noise is a challenging task. Because every noise has its own characteristics which can be combined with speech utterance to give huge variation on which the model has to operate on. Thus, a joint framework combining noise classification (NC) and speech enhancement using DNN was proposed. We first determined the noise type of contaminated speech by the voice activity detection (VAD)-DNN and the NC-DNN. Then based on the noise classification results, the corresponding SE-DNN model was applied to enhance the contaminated speech. In addition, in order to make method simpler, the structure of different DNNs was similar and the features were the same. Experimental results show that the proposed method effectively improved the performance of speech enhancement in complex noise environments. Besides, the accuracy of classification had a great influence on speech enhancement.

Deep learning for minimum mean-square error approaches to speech enhancement

Recently, the focus of speech enhancement research has shifted from minimum mean-square error (MMSE) approaches, like the MMSE short-time spectral amplitude (MMSE-STSA) estimator, to state-of-the-art masking- and mapping-based deep learning approaches. We aim to bridge the gap between these two differing speech enhancement approaches. Deep learning methods for MMSE approaches are investigated in this work, with the objective of producing intelligible enhanced speech at a high quality. Since the speech enhancement performance of an MMSE approach improves with the accuracy of the used a priori signal-to-noise ratio (SNR) estimator, a residual long short-term memory (ResLSTM) network is utilised here to accurately estimate the a priori SNR. MMSE approaches utilising the ResLSTM a priori SNR estimator are evaluated using subjective and objective measures of speech quality and intelligibility. The tested conditions include real-world non-stationary and coloured noise sources at multiple SNR levels. MMSE approaches utilising the proposed a priori SNR estimator are able to achieve higher enhanced speech quality and intelligibility scores than recent masking- and mapping-based deep learning approaches. The results presented in this work show that the performance of an MMSE approach to speech enhancement significantly increases when utilising deep learning.Availability: The proposed a priori SNR estimator is available at: https://github.com/anicolson/DeepXi.

Disentangled Feature Learning for Noise-Invariant Speech Enhancement

Most of the recently proposed deep learning-based speech enhancement techniques have focused on designing the neural network architectures as a black box. However, it is often beneficial to understand what kinds of hidden representations the model has learned. Since the real-world speech data are drawn from a generative process involving multiple entangled factors, disentangling the speech factor can encourage the trained model to result in better performance for speech enhancement. With the recent success in learning disentangled representation using neural networks, we explore a framework for disentangling speech and noise, which has not been exploited in the conventional speech enhancement algorithms. In this work, we propose a novel noise-invariant speech enhancement method which manipulates the latent features to distinguish between the speech and noise features in the intermediate layers using adversarial training scheme. To compare the performance of the proposed method with other conventional algorithms, we conducted experiments in both the matched and mismatched noise conditions using TIMIT and TSPspeech datasets. Experimental results show that our model successfully disentangles the speech and noise latent features. Consequently, the proposed model not only achieves better enhancement performance but also offers more robust noise-invariant property than the conventional speech enhancement techniques.