Speech Enhancement Applications Research Articles

Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network.

Long short-term memory (LSTM) has been effectively used to represent sequential data in recent years. However, LSTM still struggles with capturing the long-term temporal dependencies. In this paper, we propose an hourglass-shaped LSTM that is able to capture long-term temporal correlations by reducing the feature resolutions without data loss. We have used skip connections in non-adjacent layers to avoid gradient decay. In addition, an attention process is incorporated into skip connections to emphasize the essential spectral features and spectral regions. The proposed LSTM model is applied to speech enhancement and recognition applications. The proposed LSTM model uses no future information, resulting in a causal system suitable for real-time processing. The combined spectral feature sets are used to train the LSTM model for improved performance. Using the proposed model, the ideal ratio mask (IRM) is estimated as a training objective. The experimental evaluations using short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) have demonstrated that the proposed model with robust feature representation obtained higher speech intelligibility and perceptual quality. With the TIMIT, LibriSpeech, and VoiceBank datasets, the proposed model improved STOI by 16.21%, 16.41%, and 18.33% over noisy speech, whereas PESQ is improved by 31.1%, 32.9%, and 32%. In seen and unseen noisy situations, the proposed model outperformed existing deep neural networks (DNNs), including baseline LSTM, feedforward neural network (FDNN), convolutional neural network (CNN), and generative adversarial network (GAN). With the Kaldi toolkit for automated speech recognition (ASR), the proposed model significantly reduced the word error rates (WERs) and reached an average WER of 15.13% in noisy backgrounds.

Comparative Studies of Single-Channel Speech Enhancement Techniques

Several speech enhancement techniques like Spectral Subtraction, MMSE, Log-MMSE, -order MMSE, adaptive -order MMSE and compressive sensing methods are developed worldwide. Scientists, engineers and researchers have implemented, evaluated and tested all methods individually with the different speech corpus. However, we found few articles on comparative studies of various speech enhancement techniques. In the present paper, several speech enhancement techniques have been studied, and their performance in terms of speech quality measures is compared objectively and subjectively. The results have been evaluated not only through speech quality measures but also in terms of waveform and spectrogram for speech enhancement applications. For this, MATLAB is used for the simulation of all methods. After getting the enhanced speech signals, we evaluated their enhanced speech signals of the methods. Results in terms of objective evaluation parameters indicated that the adaptive -order MMSE-based method produces good-quality speech signals compared to the other methods. Also, we evaluated their enhanced speech signals using a listening test, i.e. subjective evaluation. In the subjective quality test through mean opinion score (using the listening test), the performance of the adaptive -order MMSE method and GOMP are equal. In the case of waveform and spectrogram, the visualisation of enhanced speech signal obtained from GOMP-based compressive algorithms is very close to clean speech signal.

Microphone Array Speech Enhancement Via Beamforming Based Deep Learning Network

In general, in-car speech enhancement is an application of the microphone array speech enhancement in particular acoustic environments. Speech enhancement inside the moving cars is always an interesting topic and the researchers work to create some modules to increase the quality of speech and intelligibility of speech in cars. The passenger dialogue inside the car, the sound of other equipment, and a wide range of interference effects are major challenges in the task of speech separation in-car environment. To overcome this issue, a novel Beamforming based Deep learning Network (Bf-DLN) has been proposed for speech enhancement. Initially, the captured microphone array signals are pre-processed using an Adaptive beamforming technique named Least Constrained Minimum Variance (LCMV). Consequently, the proposed method uses a time-frequency representation to transform the pre-processed data into an image. The smoothed pseudo-Wigner-Ville distribution (SPWVD) is used for converting time-domain speech inputs into images. Convolutional deep belief network (CDBN) is used to extract the most pertinent features from these transformed images. Enhanced Elephant Heard Algorithm (EEHA) is used for selecting the desired source by eliminating the interference source. The experimental result demonstrates the effectiveness of the proposed strategy in removing background noise from the original speech signal. The proposed strategy outperforms existing methods in terms of PESQ, STOI, SSNRI, and SNR. The PESQ of the proposed Bf-DLN has a maximum PESQ of 1.98, whereas existing models like Two-stage Bi-LSTM has 1.82, DNN-C has 1.75 and GCN has 1.68 respectively. The PESQ of the proposed method is 1.75%, 3.15%, and 4.22% better than the existing GCN, DNN-C, and Bi-LSTM techniques. The efficacy of the proposed method is then validated by experiments.

Single line noise cancellation using derivative of normalized least mean square algorithm

Suppression of noise in noisy speech signal is required in many speech enhancement applications like signal recording and transmission from one place to other. In this paper a novel single line noise cancellation system is proposed using derivative of normalized least mean spare algorithm. The proposed system has two phases. The first phase is generation of secondary reference signal from incoming primary signal itself at initial silence period and pause between two words, which is essential while adaptive filter using as noise canceller. Second phase is noise cancellation using proposed modified error data normalized step size algorithm. The performance of the proposed algorithm is compared with normalized least mean square (NLMS) algorithm and original error data normalized step size (EDNSS) algorithm using standard IEEE sentence (SP23) of Noizeus data base with different types of real world noise at different level of signal to noise ratio (SNR). The output of proposed, NLMS and EDNSS algorithm are measured with output SNR, excessive mean square error (EMSE) and misadjustment (M). The results clearly illustrates that the proposed algorithm gives improved result over conventional NLMS and EDNSS algorithm. The speed of convergence is also maintained as same conventional NLMS algorithm.

Performance analysis of the speech enhancement application with wavelet transform domain adaptive filters

Performance analysis of the speech enhancement application with wavelet transform domain adaptive filters

Differentiable Mean Opinion Score Regularization for Perceptual Speech Enhancement

Many speech enhancement methods require perceptual quality metrics for evaluation. The “holy grail” of perceptual speech quality assessment is human subjective ratings, known as the mean opinion score. However, acquiring human ratings is time-consuming, laborious, and expensive. Existing objective quality metrics, on the other hand, are efficient and easy to compute but do not correlate well with human ratings. In this paper, we propose a relatively lightweight deep-learning-based model to predict the human ratings of speech signals. Since it is differentiable, it can be easily employed as a perceptual regularization to improve existing deep-learning-based speech enhancement methods. Experimental results demonstrate that the predictions of our proposed model correlate well with human judgments. We present application in speech enhancement and show that, interestingly, while there is a degradation in performance in terms of traditional objective metrics, there is a significant improvement in the perceptual quality and the naturalness of the enhanced speech.

A Compact CNN-Based Speech Enhancement With Adaptive Filter Design Using Gabor Function and Region-Aware Convolution

Speech enhancement (SE) is used in many applications, such as hearing devices, to improve speech intelligibility and quality. Convolutional neural network-based (CNN-based) SE algorithms in literature often employ generic convolutional filters that are not optimized for SE applications. This paper presents a CNN-based SE algorithm with an adaptive filter design (named ‘CNN-AFD’) using Gabor function and region-aware convolution. The proposed algorithm incorporates fixed Gabor functions into convolutional filters to model human auditory processing for improved denoising performance. The feature maps obtained from the Gabor-incorporated convolutional layers serve as learnable guided masks (tuned at backpropagation) for generating adaptive custom region-aware filters. The custom filters extract features from speech regions (i.e., ‘region-aware’) while maintaining translation-invariance. To reduce the high cost of inference of the CNN, skip convolution and activation analysis-wise pruning are explored. Employing skip convolution allowed the training time per epoch to be reduced by close to 40%. Pruning of neurons with high numbers of zero activations complements skip convolution and significantly reduces model parameters by more than 30%. The proposed CNN-AFD outperformed all four CNN-based SE baseline algorithms (i.e., a CNN-based SE employing generic filters, a CNN-based SE without region-aware convolution, a CNN-based SE trained with complex spectrograms and a CNN-based SE processing in the time-domain) with an average of 0.95, 1.82 and 0.82 in short-time objective intelligibility (STOI), perceptual evaluation of speech quality (PESQ) and logarithmic spectral distance (LSD) scores, respectively, when tasked to denoise speech contaminated with NOISEX-92 noises at -5, 0 and 5 dB signal-to-noise ratios (SNRs).

Smartphone-based single-channel speech enhancement application for hearing aids.

This work presents a single-channel speech enhancement (SE) framework based on the super-Gaussian extension of the joint maximum a posteriori (SGJMAP) estimation rule. The developed SE algorithm is an open-source research smartphone-based application for hearing improvement studies. In this algorithm, the SGJMAP-based estimation for noisy speech mixture is smoothed along the frequency axis by a Mel filter-bank, resulting in a Mel-warped frequency-domain SGJMAP estimation. The impulse response of this Mel-warped estimation is obtained by applying a Mel-warped inverse discrete cosine transform (Mel-IDCT). This helps in filtering out the background noise and enhancing the speech signal. The proposed application is implemented on an iPhone (Apple, Cupertino, CA) to operate in real time and tested with normal-hearing (NH) and hearing-impaired (HI) listeners with different types of hearing aids through wireless connectivity. The objective speech quality and intelligibility test results are used to compare the performance of the proposed algorithm to existing conventional single-channel SE methods. Additionally, test results from NH and HI listeners show substantial improvement in speech recognition with the developed method in simulated real-world noisy conditions at different signal-to-noise ratio levels.

Online Construction of Variable Span Linear Filters Using a Fixed-Point Approach

The variable span linear filters (VSLFs) constitute a unified framework of conventional subspace and linear filtering techniques for noise reduction. The construction of VSLFs, however, relies on the generalized eigendecomposition (GEVD) methods, which are computationally expensive. This in turn stymies the employment of such filters in practical online processing problems. To address this issue, we first propose in this paper a fixed-point iteration technique to extract the generalized eigenvectors. It is based on maximizing the pre-whitened generalized Rayleigh quotient (GRQ). We then integrate this technique with online statistic estimation to construct VSLFs. Our proposed method is computationally efficient and can also harness parallel architectures. To show its effectiveness, we consider a speech enhancement application and compare the results with those of several existing methods.

Towards More Efficient DNN-Based Speech Enhancement Using Quantized Correlation Mask

Many studies on deep learning-based speech enhancement (SE) utilizing the computational auditory scene analysis method typically employs the ideal binary mask or the ideal ratio mask to reconstruct the enhanced speech signal. However, many SE applications in real scenarios demand a desirable balance between denoising capability and computational cost. In this study, first, an improvement over the ideal ratio mask to attain more superior SE performance is proposed through introducing an efficient adaptive correlation-based factor for adjusting the ratio mask. The proposed method exploits the correlation coefficients among the noisy speech, noise and clean speech to effectively re-distribute the power ratio of the speech and noise during the ratio mask construction phase. Second, to make the supervised SE system more computationally-efficient, quantization techniques are considered to reduce the number of bits needed to represent floating numbers, leading to a more compact SE model. The proposed quantized correlation mask is utilized in conjunction with a 4-layer deep neural network (DNN-QCM) comprising dropout regulation, pre-training and noise-aware training to derive a robust and high-order mapping in enhancement, and to improve generalization capability in unseen conditions. Results show that the quantized correlation mask outperforms the conventional ratio mask representation and the other SE algorithms used for comparison. When compared to a DNN with ideal ratio mask as its learning targets, the DNN-QCM provided an improvement of approximately 6.5% in the short-time objective intelligibility score and 11.0% in the perceptual evaluation of speech quality score. The introduction of the quantization method can reduce the neural network weights to a 5-bit representation from a 32-bit, while effectively suppressing stationary and non-stationary noise. Timing analyses also show that with the techniques incorporated in the proposed DNN-QCM system to increase its compactness, the training and inference time can be reduced by 15.7% and 10.5%, respectively.

Robustness of Acoustic Rake Filters in Minimum Variance Beamforming

Acoustic rake filters perform coherent summation of the early room reflections using beamforming, with the aim of improving beamforming performance. This concept has been investigated for speech enhancement applications, improving noise reduction and late reverberation attenuation. Current studies typically assume that the parameters of the early reflections, such as the direction-of-arrival, delay and amplitude, are known in advance in the rake filter design. This work presents a novel investigation of the acoustic rake filter in a more practical context, focusing on the minimum variance distortionless response (MVDR) formulation. First, the sensitivity of the filter performance to perturbations in the reflection parameters is derived analytically, and investigated numerically using Monte Carlo simulations with a spherical microphone array. Then, an end-to-end example of rake filtering in a blind scenario is presented, where the reflection parameters are estimated from speech signals without any prior information. This example demonstrates for the first time the use of rake filtering in a realistic scenario.

Speech Enhancement Based on Fusion of Both Magnitude/Phase-Aware Features and Targets

Recently, supervised learning methods have shown promising performance, especially deep neural network-based (DNN) methods, in the application of single-channel speech enhancement. Generally, those approaches extract the acoustic features directly from the noisy speech to train a magnitude-aware target. In this paper, we propose to extract the acoustic features not only from the noisy speech but also from the pre-estimated speech, noise and phase separately, then fuse them into a new complementary feature for the purpose of obtaining more discriminative acoustic representation. In addition, on the basis of learning a magnitude-aware target, we also utilize the fusion feature to learn a phase-aware target, thereby further improving the accuracy of the recovered speech. We conduct extensive experiments, including performance comparison with some typical existing methods, generalization ability evaluation on unseen noise, ablation study, and subjective test by human listener, to demonstrate the feasibility and effectiveness of the proposed method. Experimental results prove that the proposed method has the ability to improve the quality and intelligibility of the reconstructed speech.

Weighted Sigmoid-Based Frequency-Selective Noise Filtering for Speech Denoising

Estimation of noise often has a major impact on the quality of enhanced signal, especially when it comes in speech enhancement applications. The non-stationary noise statistics vary with time, making decision of speech active/inactive frame is however difficult. Further, since there is no prior information of noise distribution, the estimators use the recursive averaging with a fixed smoothing coefficient ranging from 0.70 to 0.99. This fixed smoothing coefficient actually correlates the previous frames of noise statistics. Unfortunately, using fixed smoothing coefficient, the estimator treats both speech active/inactive frames equally which may cause the leakage of speech/noise power and results in loss of speech intelligibility. To address this problem and to increase the noise estimation accuracy, this paper proposes a posteriori SNR and frequency dependent adaptive smoothing coefficient. Further, this paper investigates the performance of proposed weighted sigmoid function (WSIG) noise estimator. From both objective and subjective quality assessments, it is clearly evident that the proposed noise estimator yields considerably better tracking of noise spectral variations compared to the existing state of the art methods.

On Estimation of Time-Varying Variances of Source and Noise for Sensor Array Processing

Estimation of time-varying variances of signals for beamforming in sensor arrays is a challenging problem. Based on the assumption that the array manifold vector and the noise pseudo-coherence matrix are known a priori or are well estimated, we present in this paper two estimators for estimating the time-varying variances of the source signal of interest and the noise. These two estimators are then extended to deal with the following situations: 1) there are multiple candidates of the noise pseudo-coherence matrix or the noise pseudo-coherence matrix is a linear combination of some base pseudo-coherence matrices, and 2) the estimation variance is large and smoothing is needed. Simulations for speech enhancement applications are performed and the results show that the proposed estimators can well track the time-varying variances of both the speech and noise signals. It is also demonstrated that the optimal beamformer using the variance parameters estimated with the presented estimators outperforms the widely used traditional optimal beamformers in terms of improvement in both the signal-to-noise ratio (SNR) and the log-spectral distortion (LSD).

A Smart Binaural Hearing Aid Architecture Leveraging a Smartphone APP With Deep-Learning Speech Enhancement

This paper presents a smartphone-based binaural hearing aid architecture for improving the speech intelligibility of hearing aid users. The proposed system consists of an earpiece, a smartphone and an application that performs real-time speech enhancement. The speaker’s voice, which is picked up by the microphone of the earpiece that is worn on the ear, is transmitted to the smartphone via wireless technology. After the speech intelligibility is improved in real time by the deep learning speech enhancement application, it is returned to the earpiece and generates sound. Deep learning speech enhancement algorithms can be used without performing burdensome calculations on the processors in the hearing aid. The results showed that the average usage of the central processing unit in the smartphone was approximately 26%, and the signal-to-noise ratios improve by at least 20%. The presented objective and subjective results show that the proposed method achieves comparatively more noise suppression without distorting the audio.

Increasing Compactness of Deep Learning Based Speech Enhancement Models With Parameter Pruning and Quantization Techniques

Most recent studies on deep learning based speech enhancement (SE) focused on improving denoising performance. However, successful SE applications require striking a desirable balance between denoising performance and computational cost in real scenarios. In this study, we propose a novel parameter pruning (PP) technique, which removes redundant channels in a neural network. In addition, a parameter quantization (PQ) technique was applied to reduce the size of a neural network by representing weights with fewer cluster centroids. Because the techniques are derived based on different concepts, the PP and PQ can be integrated to provide even more compact SE models. The experimental results show that the PP and PQ techniques produce a compacted SE model with a size of only 10.03% compared to that of the original model, resulting in minor performance losses of 1.43% (from 0.70 to 0.69) for STOI and 3.24% (from 1.85 to 1.79) for PESQ. The promising results suggest that the PP and PQ techniques can be used in a SE system in devices with limited storage and computation resources.

New symmetric decorrelating set-membership NLMS adaptive algorithms for blind speech intelligibility enhancement

This paper addresses the problem of speech enhancement and acoustic noise reduction by partial and set-membership adaptive algorithms combined with the symmetric decorrelating adaptive (SAD) algorithm structure. In this paper, we propose two new adaptive algorithms based on set-membership principle that improve the original set-membership algorithm behavior in speech enhancement applications. The first proposed algorithm (called Proposed 1), is based on the combination of the SAD algorithm structure with a smart control that uses decorrelating properties between the output and the mixing signal to control and update the SAD adaptive filters. The second proposed algorithm (called Proposed 2) is a modification of Proposed 1 and based on a new regularization relations of the SAD adaptive filters that use a combination between the variance of the mixing and the output signals of the SAD structures. These two proposed algorithms (Proposed 1 and Proposed 2) aim to improve the convergence speed performance and the output signal-to-noise-ratio of the original SAD algorithm when no smart control of the adaptive filters is used. The proposed algorithms have very interesting properties with non-stationary signal like speech when the SAD algorithm is, used alone, fails. The simulation results that are obtained by the comparison between the proposed algorithms (Proposed 1 and Proposed 2) and the original two-channel set-membership NLMS algorithm have shown the best performances of Proposed 1 and Proposed 2 in terms of the following criteria: systems mismatch, segmental SNR and segmental men square error.

A Smart Binaural Hearing Aid Architecture Based on a Mobile Computing Platform

This paper presents a new structure for hearing aids. Normally, the power consumption and user experience are contradictory. The proposed hearing aid structure mainly consists of three parts: the earpieces, the mobile computing platform, and the real-time speech-enhancement application. It can run complex algorithms without carrying out heavy calculations on the processors in the hearing aid. Thus, the binaural algorithm is utilized without being limited by complexity and power consumption to improve the user experience. Moreover, the speech-enhancement algorithm can be updated much more easily than in traditional built-in digital signal process hearing aids. A good level of user experience is achieved by combining the hearing aid and mobile computing platform with a 400-MHz transceiver; furthermore, the 400-MHz transceiver can reduce path loss around the body. The concept verification process showed that the overall usage of the central processing unit in the smartphone is around 16%, the signal-to-noise ratios show at least a 30% improvement in some environments, and the whole system delay is 8.8 ms. The presented objective and subjective results show significant improvements regarding user experience and usability brought about by the proposed structure.

A new dual subband fast NLMS adaptive filtering algorithm for blind speech quality enhancement and acoustic noise reduction

This paper discusses the problem of acoustic noise reduction and speech enhancement through the forward blind source separation structure. Recently we have proposed a new combination between the forward blind source separation structure and the fast normalized least mean square algorithm that provides an efficient dual algorithm for noise reduction and speech enhancement applications. In this paper we propose a new subband implementation of this recent dual algorithm, this last allows improving the speed convergence behavior of the previous proposed algorithm in its fullband form. The performance of the proposed dual subband algorithm is compared with its fullband version of the dual fast normalized least mean square algorithm and the classical fullband dual normalized least mean square algorithm, and the two channel subband forward algorithm in terms of several objective criteria. The obtained results show the good performances of the proposed dual subband algorithm.

New insights on the optimality of parameterized Wiener filters for speech enhancement applications

This work presents a unified framework for defining a family of noise reduction techniques for speech enhancement applications. The proposed approach provides a unique theoretical foundation for some widely-applied soft and hard time-frequency masks, which encompasses the well-known Wiener filter and the heuristically-designed Binary mask. These techniques can now be considered as optimal solutions of the same minimization problem. The proposed cost function is defined by two design parameters that not only establish a desired trade-off between noise reduction and speech distortion, but also provide an insightful relationship with the mask morphology. Such characteristic may be useful for applications that require online adaptation of the suppression function according to variations of the acoustic scenario. Simulation examples indicate that the derived conformable suppression mask has approximately the same quality and intelligibility performance capability of the classical heuristically-defined parametric Wiener filter. The proposed approach may be of special interest for real-time embedded speech enhancement applications such as hearing aids and cochlear implants.