Short-time Objective Intelligibility Research Articles

Combining adaptive sparse NMF feature extraction and soft mask to optimize DNN for speech enhancement

In masking-based deep neural network (DNN) speech enhancement, the time–frequency masking value cannot be estimated accurately because the potential structure information of speech is ignored. In this paper, a speech enhancement method is proposed by combining adaptive sparse non-negative matrix factorization (NMF) feature extraction and soft mask to optimize DNN, using the advantages of the sparse matrix in catching the protruding structure of speech and combining with optimized masking-based prediction. First, considering the dominance of speech and noise interference in different noisy speech signals, this paper proposes a new method for estimating soft mask value, and the initial soft mask value is estimated by using speech cochleagram and noise cochleagram. Then, speech cochleagram and noise cochleagram are learned separately by the sparse NMF (SNMF) to obtain a joint dictionary. The noisy speech is sparsely represented on the joint dictionary, and the adaptive adjustment factor related to the changes of speech and noise dictionary is added to obtain the sparse coefficient. The sparse coefficient is used as the input of the DNN model, and the initial soft mask value is used as the learning label to estimate the final soft mask value. Finally, the estimated soft mask value is combined with the noisy speech cochleagram to obtain enhanced speech. Compared with other methods, the results show that 1.6039 dB increases the average signal-to-noise ratio (SNR) of the proposed method, the average perceptual evaluation of speech quality (PESQ) is increased by 0.1994, and the average short-time objective intelligibility (STOI) is improved by 0.0271, which fully illustrate the superiority of the proposed algorithm.

Real-time speech enhancement algorithm for transient noise suppression

To effectively restrain stationary noise and transient noise, a real-time single-channel speech enhancement algorithm is proposed. First, to evaluate stationary noise, the quantile noise estimation method is used to obtain the spectrum of stationary noise. Then, based on the normalized variance and gravity center of the signal, the transient noise detection method is proposed to modify the spectrum of stationary noise. Next, the speech presence probability is estimated based on the speech features and harmonic analysis. Finally, the optimized-modified log-spectral amplitude (OM-LSA) estimator is adopted for speech enhancement. The experimental noise contains 115 environmental sounds with the SNR of −10 to 10 dB. The experimental results show that the performance of the proposed algorithm is comparable to the OM-LSA algorithm which has good denoising performance, but the real-time performance of the former is much better. Compared with the Webrtc real-time algorithm, under the overall performance of stationary noise and transient noise, the overall speech quality indicators of the improved algorithm increased by 7.5%, 7.8% and 5.0%, respectively. And the short-time objective intelligibility increased by 2.4%, 2.4% and 2.0%, respectively. Even compared with the recurrent neural network(RNN) algorithm, the suppression performance of the transient noise is better. Besides, the real-time experiment base on the hardware platform shows that the runtime of processing a 10 ms frame is 4.3 ms.

Performance analysis of neural network, NMF and statistical approaches for speech enhancement

Bayesian Estimators are very useful in speech enhancement and noise reduction. But, it is noted that the traditional estimators process only amplitudes and the phase is left unprocessed. Among the Bayesian estimators, Super- Gaussian based estimators provide improved noise reduction. Super-Gaussian Bayesian estimators, which uses processed phase information for estimation of amplitudes provides further improved results. In this work, the Complex speech coefficients given Uncertain Phase (CUP) based Bayesian estimators like CUP-GG (CUP Estimator with speech spectral coefficients assumed as Gamma and noise spectral coefficients as Generalized Gamma), CUP-NG (Speech as Nakagami) are compared under white noise, pink noise, Babble noise and Non-Stationary factory noise conditions. The statistical estimators show less effective results under completely non-stationary assumptions like non-stationary factory noise, babble noise etc. Non-negative Matrix Factorization (NMF) based algorithms show better performance for non stationary noises. The drawback of NMF is, it requires apriori knowledge about speech. This drawback can be overcome by taking the advantages of both statistical approaches and NMF approaches. NR-NMF and WR-NMF speech enhancement methods are developed by providing posteriori regularization based on statistical assumption of speech and noise DFT coefficients distribution. Also a speech enhancement method which uses CUP-GG estimator and NMF with online noise bases update are considered for comparison. The progress in neural network based approaches for speech enhancement further shown that with large dataset and better training, the speech enhancement algorithms results in improved results. In this work, the neural network approach for speech enhancement is implemented and compared the method with traditional estimators and NMF approaches. For generalization of unseen noise types the proposed neural network approach uses dropout. Also for training the network, the features obtained from apriori SNR and aposteriori SNR is used in this method. The objective of this paper is to analyze the performance of speech enhancement methods based on Neural Network, NMF and statistical based. The objective performance measures Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), Signal to Noise Ratio (SNR), Segmental SNR (Seg SNR) are considered for comparison.

Speech Enhancement for Hearing Aids with Deep Learning on Environmental Noises

Hearing aids are small electronic devices designed to improve hearing for persons with impaired hearing, using sophisticated audio signal processing algorithms and technologies. In general, the speech enhancement algorithms in hearing aids remove the environmental noise and enhance speech while still giving consideration to hearing characteristics and the environmental surroundings. In this study, a speech enhancement algorithm was proposed to improve speech quality in a hearing aid environment by applying noise reduction algorithms with deep neural network learning based on noise classification. In order to evaluate the speech enhancement in an actual hearing aid environment, ten types of noise were self-recorded and classified using convolutional neural networks. In addition, noise reduction for speech enhancement in the hearing aid were applied by deep neural networks based on the noise classification. As a result, the speech quality based on the speech enhancements removed using the deep neural networks—and associated environmental noise classification—exhibited a significant improvement over that of the conventional hearing aid algorithm. The improved speech quality was also evaluated by objective measure through the perceptual evaluation of speech quality score, the short-time objective intelligibility score, the overall quality composite measure, and the log likelihood ratio score.

A real-world noise removal with wavelet speech feature

Real-world noise signals are non-stationary, and these signals are a mixture of more than one non-stationary noise signal. Most of the conventional speech enhancement algorithms (SEAs) focus primarily on a single noise corrupted speech signal, and it is far from real-world environments. In this article, we discuss speech enhancement in real-world environments with a new speech feature. The novelty of this article is three-fold, (1) The proposed model analyzed in real-world environments. (2) The proposed model uses a discrete wavelet transform (DWT) coefficients as input features. (3) The proposed Deep Denoising Autoencoder (DDAE) designed experimentally. The result of the proposed feature compares with conventional speech features like FFT-Amplitude, Log-Magnitude, Mel frequency cepstral coefficients (MFCCs), and the Gammatone filter cepstral coefficients (GFCCs). The performance of the proposed method compared with conventional speech enhancement methods. The enhanced signal evaluated with speech quality measures, like, perceptual evaluation speech quality (PESQ), weighted spectral slope (WSS), and Log-likelihood ratio (LLR). Similarly, speech intelligibility measured with short-time objective intelligibility (STOI). The results show that the proposed SEA model with the DWT feature improves quality and intelligibility in all real-world environmental Signal-to-Noise ratio (SNR) conditions.

Improving speech enhancement by focusing on smaller values using relative loss

The task of single-channel speech enhancement is to restore clean speech from noisy speech. Recently, speech enhancement has been greatly improved with the introduction of deep learning. Previous work proved that using ideal ratio mask or phase-sensitive mask as intermediation to recover clean speech can yield better performance. In this case, the mean square error is usually selected as the loss function. However, after conducting experiments, the authors find that the mean square error has a problem. It considers absolute error values, meaning that the gradients of the network depend on absolute differences between estimated values and true values, so the points in magnitude spectra with smaller values contribute little to the gradients. To solve this problem, they propose relative loss, which pays more attention to relative differences between magnitude spectra, rather than the absolute differences, and is more in accordance with human sensory characteristics. The perceptual evaluation of speech quality, the short-time objective intelligibility, the signal-to-distortion ratio, and the segmental signal-to-noise ratio are used to evaluate the performance of the relative loss. Experimental results show that it can greatly improve speech enhancement by focusing on smaller values.

Speech intelligibility enhancement: a hybrid wiener approach

Speech enhancement primarily focuses on improving the intelligibility and quality of the speech signal by using various algorithms and techniques. Processing of a speech signal refers to applying efficient mechanisms to reduce noise in the way of extracting the intended speech signal from the corrupted signal. Noise reduction techniques such as kalman filtering, spectral subtraction and adaptive wiener filtering etc. are used in different enhancement scenarios in speech processing. In the proposed method, the combination of wiener filter and Karhunen–Loeve Transform is used to remove noise and enhance the noisy speech signal. This paper presents the performance evaluation of the proposed hybrid algorithm by estimating Signal to Noise Ratio, Perceptual Evaluation of Speech Quality, Short-Time Objective Intelligibility and Extended STOI values. This algorithm has been implemented in varied noisy conditions and the results proved the fruitfulness of this method. Subjective listening evaluation is also done and both the objective and subjective results confirmed the significant improvement in speech intelligibility in the proposed method.

GEDI: Gammachirp envelope distortion index for predicting intelligibility of enhanced speech

In this study, we propose a new concept, the gammachirp envelope distortion index (GEDI), based on the signal-to-distortion ratio in the auditory envelope, SDRenv, to predict the intelligibility of speech enhanced by nonlinear algorithms. The objective of GEDI is to calculate the distortion between enhanced and clean-speech representations in the domain of a temporal envelope extracted by the gammachirp auditory filterbank and modulation filterbank. We also extend GEDI with multi-resolution analysis (mr-GEDI) to predict the speech intelligibility of sounds under non-stationary noise conditions. We evaluate GEDI in terms of the speech intelligibility predictions of speech sounds enhanced by a classic spectral subtraction and a Wiener filtering method. The predictions are compared with human results for various signal-to-noise ratio conditions with additive pink and babble noises. The results showed that mr-GEDI predicted the intelligibility curves better than short-time objective intelligibility (STOI) measure, extended-STOI (ESTOI) measure, and hearing-aid speech perception index (HASPI) under pink-noise conditions, and better than HASPI under babble-noise conditions. The mr-GEDI method does not present an overestimation tendency and is considered a more conservative approach than STOI and ESTOI. Therefore, the evaluation with mr-GEDI may provide additional information in the development of speech enhancement algorithms.

A 1.5 mW Programmable Acoustic Signal Processor for Hearing Assistive Devices With Speech Intelligibility Enhancement

This paper presents a low-power, programmable acoustic signal processor for hearing assistive devices with speech intelligibility enhancement. The reprogrammable design provides considerable flexibility for the devices to deal with personal conditions of hearing loss. A spectral-change enhancement (SCE) algorithm is implemented to improve speech intelligibility. The power consumption is minimized by adding dedicated hardware accelerators. The short-time objective intelligibility (STOI) measure is utilized for optimizing the datapath architecture. Optimization on the critical MAC operations results in 34% power and area reductions when compared to the direct-mapped design. A 50% reduction in SRAM storage is also achieved owing to the reduced memory storage for the associated MAC operations. With the aid of the optimized MAC unit and data buffer, the overall execution time is reduced by 99.2%. Designed in a 40-nm CMOS technology, the processor integrates 431k gates in area of 0.3 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . The design dissipates 1.5 mW at a clock frequency of 10.5 MHz from a 0.7V supply, with a processing latency of 1.05 ms.

Multiresolution Speech Enhancement Based on Proposed Circular Nested Microphone Array in Combination with Sub-Band Affine Projection Algorithm

Speech enhancement is one of the most important fields in audio and speech signal processing. The speech enhancement methods are divided into the single and multi-channel algorithms. The multi-channel methods increase the speech enhancement performance by providing more information with the use of more microphones. In addition, spatial aliasing is one of the destructive factors in speech enhancement strategies. In this article, we first propose a uniform circular nested microphone array (CNMA) for data recording. The microphone array increases the accuracy of the speech processing methods by increasing the information. Moreover, the proposed nested structure eliminates the spatial aliasing between microphone signals. The circular shape in the proposed nested microphone array implements the speech enhancement algorithm with the same probability for the speakers in all directions. In addition, the speech signal information is different in frequency bands, where the sub-band processing is proposed by the use of the analysis filter bank. The frequency resolution is increased in low frequency components by implementing the proposed filter bank. Then, the affine projection algorithm (APA) is implemented as an adaptive filter on sub-bands that were obtained by the proposed nested microphone array and analysis filter bank. This algorithm adaptively enhances the noisy speech signal. Next, the synthesis filters are implemented for reconstructing the enhanced speech signal. The proposed circular nested microphone array in combination with the sub-band affine projection algorithm (CNMA-SBAPA) is compared with the least mean square (LMS), recursive least square (RLS), traditional APA, distributed multichannel Wiener filter (DB-MWF), and multichannel nonnegative matrix factorization-minimum variance distortionless response (MNMF-MVDR) in terms of the segmental signal-to-noise ratio (SegSNR), perceptual evaluation of speech quality (PESQ), mean opinion score (MOS), short-time objective intelligibility (STOI), and speed of convergence on real and simulated data for white and colored noises. In all scenarios, the proposed method has high accuracy at different levels and noise types by the lower distortion in comparison with other works and, furthermore, the speed of convergence is higher than the compared researches.

Speech enhancement - an enhanced principal component analysis (EPCA) filter approach

Speech enhancement aims at improving the quality of speech in a noisy environment, also in applications such as speech recognition systems, hearing aids, teleconferencing, etc. Researchers have suggested various techniques to improve the quality of speech signal in a noisy environment. Kalman filtering technique showed better performance and was found remarkable in removing noise components. Therefore, this proposed method employs an improved version of the Kalman filter for the removal of noise in speech signals. A Principal Component Analysis (PCA) developed Kalman filter is proposed for enhancing speech intelligibility and quality. The simulation results prove the recommended technique of enhancing speech signals. It is better in performance compared with Modulation Compressive Sensing, Compressed Sensing Frequency, Wiener filter, and Log Minimum Mean Square Error (LogMMSE) in terms of Short-Time Objective Intelligibility (STOI), Segmental Signal to Noise Ratio (SSNR) and Perceptual Evaluation of Speech Quality (PESQ) for different noise levels.

Deep neural networks based binary classification for single channel speaker independent multi-talker speech separation

Speech separation is an important task of separating a target speech from the mixture signals. Speaker-independent multi-talker speech separation is a challenging task due to unpredictability of the target and interfering speech in the target-interference mixtures. Conventionally, speech separation is used as a signal processing problem, but recently it is formulated as a deep learning problem and discriminative patterns of the speech are learned from the training data. In this paper, we consider the ideal binary mask (IBM) as a supervised binary classification training-target by using fully connected deep neural networks (DNN) for single-channel speaker-independent multi-talker speech separation. The train DNNs is used to estimate IBM training-target. The mean square error (MSE) is used as an objective cost function. Standard backpropagation and Monte-Carlo dropout regularization approaches are used for better generalization and overfitting during training. The estimated training-target is applied to the mixtures to obtain the separated target speech. We have addressed the over-smoothing problem and performed equalization of spectral variances to match the estimated and clean speech features. Our experimental results in various evaluating conditions report that the proposed method outperformed the competing methods in terms of the Perceptual Evaluation of Speech Quality (PESQ), Segmental SNR (SNRSeg), Short-time objective intelligibility (STOI), normalized Frequency weighted SNRSeg (nFwSNRSeg) and HIT-FA rates.

Multi-Task Learning U-Net for Single-Channel Speech Enhancement and Mask-Based Voice Activity Detection

In this paper, a multi-task learning U-shaped neural network (MTU-Net) is proposed and applied to single-channel speech enhancement (SE). The proposed MTU-based SE method estimates an ideal binary mask (IBM) or an ideal ratio mask (IRM) by extending the decoding network of a conventional U-Net to simultaneously model the speech and noise spectra as the target. The effectiveness of the proposed SE method was evaluated under both matched and mismatched noise conditions between training and testing by measuring the perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI). Consequently, the proposed SE method with IRM achieved a substantial improvement with higher average PESQ scores by 0.17, 0.52, and 0.40 than other state-of-the-art deep-learning-based methods, such as the deep recurrent neural network (DRNN), SE generative adversarial network (SEGAN), and conventional U-Net, respectively. In addition, the STOI scores of the proposed SE method are 0.07, 0.05, and 0.05 higher than those of the DRNN, SEGAN, and U-Net, respectively. Next, voice activity detection (VAD) is also proposed by using the IRM estimated by the proposed MTU-Net-based SE method, which is fundamentally an unsupervised method without any model training. Then, the performance of the proposed VAD method was compared with the performance of supervised learning-based methods using a deep neural network (DNN), a boosted DNN, and a long short-term memory (LSTM) network. Consequently, the proposed VAD methods show a slightly better performance than the three neural network-based methods under mismatched noise conditions.

Quadratic approach for single-channel noise reduction

In this paper, we introduce a quadratic approach for single-channel noise reduction. The desired signal magnitude is estimated by applying a linear filter to a modified version of the observations’ vector. The modified version is constructed from a Kronecker product of the observations’ vector with its complex conjugate. The estimated signal magnitude is multiplied by a complex exponential whose phase is obtained using a conventional linear filtering approach. We focus on the linear and quadratic maximum signal-to-noise ratio (SNR) filters and demonstrate that the quadratic filter is superior in terms of subband SNR gains. In addition, in the context of speech enhancement, we show that the quadratic filter is ideally preferable in terms of perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) scores. The advantages, compared to the conventional linear filtering approach, are particularly significant for low input SNRs, at the expanse of a higher computational complexity. The results are verified in practical scenarios with nonstationary noise and in comparison to well-known speech enhancement methods. We demonstrate that the quadratic maximum SNR filter may be superior, depending on the nonstationary noise type.

Enhancement of Speech Intelligibility using Binary Mask Based on channel selection criteria

Most of the existing noise reduction algorithms used in hearing aid applications apply a gain function in order to reduce the noise intervention. In the present paper, we study the effect of the two types of speech distortions introduced by the gain functions. If these distortions are properly controlled large gains in intelligibility can be obtained. The sentences were corrupted by various kinds of noises i.e. babble noise, car noise, helicopter noise and random noise and processed through a noise-reduction algorithm. Subjective tests were conducted with normal hearing listeners by presenting the processed speech with controlled distortions. The method proposed by Kim et al uses the wiener filter. Here in this paper, we have used the parametric wiener filter. The experimental results clearly indicated improvement in intelligibility at 0dB, -5dB, +5dB and 10dB input signal-to-noise (SNR) values in short-time objective intelligibility (STOI) and Segmental signal-to-noise ratio (SSNR) objective measures.

A Hybrid Approach for Single Channel Speech Enhancement using Deep Neural Network and Harmonic Regeneration Noise Reduction

This paper presents a hybrid approach for single channel speech enhancement using deep neural network (DNN) and harmonic regeneration noise reduction (HRNR). The DNN was used as a supervised algorithm to predict new target mask such as constrained Wiener Filter (cWF) target mask from noisy mixture signal that was transformed into gammatone filter bank features. Meanwhile, HRNR algorithm was applied in the post-filtering strategy to eliminate residual noise. The DNN algorithm is an emerging supervised speech enhancement to overcome heavy nonstationary noise and low signal-to-noise ratio (SNR) issues. To validate the proposed algorithm with new target mask, 600 Malay utterances combining male and female speakers were used in a training session while 120 Malay utterances were used in a prediction session. The short time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) scores were calculated as the performance metrics. In this work, the proposed target mask outperformed other baseline target masks. Thus, PESQ and STOI scores for the hybrid speech enhancement algorithm is 1.17 and 0.79, respectively, at - 5 dB babble noise SNR.

Transfer Learning Algorithm for Enhancing the Unlabeled Speech

To improve the generalization ability of speech enhancement algorithms for unlabeled noisy speech, a speech enhancement transfer learning model based on the feature-attention multi-kernel maximum mean discrepancy (FA-MK-MMD) is proposed. To obtain a representation of the shared subspace (the part related with clean speech in the feature extracted by shared encoder) between source domain (speech with known noise and labels) and target domain (speech with unknown noise and no labels), the algorithm takes MK-MMD as loss function for reducing distribution differences between these two domains, which could improve the adaptability to the unknown noise. Furthermore, considering that different noise have different influence on the representation of shared subspace, the attention mechanism is applied to feature dimension to screen out the information less polluted by noise, which is helpful for reconstructing the clean speech. In the term of speech with unknown noise and no labels, the experiments demonstrate that the proposed algorithm has improved the frequency-weighed segmental signal-to-noise ratio (fwsegSNR), the perceptual evaluation of the speech quality (PESQ) and the short time objective intelligibility (STOI) compared with the baseline algorithm.

Performance Improvement of Adaptive Wavelet Thresholding for Speech Enhancement Using Generalized Gaussian Priors and Frame-Wise Context Modeling

This work aims at developing an adaptive wavelet thresholding algorithm for speech enhancement with significant performance improvement over other wavelet-based counterparts. This is accomplished through the formulation of the optimum threshold for noise reduction, based on the generalized Gaussian priors to fully characterize the statistics of speech and noise wavelet coefficients. In addition, through the frame-wise context modeling which enables tracking of the statistical characteristics of each individual coefficient on the frame-wise basis, the optimum threshold is accurate and adaptive at both the coefficient level and frame level. The frame-wise context model is formulated by virtue of the context subspace projection of the wavelet coefficients, with the context index employed as the invariant correspondence between successive frame parameters, thereby enabling the frame-wise tracking at the coefficient level. Simulation results show significant improvement over the wavelet-based speech enhancement algorithms in terms of the segmental signal-to-noise ratio improvement by as much as 226%, the perceptual evaluation of speech quality by 36%, the short-time objective intelligibility by 17.8% and the cepstral distance by 33.3%. When benchmarked with the well-established short-time-Fourier-transform-based counterparts, the proposed wavelet thresholding algorithm offers favorable and more robust performances, particularly under non-stationary noise conditions, with no adverse musical noise effect.

End-to-End Post-Filter for Speech Separation With Deep Attention Fusion Features

In this article, we propose an end-to-end post-filter method with deep attention fusion features for monaural speaker-independent speech separation. At first, a time-frequency domain speech separation method is applied as the pre-separation stage. The aim of pre-separation stage is to separate the mixture preliminarily. Although this stage can separate the mixture, it still contains the residual interference. In order to enhance the pre-separated speech and improve the separation performance further, the end-to-end post-filter (E2EPF) with deep attention fusion features is proposed. The E2EPF can make full use of the prior knowledge of the pre-separated speech, which contributes to speech separation. It is a fully convolutional speech separation network and uses the waveform as the input features. Firstly, the 1-D convolutional layer is utilized to extract the deep representation features for the mixture and pre-separated signals in the time domain. Secondly, to pay more attention to the outputs of the pre-separation stage, an attention module is applied to acquire deep attention fusion features, which are extracted by computing the similarity between the mixture and the pre-separated speech. These deep attention fusion features are conducive to reduce the interference and enhance the pre-separated speech. Finally, these features are sent to the post-filter to estimate each target signals. Experimental results on the WSJ0-2mix dataset show that the proposed method outperforms the state-of-the-art speech separation method. Compared with the pre-separation method, our proposed method can acquire 64.1%, 60.2%, 25.6% and 7.5% relative improvements in scale-invariant source-to-noise ratio (SI-SNR), the signal-to-distortion ratio (SDR), the perceptual evaluation of speech quality (PESQ) and the short-time objective intelligibility (STOI) measures, respectively.

Listening difficulty estimation model using short-time objective intelligibility measure for outdoor public address systems

Listening difficulty estimation model using short-time objective intelligibility measure for outdoor public address systems