Input Noisy Speech Research Articles

End‐to‐end speech‐denoising deep neural network based on residual‐attention gated linear units

AbstractIn this letter, an improved gated linear unit (GLU) structure for end‐to‐end (E2E) speech enhancement is proposed. In the U‐Net structure, which is widely used as the foundational architecture for E2E deep neural network‐based speech denoising, the input noisy speech signal undergoes multiple layers of encoding and is compressed into essential potential representative information at the bottleneck. The latent information is then transmitted to the decoder stage for the restoration of the target clean speech. Among these approaches, CleanUNet, a prominent state‐of‐the‐art (SOTA) method, enhances temporal attention in latent space by employing multi‐head self‐attention. However, unlike the approach of applying the attention mechanism to the potentially compressed representative information of the bottleneck layer, the proposed method instead assigns the attention module to the GLU of each encoder/decoder block layer. The proposed method is validated by measuring short‐term objective speech intelligibility and sound quality. The objective evaluation results indicated that the proposed method using residual‐attention GLU outperformed existing methods using SOTA models such as FAIR‐denoiser and CleanUNet across signal‐to‐noise ratios ranging from 0 to 15 dB.

Deep Neural Network for Supervised Single-Channel Speech Enhancement

Speech enhancement is fundamental for various real time speech applications and it is a challenging task in the case of a single channel because practically only one data channel is available. We have proposed a supervised single channel speech enhancement algorithm in this paper based on a deep neural network (DNN) and less aggressive Wiener filtering as additional DNN layer. During the training stage the network learns and predicts the magnitude spectrums of the clean and noise signals from input noisy speech acoustic features. Relative spectral transform-perceptual linear prediction (RASTA-PLP) is used in the proposed method to extract the acoustic features at the frame level. Autoregressive moving average (ARMA) filter is applied to smooth the temporal curves of extracted features. The trained network predicts the coefficients to construct a ratio mask based on mean square error (MSE) objective cost function. The less aggressive Wiener filter is placed as an additional layer on the top of a DNN to produce an enhanced magnitude spectrum. Finally, the noisy speech phase is used to reconstruct the enhanced speech. The experimental results demonstrate that the proposed DNN framework with less aggressive Wiener filtering outperforms the competing speech enhancement methods in terms of the speech quality and intelligibility.

CAQoE: A Novel No-Reference Context-aware Speech Quality Prediction Metric

The quality of speech degrades while communicating over Voice over Internet Protocol applications, for example, Google Meet, Microsoft Skype, and Apple FaceTime, due to different types of background noise present in the surroundings. It reduces human perceived Quality of Experience (QoE). Along this line, this article proposes a novel speech quality prediction metric that can meet human’s desired QoE level. Our motivation is driven by the lack of evidence showing speech quality metrics that can distinguish different noise degradations before predicting the quality of speech. The quality of speech in noisy environments is improved by speech enhancement algorithms, and for measuring and monitoring the quality of speech, objective speech quality metrics are used. With the integration of these components, a novel no-reference context-aware QoE prediction metric (CAQoE) is proposed in this article, which initially identifies the context or noise type or degradation type of the input noisy speech signal and then predicts context-specific speech quality for that input speech signal. It will have of great importance in deciding the speech enhancement algorithms if the types of degradations causing poor speech quality are known along with the quality metric. Results demonstrate that the proposed CAQoE metric outperforms in different contexts as compared to the metric where contexts are not identified before predicting the quality of speech, even in the presence of limited size speech corpus having different contexts available from the NOIZEUS speech database.

End-to-End Neural Based Modification of Noisy Speech for Speech-in-Noise Intelligibility Improvement

Intelligibility of speech can be significantly reduced when it is presented in adverse near-end listening conditions, like background noise. Multiple approaches have been suggested to improve the perception of speech in such conditions. However, most of these approaches were designed to work with clean input speech. Therefore, they have serious limitations when deployed in real world applications like telephony and hearing aids, where noisy input speech is quite common. In this paper we present an end-to-end neural network approach for the above problem, which effectively reduces the input noise and improves the intelligibility for listeners in adverse conditions. To that end, a convolutional neural network topology with variable dilation factors is proposed and evaluated both in a causal and a non-causal configuration using raw speech as input. A Teacher-Student training strategy is employed, where the Teacher is a well-established speech-in-noise intelligibility enhancer based on spectral shaping followed by dynamic range compression (SSDRC). The evaluation is performed both objectively using the speech intelligibility in bits metric (SIIB), and subjectively on the Greek Harvard corpus. A noise robust multi-band version of SSDRC was used as a baseline. Compared with the baseline, at 0 dB input SNR, the suggested neural network system achieved about 380% and 230% relative SIIB improvements in fluctuating and stationary backgrounds, respectively. Subjectively, the suggested model increased listeners’ keyword correct rate in stationary noise from 25% to 60% at 0 dB input SNR, and from about 52% to 75% at 5 dB input SNR, compared with the baseline.

Noise-Robust Voice Conversion Using High-Quefrency Boosting via Sub-Band Cepstrum Conversion and Fusion

This paper presents a noise-robust voice conversion method with high-quefrency boosting via sub-band cepstrum conversion and fusion based on the bidirectional long short-term memory (BLSTM) neural networks that can convert parameters of vocal tracks of a source speaker into those of a target speaker. With the implementation of state-of-the-art machine learning methods, voice conversion has achieved good performance given abundant clean training data. However, the quality and similarity of the converted voice are significantly degraded compared to that of a natural target voice due to various factors, such as limited training data and noisy input speech from the source speaker. To address the problem of noisy input speech, an architecture of voice conversion with statistical filtering and sub-band cepstrum conversion and fusion is introduced. The impact of noises on the converted voice is reduced by the accurate reconstruction of the sub-band cepstrum and the subsequent statistical filtering. By normalizing the mean and variance of the converted cepstrum to those of the target cepstrum in the training phase, a cepstrum filter was constructed to further improve the quality of the converted voice. The experimental results showed that the proposed method significantly improved the naturalness and similarity of the converted voice compared to the baselines, even with the noisy inputs of source speakers.

Bark scaled oversampled WPT based speech recognition enhancement in noisy environments

The performance of speech recognition system degrades significantly in real-world environment, is a case of the acoustic mismatch between the training and operating conditions. This paper presents a two-stage approach to make a speech recognition system immune to additive and uncorrelated background noise i.e. robust. In the first stage, an oversampled wavelet packet decomposes the entire input noisy speech into seventeen nonlinear frequency subbands like the Bark scale of the human hearing system and the adaptive noise estimation based spectral subtraction filters the noisy speech from each subband signal. The oversampled WPT is linear and advantageous as it causes to overcome the shift-invariance complexity by removing the decimation after the filtering at each decomposition level. In the second stage, a nonparametric approach is used for feature extraction from filtered speech, and the parameters from the feature extraction stage are compared with the parameters extracted from speech signals stored in a template to recognize the utterance. A series of experiments are carried out to evaluate the performance of the proposed two-stage system in a variety of real environments, with and without the use of the first stage. Recognition accuracy is evaluated at the word level in a wide range of SNRs for various types of noisy environments. The experimental results show significant improvement in recognition performance at low SNR using the proposed system.

Variance based time-frequency mask estimation for unsupervised speech enhancement

Variance based two dimensional time-frequency mask estimation for unsupervised speech enhancement is proposed to improve the speech quality and intelligibility by reducing the low-frequency residual noise distortion in the noisy speech signals. Unlike conventional speech enhancement methods, the proposed method is able to reduce the residual noise distortion by utilizing benefits of the less aggressive Wiener gain and variance based two dimensional time-frequency mask to establish a two-stage speech enhancement method. In the first stage, the less aggressive Wiener gain with modified a priori signal-to-noise (SNR) estimate is applied to the input noisy speech to obtain a reduced noise pre-processed speech signal. In the second stage, variance based features are extracted from the pre-processed speech and compared to a nonparametric adaptive threshold to construct a two dimensional time-frequency mask. The estimated mask is then applied to the pre-processed speech from the first stage to suppress the annoying residual noise distortion. A comparative performance study is included to demonstrate the effectiveness of the proposed method in various noisy conditions. The experimental results showed large improvements in terms of the perceptual evaluation of speech quality (PESQ), segmental SNR (SegSNR), residual noise distortion (BAK) and speech distortion (SIG) over that achieved with competing methods at different input SNRs. To measure the understanding of enhanced speech in different noisy conditions, short-time intelligibility prediction (STOI) is used which reinforced a better performance of the proposed method in terms of the speech intelligibility. The time-varying spectral analysis validated significant reduction of the residual noise components in the enhanced speech.

Performance improvement of monaural speech separation system using image analysis techniques

This research work proposes an image analysis-based algorithm to enhance the time–frequency (T–F) mask obtained in the initial segmentation of CASA-based monaural speech separation system to improve speech quality and intelligibility. It consists of labelling the initial segmentation mask, boundary extraction, active pixel detection and eliminating the non-active pixels related to noise. In labelling, the T–F mask obtained is labelled as periodicity pixel ( P ) matrix and non-periodicity pixel ( NP ) matrix. Next speech boundaries are created by connecting all the possible nearby P and NP matrix. Some speech boundary may include noisy T–F units as holes; these holes are treated using the proposed algorithm. The proposed algorithm is evaluated with the quality and intelligibility measures such as signal to noise ratio (SNR), perceptual evaluation of speech quality, P EL , P NR , coherence speech intelligibility index (CSII), normalised covariance metric (NCM), and short-time objective intelligibility (STOI). The experimental results show that the proposed algorithm improves the speech quality by increasing the SNR with an average value of 9.91 dB and reduces the P NR by an average value of 25.6% and also improves the speech intelligibility in terms of CSII, NCM, and STOI when compared with the input noisy speech mixture

An Auto-Focusing-Noise Suppressor for Cell-Phone Videos Based on Multiple Noise References and Spectral Gain Selection

This paper proposes an auto-focusing (AF)-noise suppressor for cell-phone videos based on multiple noise references and spectral gain selection. AF noise generated by a lens actuator, a cam, or a helicoid is suppressed with a spectral gain which is calculated from a combination of multiple noise references prepared in advance. For quickly adjusting the spectral gain to product-dependent noise characteristics, a mixing ratio of noise references is adaptively determined based on an analysis result of the input noisy speech. The spectral gain for AF noise or that for background noise, whichever is smaller, is applied to the input noisy speech for actual suppression so that both AF noise and background noise are suppressed. Simulation results demonstrate that an AF noise recorded with a commercial cell-phone handset is successfully suppressed with little distortion in the target speech. The PESQ score is improved by as much as 2.9 compared to a conventional noise suppressor designed for ordinary background noise. This paper has been employed in commercial products which are shipped worldwide.

Robust Recognition of English Speech in Noisy Environments Using Frequency Warped Signal Processing

The performance level of speech recognizer drops significantly when there is an acoustic mismatch between training and operational environments. A speech recognizer is called robust if it preserves good recognition accuracy even in the mismatch conditions. Present study addresses the recognition of English speech in noisy environments and presents the comparative study of various frequency scales used in parameterization based on the average recognition rate. For the robust automatic speech reorganization, a front end signal enhancement component, spectral subtraction algorithm, is used to prefilter the noisy input speech prior fed to the recognizer. A number of frequency warped scales namely, perceptual scales viz, Mel scale, Bark scale, equivalent rectangular bandwidth rate scale, and a non-perceptual scale called uniform scale are used in the parameterization for feature extraction from enhanced speech. A suite of experiments is carried out to evaluate the performance of the speech recognizer, with and without the use of a front end signal enhancement component, in a variety of noisy environments. Recognition accuracy is tested in terms of word linguistic levels on a wide range of signal to noise ratios for both stationary and non-stationary noises.

A wavelet- based transform method for quality improvement in noisy speech patterns of Arabic language

This paper addresses the problem of single-channel speech enhancement of low (negative) SNR of Arabic noisy speech signals. For this aim, a binary mask thresholding function based coiflet5 mother wavelet transform is proposed for Arabic speech enhancement. The effectiveness of binary mask thresholding function based coiflet5 mother wavelet transform is compared with Wiener method, spectral subtraction, log-MMSE, test-PSC and p-mmse in presence of babble, pink, white, f-16 and Volvo car interior noise. The noisy input speech signals are processed at various levels of input SNR range from ź5 to ź25 dB. Performance of the proposed method is evaluated with the help of PESQ, SNR and cepstral distance measure. The results obtained by proposed binary mask thresholding function based coiflet5 wavelet transform method are very encouraging and shows that the proposed method is much helpful in Arabic speech enhancement than other existing methods.

최대우도를 부가한 주파수 변이 PMC 방법의 잡음 음성 인식 성능개선

잡음에 강건한 음성 인식을 위한 주파수 변이를 이용한 PMC( Parallel Model Compensation Using Frequency-variant, FV-PMC) 방법은 인식시 입력음성에 혼입이 예상되는 잡음들을 평균 주파수 변이도를 임계치로 하여 몇 가지 잡음 군으로 분류한 후 각 잡음 군 별로 인식을 수행하는 방법이다. 이 방법은 기준 임계치를 이용하여 양호하게 분류된 잡음 음성들에 대해서는 매우 우수한 성능을 보이나, 미 분류된 잡음 음성들에 대해서는 기존의 PMC 방법에서와 같이 무잡음 모델과 결합하여 음성 인식을 수행함으로 인해 평균 음성 인식률이 낮아지는 문제점이 있다. 이러한 문제점을 해결하기 위하여 본 논문에서는 기존의 방법에서 사용하였던 평균주파수 임계치 방법 대신에 최대 우도를 부가하여 미분류를 방지함으로써 입력 잡음음성에 포함되는 잡음의 군별 잡음 분류 율을 높여 인식률을 제고하는 개선된 주파수 변이 PMC 인식방법을 제안하였다. Aurora 2.0 데이터베이스를 이용한 인식실험결과, 기존의 FV-PMC 방법에 비해 향상된 결과를 확인할 수 있었다. The Parallel Model Compensation Using Frequency-variant: FV-PMC for noise-robust speech recognition is a method to classify the noises, which are expected to be intermixed with input speech when recognized, into several groups of noises by setting average frequency variant as a threshold value; and to recognize the noises depending on the classified groups. This demonstrates the excellent performance considering noisy speech categorized as good using the standard threshold value. However, it also holds a problem to decrease the average speech recognition rate with regard to unclassified noisy speech, for it conducts the process of speech recognition, combined with noiseless model as in the existing PMC. To solve this problem, this paper suggests a enhanced method of recognition to prevent the unclassified through improving the extent of rating scales with use of maximum likelihood so that the noise groups, including input noisy speech, can be classified into more specific groups, which leads to improvement of the recognition rate. The findings from recognition experiments using Aurora 2.0 database showed the improved results compared with those from the method of the previous FV-PMC.

Speech enhancement based on soft audible noise masking and noise power estimation

This paper presents a perceptual model based speech enhancement algorithm. The proposed algorithm measures the amount of the audible noise in the input noisy speech based on estimation of short-time spectral power of noise signal, and masking threshold calculated from the estimated spectrum of clean speech. An appropriate amount of noise reduction is chosen based on the result to achieve good noise suppression without introducing significant distortion to the clean speech. To mitigate the problem of “musical noise”, the amount of noise reduction is linked directly to the estimation of short-term noise spectral amplitude instead of noise variance so that the spectral peaks of noise can be better suppressed. Good performance of the proposed speech enhancement system is confirmed through objective and subjective tests.

Statistical voice activity detection based on sparse representation over learned dictionary

Abstract In this paper, we present a novel approach to voice activity detection (VAD) based on the sparse representation of an input noisy speech over a learned dictionary. First, we investigate the relationship between the signal detection and the sparse representation based on the Bayesian framework. Second, we derive the decision rule and an adaptive threshold based on a likelihood ratio test, by modeling the non-zero elements in the sparse representation as a Gaussian distribution. The experimental results show that the proposed approach outperforms the current statistical model-based methods, such as Gaussian, Laplacian, and Gamma, under white, babble, and vehicle noise conditions.

Dealing with noise in automatic speech recognition.

While automatic speech recognition (ASR) can work very well for clean speech, recognition accuracy often degrades significantly when the speech signal is subject to corruption, as occurs in many communication channels. This paper will survey recent methods for handling various distortions in practical ASR. The problem is often presented as an issue of mismatch between the models that are created during prior training phases and unforeseen environmental acoustic conditions that occur during the normal test phase. As one can never anticipate all possible future conditions, ASR analysis must be able to adapt to a wide variety of distortions. Human listeners furnish a useful standard of comparison for ASR in that humans are much more flexible in handling unexpected acoustic distortions than current ASR is. Methods that adapt ASR features and models will be compared against ASR methods that enhance the noisy input speech. Other topics to be discussed will include estimation of noise and channel parameters, RAS...

A Subtractive-Type Speech Enhancement Using the Perceptual Frequency-Weighting Function

The present paper describes quality enhancement of speech corrupted by an additive background noise in a single-channel system. The proposed approach is based on the introduction of a perceptual criterion using a frequency-weighting filter in a subtractive-type enhancement process. Although this subtractive-type method is very attractive because of its simplicity, it produces an unnatural and unpleasant residual noise. Thus, it is difficult to select fixed optimized parameters for all speech and noise conditions. A new and effective algorithm is thus developed based on the masking properties of the human ear. This newly developed algorithm allows for an automatic adaptation in the time and frequency of the enhancement system and determines a suitable noise estimate according to the frequency of the noisy input speech. Experimental results demonstrate that the proposed approach can efficiently remove additive noise related to various kinds of noise corruption.

A noise reduction technique of speech signal using ICA and spectral analysis

This paper discusses the new method on noise reduction exploiting the combined effects of wavelet decomposition, ICA and spectral analysis on noisy speech. The input noisy speech is wavelet decomposed into two signals. Wavelet entropy is computed based on the modified probability density function for the signal derived from the approximation coefficients during wavelet decomposition. By proper entropy comparison, the starting frame is detected. Between the two signals obtained from the wavelet decomposition, one is speech combined with noise and another one is noise alone. These two signals are analysed in independent component analysis (ICA) domain, in order to generate an enhanced speech. Zero-crossing rate is computed and used to discriminate between speech and noise. Then, spectral analysis is performed on the noise prior to starting frame and noisy speech. Elimination of noise frequencies in the noisy speech leads to noise reduced speech. Subjective analysis and experimental results show the considerable noise reduction capability of the proposed algorithm.

Dynamic Bayesian Network Inversion for Robust Speech Recognition

This paper presents an inversion algorithm for dynamic Bayesian networks towards robust speech recognition, namely DBNI, which is a generalization of hidden Markov model inversion (HMMI). As a dual procedure of expectation maximization (EM)-based model reestimation, DBNI finds the 'uncontaminated' speech by moving the input noisy speech to the Gaussian means under the maximum likelihood (ML) sense given the DBN models trained on clean speech. This algorithm can provide both the expressive advantage from DBN and the noise-removal feature from model inversion. Experiments on the Aurora 2.0 database show that the hidden feature model (a typical DBN for speech recognition) with the DBNI algorithm achieves superior performance in terms of word error rate reduction.

A Comprehensive Noise Robust Speech Parameterization Algorithm Using Wavelet Packet Decomposition-Based Denoising and Speech Feature Representation Techniques

This paper concerns the problem of automatic speech recognition in noise-intense and adverse environments. The main goal of the proposed work is the definition, implementation, and evaluation of a novel noise robust speech signal parameterization algorithm. The proposed procedure is based on time-frequency speech signal representation using wavelet packet decomposition. A new modified soft thresholding algorithm based on time-frequency adaptive threshold determination was developed to efficiently reduce the level of additive noise in the input noisy speech signal. A two-stage Gaussian mixture model (GMM)-based classifier was developed to perform speech/nonspeech as well as voiced/unvoiced classification. The adaptive topology of the wavelet packet decomposition tree based on voiced/unvoiced detection was introduced to separately analyze voiced and unvoiced segments of the speech signal. The main feature vector consists of a combination of log-root compressed wavelet packet parameters, and autoregressive parameters. The final output feature vector is produced using a two-staged feature vector postprocessing procedure. In the experimental framework, the noisy speech databases Aurora 2 and Aurora 3 were applied together with corresponding standardized acoustical model training/testing procedures. The automatic speech recognition performance achieved using the proposed noise robust speech parameterization procedure was compared to the standardized mel-frequency cepstral coefficient (MFCC) feature extraction procedures ETSI ES 201 108 and ETSI ES 202 050.

Piecewise-linear transformation-based HMM adaptation for noisy speech

This paper proposes a new method using piecewise-linear transformation for adapting phone HMMs to noisy speech. Various noises are clustered according to their spectral property, and a noisy speech HMM corresponding to each clustered noise and SNR condition is made. Based on the likelihood maximization criterion, an HMM that best matches an input noisy speech is selected and further adapted using linear transformation. The proposed method is evaluated by its ability to recognize noisy broadcast-news speech. It is confirmed that the proposed method is effective in recognizing numerically noise-added speech and actual noisy speech under various noise conditions. The proposed method minimizes mismatches between noisy input speech and the HMM’s, sentence by sentence, without requiring online noise spectrum/model estimation. The proposed method is therefore easily applicable to real world conditions with frequently changing noise.