Noisy Reverberant Environments Research Articles

Joint dereverberation and blind source separation using a hybrid autoregressive and convolutive transfer function-based model

Most frequency-domain blind source separation (BSS) methods are based on the multiplicative narrowband assumption, which is not valid in long reverberation environments. In contrast, convolutive transfer function (CTF)-based BSS methods do not rely on the narrowband assumption, and the separation performance is significantly improved compared to the traditional algorithms in long reverberation environments. However, the CTF-based BSS methods and their variants, e.g., autoregressive (AR) BSS methods, introduce modeling errors to some extent, due to the truncation or approximation during the optimization process. To address this problem, we propose a frequency-domain BSS method employing a hybrid AR and CTF model, which can provide more precise representations of the early reflections and late reverberations. Furthermore, we utilize the Gaussian noise model to deal with the BSS problem in noisy reverberant environments. We formulate the objective function using the maximum log-likelihood criterion, and derive an efficient iterative algorithm for parameter estimation with the block coordinate descent (BCD) method. Experimental results show that the proposed method has a better separation performance than the existing methods in long reverberation environments.

Three-stage hybrid neural beamformer for multi-channel speech enhancement.

This paper proposes a hybrid neural beamformer for multi-channel speech enhancement, which comprises three stages, i.e., beamforming, post-filtering, and distortion compensation, called TriU-Net. The TriU-Net first estimates a set of masks to be used within a minimum variance distortionless response beamformer. A deep neural network (DNN)-based post-filter is then utilized to suppress the residual noise. Finally, a DNN-based distortion compensator is followed to further improve speech quality. To characterize the long-range temporal dependencies more efficiently, a network topology, gated convolutional attention network, is proposed and utilized in the TriU-Net. The advantage of the proposed model is that the speech distortion compensation is explicitly considered, yielding higher speech quality and intelligibility. The proposed model achieved an average 2.854 wb-PESQ score and 92.57% ESTOI on the CHiME-3 dataset. In addition, extensive experiments conducted on the synthetic data and real recordings confirm the effectiveness of the proposed method in noisy reverberant environments.

Speech improvement in noisy reverberant environments using virtual microphones along with proposed array geometry

This paper proposes a novel approach for improving the speech of a single speaker in noisy reverberant environments. The proposed approach is based on using a beamformer with a large number of virtual microphones with the suggested arrangement on an open sphere. Our method takes into account virtual microphone signal synthesizing using the non-parametric sound field reproduction in the spherical harmonics domain and the popular weighted prediction error method. We obtain entirely accurate beam steering towards a known source location with more directivity. The suggested approach is proven to perform effectively not just in boosting the directivity factor but also in terms of improving speech quality as measured by subjective metrics like the PESQ. In comparison to current research in the area of speech enhancement by beamformer, our experiments reveal more noise and reverberation suppression as well as improved quality in the enhanced speech samples due to the usage of virtual beam rotation in the fixed beamformer. Text for this section.

Study on the perception of nonlinguistic information of noise-vocoded speech under noise and/or reverberation conditions

It has been known that noise and reverberation greatly affect the perception of linguistic information, in particular speech intelligibility. However, the effect of noise and reverberation on the perception of non-linguistic information has not been clarified. We investigated how these types of disturbances affect the perception of non-linguistic information (speaker individuality and vocal emotion) of noise-vocoded speech. We conducted speaker-distinction and vocal-emotion-recognition experiments using noise-vocoded speech created from the speech in noisy, reverberation, and noisy reverberant environments as stimuli. We used seven noise conditions (signal-to-noise ratio (SNR) = ∞,20,15,10,5,0,−5 dB) and six reverberation conditions (reverberation time (TR) = 0.0,0.1,0.3,0.5,1.0,2.0 s). In both speaker-distinction and vocal-emotion-recognition experiments, the main effects of noise and reverberation were significant, but the interaction was not significant. From these results, except for extremely poor sound conditions, under daily noise and reverberation conditions (an SNR of more than 10 dB and TR less than 1.0 s), there were no significant effects of noise and reverberation.

Contribution of Common Modulation Spectral Features to Vocal-Emotion Recognition of Noise-Vocoded Speech in Noisy Reverberant Environments

In one study on vocal emotion recognition using noise-vocoded speech (NVS), the high similarities between modulation spectral features (MSFs) and the results of vocal-emotion-recognition experiments indicated that MSFs contribute to vocal emotion recognition in a clean environment (with no noise and no reverberation). Other studies also clarified that vocal emotion recognition using NVS is not affected by noisy reverberant environments (signal-to-noise ratio is greater than 10 dB and reverberation time is less than 1.0 s). However, the contribution of MSFs to vocal emotion recognition in noisy reverberant environments is still unclear. We aimed to clarify whether MSFs can be used to explain the vocal-emotion-recognition results in noisy reverberant environments. We analyzed the results of vocal-emotion-recognition experiments and used an auditory-based modulation filterbank to calculate the modulation spectrograms of NVS. We then extracted ten MSFs as higher-order statistics of modulation spectrograms. As shown from the relationship between MSFs and vocal-emotion-recognition results, except for extremely high noisy reverberant environments, there were high similarities between MSFs and the vocal emotion recognition results in noisy reverberant environments, which indicates that MSFs can be used to explain such results in noisy reverberant environments. We also found that there are two common MSFs (MSKTk (modulation spectral kurtosis) and MSTLk (modulation spectral tilt)) that contribute to vocal emotion recognition in all daily environments.

VACE-WPE: Virtual Acoustic Channel Expansion Based on Neural Networks for Weighted Prediction Error-Based Speech Dereverberation

Speech dereverberation is an important issue for many real-world speech processing applications. Among the techniques developed, the weighted prediction error (WPE) algorithm has been widely adopted and advanced over the last decade, which blindly cancels out the late reverberation component from the reverberant mixture of microphone signals. In this study, we extend the neural-network-based virtual acoustic channel expansion (VACE) framework for the WPE-based speech dereverberation, a variant of the WPE that we recently proposed to enable the use of dual-channel WPE algorithm in a single-microphone speech dereverberation scenario. Based on the previous study, some ablation studies are conducted regarding the constituents of the VACE-WPE in an offline processing scenario. These studies reveal the characteristics of the system, thereby simplifying the architecture and leading to the introduction of new strategies for training the neural network for the VACE. Experimental results demonstrate that VACE-WPE (our PyTorch implementation and pre-trained models are available from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/dreadbird06/vace_wpe</uri> ) considerably outperforms its single-channel counterpart in simulated noisy reverberant environments in terms of objective speech quality and is superior to the single-channel WPE as well as several fully neural speech dereverberation methods when employed as the front-end for the far-field automatic speech recognizer.

Independent Vector Extraction for Fast Joint Blind Source Separation and Dereverberation

We address a blind source separation (BSS) problem in a noisy reverberant environment in which the number of microphones $M$ is greater than the number of sources of interest, and the other noise components can be approximated as stationary and Gaussian distributed. Conventional BSS algorithms for the optimization of a multi-input multi-output convolutional beamformer have suffered from a huge computational cost when $M$ is large. We here propose a computationally efficient method that integrates a weighted prediction error (WPE) dereverberation method and a fast BSS method called independent vector extraction (IVE), which has been developed for less reverberant environments. We show that, given the power spectrum for each source, the optimization problem of the new method can be reduced to that of IVE by exploiting the stationary condition, which makes the optimization easy to handle and computationally efficient. An experiment of speech signal separation shows that, compared to a conventional method that integrates WPE and independent vector analysis, our proposed method achieves much faster convergence while maintaining its separation performance.

Speech separation based on reliable binaural cues with two-stage neural network in noisy-reverberant environments

Background noise and room reverberation often cause a decrease in reliability of binaural cues and speech quality, especially in non-stationary environment. In order to solve these problems, we propose a novel speech separation algorithm based on two-stage neural network model and a special separation mask in noisy-reverberant environment. In this algorithm, firstly, the weight matrix is derived to construct reliable binaural cues through the first-stage neural network. The reliable binaural cues combined with complementary spectral features is used as input of separation DNN. Secondly, a special separation mask is introduced for noisy-reverberant environment, which can suppress background noise and reduce reverberation. Thirdly, the separation DNN is used as nonlinear function to estimate separation mask. Then, the two-stage neural network system is trained jointly. During the joint training process, the system adaptively adjusts the weight matrix according to the final error, which is similar to the attention mechanism introduced for binaural features. At the same time, due to the increased reliability of binaural cues, neural networks can make better use of effective information. Finally, the estimated separation mask is used to weight the noisy-reverberant speech to achieve the enhanced speech. Experimental results indicate that the proposed algorithm achieves better performance than the contrast algorithms in different scenarios with various amounts of noise and reverberation.

PSD Estimation and Source Separation in a Noisy Reverberant Environment Using a Spherical Microphone Array

In this paper, we propose an efficient technique for estimating individual power spectral density (PSD) components, i.e., PSD of each desired sound source as well as of noise and reverberation, in a multi-source reverberant sound scene with coherent background noise. We formulate the problem in the spherical harmonics domain to take the advantage of the inherent orthogonality of the spherical harmonics basis functions and extract the PSD components from the cross-correlation between the different sound field modes. We also investigate an implementation issue that occurs at the nulls of the Bessel functions and offer an engineering solution. The performance evaluation takes place in a practical environment with a commercial microphone array in order to measure the robustness of the proposed algorithm against all the deviations incurred in practice. We also exhibit an application of the proposed PSD estimator through a source septation algorithm and compare the performance with a contemporary method in terms of different objective measures.

Speech intelligibility improvement in noisy reverberant environments based on speech enhancement and inverse filtering

The speech intelligibility of indoor public address systems is degraded by reverberation and background noise. This paper proposes a preprocessing method that combines speech enhancement and inverse filtering to improve the speech intelligibility in such environments. An energy redistribution speech enhancement method was modified for use in reverberation conditions, and an auditory-model-based fast inverse filter was designed to achieve better dereverberation performance. An experiment was performed in various noisy, reverberant environments, and the test results verified the stability and effectiveness of the proposed method. In addition, a listening test was carried out to compare the performance of different algorithms subjectively. The objective and subjective evaluation results reveal that the speech intelligibility is significantly improved by the proposed method.

Joint Denoising and Dereverberation Using Exemplar-Based Sparse Representations and Decaying Norm Constraint

Exemplar-based nonnegative models, where the noisy speech is decomposed as a sparse nonnegative linear combination of the speech and noise exemplars stored in a dictionary, have been successfully used for speech denoising. This paper extends this technique for the single-channel speech enhancement in noisy reverberant environments using a novel approximation of the noisy reverberant speech in the frequency domain and nonnegative matrix deconvolution. In the proposed model, the room impulse response RIR in the magnitude short-time Fourier transform domain is defined such that its decaying structure can also be estimated from the test data itself, whereas the existing models used a suboptimal binwise clamping procedure to impose such a decaying structure that does not hold in a typical RIR. This paper presents multiplicative updates for estimating the RIR, its decay, and the underlying anechoic speech and noise. The proposed model is evaluated on a synthetically created dataset created by convolving TIMIT recordings with RIRs measured from different rooms and varying speaker-and-microphone locations, and adding background noises taken from the CHiME corpus. Simulation results show that the proposed model results in a better RIR estimate over the existing model and improves various instrumental speech quality measures.

Bayesian feature enhancement using independent vector analysis and reverberation parameter re-estimation for noisy reverberant speech recognition

Because speech recorded by distant microphones in real-world environments is contaminated by both additive noise and reverberation, the automatic speech recognition (ASR) performance is seriously degraded due to the mismatch between the training and testing environments. In the previous studies, some of the authors proposed a Bayesian feature enhancement (BFE) method with re-estimation of reverberation filter parameters for reverberant speech recognition and a BFE method employing independent vector analysis (IVA) to deal with speech corrupted by additive noise. Although both of them accomplish significant improvements in either reverberation- or noise-robust ASR, most of the real-world environments involve both additive noise and reverberation. For robust ASR in the noisy reverberant environments, in this paper, we present a hidden-Markov-model (HMM)-based BFE method using IVA and reverberation parameter re-estimation (RPR) to remove additive and reverberant distortion components in speech acquired by multi-microphones effectively by introducing Bayesian inference in the observation model of input speech features. Experimental results show that the presented method can further reduce the word error rates (WERs) compared with the BFE methods based on conventional noise and/or reverberation models and combinations of the BFE methods for reverberation- or noise-robust ASR.

Speech enhancement of instantaneous amplitude and phase for applications in noisy reverberant environments

We previously proved that restoring the instantaneous amplitude as well as instantaneous phase on the output from Gammatone filterbank plays a significant role in speech enhancement. However, dereverberation is still a challenge since the previously proposed scheme can only work in noisy environments. In this paper, we extend our previously proposed scheme to general speech enhancement for removing both the effects of noise and reverberation by restoring instantaneous amplitude and phase simultaneously. Objective and subjective experiments were conducted under various noisy reverberant conditions to evaluate the effectiveness of the extension of the proposed scheme. The signal to error ratio (SER), correlation, PESQ, and SNR loss were used in objective evaluations. The normalized mean preference score and correctness in modified rhyme test (MRT) were used in subjective evaluations. We also tested how effective our proposed scheme is as a front-end for an automatic speech recognition (ASR) system in realistic noisy reverberant environments. The results of all evaluations revealed that the proposed scheme could effectively improve quality and intelligibility of speech signals under noisy reverberant conditions.

MTF-Based Kalman Filtering with Linear Prediction for Power Envelope Restoration in Noisy Reverberant Environments

MTF-Based Kalman Filtering with Linear Prediction for Power Envelope Restoration in Noisy Reverberant Environments

Robust Voice Activity Detection Based on Concept of Modulation Transfer Function in Noisy Reverberant Environments

Most of the current voice activity detection (VAD) algorithms deal with clean (noiseless) speech or speech with additive noise conditions. They cannot work in noisy reverberant environments or work poorly if they do, because speech is smeared due to the effects of noise and reverberation. This paper proposes a robust VAD algorithm for precisely detecting speech and non-speech periods in noisy reverberant environments. The proposed VAD algorithm consists of three blocks. The first block is an estimation of the signal to noise ratio (SNR) which is used to mitigate the additive noise effect on the speech power envelope. The second block is a speech power envelope dereverberation based on the modulation transfer function concept. The last block is a threshold processing on the dereverberated speech power envelope for speech/non-speech decision. Experiments on VAD in both artificial and realistic noisy reverberant environments revealed that the proposed VAD algorithm significantly outperforms the conventional VAD algorithms.

Binaural Sound Source Localization in Noisy Reverberant Environments Based on Equalization-Cancellation Theory

Binaural Sound Source Localization in Noisy Reverberant Environments Based on Equalization-Cancellation Theory

A modulation-transfer-function-based method for restoring sub-band power envelope from noisy reverberant speech

The concept of the modulation transfer function (MTF) can be successfully applied to evaluating the quality of speech transmission in room acoustics (noisy reverberant environments) as functions of reverberation (reverberation time) and additive noise (signal to noise ratio) (Houtgast and Steeneken, J. Acoust. Soc. Am., 77, 1069-1077, 1985). This paper proposes a method of restoring the power envelope from noisy reverberant speech based on the MTF concept. The proposed method does not need the impulse response and noise conditions in room acoustics to be measured to enhance speech. The proposed approach suppresses the effects of reverberation and noise on the power envelopes by restoring the smeared MTF. We carried out massive simulations of noise-suppression and dereverberation on noisy reverberant speech to objectively evaluate the proposed method. The results revealed that the proposed method could simultaneously work well with both the suppression of noise and dereverberation. We further tested the proposed method as a front-end processor for ASR systems in noisy reverberant environments, and compared it with other methods (MFCC, CMN, spectral subtraction, and RASTA filtering on a constant-bandwidth filterbank). The results demonstrated that the improvement in recognition with the proposed method was more effective than that in extremely noisy reverberant environments.

Sound Source DOA Estimation and Localization in Noisy Reverberant Environments Using Least-Squares Support Vector Machines

This paper presents two new algorithms for mapping the time-differences-of-arrival (TDOAs) measured from the microphone pairs to sound source direction-of-arrival (DOA) and location in room environments based on the least-squares support vector machine (LS-SVM). Least squares (LS) has been widely used in the TDOA based algorithms for sound source DOA estimation or localization to map the measured TDOAs into sound source DOA or location. The drawback of LS mapping is that its performance degrades significantly in some scenarios. To combat this problem, an LS-SVM regression based algorithm for the nonlinear mapping is proposed, which outperforms the LS based algorithm in noisy reverberant rooms. Conventional approaches to sound source localization usually assume that the microphones used are ideal and that the locations of the microphones are also known a priori, which may not be well satisfied in practice. Therefore, the microphone arrays need to be calibrated carefully before use. However, it is not an easy task to calibrate microphone arrays perfectly. In this paper, we also proposed an algorithm for sound source localization based on the LS-SVM, which has the advantage that microphone array calibration is not required. The performance of the proposed algorithms is validated by the simulation results in noisy reverberant environments.

Dual-Source Transfer-Function Generalized Sidelobe Canceller

Full-duplex hands-free man/machine interface often suffers from directional nonstationary interference, such as a competing speaker, as well as stationary interferences which may comprise both directional and nondirectional signals. The transfer-function generalized sidelobe canceller (TF-GSC) exploits the nonstationarity of the speech signal to enhance it when the undesired interfering signals are stationary. Unfortunately, the assumptions leading to the derivation of the TF-GSC are violated when a nonstationary interference is present. In this paper, we propose an adaptive beamformer, based on the TF-GSC, that is suitable for cancelling nonstationary interferences in noisy reverberant environments. We modify two of the TF-GSC components to enable suppression of the nonstationary undesired signal. A modified fixed beamformer (FBF) is designed to block the nonstationary interfering signal while maintaining the desired speech signal. A modified blocking matrix (BM) is designed to block both the desired signal and the nonstationary interference. We introduce a novel method for updating the blocking matrix in double talk scenarios, which exploits the nonstationarity of both the desired and interfering speech signals. Experimental results demonstrate the performance of the proposed algorithm in noisy and reverberant environments and show its superiority over the original TF-GSC.

An evaluation of talker localization based on direction of arrival estimation and statistical sound source identification

It is very important to capture distant-talking speech for a hands-free speech interface with high quality. A microphone array is an ideal candidate for this purpose. However, this approach requires localizing the target talker. Conventional talker localization algorithms in multiple sound source environments not only have difficulty localizing the multiple sound sources accurately, but also have difficulty localizing the target talker among known multiple sound source positions. To cope with these problems, we propose a new talker localization algorithm consisting of two algorithms. One is DOA (direction of arrival) estimation algorithm for multiple sound source localization based on CSP (cross-power spectrum phase) coefficient addition method. The other is statistical sound source identification algorithm based on GMM (Gaussian mixture model) for localizing the target talker position among localized multiple sound sources. In this paper, we particularly focus on the talker localization performance based on the combination of these two algorithms with a microphone array. We conducted evaluation experiments in real noisy reverberant environments. As a result, we confirmed that multiple sound signals can be identified accurately between ‘‘speech’’ or ‘‘non-speech’’ by the proposed algorithm. [Work supported by ATR, and MEXT of Japan.]