Reverberant Environments Research Articles

Enhanced forensic speaker verification performance using the ICA-EBM algorithm under noisy and reverberant environments

Forensic speaker verification performance reduces significantly under high levels of noise and reverberation. Multiple channel speech enhancement algorithms, such as independent component analysis by entropy bound minimization (ICA-EBM), can be used to improve noisy forensic speaker verification performance. Although the ICA-EBM was used in previous studies to separate mixed speech signals under clean conditions, the effectiveness of using the ICA-EBM for improving forensic speaker verification performance under noisy and reverberant conditions has not been investigated yet. In this paper, the ICA-EBM algorithm is used to separate the clean speech from noisy speech signals. Features from the enhanced speech are obtained by combining the feature-warped mel frequency cepstral coefficients with similar features extracted from the discrete wavelet transform. The identity vector (i-vector) length normalized Gaussian probabilistic linear discriminant analysis is used as a classifier. The Australian Forensic Voice Comparison and QUT-NOISE corpora were used to evaluate forensic speaker verification performance under noisy and reverberant conditions. Simulation results demonstrate that forensic speaker verification performance based on ICA-EBM improves compared with that of the traditional independent component analysis under different types of noise and reverberation environments. For surveillance recordings corrupted with different types of noise (CAR, STREET and HOME) at − 10 dB signal to noise ratio, the average equal error rate of the proposed method based on ICA-EBM is better than that of the traditional ICA by 12.68% when the interview recordings are kept clean, and 7.25% when the interview recordings have simulated room reverberations.

USING AUTOMATIC SPEECH RECOGNITION AND SPEECH SYNTHESIS TO IMPROVE THE INTELLIGIBILITY OF COCHLEAR IMPLANT USERS IN REVERBERANT LISTENING ENVIRONMENTS.

Cochlear implant (CI) users experience substantial difficulties in understanding reverberant speech. A previous study proposed a strategy that leverages automatic speech recognition (ASR) to recognize reverberant speech and speech synthesis to translate the recognized text into anechoic speech. However, the strategy was trained and tested on the same reverberant environment, so it is unknown whether the strategy is robust to unseen environments. Thus, the current study investigated the performance of the previously proposed algorithm in multiple unseen environments. First, an ASR system was trained on anechoic and reverberant speech using different room types. Next, a speech synthesizer was trained to generate speech from the text predicted by the ASR system. Experiments were conducted in normal hearing listeners using vocoded speech, and the results showed that the strategy improved speech intelligibility in previously unseen conditions. These results suggest that the ASR-synthesis strategy can potentially benefit CI users in everyday reverberant environments.

Joint Optimization of Deep Neural Network-Based Dereverberation and Beamforming for Sound Event Detection in Multi-Channel Environments.

In this paper, we propose joint optimization of deep neural network (DNN)-supported dereverberation and beamforming for the convolutional recurrent neural network (CRNN)-based sound event detection (SED) in multi-channel environments. First, the short-time Fourier transform (STFT) coefficients are calculated from multi-channel audio signals under the noisy and reverberant environments, which are then enhanced by the DNN-supported weighted prediction error (WPE) dereverberation with the estimated masks. Next, the STFT coefficients of the dereverberated multi-channel audio signals are conveyed to the DNN-supported minimum variance distortionless response (MVDR) beamformer in which DNN-supported MVDR beamforming is carried out with the source and noise masks estimated by the DNN. As a result, the single-channel enhanced STFT coefficients are shown at the output and tossed to the CRNN-based SED system, and then, the three modules are jointly trained by the single loss function designed for SED. Furthermore, to ease the difficulty of training a deep learning model for SED caused by the imbalance in the amount of data for each class, the focal loss is used as a loss function. Experimental results show that joint training of DNN-supported dereverberation and beamforming with the SED model under the supervision of focal loss significantly improves the performance under the noisy and reverberant environments.

Robust Indoor Speaker Localization in the Circular Harmonic Domain

In this article, modal signal processing using circular sensor arrays has been shown to be an attractive way for speaker localization, due to the fact that it enables high flexibility in sound field analysis, which inherently supports wideband acoustic sources and provides a frequency invariant beampattern. However, the existing circular harmonic direction-of-arrival (DOA) estimation approaches suffer from the following problems: first, depending strongly on the characteristics of sensor arrays; second, the presence of singularity when using Bessel function of the first kind; and third, being sensitive to noisy and reverberant environments. To combat these problems, this article proposes a robust indoor speaker localization method that incorporates circular harmonic pseudointensity vector through joining the least-squares decomposition and the spatial processing with a small-sized sensor array. The root mean square error of DOA estimation of the proposed method is compared with those published approaches in simulations for different noise levels and reverberation times. In addition, the superior performance of the proposed indoor speaker localization method is validated by using speech recordings in a real-world acoustic environment.

The effect of pinna filtering in binaural transfer functions on externalization in a reverberant environment

The perception of the externalization of noise stimuli is investigated with respect to the pinna filtering effect on direct and reverberant parts of binaural transfer functions. Individual binaural room impulse responses (BRIRs) recorded with and without pinnae, called pinna BRIRs and head BRIRs respectively, are divided into the direct and reverberant parts. For frontal and lateral sources, four types of BRIRs are constructed and investigated. Two of them are measured on subjects with their pinnae exposed or not, and the other two are obtained by concatenating the direct and reverberant parts of the two types of measured BRIRs crosswise. Normal hearing listeners are presented with noise trains convolved with these four types of BRIRs. It is found that pinna filtering in the direct parts is helpful for externalization while pinna filtering in the reverberant parts produces less externalized sound images. Further analysis shows that pinna filtering can change the direct-to-reverberant ratio (DRR), the frequency-to-frequency variability (FFV), and the interaural coherence (IC). The DRR and the FFV are only partially related to the externalization ratings, but the short-term IC corresponds well to the ratings. This study provides guidance for improving externalization perception in binaural synthesis.

A General Nonstationary and Time-Varying Mixed Signal Blind Source Separation Method Based on Online Gaussian Process

Most nonstationary and time-varying mixed source separation algorithms are based on the model of instantaneous mixtures. However, the observation signal is a convolutional mixed source in reverberation environment, such as mobile voice received by indoor microphone arrays. In this paper, a time-varying convolution blind source separation (BSS) algorithm for nonstationary signals is proposed, which can separate both time-varying instantaneous mixtures and time-varying convolution mixtures. We employ the variational Bayesian (VB) inference method with Gaussian process (GP) prior for separating the nonstationary source frame by frame from the time-varying convolution signal, in which the prior information of the mixing matrix and the source signal are obtained by the Gaussian autoregressive method, and the posterior distributions of parameters (source signal and mixing matrix) are obtained by the VB learning. In the learning process, the learned parameters and hyperparameters are propagated to the next frame for VB inference as the prior which is combined with the likelihood function to get the posterior distribution. The experimental results show that the proposed algorithm is effective for separating time-varying mixed speech signals.

Audiological outcome measures with the BONEBRIDGE transcutaneous bone conduction hearing implant: impact of noise, reverberation and signal processing features

Objective: To assess the performance of an active transcutaneous implantable-bone conduction device (TI-BCD), and to evaluate the benefit of device digital signal processing (DSP) features in challenging listening environments.Design: Participants were tested at 1- and 3-month post-activation of the TI-BCD. At each session, aided and unaided phoneme perception was assessed using the Ling-6 test. Speech reception thresholds (SRTs) and quality ratings of speech and music samples were collected in noisy and reverberant environments, with and without the DSP features. Self-assessment of the device performance was obtained using the Abbreviated Profile of Hearing Aid Benefit (APHAB) questionnaire.Study sample: Six adults with conductive or mixed hearing loss.Results: Average SRTs were 2.9 and 12.3 dB in low and high reverberation environments, respectively, which improved to −1.7 and 8.7 dB, respectively with the DSP features. In addition, speech quality ratings improved by 23 points with the DSP features when averaged across all environmental conditions. Improvement scores on APHAB scales revealed a statistically significant aided benefit.Conclusions: Noise and reverberation significantly impacted speech recognition performance and perceived sound quality. DSP features (directional microphone processing and adaptive noise reduction) significantly enhanced subjects’ performance in these challenging listening environments.

Target detection method using multipath information in an underwater waveguide environment

Interface scattering in the ocean environment consistently results in reverberation and multipath echo, thereby decreasing the detection probability of moving targets. In this work, a new detection method which fully uses underwater acoustic channel characteristics to realise high detection probability for moving targets is proposed. First, this method combines fractional Fourier transform and fractional convolution theory to build a delay time-Doppler shift matrix related to multipath echoes. Second, the insensitivity of linear modulated frequency signal and distribution of multipath echo analysis are utilised to detect the moving target in a severe environment. Simulation results show that the proposed method can obtain higher detection probability and more accurate parameter estimation than the traditional method. Experimental results also verify that the proposed method has favourable performance even in strong reverberation environment.

Sound Source Localization Based on GCC-PHAT With Diffuseness Mask in Noisy and Reverberant Environments

Although sound source localization is a desirable technique in many communication systems and intelligence applications, the distortion caused by diffuse noise or reverberation makes the time delay estimation (TDE) between signals acquired by a pair of microphones a complicated and challenging problem. In this paper, we describe a method that can efficiently achieve sound source localization in noisy and reverberant environments. This method is based on the generalized cross-correlation (GCC) function with phase transform (PHAT) weights (GCC-PHAT) to achieve robustness against reverberation. In addition, to estimate the time delay robust to diffuse components and to further improve the robustness of the GCC-PHAT against reverberation, time-frequency(t-f) components of observations directly emitted by a point source are chosen by “inversed” diffuseness. The diffuseness that can be estimated from the coherent-to-diffuse power ratio (CDR) based on spatial coherence between two microphones represents the contribution of diffuse components on a scale of zero to one with direct sounds from a source modeled to be fully coherent. In particular, the “inversed” diffuseness is binarized with a very rigorous threshold to select highly reliable components for accurate TDE even in noisy and reverberant environments. Experimental results for both simulated and real-recorded data consistently demonstrated the robustness of the presented method against diffuse noise and reverberation.

Multi-Hypothesis Square-Root Cubature Kalman Particle Filter for Speaker Tracking in Noisy and Reverberant Environments

In this paper, a multi-hypothesis square-root cubature Kalman particle filter (MH-SRCKPF) is proposed for speaker tracking in noisy and reverberant environments with distributed microphone arrays. The conventional cubature Kalman particle filter (CKPF) uses the cubature Kalman filter (CKF) to generate its proposal for particle sampling. Such a proposal incorporates only one observation from a certain localization function for the state estimation, which is vulnerable to noise or reverberation, yielding the degraded tracking performance. To tackle the problem, by incorporating multiple possible observations into CKF for the proposal, a multi-hypothesis CKPF (MH-CKPF) algorithm is first developed. Furthermore, to improve the numerical stability, an MH-SRCKPF algorithm is developed, where the state estimate and the square root of the error covariance are propagated at each time. Finally, the MH-SRCKPF is applied to the speaker tracking problems in distributed microphone arrays. Experimental results demonstrate that the proposed MH-SRCKPF outperforms the competing methods in the presence of noise and reverberation. Meanwhile, by propagating the square root of the state covariance, the proposed method exhibits attractive numerical characteristics.

A Unified Deep Learning Framework for Short-Duration Speaker Verification in Adverse Environments

Speaker verification (SV) has recently attracted considerable research interest due to the growing popularity of virtual assistants. At the same time, there is an increasing requirement for an SV system: it should be robust to short speech segments, especially in noisy and reverberant environments. In this paper, we consider one more important requirement for practical applications: the system should be robust to an audio stream containing long non-speech segments, where a voice activity detection (VAD) is not applied. To meet these two requirements, we introduce feature pyramid module (FPM)-based multi-scale aggregation (MSA) and self-adaptive soft VAD (SAS-VAD). We present the FPM-based MSA to deal with short speech segments in noisy and reverberant environments. Also, we use the SAS-VAD to increase the robustness to long non-speech segments. To further improve the robustness to acoustic distortions (i.e., noise and reverberation), we apply a masking-based speech enhancement (SE) method. We combine SV, VAD, and SE models in a unified deep learning framework and jointly train the entire network in an end-to-end manner. To the best of our knowledge, this is the first work combining these three models in a deep learning framework. We conduct experiments on Korean indoor (KID) and VoxCeleb datasets, which are corrupted by noise and reverberation. The results show that the proposed method is effective for SV in the challenging conditions and performs better than the baseline i-vector and deep speaker embedding systems.

Semi-Supervised Multiple Source Localization Using Relative Harmonic Coefficients Under Noisy and Reverberant Environments

This article develops a semi-supervised algorithm to address the challenging multi-source localization problem in a noisy and reverberant environment, using a spherical harmonics domain source feature of the relative harmonic coefficients. We present a comprehensive research of this source feature, including (i) an illustration confirming its sole dependence on the source position, (ii) a feature estimator in the presence of noise, (iii) a feature selector exploiting its inherent directivity over space. Source features at varied spherical harmonic modes, representing unique characterization of the soundfield, are fused by the Multi-Mode Gaussian Process modeling. Based on the unifying model, we then formulate the mapping function revealing the underlying relationship between the source feature(s) and position(s) using a Bayesian inference approach. Another issue of the overlapped components is addressed by a pre-processing technique performing overlapped frame detection, which in turn reduces this challenging problem to a single source localization. It is highlighted that this data-driven method has a strong potential to be implemented in practice because only a limited number of labeled measurements is required. We evaluate this proposed algorithm using simulated recordings between multiple speakers in diverse environments, and extensive results confirm improved performance in comparison with the state-of-art methods. Additional assessments using real-life recordings further prove the effectiveness of the method, even at unfavorable circumstances with severe source overlapping.

Multi-Source DOA Estimation Through Pattern Recognition of the Modal Coherence of a Reverberant Soundfield

We propose a novel multi-source direction of arrival (DOA) estimation technique using a convolutional neural network algorithm which learns the modal coherence patterns of an incident soundfield through measured spherical harmonic coefficients. We train our model for individual time-frequency bins in the short-time Fourier transform spectrum by analyzing the unique snapshot of modal coherence for each desired direction. The proposed method is capable of estimating simultaneously active multiple sound sources on a $3$D space using a single-source training scheme. This single-source training scheme reduces the training time and resource requirements as well as allows the reuse of the same trained model for different multi-source combinations. The method is evaluated against various simulated and practical noisy and reverberant environments with varying acoustic criteria and found to outperform the baseline methods in terms of DOA estimation accuracy. Furthermore, the proposed algorithm allows independent training of azimuth and elevation during a full DOA estimation over $3$D space which significantly improves its training efficiency without affecting the overall estimation accuracy.

Source Localization Using Distributed Microphones in Reverberant Environments Based on Deep Learning and Ray Space Transform

In this article we present a methodology for source localization in reverberant environments from Generalized Cross Correlations (GCCs) computed between spatially distributed individual microphones. Reverberation tends to negatively affect localization based on Time Differences of Arrival (TDOAs), which become inaccurate due to the presence of spurious peaks in the GCC. We therefore adopt a data-driven approach based on a convolutional neural network, which, using the GCCs as input, estimates the source location in two steps. It first computes the Ray Space Transform (RST) from multiple arrays. The RST is a convenient representation of the acoustic rays impinging on the array in a parametric space, called Ray Space. Rays produced by a source are visualized in the RST as patterns, whose position is uniquely related to the source location. The second step consists of estimating the source location through a nonlinear fitting, which estimates the coordinates that best approximate the RST pattern obtained through the first step. It is worth noting that training can be accomplished on simulated data only, thus relaxing the need of actually deploying microphone arrays in the acoustic scene. The localization accuracy of the proposed techniques is similar to the one of SRP-PHAT, however our method demonstrates an increased robustness regarding different distributed microphones configurations. Moreover, the use of the RST as an intermediate representation makes it possible for the network to generalize to data unseen during training.

Successive Relative Transfer Function Identification Using Blind Oblique Projection

Distortionless speech extraction in a reverberant environment can be achieved by applying a beamforming algorithm, provided that the relative transfer functions (RTFs) of the sources and the covariance matrix of the noise are known. In this paper, the challenge of RTF identification in a multi-speaker scenario is addressed. We propose a successive RTF identification (SRI) technique, based on the sole assumption that sources do not become simultaneously active. That is, we address the challenge of estimating the RTF of a specific speech source while assuming that the RTFs of all other active sources in the environment were previously estimated in an earlier stage. The RTF of interest is identified by applying the blind oblique projection (BOP)-SRI technique. When a new speech source is identified, the BOP algorithm is applied. BOP results in a null steering toward the RTF of interest, by means of applying an oblique projection to the microphone measurements. We prove that by artificially increasing the rank of the range of the projection matrix, the RTF of interest can be identified. An experimental study is carried out to evaluate the performance of the BOP-SRI algorithm in various signal to noise ratio (SNR) and signal to interference ratio (SIR) conditions and to demonstrate its effectiveness in speech extraction tasks.

Square Root-Based Multi-Source Early PSD Estimation and Recursive RETF Update in Reverberant Environments by Means of the Orthogonal Procrustes Problem

Multi-channel short-time Fourier transform (STFT) domain-based processing of reverberant microphone signals commonly relies on power-spectral-density (PSD) estimates of early source images, where early refers to reflections contained within the same STFT frame. State-of-the-art approaches to multi-source early PSD estimation, given an estimate of the associated relative early transfer functions (RETFs), conventionally minimize the approximation error defined with respect to the early correlation matrix, requiring non-negative inequality constraints on the PSDs. Instead, we here propose to factorize the early correlation matrix and minimize the approximation error defined with respect to the early-correlation-matrix square root. The proposed minimization problem -- constituting a generalization of the so-called orthogonal Procrustes problem -- seeks a unitary matrix and the square roots of the early PSDs up to an arbitrary complex argument, making non-negative inequality constraints redundant. A solution is obtained iteratively, requiring one singular value decomposition (SVD) per iteration. The estimated unitary matrix and early PSD square roots further allow to recursively update the RETF estimate, which is not inherently possible in the conventional approach. An estimate of the said early-correlation-matrix square root itself is obtained by means of the generalized eigenvalue decomposition (GEVD), where we further propose to restore non-stationarities by desmoothing the generalized eigenvalues in order to compensate for inevitable recursive averaging. Simulation results indicate fast convergence of the proposed multi-source early PSD estimation approach in only one iteration if initialized appropriately, and better performance as compared to the conventional approach.

A Deep Learning Framework for Robust DOA Estimation Using Spherical Harmonic Decomposition

Spherical harmonic decomposition facilitates decomposing the sound pressure at different microphones into independent functions of frequency, azimuth and elevation of the source and microphone locations. This decomposition facilitates the extraction of two sets of features containing different information about elevation and azimuth of the source for the direction of arrival (DOA) estimation. These features can be given as input to a learning approach for the estimation of azimuth and elevation separately. This approach aims at breaking down the problem of DOA estimation into azimuth and elevation estimation separately. An advantage of this is the reduction in computational complexity when compared with the joint DOA estimation. This facilitates a straightforward extension of this approach to denser DOA search grids. The contribution of this paper is threefold. First, we propose spherical harmonic magnitude and phase features and discuss the information present in these features regarding the azimuth and elevation of the source. Second, we propose the convolutional neural network architectures for DOA estimation. Third, we analyse the training, run-time computational complexities and propose to extend the DOA estimation approach to dense DOA search grid rather than restricting to a sparse DOA search grid. The performance of conventional DOA estimation approaches degrades in case of a noisy and reverberant environment. Several advancements to the existing DOA estimation approaches have been recently proposed. However, to the best of the authors’ knowledge, learning approaches to DOA estimation with dense DOA search grids with few frames in the context of spherical arrays have not been proposed. Performance evaluation is carried out using simulated as well as real datasets. The proposed approach is also evaluated on LOCATA dataset in the context of a moving source. The results are motivating enough to consider the application of the proposed method in practical scenarios.

Analytical Geometry Calibration for Acoustic Transceiver Arrays

There are many intelligent devices around us that have the ability to emit and receive audio signals, such as smartphones, smartwatches, and tablets. If they constitute an acoustic transceiver network, it can boost the performance of many audio processing tasks. The speaker localization and tracking algorithms using such a network require the prior information of node positions. To acquire this knowledge, we derive an analytical solution for geometry calibration of acoustic transceiver networks where each node consists of a microphone array and a loudspeaker. Specifically, a linear cost function for node orientations is first established using the delivered direction-of-arrival (DoA) measurements, and its analytical solution is derived. Next, based on the DoA and time-of-arrival (ToA) measurements, another cost function for node positions is formulated, which can also be solved in the closed form. Experimental results show that the proposed method can successfully estimate the geometry structure of acoustic transceiver networks in noisy and reverberant environments. In contrast to most state-of-the-art geometry calibration algorithms, which iteratively solve the calibration problem by optimization methods, the proposed method can decrease the computational complexity greatly.

Diagonal Unloading Beamforming in the Spherical Harmonic Domain for Acoustic Source Localization in Reverberant Environments

Spherical microphone arrays allow the sound field analysis in three dimensions with the advantage of having the same resolution in all directions. By considering the frequency-independent character of the steering vectors in the spherical harmonic (SH) domain, we propose a very low-complexity SH diagonal unloading (DU) beamforming with a novel frequency smoothing power transform (FSPT) of the covariance matrices. We consider the direction of arrival (DOA) estimation problem of acoustic sources in reverberant conditions. The DU beamforming provides high resolution directional response since it exploits the subspace orthogonality property of the covariance matrix by the removal or the attenuation of the signal subspaces, obtained through the subtraction of an opportune diagonal matrix from the covariance matrix. The FSPT aims at smoothing the narrowband covariance matrices of the entire set of frequency domain components, and it pursues this goal by minimizing the narrowband error contributions due to reverberation in the broadband frequency smoothing covariance matrix. We analyze the DOA estimation performance using speech signals with simulations and real acoustic data in reverberant conditions. The results show that the proposed SH-DU-FSPT has a DOA estimation performance comparable to that of high resolution state-of-the-art methods with a significant reduction of the computational cost, since the steering directional responses are computed on the broadband frequency smoothing covariance matrix.

A Parametric Approach to Virtual Miking for Sources of Arbitrary Directivity

In this article we propose a methodology for the reconstruction of sound fields in arbitrary locations based on the signals acquired by a spatial distribution of compact microphone arrays (virtual miking). The proposed method is suitable for operating in reverberant environments, thanks to a two-stage analysis process, the former of which aims at separating the direct and the diffuse components of the sound field. The method that we propose is inherently parametric, as the sources of the acoustic scene are characterized by parameters describing location and directivity (spherical harmonics expansion), which are extracted from the exterior model of the direct component of the sound field. Once the parameters of the sources are extracted, the direct sound field at an arbitrary location is reconstructed. The diffuse component is reconstructed from the joint knowledge of the diffuse component at the locations of the distributed microphone arrays, under the assumption of isotropic behavior. Results show that the proposed technique is able to analyze the sound field and reconstruct the parameters of the sources that are active in the scene. In addition, the synthesis of the signals at the virtual microphone locations turns out to accurately match (in terms of spatial cues) the actual sound field, as measured by a microphone places in the desired location.