Simulated Room Impulse Responses Research Articles

Effective Acoustic Model-Based Beamforming Training for Static and Dynamic Hri Applications.

Human-robot collaboration will play an important role in the fourth industrial revolution in applications related to hostile environments, mining, industry, forestry, education, natural disaster and defense. Effective collaboration requires robots to understand human intentions and tasks, which involves advanced user profiling. Voice-based communication, rich in complex information, is key to this. Beamforming, a technology that enhances speech signals, can help robots extract semantic, emotional, or health-related information from speech. This paper describes the implementation of a system that provides substantially improved signal-to-noise ratio (SNR) and speech recognition accuracy to a moving robotic platform for use in human-robot interaction (HRI) applications in static and dynamic contexts. This study focuses on training deep learning-based beamformers using acoustic model-based multi-style training with measured room impulse responses (RIRs). The results show that this approach outperforms training with simulated RIRs or matched measured RIRs, especially in dynamic conditions involving robot motion. The findings suggest that training with a broad range of measured RIRs is sufficient for effective HRI in various environments, making additional data recording or augmentation unnecessary. This research demonstrates that deep learning-based beamforming can significantly improve HRI performance, particularly in challenging acoustic environments, surpassing traditional beamforming methods.

A cloud-based simulation framework for high-fidelity room impulse response dataset generation: case study, validation and computational performance

Increased cost-effective computational resources in the cloud in combination with state-of-the-art simulation algorithms has opened doors for generating large quantities of simulated high-fidelity room impulse responses. Such datasets can, e.g., be used for training machine learning based audio signal processing algorithms such as speech enhancement or dereverberation, as well as for carrying out virtual prototyping of audio devices in different acoustic conditions. The authors have previously presented a cloud-based simulation engine that is well suited for such large-scale dataset generation tasks. The simulation engine leverages a hybrid wave/geometrical solver, in combination with various geometry pre-processing technology and spatial audio post-processing technology. In this paper, we present a dataset generation case study using this simulation engine, as well as a novel validation study into certain aspects of the spatial audio post-processing. Finally, we present an analysis into computational performance when leveraging multi-node computational architectures to run extremely large wave-based simulations using the simulation framework. To the best of our knowledge, this is the first time a multi-node wave-based acoustics solver presented, enabling wave-based simulations with billions of degrees of freedom.

Differences between measured and simulated room impulse responses

Simulation models based on geometrical acoustics mostly do not immediately deliver a simulated room impulse response, but intermediate results such as an energy histogram or an energy decay curve. At this point, further models are required to generate a room impulse response of the simulated environment, which are essential for the process of auralization. While for various simulated scenarios, the application of a reflection model based on a basic theory is sufficient, detailed comparisons of simulated and measured room impulse responses reveal differences which are attributed to the lack of diffuse reflections in the applied reflection model. These deviations can also substantially affect derived room acoustic parameters. This work presents and explains the shortcomings and discusses potential improvements to be considered in the simulation process.

Fast processing models effects of reflections on binaural unmasking

Sound reflections and late reverberation alter energetic and binaural cues of a target source, thereby affecting its detection in noise. Two experiments investigated detection of harmonic complex tones, centered around 500 Hz, in noise, in a virtual room with different modifications of simulated room impulse responses (RIRs). Stimuli were auralized using the Simulated Open Field Environment’s (SOFE’s) loudspeakers in anechoic space. The target was presented from the front (0°) or 60° azimuth, while an anechoic noise masker was simultaneously presented at 0°. In the first experiment, early reflections were progressively added to the RIR and detection thresholds of the reverberant target were measured. For a frontal sound source, detection thresholds decreased while adding early reflections within the first 45 ms, whereas for a lateral sound source, thresholds remained constant. In the second experiment, early reflections were removed while late reflections were kept along with the direct sound. Results for a target at 0° show that even reflections as late as 150 ms reduce detection thresholds compared to only the direct sound. A binaural model with a sluggishness component following the computation of binaural unmasking in short windows predicts measured and literature results better than when large windows are used.

Sound Source Localization Using a Convolutional Neural Network and Regression Model.

In this research, a novel sound source localization model is introduced that integrates a convolutional neural network with a regression model (CNN-R) to estimate the sound source angle and distance based on the acoustic characteristics of the interaural phase difference (IPD). The IPD features of the sound signal are firstly extracted from time-frequency domain by short-time Fourier transform (STFT). Then, the IPD features map is fed to the CNN-R model as an image for sound source localization. The Pyroomacoustics platform and the multichannel impulse response database (MIRD) are used to generate both simulated and real room impulse response (RIR) datasets. The experimental results show that an average accuracy of 98.96% and 98.31% are achieved by the proposed CNN-R for angle and distance estimations in the simulation scenario at SNR = 30 dB and RT60 = 0.16 s, respectively. Moreover, in the real environment, the average accuracies of the angle and distance estimations are 99.85% and 99.38% at SNR = 30 dB and RT60 = 0.16 s, respectively. The performance obtained in both scenarios is superior to that of existing models, indicating the potential of the proposed CNN-R model for real-life applications.

The effect of low frequency reverberation on Chinese speech intelligibility in two classrooms

The previous study had investigated the students’ speech perceptions in two primary school classrooms with similar reverberation time and found that the speech intelligibility in a classroom was better than that in other one. It is speculated that due to difference in the improvement in low frequency reverberation. In this study, the reverberation time and speech transmission index were obtained and compared from these two classrooms by simulated. The subjective Chinese speech intelligibility was evaluated through simulated binaural room impulse responses. The effect of low frequency reverberation on Chinese speech intelligibility in two classrooms was discussed. The result shows that improving the low frequency characteristic of reverberation time can be helpful to improve the Chinese speech intelligibility in classroom.

Investigation on Chinese speech reception threshold of the elderly in noise and reverberation

The elderly is more vulnerable to the detrimental effects of noise and reverberation. In the study, five simulated binaural room impulse responses with different reverberation time (RT) were convolved with test speech signals and background noise. Speech reception threshold (SRT) test was conducted on 22 elderly with different hearing levels (HLs) under different RT and background noise level (BNL) conditions. The results show that the SRT increases with the increase of age and deterioration of HL. The elderly have higher SRT than young adults. There is a high linear correlation between SRT and RT in different age groups and HL groups. Under 40 dBA and 55 dBA BNL conditions, the slope of linear fitting relationship between SRT and RT increases with the increase of age, and increases with the increase of pure tone average auditory. It also shows that the difference between different groups increases with the increase of RT. Under the same RT condition, the SRT in 40 dBA BNL is higher than that in 55 dBA BNL. However, the actual needed speech sound pressure level in 40 dBA BNL to obtain the same speech intelligibility is lower than that in 55 dBA BNL. It is more beneficial to improve the speech intelligibility of the elderly under a lower BNL condition with a sufficient sound pressure level. The architects should take reasonable acoustical measures to reduce the BNL and RT to improve the speech communication of the elderly in rooms.

Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation.

We propose multi-microphone complex spectral mapping, a simple way of applying deep learning for time-varying non-linear beamforming, for speaker separation in reverberant conditions. We aim at both speaker separation and dereverberation. Our study first investigates offline utterance-wise speaker separation and then extends to block-online continuous speech separation (CSS). Assuming a fixed array geometry between training and testing, we train deep neural networks (DNN) to predict the real and imaginary (RI) components of target speech at a reference microphone from the RI components of multiple microphones. We then integrate multi-microphone complex spectral mapping with minimum variance distortionless response (MVDR) beamforming and post-filtering to further improve separation, and combine it with frame-level speaker counting for block-online CSS. Although our system is trained on simulated room impulse responses (RIR) based on a fixed number of microphones arranged in a given geometry, it generalizes well to a real array with the same geometry. State-of-the-art separation performance is obtained on the simulated two-talker SMS-WSJ corpus and the real-recorded LibriCSS dataset.

Adding air absorption to simulated room acoustic models.

Room acoustics models are often used in auditory research. The simplest implementations of these models ignore air absorption, and this lack can affect the room response, especially at high frequencies and for large simulated spaces. This letter proposes a method for adding air absorption to a simulated room impulse response calculated without air absorption. The procedure creates a time-varying lowpass filter that approximates air absorption as a function of distance. The existing absorption-free room response is passed through the filter to create a new response that incorporates air absorption effects.

Multi-Modal Multi-Channel Target Speech Separation

Target speech separation refers to extracting a target speaker's voice from an overlapped audio of simultaneous talkers. Previously the use of visual modality for target speech separation has demonstrated great potentials. This work proposes a general multi-modal framework for target speech separation by utilizing all the available information of the target speaker, including his/her spatial location, voice characteristics and lip movements. Also, under this framework, we investigate on the fusion methods for multi-modal joint modeling. A factorized attention-based fusion method is proposed to aggregate the high-level semantic information of multi-modalities at embedding level. This method firstly factorizes the mixture audio into a set of acoustic subspaces, then leverages the target's information from other modalities to enhance these subspace acoustic embeddings with a learnable attention scheme. To validate the robustness of proposed multi-modal separation model in practical scenarios, the system was evaluated under the condition that one of the modalities is temporarily missing, invalid or corrupted. Experiments are conducted on a large-scale audio-visual dataset collected from YouTube (to be released) that spatialized by simulated room impulse responses (RIRs). Experiment results illustrate that our proposed multi-modal framework significantly outperforms single-modal and bi-modal speech separation approaches, while can still support real-time processing.

Computer Modeling of Barrel-Vaulted Sanctuary Exhibiting Flutter Echo with Comparison to Measurements

Computer modeling in acoustics allows for the prediction of acoustical defects and the evaluation of potential remediations. In this article, computer modeling is applied to the case of a barrel-vaulted sanctuary whose architectural design and construction led to severe flutter echoes along the main aisle, which was later mitigated through acoustical remediations. State-of-the-art geometrical acoustics and wave-based simulations are carried out to analyze the acoustics of this space, with a particular focus on the flutter echoes along the main aisle, before and after remediations. Multi-resolution wavelet and spectrogram analyses are carried out to isolate and characterize flutter echoes within measurements and computer-simulated room impulse responses. Comparisons of simulated responses to measurements are also made in terms of decay times and curves. Simulated room impulse responses from both geometrical acoustics and wave-based methods show evidence of flutter echoes matching measurements, to varying degrees. Time-frequency analyses isolating flutter echoes demonstrate better matches to measurements from wave-based simulated responses, at the cost of longer simulation times than geometrical acoustics simulations. This case study highlights the importance of computer modeling of acoustics in early design phases of architectural planning of worship spaces.

The Acoustic Environment of York Minster’s Chapter House

York Minster is the largest medieval Gothic cathedral in Northern Europe, renowned for its magnificent architecture and its stained glass windows. Both acoustic measurements and simulation techniques have been used to analyse the acoustic environment of its Chapter House, which dates from the 13th-century and features an octagonal geometry with Gothic Decorated stone walls replete of geometric patterns and enormous stained glass windows, covered by a decorated wooden vault. Measured and simulated room impulse responses served to better understand how their architectural features work together to create its highly reverberant acoustic field. The authors start by analysing its acoustic characteristics in relation to its original purpose as a meeting place of the cathedral’s Chapter, and end by reflecting on its modern use for a variety of cultural events, such as concerts and exhibitions. This work is part of the “Cathedral Acoustics” project, funded by the EC through the Marie-Sklodowska-Curie scheme.

Deep Learning Based Target Cancellation for Speech Dereverberation.

This study investigates deep learning based single- and multi-channel speech dereverberation. For single-channel processing, we extend magnitude-domain masking and mapping based dereverberation to complex-domain mapping, where deep neural networks (DNNs) are trained to predict the real and imaginary (RI) components of the direct-path signal from reverberant (and noisy) ones. For multi-channel processing, we first compute a minimum variance distortionless response (MVDR) beamformer to cancel the direct-path signal, and then feed the RI components of the cancelled signal, which is expected to be a filtered version of non-target signals, as additional features to perform dereverberation. Trained on a large dataset of simulated room impulse responses, our models show excellent speech dereverberation and recognition performance on the test set of the REVERB challenge, consistently better than single- and multi-channel weighted prediction error (WPE) algorithms.

Three-dimensional reflector localisation and room geometry estimation using a spherical microphone array.

The analysis of room impulse responses to localise reflecting surfaces and estimate room geometry is applicable in numerous aspects of acoustics, including source localisation, acoustic simulation, spatial audio, audio forensics, and room acoustic treatment. Geometry inference is an acoustic analysis problem where information about reflections extracted from impulse responses are used to localise reflective boundaries present in an environment, and thus estimate the geometry of the room. This problem, however, becomes more complex when considering non-convex rooms, as room shape cannot be constrained to a subset of possible convex polygons. This paper presents a geometry inference method for localising reflective boundaries and inferring the room's geometry for convex and non-convex room shapes. The method is tested using simulated room impulse responses for six scenarios, and real-world room impulse responses measured in a cuboid-shaped room, using a spherical microphone array containing multiple spatially distributed channels capable of capturing both time- and direction-of-arrival. Results show that the general shape of the rooms is inferred for each case, with a higher degree of accuracy for convex shaped rooms. However, inaccuracies generally arise as a result of the complexity of the room being inferred, or inaccurate estimation of time- and direction-of-arrival of reflections.

A Novel Scheme for Single-Channel Speech Dereverberation

This paper presents a novel scheme for speech dereverberation. The core of our method is a two-stage single-channel speech enhancement scheme. Degraded speech obtains a sparser representation of the linear prediction residual in the first stage of our proposed scheme by applying orthogonal matching pursuit on overcomplete bases, trained by the K-SVD algorithm. Our method includes an estimation of reverberation and mixing time from a recorded hand clap or a simulated room impulse response, which are used to create a time-domain envelope. Late reverberation is suppressed at the second stage by estimating its energy from the previous envelope and removed with spectral subtraction. Further speech enhancement is applied on minimizing the background noise, based on optimal smoothing and minimum statistics. Experimental results indicate favorable quality, compared to two state-of-the-art methods, especially in real reverberant environments with increased reverberation and background noise.

Speech Dereverberation With Context-Aware Recurrent Neural Networks

In this paper, we propose a model to perform speech dereverberation by estimating its spectral magnitude from the reverberant counterpart. Our models are capable of extracting features that take into account both short- and long-term dependencies in the signal through a convolutional encoder (which extracts features from a short, bounded context of frames) and a recurrent neural network for extracting long-term information. Our model outperforms a recently proposed model that uses different context information depending on the reverberation time, without requiring any sort of additional input, yielding improvements of up to 0.4 on perceptual evaluation of speech quality, 0.3 on short-time objective intelligibility, and 1.0 on perceptual objective listening quality assessment relative to reverberant speech. We also show our model is able to generalize to real room impulse responses even when only trained with simulated room impulse responses, different speakers, and high reverberation times. Finally, listening tests show the proposed method outperforming benchmark models in reduction of perceived reverberation.

Supervised single-channel speech dereverberation and denoising using a two-stage model based sparse representation

In many acoustic conditions, a single-channel recorded speech signal may be severely affected by reverberation and noise, leading to a reduced speech quality and intelligibility. This paper focuses on proposing a novel two-stage model scheme by decomposing room impulse responses (RIRs) into two convolution parts for single-channel speech dereverberation and denoising. Similar as previous methods, the proposed two-stage model uses non-negative approximations of the convolutive transfer function (NCTF) to simultaneously estimate the magnitude spectrograms of the speech and the RIR. It focuses on iteratively updating model parameters to estimate a less reverberant speech signal and a short RIR at first stage, then the clean speech signal and the other short RIR are estimated by iteratively renewing at the second stage. There are always denosing processing steps existing in both stages to denoise more thoroughly. A straightforward method based on the scheme is built to enhance the speech from the noisy reverberant signal, then two fusion methods inspired by ensemble learning are proposed for speech enhancement. The advantages of our proposed methods are more capable to enhance the speech and more time-saving through decomposing the long RIRs into two shorter ones. Additionally, the optimal estimator is derived based on temporal stacking to utilize speech temporal dynamics. Experiments are performed on two simulated RIRs and a real RIR to compare the performances of the proposed methods with a state-of-the-art method and the results show that the proposed methods have achieved either better or comparable performances in most measures but phone error rate.

Time-Frequency Masking in the Complex Domain for Speech Dereverberation and Denoising.

In real-world situations, speech is masked by both background noise and reverberation, which negatively affect perceptual quality and intelligibility. In this paper, we address monaural speech separation in reverberant and noisy environments. We perform dereverberation and denoising using supervised learning with a deep neural network. Specifically, we enhance the magnitude and phase by performing separation with an estimate of the complex ideal ratio mask. We define the complex ideal ratio mask so that direct speech results after the mask is applied to reverberant and noisy speech. Our approach is evaluated using simulated and real room impulse responses, and with background noises. The proposed approach improves objective speech quality and intelligibility significantly. Evaluations and comparisons show that it outperforms related methods in many reverberant and noisy environments.

Study of the perception of warmth in concert halls and correlation with room acoustics metrics

Challenges in concert hall acoustics research occur in bridging the domain between subjective attributes and objective metrics. Several studies have shown that the primary attributes associated with concert hall acoustic preference ratings are strength, clarity, and timbre. The purpose of this study was to investigate the correlation of warmth, which is closely associated with timbre, with the objective metric of bass ratio (BR), which is a measure of the ratio of acoustic information (such as reverberation time or strength) at low-frequencies to that at mid-frequencies. The stimuli for this study were simulated binaural room impulse responses (BRIRs) obtained in ODEON and convolved with a concise orchestral anechoic motif. The BRIRs were simulated using models of five European halls. A model of the Vienna Musikverein with multiple absorption settings was also used to achieve different values of BR within a single hall geometry. The stimuli were presented to participants over headphones and they were asked to rate warmth, low- and high-frequency reverberance on continuous five-point scales. Participants rated each of the attributes in separate sets to avoid potential bias in the ratings. A repeated measures analysis of variance was conducted to examine the relationship between the subjective ratings and BR.

A Single-Channel Non-Intrusive C50 Estimator Correlated With Speech Recognition Performance

Several intrusive measures of reverberation can be computed from measured and simulated room impulse responses, over the full frequency band or for each individual mel-frequency subband. It is initially shown that full-band clarity index ${\rm{C}}_{50}$ is the most correlated measure on average with reverberant speech recognition performance. This corroborates previous findings but now for the dataset to be used in this study. We extend the previous findings to show that ${\rm{C}}_{50}$ also exhibits the highest mutual information on average. Motivated by these extended findings, a nonintrusive room acoustic (NIRA) estimation method is proposed to estimate ${\rm{C}}_{50}$ from only the reverberant speech signal. The NIRA method is a data-driven approach based on computing a number of features from the speech signal and it employs these features to train a model used to perform the estimation. The choice of features and learning techniques are explored in this work using an evaluation set which comprises approximately 100 000 different reverberant signals (around 93 h of speech) including reverberation from measured and simulated room impulse responses. The feature importance of each feature with respect to the estimation of the target ${\rm{C}}_{50}$ is analysed following two different approaches. In both cases, the newly chosen set of features shows high importance for the target. The best ${\rm{C}}_{50}$ estimator provides a root-mean-square deviation around 3 dB on average for all reverberant test environments.