Binaural Room Impulse Responses Research Articles

Localization based stereo speech source separation using probabilistic time-frequency masking and deep neural networks

Time-frequency (T-F) masking is an effective method for stereo speech source separation. However, reliable estimation of the T-F mask from sound mixtures is a challenging task, especially when room reverberations are present in the mixtures. In this paper, we propose a new stereo speech separation system where deep neural networks are used to generate soft T-F mask for separation. More specifically, the deep neural network, which is composed of two sparse autoencoders and a softmax regression, is used to estimate the orientations of the dominant source at each T-F unit, based on low-level features, such as mixing vector (MV), interaural level, and phase difference (IPD/ILD). The dataset for training the networks was generated by the convolution of binaural room impulse responses (RIRs) and clean speech signals positioned in different angles with respect to the sensors. With the training dataset, we use unsupervised learning to extract high-level features from low-level features and use supervised learning to find the nonlinear functions between high-level features and the orientations of dominant source. By using the trained networks, the probability that each T-F unit belongs to different sources (target and interferers) can be estimated based on the localization cues which is further used to generate the soft mask for source separation. Experiments based on real binaural RIRs and TIMIT dataset are provided to show the performance of the proposed system for reverberant speech mixtures, as compared with a model-based T-F masking technique proposed recently.

Natural Listening over Headphones in Augmented Reality Using Adaptive Filtering Techniques

Augmented reality (AR), which composes of virtual and real world environments, is becoming one of the major topics of research interest due to the advent of wearable devices. Today, AR is commonly used as assistive display to enhance the perception of reality in education, gaming, navigation, sports, entertainment, simulators, etc. However, most of the past works have mainly concentrated on the visual aspects of AR. Auditory events are one of the essential components in human perceptions in daily life but the augmented reality solutions have been lacking in this regard till now compared to visual aspects. Therefore, there is a need of natural listening in AR systems to give a holistic experience to the user. A new headphones configuration is presented in this work with two pairs of binaural microphones attached to headphones (one internal and one external microphone on each side). This paper focuses on enabling natural listening using open headphones employing adaptive filtering techniques to equalize the headset such that virtual sources are perceived as close as possible to sounds emanating from the physical sources. This would also require a superposition of virtual sources with the physical sound sources, as well as ambience. Modified versions of the filtered-x normalized least mean square algorithm (FxNLMS) are proposed in the paper to converge faster to the optimum solution as compared to the conventional FxNLMS. Measurements are carried out with open structure type headphones to evaluate their performance. Subjective test was conducted using individualized binaural room impulse responses (BRIRs) to evaluate the perceptual similarity between real and virtual sounds.

Interactive room auralization of sound sources with exchangeable directivities

Although established measurement methods and simulation models for source directivities are available, there is a lack of tools to directly experience the audible effects of virtual sound sources in different conditions. Only an auralization in connection with psychoacoustic experiments can answer questions regarding the required solution and quality of sound source directivities. This paper presents a real-time simulation engine based on geometrical acoustics which accounts for both source directivities and the room acoustics of the environment. A hybrid simulation model is applied combining an image source model and a ray tracing algorithm to generate a binaural room impulse response, which is then convolved with an anechoic sound file (e.g., of a musical instrument). A convenient user interface is introduced by integrating the simulation engine into the 3D modeling program SketchUp. This makes it possible to easily position virtual sound sources and select various source directivities inside a modifiable room model. The listener receives immediate perceptual feedback and is able to examine the effects of varying virtual sound source locations and source directivities in different room acoustic environments.

Scalable Multiband Binaural Renderer for MPEG-H 3D Audio

To provide immersive 3D multimedia service, MPEG has launched MPEG-H, ISO/IEC 23008, “High Efficiency Coding and Media Delivery in Heterogeneous Environments.” As part of the audio, MPEG-H 3D Audio has been standardized based on a multichannel loudspeaker configuration (e.g., 22.2). Binaural rendering is a key application of 3D audio; however, previous studies focus on binaural rendering with low complexity such as IIR filter design for HRTF or pre-/post-processing to solve in-head localization or front-back confusion. In this paper, a new binaural rendering algorithm is proposed to support the large number of input channel signals and provide high-quality in terms of timbre, parts of this algorithm were adopted into the MPEG-H 3D Audio. The proposed algorithm truncates binaural room impulse response at mixing time, the transition point from the early-reflections to the late reverberation part. Each part is processed independently by variable order filtering in frequency domain (VOFF) and parametric late reverberation filtering (PLF), respectively. Further, a QMF domain tapped delay line (QTDL) is proposed to reduce complexity in the high-frequency band, based on human auditory perception and codec characteristics. In the proposed algorithm, a scalability scheme is adopted to cover a wide range of applications by adjusting the threshold of mixing time. Experimental results show that the proposed algorithm is able to provide the audio quality of a binaural rendered signal using full-length binaural room impulse responses. A scalability test also shows that the proposed scalability scheme smoothly compromises between audio quality and computational complexity.

Equivalence of plane wave and spherical harmonics rendering of binaural room impulse response

Binaural technology has various applications in virtual acoustics, architectural acoustics, tele-communications, and auditory science. One key element in binaural technology is the binaural room impulse response (BRIR), which represents a continuum of plane waves spatially filtered by head related transfer functions (HRTFs). Such BRIRs can be rendered from spherical microphone array recordings and free-field HRTFs, either in the space domain using plane-wave composition or in the spherical-harmonics domain using order-limited spherical harmonics representation of the sound field. While these approaches have been individually employed in a number of recent studies, it appears that the literature does not offer a comprehensive analysis or a theoretical framework relating the two representations with respect to binaural reproduction and perception. In this paper, we provide a mathematical analysis showing that when certain sampling conditions are maintained, the plane-wave and spherical-harmonics representations are equivalent. Further, we show that under these conditions, resulting binaural signals are independent of the employed spatial sampling schemes. The analysis is complemented by a listening experiment, in which both plane-wave and spherical-harmonics representations are perceptually evaluated for different spatial sampling schemes and spherical harmonic orders.

Just noticeable differences of listener envelopment and apparent source width

Listener envelopment (LEV) and apparent source width (ASW) are attributes which describe the spatial perception of a sound in a room. They are linked to several parameters of physically measurable binaural room impulse responses (BRIR), e.g., the interaural cross-correlation (IACC). Both the LEV and ASW decrease with increasing IACC. For this study the IACC of the BRIR of different rooms was directly manipulated to systematically change the spatial impression of a sound in a room. The BRIRs were convolved with anechoic music signals, and the resulting stimuli were presented to subjects in a psychoacoustic experiment to estimate the just noticeable differences of LEV and ASW. For that purpose, the subjects had to rate the different stimuli for both perceptual attributes in seven steps from “no envelopment” to “completely enveloped” or from “small” to “wide” respectively. In a second step, the stimuli which were rated the same, were used in a paired comparison paradigm to obtain a more precise estimation of the ability to distinguish between similar stimuli. The results of this study are used in a psychoacoustically motivated model to estimate differences in perceived spaciousness of sounds in rooms by the prediction of the perceived envelopment and source width.

The effect of spatial separation in distance on the intelligibility of speech in rooms.

The influence of spatial separation in source distance on speech reception thresholds (SRTs) is investigated. In one scenario, the target was presented at 0.5 m distance, and the masker varied from 0.5 m distance up to 10 m. In a second scenario, the masker was presented at 0.5 m distance and the target distance varied. The stimuli were synthesized using convolution with binaural room impulse responses (BRIRs) measured on a dummy head in a reverberant auditorium, and were equalized to compensate for distance-dependent spectral and intensity changes. All sources were simulated directly in front of the listener. SRTs decreased monotonically when the target was at 0.5 m and the speech-masker was moved further away, resulting in a SRT improvement of up to 10 dB. When the speech masker was at 0.5 m and the target was moved away, a large variation across subjects was observed. Neither short-term signal-to-noise ratio (SNR) improvements nor cross-ear glimpsing could account for the observed improvement in intelligibility. However, the effect might be explained by an improvement in the SNR in the modulation domain and a decrease in informational masking. This study demonstrates that distance-related cues can play a significant role when listening in complex environments.

Auditory/visual distance estimation: accuracy and variability.

Past research has shown that auditory distance estimation improves when listeners are given the opportunity to see all possible sound sources when compared to no visual input. It has also been established that distance estimation is more accurate in vision than in audition. The present study investigates the degree to which auditory distance estimation is improved when matched with a congruent visual stimulus. Virtual sound sources based on binaural room impulse response (BRIR) measurements made from distances ranging from approximately 0.3 to 9.8 m in a concert hall were used as auditory stimuli. Visual stimuli were photographs taken from the participant's perspective at each distance in the impulse response measurement setup presented on a large HDTV monitor. Participants were asked to estimate egocentric distance to the sound source in each of three conditions: auditory only (A), visual only (V), and congruent auditory/visual stimuli (A+V). Each condition was presented within its own block. Sixty-two participants were tested in order to quantify the response variability inherent in auditory distance perception. Distance estimates from both the V and A+V conditions were found to be considerably more accurate and less variable than estimates from the A condition.

Effects of virtual speaker density and room reverberation on spatiotemporal thresholds of audio-visual motion coherence.

The present study examined the effects of spatial sound-source density and reverberation on the spatiotemporal window for audio-visual motion coherence. Three different acoustic stimuli were generated in Virtual Auditory Space: two acoustically “dry” stimuli via the measurement of anechoic head-related impulse responses recorded at either 1° or 5° spatial intervals (Experiment 1), and a reverberant stimulus rendered from binaural room impulse responses recorded at 5° intervals in situ in order to capture reverberant acoustics in addition to head-related cues (Experiment 2). A moving visual stimulus with invariant localization cues was generated by sequentially activating LED's along the same radial path as the virtual auditory motion. Stimuli were presented at 25°/s, 50°/s and 100°/s with a random spatial offset between audition and vision. In a 2AFC task, subjects made a judgment of the leading modality (auditory or visual). No significant differences were observed in the spatial threshold based on the point of subjective equivalence (PSE) or the slope of psychometric functions (β) across all three acoustic conditions. Additionally, both the PSE and β did not significantly differ across velocity, suggesting a fixed spatial window of audio-visual separation. Findings suggest that there was no loss in spatial information accompanying the reduction in spatial cues and reverberation levels tested, and establish a perceptual measure for assessing the veracity of motion generated from discrete locations and in echoic environments.

Relationship Between Chinese Speech Intelligibility and Speech Transmission Index Under Reproduced General Room Conditions

The subjective Chinese (Mandarin) articulation scores of a total of 50 sound conditions, namely at 12 receiver positions in four rooms, were obtained by expert listeners based on in-situ measured binaural room impulse responses (BRIRs) and binaural technology for headphone reproduction from a phonetically balanced test. The relationship between Chinese speech articulation scores and the speech transmission index (STI) according to IEC 60268-16 was established. The spectrum difference between the average spectrum of Chinese and that recommended by IEC may noticeably influence the measurement results with a STI difference of up to 0.054. However, the STI method can evaluate Chinese intelligibility without modulating the correction factor values recommended by IEC; the standard deviation between the articulation scores and the curve was 5.70%, which is relatively small.

Influence of manipulated early reflections on room acoustical perception

The binaural room impulse response (BRIR) can be used to study the perception of room acoustics and consists of different parts such as the direct sound, early and late reflections, and a diffuse reverberant tail. For this study, the contribution of early reflections to several perceptual attributes was investigated. For this purpose, the strength of the early reflections of recorded BRIRs were either increased or decreased. The resulting BRIRs as well as the original BRIRs have then been convolved with anechoic music signals to obtain the stimuli that were presented to the subjects in a psychoacoustic experiment. The subjects had to rate the spatial impression of these manipulated and none-manipulated signals for the perceptual attributes “Listener envelopment,”” Apparent source width,” “Distance” and “Presence.” In addition to determining the influence of manipulated early reflections on the spatial impression of a room, also a comparison was made with the perceptual effect of an artificial increase of the interaural cross-correlation. Results indicate that the perceived source width increases with increasing level of the early reflections. The effect of the level manipulations on the listener envelopment seems to be small compared to the influence of the cross-correlation.

Uncertainty in acoustic metrics due to spatial variations of the non-diffuse sound field measured in a variable acoustics classroom

Acoustic metrics are commonly expressed as single numbers in classroom acoustical designs, often neglecting the physical quantity's uncertainty due to the non-diffuse sound field in the seating area. A database of measured monaural and binaural room impulse responses (RIR) was previously gathered from a fully furnished mock-up classroom. Different wall and ceiling absorption configurations were used to alter the mid-frequency reverberation times (RT) in five scenarios between 0.4 and 1.1 s. The middle three RT scenarios were additionally created from two different material configurations. For each material configuration (eight in total), two furniture orientations were utilized. RIRs were measured at 9 to 10 receiver positions for each material/furniture configuration to document the spatial variation in the resulting sound field. Diffuseness has been calculated for each receiver position utilizing the measured RIRs by following Hanyu's (2013) method using normalized decay-canceled impulse responses. Variations in diffuseness and in the assorted acoustic metrics calculated from the measured RIRs are investigated across different receiver positions. These acoustic metrics, pertinent to classroom acoustical designs, include RT, speech transmission index, clarity, and interaural cross-correlation. Means to quantify uncertainty in these metrics due to spatial variation in the non-diffuse sound field will be discussed.

Interactive Real-Time Simulation and Auralization for Modifiable Rooms

To study the effects of any changes to a room or setting on the room acoustics, a framework was developed that enables immediate acoustic and visual feedback to the user. This is achieved by running interactive room acoustics simulations and auralizations in real-time. Physically based binaural room impulse responses (BRIRs) are calculated using the image source method and ray tracing and are divided into direct sound, early reflections and late reverberation. Any part of the BRIR is updated as quickly as possible depending on the user's interaction with the scene. This includes changes to sources and receivers (positions/orientations/directivities/HRTF), to surface materials and to the room geometry itself. Using streaming low-latency convolution, an immediate feedback is provided to the user. A parallelization concept features multi-threading and networked PC-clusters, so that the workload can be effectively distributed, offering a scalability to simulate small to huge scenes, depending on the available computation power. For convenient scene design and interaction, a plug-in for Trimble SketchUp was developed that enables real-time room acoustics and room acoustics parameter visualization to this easy-to-use CAD modeling tool.

Effect of room acoustics on timbral brightness of clarinet tones: Experimental investigation with two binaural room impulse responses

This study is aimed at investigating how room acoustics affect the timbral brightness of clarinet tones. For semi-anechoic stimuli, nine natural clarinet tones produced at three different dynamic levels and three different notes were used. Reverberant stimuli were generated by convolving each semi-anechoic tone with two different binaural room impulse responses. The stimuli were presented dichotically over headphones to the fifteen participants at equal loudness. The scale values of timbral brightness for each stimulus at equal notes were obtained through Scheffe's paired- comparison test. The results showed that timbral brightness was significantly varied depending on room acoustic conditions. The results also showed that the spectral centroid of tones produced at equal dynamic levels and equal notes was varied depending on room acoustic conditions. The variation range was equivalent to 48-52% of the overall range of that caused by varying both room acoustic conditions and dynamic levels at each note. Timbral brightness was found to linearly increase with the increase in the spectral centroid, irrespective of room acoustics. The slope of timbral brightness corresponding to the spectral centroid was about 1.5-1.6 times steeper for the reverberant tones than for the semi-anechoic tones.

The Contribution of Head Movement to the Externalization and Internalization of Sounds

BackgroundWhen stimuli are presented over headphones, they are typically perceived as internalized; i.e., they appear to emanate from inside the head. Sounds presented in the free-field tend to be externalized, i.e., perceived to be emanating from a source in the world. This phenomenon is frequently attributed to reverberation and to the spectral characteristics of the sounds: those sounds whose spectrum and reverberation matches that of free-field signals arriving at the ear canal tend to be more frequently externalized. Another factor, however, is that the virtual location of signals presented over headphones moves in perfect concert with any movements of the head, whereas the location of free-field signals moves in opposition to head movements. The effects of head movement have not been systematically disentangled from reverberation and/or spectral cues, so we measured the degree to which movements contribute to externalization.Methodology/Principal FindingsWe performed two experiments: 1) Using motion tracking and free-field loudspeaker presentation, we presented signals that moved in their spatial location to match listeners’ head movements. 2) Using motion tracking and binaural room impulse responses, we presented filtered signals over headphones that appeared to remain static relative to the world. The results from experiment 1 showed that free-field signals from the front that move with the head are less likely to be externalized (23%) than those that remain fixed (63%). Experiment 2 showed that virtual signals whose position was fixed relative to the world are more likely to be externalized (65%) than those fixed relative to the head (20%), regardless of the fidelity of the individual impulse responses.Conclusions/SignificanceHead movements play a significant role in the externalization of sound sources. These findings imply tight integration between binaural cues and self motion cues and underscore the importance of self motion for spatial auditory perception.

Auditory and Visual Contribution to Egocentric Distance and Room Size Perception

The perception of rooms involves various unimodal and multimodal aspects on different perceptual levels. Rather abstract yet self-evident aspects are the source distance and the room size. We investigated to what extent the perceived room size and egocentric source distance as supramodal aspects are based on the auditory and the visual modality, i.e. experimentally influenced by the acoustic and optical stimulus. The statistical determination of the respective contributions demands the mutually independent variation of optical and acoustical room properties, usually referred to as conflicting stimulus paradigm. Simulation data of four rooms were collected acoustically by acquiring binaural room impulse responses for different head orientations and optically by acquiring stereoscopic images of the rooms including the electroacoustic sound source. In the laboratory, respective acoustic scenes were played back applying dynamic binaural synthesis, whereas the optical scenes were presented by the use of a stereoscopic display. Test participants were asked to assess the source distance and the room size. Results show main effects of the modalities rather than an interaction effect. It was found that distance perception in rooms is predominantly based on the acoustic stimulus characteristics whereas room size perception predominantly relies on optical information. There is no evidence in favour of the proximity image effect hypothesis. Results do also not confirm an auditory or audio-visual underestimation of the source distance but show a general underestimation of room size and indicate an audio-visual asymmetry regarding the accuracy of room size perception. Maximum accuracy and cross-modal consistency of judgements were distinctively observed in low-absorbent rooms.

Intelligibility assessment in elementary school classrooms from binaural room impulse responses measured with a childlike dummy head

In a classroom, especially for those intended for small children, speech intelligibility is a basic issue for learning efficacy. As it is well known, applying articulation tests (AT’s) is the best way to assess room’s speech intelligibility. However, the technique is expensive and takes a rather long time to be completed. In this paper, a new technique to assess classroom speech intelligibility is presented, which does not require the application of AT’s inside the classroom. The room binaural impulse response, produced by a sweep sine signal and emitted by an artificial speaker (AS), is recorded with a dummy head. Since small children are involved, the binaural recording was performed with a childlike dummy head, with a size corresponding, roughly, to that of a 10years old child. The binaural measurements were also made using an adult dummy head for comparison purposes. The measured binaural impulse responses were convolved with anechoic signals containing the enunciation of monosyllables. The background noise measured in the room is then added and the result is convolved again with a headphone impulse response, generating an acoustical virtual reality – the auralizations. The resulting binaural signals are applied to subjects of the same age in virtual AT’s. To validate the technique, actual AT’s using the same anechoic lists of monosyllables emitted by the AS inside the classroom were performed. The technique was applied in three schools to students of different level classrooms. The results show that the proposed technique presents good reliability to assess classroom intelligibility when the measurements are made with the childlike dummy head. For virtual AT’s obtained with the adult dummy head, however, it was verified that the children ability to recognize the monosyllables was much lower than that with the childlike dummy head. This is a consequence of the pronounced differences between the head related impulse responses for an adult and a child, resulting in distinct binaural impulse responses.

Perceptual limits for detecting interaural-cue manipulations measured in reverberant settings

In this study, binaural room impulse responses (BRIRs) were manipulated to determine the just noticeable differences in the interaural time delay (ITD), interaural level difference (ILD), and interaural cross-correlation (ICC) in reverberant settings. The BRIR were split in two sections, the first 75–150 ms of the BRIR was found to be direction dependent, and for this first part either an extra ITD or ILD was applied. These manipulations were expected to change the perceived direction of the sound source. Changes in the ICC were applied to the remaining part of the BRIR, which was expected to change the overall spatial impression, but not the perceived location. Each of these three differently manipulated BRIRs was convolved with an anechoic musical instrument, and the just noticeable change in ITD, ILD, or ICC was determined in a listening experiment. Due to the convolution with a temporally varying musical instrument stimulus, a complex spectrotemporal pattern of binaural cues is created. An analysis of these cues will be presented and it will be compared to the listening test results. This analysis will be based on a model of human auditory processing, which predicts perceptual cues related to room acoustic perception.

Modeling binaural receivers in finite difference simulation of room acoustics

Binaural room impulse responses are important for auralisation as well as for objective research in room acoustics. In geometrical room simulation methods, obtaining such responses is easily achieved by convolving each computed reflection tap with a corresponding pre-measured angle-dependent head-related impulse response. Unfortunately, employing such an approach in wave based methods is challenging due to temporal overlap of room reflections in the calculated response. One alternative is to physically embed a listener geometry in the grid. Whilst this method is straightforward, it requires voxelization of a geometrically-complex object. Furthermore, with non-conformal boundary conditions, the voxelized geometry is sample-rate dependent, meaning that numerical consistency may become compromised. In this paper we discuss the merits and drawbacks of embedding different listener geometries in the grid, ranging from a simple rigid sphere to a fully featured laser-scan of a Kemar mannikin. Results are analysed in terms of generated auditory cues, and are discussed in light of room acoustics modelling using the FDTD method.

Release from masking through spatial separation in distance in hearing impaired listeners

It is widely accepted that speech intelligibility improves as a speech signal and interfering masker are separated spatially in azimuth. In a previous study [Westermann et al. IHCON (2012)] a similarly strong improvement was found for normal hearing (NH) listeners when target and masker are separated in distance. In this study speech reception thresholds (SRTs) were measured for 16 hearing impaired (HI) listeners using the Listening in Spatialized Noise-Sentences Test (LiSN-S) and the Coordinate Response Measure (CRM). Acoustic scenarios were auralized via headphones using binaural room impulse responses recorded in an auditorium. In the first scenario, the target was presented at a distance of 0.5 m from the center of the listener’s head and the interferer at a distance of 0.5 m or 10 m. In a second setup, the interferer’s location was fixed and the target’s location was varied. HI listeners showed a substantial release from masking as target and interferer were separated in distance. This effect was consistent for both LiSN-S and CRM, but less pronounced than for NH listeners. This study suggests that distance related cues play a significant role when listening in complex environments and are also to some extent available to HI listeners.