Virtual Auditory Environment Research Articles

Spatial release of masking in children and adults in non-individualized virtual environments.

The spatial release of masking (SRM) is often measured in virtual auditory environments created from head-related transfer functions (HRTFs) of a standardized adult head. Adults and children, however, differ in head dimensions and mismatched HRTFs are known to affect some aspects of binaural hearing. So far, there has been little research on HRTFs in children and it is unclear whether a large mismatch of spatial cues can degrade speech perception in complex environments. In two studies, the effect of non-individualized virtual environments on SRM accuracy in adults and children was examined. The SRMs were measured in virtual environments created from individual and non-individualized HRTFs and the equivalent real anechoic environment. Speech reception thresholds (SRTs) were measured for frontal target sentences and symmetrical speech maskers at 0° or ±90° azimuth. No significant difference between environments was observed for adults. In 7 to 12-year-old children, SRTs and SRMs improved with age, with SRMs approaching adult levels. SRTs differed slightly between environments and were significantly worse in a virtual environment based on HRTFs from a spherical head. Adult HRTFs seem sufficient to accurately measure SRTs in children even in complex listening conditions.

Head tracker using webcam for auralization

Binaural rendering is a technique that seeks to generate virtual auditory environments that replicate the natural listening experience, including the three-dimensional perception of spatialized sound sources. As such, real-time knowledge of the listener's position, or more specifically, their head and ear orientations allow the transfer of movement from the real world to virtual spaces, which consequently enables a richer immersion and interaction with the virtual scene. This study presents the use of a simple laptop integrated camera (webcam) as a head tracker sensor, disregarding the necessity to mount any hardware to the listener's head. The software was built on top of a state-of-the-art face landmark detection model, from Google's MediaPipe library for Python. Manipulations to the coordinate system are performed, in order to translate the origin from the camera to the center of the subject's head and adequately extract rotation matrices and Euler angles. Low-latency communication is enabled via User Datagram Protocol (UDP), allowing the head tracker to run in parallel and asynchronous with the main application. Empirical experiments have demonstrated reasonable accuracy and quick response, indicating suitability to real-time applications that do not necessarily require methodical precision.

Using a linear microphone array to evaluate inhomogeneous sound-fields of recreated enclosures

Auditory Virtual Environments (AVE) typically recreate the sound at a specific listener position. Extended loudspeaker-based systems typically use the rich capabilities to enlarge the sweet spot, so a broader audience can experience the sound from the same virtual listener position. An alternative approach is taken with Rensselaer's Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab). Using 128 horizontally arranged loudspeakers, inhomogeneous sound fields are produced over a user area of 10 × 12 sqm, recreating the sound of acoustical enclosures using a ray-tracing method. A linear microphone array, consisting of 40 omnidirectional microphones, spaced 15 cm apart, is used to measure the fidelity of the reproduced inhomogeneous sound fields. This is accomplished by comparing the spatial room impulse responses measured in the CRAIVE-Lab to those generated synthetically (or computationally). The arrival times of direct sound and reflections are compared for both simulation methods. The analysis focuses on the spatial properties of the arriving wave fronts, including the direction of arrival and curvature, to determine the degree of accuracy to which the inhomogeneous properties of the sound field are reproduced. [Work supported by NSF #1909229.]

Stowaway City: An Immersive Audio Experience for Multiple Tracked Listeners in a Hybrid Listening Environment

Stowaway City is an immersive audio experience that combines electroacoustic composition and storytelling with extended reality. The piece was designed to accommodate multiple listeners in a shared auditory virtual environment. Each listener, based on their tracked position and rotation in space, wirelessly receives an individual binaurally decoded sonic perspective via open-back headphones. The sounds and unfolding narrative are mapped to physical locations in the performance area, which are only revealed through exploration and physical movement. Spatial audio is simultaneously presented to all listeners via a spherical loudspeaker array that supplements the headphone audio, thus forming a hybrid listening environment. The work is presented as a conceptual and technical design paradigm for creative sonic application of the technology in this medium. The author outlines a set of strategies that were used to realize the composition and technical affordances of the system.

A Computational Model to Implement Binaural Synthesis in a Hard Real-Time Auditory Virtual Environment

There is a growing interest in the development and the evaluation of real-time auditory virtual environments (AVE). The implementation of this type of simulation system in general purpose computers is a still a challenge, and there are few studies that evaluated the perceived quality of synthetized sounds of simulated acoustic scenes. To evoke in the listener a correct image of the modeling space, the system must be dynamic and interactive. That is, it must respond to the changes in the acoustic scenario produced by the listener movement, in a perceptually acceptable time and with an update rate that guarantees continuity in the reproduction of sound events. Hard real-time systems ensure that a given task runs within a given time interval, providing deterministic behavior for applications with time restrictions. In the current article, a computational model to implement binaural synthesis in a hard real-time AVE is presented and evaluated. The computer model was implemented in an open-source auralization system. Measurements and real-time simulations on a university classroom were carried out to perform a reverberation time parameters validation and a system performance evaluation. Also, measured and simulated binaural soundtracks (composed from anechoic stimuli) were compared in terms of three selected perceptual attributes for subjective evaluations of static positions. The results showed that real-time performance was acceptable according to values previously reported in the literature and that computer prediction errors for the measured parameters were within the subjective difference limens. The computational model was able to generate an AVE with an acceptable overall perceptual quality.

Polyphonic pitch perception in rooms using deep learning networks with data rendered in auditory virtual environments

This paper proposes methods for generation and implementation of uniform, large-scale data from auralized MIDI music files for use with deep learning networks for polyphonic pitch perception and impulse response recognition. This includes synthesis and sound source separation of large batches of multitrack MIDI files in non-real time, convolution with artificial binaural room impulse responses, and techniques for neural network training. Using ChucK, individual tracks for each MIDI file, containing the ground truth for pitch and other parameters, are processed concurrently with variable Synthesis ToolKit (STK) instruments, and the audio output is written to separate wave files in order to create multiple incoherent sound sources. Then, each track is convolved with a measured or synthetic impulse response that corresponds to the virtual position of the instrument in the room before all tracks are digitally summed. The database now contains the symbolic description in the form of MIDI commands and the auralized music performances. A polyphonic pitch model based on an array of autocorrelation functions for individual frequency bands is used to train a neural network and analyze the data [Work supported by IBM AIRC grant and NSF BCS-1539276.]

The impact of reverberation on speech intelligibility in cochlear implant recipients

Listening to speech in an environment with reverberation can be challenging for both the normal and impaired auditory system. However, it has been shown for both normal- and impaired-hearing listeners that it is the late reflections that are responsible for degrading intelligibility, whereas early reflections actually aid intelligibility by increasing the effective signal-to-noise ratio. Contrastingly, studies conducted with cochlear implant (CI) recipients have suggested that CI recipients have almost no tolerance for reverberation and that they are negatively impacted by both early and late reflections. The main objective of the current study is to re-evaluate the influence of reverberation on speech intelligibility in CI recipients using more authentic virtual auditory environments. Unlike previous studies in this area, this study was conducted using a loudspeaker-based auralization system rather than non-individualized binaural room simulations. Speech intelligibility was measured in simulations of a range of actual physical rooms with plausible source-receiver distances, both with and without late reflections. The results show that the effect of reverberation is much smaller than previously suggested, especially with short source-receiver distances. Furthermore, the results suggest that, in contrast to previous literature, early reflections may not actually be detrimental to CI recipients.

Perceptual thresholds of spatial audio update latency in virtual auditory and audiovisual environments

When observers in a virtual sound environment are in motion relative to a source, the virtual-auditory display must rapidly track the users head position and update the location-cueing acoustic filters—known as head-related transfer functions (HRTFs)—in order to accurately reflect the source’s location relative to the current head orientation and position. The end-to-end spatial audio system latency (SASL) is the time elapsed between the listener assuming a position and sound being delivered to the ears with an HRTF corresponding to that same position. To maintain the stability and plausibility of a virtual sound “object,” the SASL must lie below a perceptual temporal threshold. The current study sought to rigorously probe the threshold of SASL detectability in human observers at three different rotational head velocities and at various source locations. These conditions were repeated in an audiovisual experiment in which the sound stimulus was now accompanied by a zero-latency visual stimulus. Thus, we characterized SASL thresholds in environments where cross-modal interactions occur due to visual information accompanying sound as is often the case in virtual reality (VR) applications. Initial results reveal a rich interaction between the contextual factors of the virtual environment and listeners’ sensitivity to SASL.

Multimodal Perceptual Training for Improving Spatial Auditory Performance in Blind and Sighted Listeners

Abstract The use of individualised Head Related Transfer Functions (HRTF) is a fundamental prerequisite for obtaining an accurate rendering of 3D spatialised sounds in virtual auditory environments. The HRTFs are transfer functions that define the acoustical basis of auditory perception of a sound source in space and are frequently used in virtual auditory displays to simulate free-field listening conditions. However, they depend on the anatomical characteristics of the human body and significantly vary among individuals, so that the use of the same dataset of HRTFs for all the users of a designed system will not offer the same level of auditory performance. This paper presents an alternative approach to the use on non-individualised HRTFs that is based on a procedural learning, training, and adaptation to altered auditory cues.We tested the sound localisation performance of nine sighted and visually impaired people, before and after a series of perceptual (auditory, visual, and haptic) feedback based training sessions. The results demonstrated that our subjects significantly improved their spatial hearing under altered listening conditions (such as the presentation of 3D binaural sounds synthesised from non-individualized HRTFs), the improvement being reflected into a higher localisation accuracy and a lower rate of front-back confusion errors.

A cross-cultural comparison of salient perceptual characteristics of height channels for a virtual auditory environment

Perceptual characteristics of virtual auditory environments from three listener groups were compared. To generate convincing and pleasing virtual auditory environments, acoustic impulse responses were measured in two venues using an innovative microphone array and convolved with two anechoic recordings. Subsequently, the convolved sound sources were assigned to loudspeakers (five horizontal channels and four height channels), and inter-channel level balances were optimized. The authors conducted a controlled listening test with two variables: height-channel configurations (eight conditions) and stimuli (four conditions--two musical selections times and two target venues) to determine the influence of (1) two control variables on the perceived appropriateness of virtual auditory environments and (2) the cultural background of three listener groups composed of participants from Canada (group 1, 11 subjects), the USA (group 2, 12 subjects), and Japan (group 3, 14 subjects). The data analysis revealed that the configuration variable (the height position of the loudspeakers) has a greater influence on perceived appropriateness than the stimulus variable for all three groups. In addition, the results showed that although group 1 data had a similar listening response pattern to group 2, the response of group 3 was different. A subsequent analysis of reported descriptors found that groups 1 and 2 chose height configurations that generated a "frontal" and "narrow" impression as a more appropriate virtual auditory environment, while group 3 chose the same characteristics but as a less appropriate environment. Groups 1 and 2 also described a less appropriate auditory environment with "wide, spacious, and surrounding" images that again were described by group 3 as more appropriate. While room acoustics and loudspeaker size also contributed to the overall modulation of listeners' judgment, the findings support the idea that cultural background affects perceptual responses to spatial sound and is therefore important in rendering a homogeneous experience of a virtual auditory environment for listeners in remote spaces.

The Effects of Spatialized Sounds on the Sense of Presence in Auditory Virtual Environments: A Psychological and Physiological Study

Although many studies have indicated that spatialized sounds increase the subjective sense of presence in virtual environments, few studies have examined the effects of sounds objectively. In this study, we examined whether three-dimensional reproduced sounds increase the sense of presence in auditory virtual environments by using physiological and psychological measures. We presented the sounds of people approaching the listener through a three-dimensional reproduction system using 96 loudspeakers. There were two spatial sound conditions, spatialized and non-spatialized, which had different spatial accuracy of the reproduction. The experimental results showed that presence ratings for spatialized sounds were greater than for non-spatialized sounds. Further, the results of the physiological measures showed that the sympathetic nervous system was activated to a greater extent by the spatialized sounds compared with the non-spatialized sounds, and the responses to the three-dimensional reproduced sounds were similar to those that occur during intrusions into personal space in the real world. Additionally, a correlation was found between the psychological and the physiological responses in the spatialized sound condition. These results suggest that the physiological measures correlate to the perceived presence in acoustic environments.

Emotion modulates activity in the ‘what’ but not ‘where’ auditory processing pathway

Auditory cortices can be separated into dissociable processing pathways similar to those observed in the visual domain. Emotional stimuli elicit enhanced neural activation within sensory cortices when compared to neutral stimuli. This effect is particularly notable in the ventral visual stream. Little is known, however, about how emotion interacts with dorsal processing streams, and essentially nothing is known about the impact of emotion on auditory stimulus localization. In the current study, we used fMRI in concert with individualized auditory virtual environments to investigate the effect of emotion during an auditory stimulus localization task. Surprisingly, participants were significantly slower to localize emotional relative to neutral sounds. A separate localizer scan was performed to isolate neural regions sensitive to stimulus location independent of emotion. When applied to the main experimental task, a significant main effect of location, but not emotion, was found in this ROI. A whole-brain analysis of the data revealed that posterior-medial regions of auditory cortex were modulated by sound location; however, additional anterior-lateral areas of auditory cortex demonstrated enhanced neural activity to emotional compared to neutral stimuli. The latter region resembled areas described in dual pathway models of auditory processing as the ‘what’ processing stream, prompting a follow-up task to generate an identity-sensitive ROI (the ‘what’ pathway) independent of location and emotion. Within this region, significant main effects of location and emotion were identified, as well as a significant interaction. These results suggest that emotion modulates activity in the ‘what,’ but not the ‘where,’ auditory processing pathway.

Simplification of head-related impulse response in early reflection simulation

In virtual auditory environment, early reflections are usually simulated by the image-source method, and binaural signals are synthesized by convolving the input stimulus with corresponding head-related impulse responses (HRIRs). Considering the limited resolution of the human hearing, this work investigates the minimal length of HRIRs needed in early reflection simulation via a simple model consisting of a single direct sound and a reflection. The direct sound is synthesized using 512-point HRIR and fixed at the position directly in front of the subject, while the reflection is synthesized by HRIRs at various directions with four time-domain lengths (512, 256, 128, and 64 points, at a sampling frequency of 44.1 kHz) as well as five time delays relative to the direct sound (from 10 to 50 ms at intervals of 10 ms). A three-interval, two-alternative forced-choice paradigm is employed in this work. Results indicate that for most spatial directions the HRIR with a length of 64-point is perceptually adequate in early reflection simulation. [Work supported by State Key Laboratory of Subtropical Building Science, South China University of Technology, Grant No. 2013KB23.]

Platform for dynamic virtual auditory environment real-time rendering system

This paper reports the recent works and progress on a PC and C++ language-based virtual auditory environment (VAE) system platform. By tracing the temporary location and orientation of listener’s head and dynamically simulating the acoustic propagation from sound source to two ears, the system is capable of recreating free-field virtual sources at various directions and distances as well as auditory perception in reflective environment via headphone presentation. Schemes for improving VAE performance, including PCA-based (principal components analysis) near-field virtual source synthesis, simulating six degrees of freedom of head movement, are proposed. Especially, the PCA-based scheme greatly reduces the computational cost of multiple virtual sources synthesis. Test demonstrates that the system exhibits improved performances as compared with some existing systems. It is able to simultaneously render up to 280 virtual sources using conventional scheme, and 4500 virtual sources using the PCA-based scheme. A set of psychoacoustic experiments also validate the performance of the system, and at the same time, provide some preliminary results on the research of binaural hearing. The functions of the VAE system is being extended and the system serves as a flexible and powerful platform for future binaural hearing researches and virtual reality applications.

Platform for virtual auditory environment real time rendering system

By dynamically synthesizing binaural signals in free-field and reflective environment, a virtual auditory environment (VAE) real time rendering system recreates realistic auditory events or perceptions for listeners. VAE systems have been applied in various fields, such as the research of binaural hearing, multimedia and virtual reality, among others. In present work, a PC and C++ language-based VAE system is designed and implemented. Schemes for improving the performances of the system, including multiple virtual source synthesis, auditory distance perception, dynamic information simulation for multiple degrees of freedom of listener, as well as dynamic characters of the system, are proposed. The results from measurement indicate that the system is capable of synthesizing 280 virtual sound sources (including free-field sources and image sources for reflections) simultaneously in conventional working mode, or 4500 virtual sources in proposed PCA (principal components analysis) working mode. The update rate is 120 Hz, and the system latency is 25.4ms. A set of psychoacoustic experiments also validate the performance of the system. The function extension of VAE can serve as a flexible and powerful platform for binaural and virtual reality research.

Rapid head-related transfer function adaptation using a virtual auditory environment

The paper reports on the ability of people to rapidly adapt in localizing virtual sound sources in both azimuth and elevation when listening to sounds synthesized using non-individualized head-related transfer functions (HRTFs). Participants were placed within an audio-kinesthetic Virtual Auditory Environment (VAE) platform that allows association of the physical position of a virtual sound source with an alternate set of acoustic spectral cues through the use of a tracked physical ball manipulated by the subject. This set-up offers a natural perception-action coupling, which is not limited to the visual field of view. The experiment consisted of three sessions: an initial localization test to evaluate participants' performance, an adaptation session, and a subsequent localization test. A reference control group was included using individual measured HRTFs. Results show significant improvement in localization performance. Relative to the control group, participants using non-individual HRTFs reduced localization errors in elevation by 10° with three sessions of 12 min. No significant improvement was found for azimuthal errors or for single session adaptation.

Auditory localization in realistic environments by normal-hearing and hearing-impaired listeners

The ability to correctly localize sounds is important for general awareness of the auditory scene and communication in adverse acoustic conditions. However, most localization studies are performed in rather simple and artificial conditions. In particular, very few studies have considered localization in reverberant environments or in the presence of complex interferers, and no studies have systematically investigated the effect of distance. In the present study, localization performance was therefore measured as a function of source-receiver distance using a virtual auditory environment. With increasing source-receiver distance the direct-to-reverberation energy ratio decreases and the auditory system increasingly relies on mechanisms related to the precedence effect. Both aspects may be particularly problematic for hearing-impaired listeners. The acoustics of a cafeteria was simulated with the room acoustic software ODEON for a large number of source-receiver locations. Signals were generated for a 3D ar...

Distance Perception in Interactive Virtual Acoustic Environments using First and Higher Order Ambisonic Sound Fields

In this paper, we present an investigation into the perception of source distance in interactive virtual auditory environments in the context of First (FOA) and Higher Order Ambisonic (HOA) reproduction. In particular, we investigate the accuracy of sound field reproduction over virtual loudspeakers (headphone reproduction) with increasing Ambisonic order. Performance of 1st, 2nd and 3rd order Ambisonics in representing distance cues is assessed in subjective audio perception tests. Results demonstrate that 1st order sound fields can be sufficient in representing distance cues for Ambisonic-to-binaural decodes.

Rapid Auditory System Adaptation Using a Virtual Auditory Environment

Various studies have highlighted plasticity of the auditory system from visual stimuli, limiting the trained field of perception. The aim of the present study is to investigate auditory system adaptation using an audio-kinesthetic platform. Participants were placed in a Virtual Auditory Environment allowing the association of the physical position of a virtual sound source with an alternate set of acoustic spectral cues or Head-Related Transfer Function (HRTF) through the use of a tracked ball manipulated by the subject. This set-up has the advantage to be not being limited to the visual field while also offering a natural perception-action coupling through the constant awareness of one's hand position. Adaptation process to non-individualized HRTF was realized through a spatial search game application. A total of 25 subjects participated, consisting of subjects presented with modified cues using non-individualized HRTF and a control group using individual measured HRTFs to account for any learning effect...

The influence of spectral characteristics of early reflections on speech intelligibility

The auditory system takes advantage of early reflections (ERs) in a room by integrating them with the direct sound (DS) and thereby increasing the effective speech level. In the present paper the benefit from realistic ERs on speech intelligibility in diffuse speech-shaped noise was investigated for normal-hearing and hearing-impaired listeners. Monaural and binaural speech intelligibility tests were performed in a virtual auditory environment where the spectral characteristics of ERs from a simulated room could be preserved. The useful ER energy was derived from the speech intelligibility results and the efficiency of the ERs was determined as the ratio of the useful ER energy to the total ER energy. Even though ER energy contributed to speech intelligibility, DS energy was always more efficient, leading to better speech intelligibility for both groups of listeners. The efficiency loss for the ERs was mainly ascribed to their altered spectrum compared to the DS and to the filtering by the torso, head, and pinna. No binaural processing other than a binaural summation effect could be observed.