Binaural Synthesis Research Articles

Effect of discretization on off-axis listeners in binaural synthesis based on the optimal source distribution

3D sound reproduction based on binaural synthesis over loudspeakers is one of the most important techniques in immersive spatial audio. The system inversion involved in the binaural synthesis gives rise to several problems such as dynamic range loss, distortion of the reproduction transfer function, and a lack of robustness to room reflections. The principle of Optimal Source Distribution was proposed to overcome these problems. This utilizes a pair of conceptual monopole transducers whose azimuthal location varies continuously as a function of frequency. Another potential of the principle is that it can provide listening positions for multiple off-axis listeners, in addition to the on-axis listener whose inverse matrix is given, since the phase interference state is aligned at all frequencies. However, these continuously varying transducers are too conceptual to realize in practice. It is necessary to provide discretized sound sources with appropriate frequency ranges according to the principle. In this paper, the effect of discretization of the principle on off-axis listeners' performance is numerically investigated to reveal how those off-axis sweet spots are formed as the discretization becomes finer. An experiment is also carried out using an actual discretized system and the effect of the head-related transfer functions is considered.

Validation of sound source localization quality in binaural audio synthesis with head-tracking

Binaural synthesis with head tracking has become a highly important tool in creating an auditory experience for users that is as similar as possible to their perception of sound in the real world. This is crucial in many different applications that rely on accurate representation of real or virtual sound environments, ranging from scientific research to gaming applications. The quality of the binaural synthesis process and the accuracy and naturalness of the produced result depend on numerous factors, ranging from the selection of suitable IMU sensor technology used for head tracking to the choice of methods for implementing binaural synthesis using generic or individual sets of Head-Related Transfer Functions (HRTF). This is especially significant in Virtual Reality (VR) systems with three or six degrees of freedom that fully immerse the users in the virtual world through visual and auditory stimuli. This paper presents the authors' experiences in addressing a variety of challenges that arose during the design of such spatial audio systems and the testing of sound source localization quality. Strong emphasis is put on the testing of sound source localization quality, where the position of the source dynamically changes in relation to the listener.

Investigating the after effect of head-tracked movement on externalization for varying sound sources in binaural synthesis.

Binaural audio reproduction aims to create a realistic auditory experience for the listener through headphones. However, front and rear sound sources are often internalized (perceived inside the head) rather than externalized (perceived outside the head). Previous studies have shown that large head movements can significantly improve externalization, and that this improvement can persist even after head movements have stopped. This study examined how head-tracked movements influence the perception of sound externalization depending on the nature of the sound sources. The stimuli consisted of a first ``head movement" part during which the participants had to either perform a head movement or stay still, followed by a "rating" part where they evaluated externalization without moving. The sound sources during the ``rating" part could be similar, slightly different, or very different from the source during the head movement part. Results demonstrated that head-tracked movements effectively heightened externalization perception for all types of sources immediately after the movement. The type of source (female voice, male voice, or pink noise) did not affect perceived externalization. These findings suggest that the externalization enhancement induced by head-tracked movement persists after the movement ends even when encountering a different sound source.

Capturing differences in perception and aesthetic judgment of live or medially presented music: development of a self-report instrument.

Nowadays there are multiple ways to perceive music, from attending concerts (live) to listening to recorded music through headphones (medial). In between there are many mixed modes, such as playback performances. In empirical music research, this plurality of performance forms has so far found little recognition. Until now no measuring instrument has existed that could adequately capture the differences in perception and aesthetic judgment. The purpose of our empirical investigation was to capture all dimensions relevant to such an assessment. Using 3D-simulations and dynamic binaural synthesis, various live and medial situations were simulated. A qualitative survey was conducted at the Department of Audio Communication of the Technical University of Berlin (TU Berlin). With the help of the repertory grid technique, a data pool of approximately 400 attribute pairs was created and individual rating data were collected. Our first study served to create a semantic differential. In a second study, this semantic differential was evaluated. The development of the semantic differential was carried out by first using a mixed-method approach to qualitative analysis according to grounded theory. Thereafter, a principal component analysis reduced the attribute pairs to 67 items in four components. The semantic differential consists of items concerning acoustic, visual and audio-visual interaction as well as items with an overarching assessment of the stimuli. The evaluation study, comprising 45 participants (23 male and 22 female, M = 42.56 years, SD = 17.16) who rated 12 stimuli each, reduced the items to 61 and resulted in 18 subscales and nine single items. Because the survey used simulations, the social component may be underrepresented. Nevertheless, the questionnaire we created enables the evaluation of music performances (especially for classical concerts) in a new scope, thus opening many opportunities for further research. For example, in a live concert context, we observed not only that seating position influences the judgment of sound quality but also that visual elements influence immersion and felt affect. In the future, the differential could be reviewed for a larger stimulus pool, extended or used modularly for different research questions.

Sound field recording using distributed spherical microphone arrays based on a virtual spherical model

Binaural synthesis helps provide rich spatial information to users. Spherical microphone arrays and head-related transfer functions significantly aid in realizing the aforementioned approach. In a typical case, a spherical microphone array is set at the recording point, and listeners experience the sound binaurally as if they are at the recording point. However, in real situations, setting a microphone array at the recording point is not always feasible. In this study, we investigate binaural synthesis using an Auditory Display based on VIrtual SpherE (ADVISE) model with distributed spherical microphone arrays. In the proposed ADVISE-based method, the virtual sphere is set around the target point, and the sound pressure on the boundary of the virtual sphere is calculated based on recorded signals using spherical microphone arrays. The simulation results indicate that the proposed method can robustly synthesize sound fields even when the target point is far from the spherical microphone arrays.

Shaping the auditory peripersonal space with motor planning in immersive virtual reality

AbstractImmersive audio technologies require personalized binaural synthesis through headphones to provide perceptually plausible virtual and augmented reality (VR/AR) simulations. We introduce and apply for the first time in VR contexts the quantitative measure called premotor reaction time (pmRT) for characterizing sonic interactions between humans and the technology through motor planning. In the proposed basic virtual acoustic scenario, listeners are asked to react to a virtual sound approaching from different directions and stopping at different distances within their peripersonal space (PPS). PPS is highly sensitive to embodied and environmentally situated interactions, anticipating the motor system activation for a prompt preparation for action. Since immersive VR applications benefit from spatial interactions, modeling the PPS around the listeners is crucial to reveal individual behaviors and performances. Our methodology centered around the pmRT is able to provide a compact description and approximation of the spatiotemporal PPS processing and boundaries around the head by replicating several well-known neurophysiological phenomena related to PPS, such as auditory asymmetry, front/back calibration and confusion, and ellipsoidal action fields.

A real-time auralization framework for scientific experiments

The software Virtual Acoustics (VA) is an open-source auralization framework developed at the Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany. Using physics-based models for source emission and sound propagation, it allows plausible rendering of virtual scenes based on purely synthetic data. Its modular design provides a variety of rendering modules using different assumptions for the sound propagation. This allows the rendering of various indoor and outdoor scenarios with adjustable degree of complexity. Based on a similarly modular concept, VA allows a flexible way of reproducing the sound via headphones or loudspeaker arrays. For this purpose, the signals can be spatialized using binaural synthesis, higher-order ambisonics or VBAP. The respective scenarios, including source and receiver movement, can be controlled using the C + +, C#, Matlab, Python programming languages. Additionally, there are interfaces for the Unreal and Unity game engines allowing the combination with sophisticated visual cues, e.g., using HMDs. The software’s flexible structure and configuration possibilities enable the controlled application in listening experiments while providing an adequate level of plausibility. This work discusses the options as well as the limitations of the software and presents application examples in scientific experiments.

Significance of Binaural Synthesis on the Spatial Acoustic Impression of Enclosures

Listening tests have been carried out to quantify the significance of binaural auralization over monaural auralization in accordance with the acoustic properties of the enclosure. To this end, acoustic rendering of three different rooms were generated based on synthesized monaural (two channels with the same audio material) and binaural room impulse responses. The auralizations were evaluated by means of subjective tests using headphones with non-individualized equalization. Parameters, such as localization, spatial impression and realism, were taken into consideration to determine the relevance of providing binaural information for the auralization of a given room. The analysis of the data has been conducted following a statistical approach based on ANOVA and Pearson correlation. The results indicate that spatial perception is strongly dependent on the acoustic characteristics of the rooms and on the listening condition of the audio material. Furthermore, as expected, advantages of binaural rendering in terms of source localization was also confirmed.

Playing With Others Using Headphones: Musicians Prefer Binaural Audio With Head Tracking Over Stereo

Immersive listening systems have grown significantly over the past decade and are now an established area of scientific, artistic, and industrial research. However, scarce research has been conducted on musicians' preferences for playing through headphones over binaural spatialization systems with the addition of head tracking, as opposed to classical stereophonic systems. This comparison is essential to optimally support the playing experience with others for cases of remote collaborative playing, individual instrumental practice, individual recreational music-making using backing tracks, and studio recording sessions. In this article, we study the preferences of playing musicians for a stereophonic system versus a binaural head-tracking system composed of Ambisonics technology and binaural synthesis with generalized head-related transfer functions. We conducted two experiments, each with 30 expert musicians, where participants were asked to rate and compare the 2 listening conditions while playing their instrument either seated or standing. Overall, the quantitative and qualitative results indicated a generalized preference for the binaural system with head tracking over the stereophonic system, with higher ratings for localization, immersion, social presence, realism, and connection with other musicians. Moreover, participants moved their heads significantly more in the binaural conditions. This phenomenon may be explained by the higher engagement and arousal due to the improved auditory experience, or alternatively by the presence of embodied music cognition mechanisms that cause a higher degree of exploration to better understand the action–perception loop. These findings highlight the need for progressing current commercial hardware and software systems used by musicians while playing over headphones.

The Accuracy of Dynamic Sound Source Localization and Recognition Ability of Individual Head-Related Transfer Functions in Binaural Audio Systems with Head Tracking

The use of audio systems that employ binaural synthesis with head tracking has become increasingly popular, particularly in virtual reality gaming systems. The binaural synthesis process uses the Head-Related Transfer Functions (HRTF) as an input required to assign the directions of arrival to sounds coming from virtual sound sources in the created virtual environments. Generic HRTFs are often used for this purpose to accommodate all potential listeners. The hypothesis of the research is that the use of individual HRTF in binaural synthesis instead of generic HRTF leads to improved accuracy and quality of virtual sound source localization, thus enhancing the user experience. A novel methodology is proposed that involves the use of dynamic virtual sound sources. In the experiments, the test participants were asked to determine the direction of a dynamic virtual sound source in both the horizontal and vertical planes using both generic and individual HRTFs. The gathered data are statistically analyzed, and the accuracy of localization is assessed with respect to the type of HRTF used. The individual HRTFs of the test participants are measured using a novel and efficient method that is accessible to a broad range of users.

Virtual sound source perception challenges of binaural audio systems with head-tracking

With the recent leaps in spatial audio technology, the use of binaural head-tracking for spatial audio can revolutionize the way how music and audio are experienced. Moreover, to research noise-related perception in laboratories, binaural head-tracking is frequently used as an audio reproduction system. Although this technology is getting significantly better in recent years, signal processing of binaural audio is often experiencing problems such as non-adequate sound source externalization and unacceptably high values of system response time since it influences the usability of the technology at fast head rotation. In this paper, the results of an experiment are presented, in which test subjects are determining the direction of a virtual sound source in the horizontal plane with multiple parameter changes. The experiment is done in a controlled environment in a listening room with known acoustical properties. Parameter variation includes: hardware head-tracker variation (commercial one and head-tracker based on simple embedded system), various software solutions for real-time binaural synthesis, and variation in the used Head-Related Transfer Functions. The problem of externalization of a virtual sound source via headphones is discussed, and possible solutions to the problem are given. Results of the experiment and recorded data are presented.

The Performance of Inertial Measurement Unit Sensors on Various Hardware Platforms for Binaural Head-Tracking Applications

Binaural synthesis with head tracking is often used in spatial audio systems. The devices used for head tracking must provide data on the orientation of the listener's head. These data need to be highly accurate, and they need to be provided as fast and as frequently as possible. Therefore, head-tracking devices need to be equipped with high-quality inertial measurement unit (IMU) sensors. Since IMUs readily include triaxial accelerometers, gyroscopes, and magnetometers, it is crucial that all of these sensors perform well, as the head orientation is calculated from all sensor outputs. This paper discusses the challenges encountered in the process of the performance assessment of IMUs through appropriate measurements. Three distinct hardware platforms were investigated: five IMU sensors either connected to Arduino-based embedded systems or being an integral part of one, five smartphones across a broad range of overall quality with integrated IMUs, and a commercial virtual reality unit that utilizes a headset with integrated IMUs. An innovative measurement method is presented and proposed for comparing the performance of sensors on all three platforms. The results of the measurements performed using the proposed method show that all three investigated platforms are adequate for the acquisition of the data required for calculating the orientation of a device as the input to the binaural synthesis process. Some limitations that have been observed during the measurements, regarding data acquisition and transfer, are discussed.

Points2Sound: from mono to binaural audio using 3D point cloud scenes

For immersive applications, the generation of binaural sound that matches its visual counterpart is crucial to bring meaningful experiences to people in a virtual environment. Recent studies have shown the possibility of using neural networks for synthesizing binaural audio from mono audio by using 2D visual information as guidance. Extending this approach by guiding the audio with 3D visual information and operating in the waveform domain may allow for a more accurate auralization of a virtual audio scene. We propose Points2Sound, a multi-modal deep learning model which generates a binaural version from mono audio using 3D point cloud scenes. Specifically, Points2Sound consists of a vision network and an audio network. The vision network uses 3D sparse convolutions to extract a visual feature from the point cloud scene. Then, the visual feature conditions the audio network, which operates in the waveform domain, to synthesize the binaural version. Results show that 3D visual information can successfully guide multi-modal deep learning models for the task of binaural synthesis. We also investigate how 3D point cloud attributes, learning objectives, different reverberant conditions, and several types of mono mixture signals affect the binaural audio synthesis performance of Points2Sound for the different numbers of sound sources present in the scene.

Human Head Yaw Estimation Based on Two 3-Axis Accelerometers

The estimation of the orientation of an object, and a human head in particular, can be defined by the Euler angles: the yaw, pitch and roll. The robust and drift-free estimation of those angles is usually achieved with the data from several sensors such as accelerometers, gyroscopes and magnetometers, processed with sensor fusion algorithms. However, wearable devices such as hearing instruments are rarely equipped with all those sensors, and might usually only have a single accelerometer embedded per device. While it is possible to retrieve a correct estimation of the roll and pitch using only a single accelerometer, estimating the yaw is a more challenging task as accelerations around the gravity vector cannot be detected by an accelerometer. In the context of binaural communication devices and spatial hearing, the yaw of the head is a key information that helps achieving dynamic binaural synthesis. This work proposes an algorithm that aims at estimating the yaw of a human head by using only two 3-axis accelerometers that are placed at each side of the head. The algorithm is evaluated with acceleration measurements of human subjects achieving a realistic task. The results suggest that the strategy used is promising as a good performance can be achieved for a majority of the measures as long as an individualized set of parameters can be selected. The results also suggest that the performance of the current implementation is sensitive to sensor displacement after calibration or a wrong estimation during initialization.

Perceptual evaluation of approaches for binaural reproduction of non-spherical microphone array signals

Microphone arrays consisting of sensors mounted on the surface of a rigid, spherical scatterer are popular tools for the capture and binaural reproduction of spatial sound scenes. However, microphone arrays with a perfectly spherical body and uniformly distributed microphones are often impractical for the consumer sector, in which microphone arrays are generally mounted on mobile and wearable devices of arbitrary geometries. Therefore, the binaural reproduction of sound fields captured with arbitrarily shaped microphone arrays has become an important field of research. In this work, we present a comparison of methods for the binaural reproduction of sound fields captured with non-spherical microphone arrays. First, we evaluated equatorial microphone arrays (EMAs), where the microphones are distributed on an equatorial contour of a rigid, spherical 1. Second, we evaluated a microphone array with six microphones mounted on a pair of glasses. Using these two arrays, we conducted two listening experiments comparing four rendering methods based on acoustic scenes captured in different rooms2. The evaluation includes a microphone-based stereo approach (sAB stereo), a beamforming-based stereo approach (sXY stereo), beamforming-based binaural reproduction (BFBR), and BFBR with binaural signal matching (BSM). Additionally, the perceptual evaluation included binaural Ambisonics renderings, which were based on measurements with spherical microphone arrays. In the EMA experiment we included a fourth-order Ambisonics rendering, while in the glasses array experiment we included a second-order Ambisonics rendering. In both listening experiments in which participants compared all approaches with a dummy head recording we applied non-head-tracked binaural synthesis, with sound sources only in the horizontal plane. The perceived differences were rated separately for the attributes timbre and spaciousness. Results suggest that most approaches perform similarly to the Ambisonics rendering. Overall, BSM, and microphone-based stereo were rated the best for EMAs, and BFBR and microphone-based stereo for the glasses array.

An Algorithm for Generating Virtual Sources in Dynamic Virtual Auditory Display Based on Tensor Decomposition of Head-Related Impulse Responses

Dynamic virtual auditory displays (VADs) are increasingly used for generating various auditory objects and scenes in virtual and augmented reality. Dynamic VADs are required to generate virtual sources in various directions and distances by using HRTF- or HRIR-based binaural synthesis. In the present work, an algorithm for improving the efficiency and performance of binaural synthesis in dynamic VAD is proposed. Based on tensor decomposition, a full set of near-field HRIRs is decomposed as a combination of distance-, direction-, and time-related modes. Then, binaural synthesis in VAD can be implemented by a common set of time mode-related convolvers or filters associated with direction- and distance-related weights. Dynamic binaural signals are created by updating the weights rather than updating the HRIR-based convolvers, which enables the independent control of virtual source distance and direction and avoids the audible artifact caused by updating the HRIR-based convolvers. An example of implementation indicates that a set of eight common convolvers or filters for each ear is enough to synthesize the binaural signals with sufficient accuracy. The computational efficiency of simultaneously generating multiple virtual sources is improved when the number of virtual sources is larger than eight. A virtual-source localization experiment validates the algorithm.

Spatial auditory presentation of a partner\u2019s presence induces the social Simon effect

Social presence is crucial for smooth communications in virtual reality (VR). Current telecommunication systems rarely submit spatial auditory information originating from remote people. However, such information may enhance social presence in VR. In this study, we constructed a dynamic binaural synthesis system and investigated the effect of spatial auditory information of a remote partner on a participant’s behavior using the social Simon effect (SSE). The SSE is a spatial stimulus–response compatibility effect between two persons. The SSE occurs when one perceives that their partner is present. Several studies have confirmed the SSE in actual environments. We presented partner sounds diotically (i.e., without spatial information) to one group or binaurally (i.e., with spatial information) to another group through headphones without providing visual information about the partner. The results showed that the SSE was induced only in the binaural group in the current auditory VR (Experiment 1), whereas both groups exhibited the SSE in an actual environment (Experiment 2). These results suggest that the auditory spatial information of remote people is sufficient to induce the SSE and has a potential to enhance social presence.

Binaural reproduction of dummy head and spherical microphone array data-A perceptual study on the minimum required spatial resolution.

Dynamic binaural synthesis requires binaural room impulse responses (BRIRs) for each head orientation of the listener. Such BRIRs can either be measured with a dummy head or calculated from the spherical microphone array (SMA) data. Because the dense dummy head measurements require enormous effort, alternatively sparse measurements can be performed and then interpolated in the spherical harmonics domain. The real-world SMAs, on the other hand, have a limited number of microphones, resulting in spatial undersampling artifacts. For both of the methods, the spatial order N of the underlying sampling grid influences the reproduction quality. This paper presents two listening experiments to determine the minimum spatial order for the direct sound, early reflections, and reverberation of the dummy head or SMA measurements required to generate the horizontally head-tracked binaural synthesis perceptually indistinguishable from a high-resolution reference. The results indicate that for direct sound, N = 9-13 is required for the dummy head BRIRs, but significantly higher orders of N = 17-20 are required for the SMA BRIRs. Furthermore, significantly lower orders are required for the late parts with N = 4-5 for the early reflections and reverberation of the dummy head BRIRs but N = 12-13 for the early reflections and N = 6-9 for the reverberation of the SMA BRIRs.

The Availability of a Hidden Real Reference Affects the Plausibility of Position-Dynamic Auditory AR

This study examines the plausibility of Auditory Augmented Reality (AAR) realized with position-dynamic binaural synthesis over headphones. An established method to evaluate the plausibility of AAR asks participants to decide whether they are listening to the virtual or real version of the sound object. To date, this method has only been used to evaluate AAR systems for seated listeners. The AAR realization examined in this study instead allows listeners to turn to arbitrary directions and walk towards, past, and away from a real loudspeaker that reproduced sound only virtually. The experiment was conducted in two parts. In the first part, the subjects were asked whether they are listening to the real or the virtual version, not knowing that it was always the virtual version. In the second part, the real versions of the scenes where the loudspeaker actually reproduced sound were added. Two different source positions, three different test stimuli, and two different sound levels were considered. Seventeen volunteers, including five experts, participated. In the first part, none of the participants noticed that the virtual reproduction was active throughout the different test scenes. The inexperienced listeners tended to accept the virtual reproduction as real, while experts distributed their answers approximately equally. In the second part, experts could identify the virtual version quite reliably. For inexperienced listeners, the individual results varied enormously. Since the presence of the headphones influences the perception of the real sound field, this shadowing effect had to be considered in the creation of the virtual sound source as well. This requirement still limits test methods considering the real version in its ecological validity. Although the results indicate that the availability of a hidden real reference leads to a more critical evaluation, it is crucial to be aware that the presence of the headphones slightly distorts the reference. This issue seems more vital to the plausibility estimates achieved with this evaluation method than the increased freedom in motion.

Precision of inertial measurement unit sensors in head-tracking systems used for binaural synthesis

Binaural synthesis is the most used sound system for diverse virtual and augmented reality systems nowadays, given its simplicity of implementation and the need of using only two audio channels. It is widely used in computer games, auralization and even audio production. To achieve the most natural sound field recreation, systems used for binaural synthesis must include a head-tracking sensor to dynamically calculate the binaural signal for the head orientation at any given moment. This is done by inertial measurement unit (IMU) sensors, specifically the triaxial accelerometers, gyroscopes, and magnetometers. Simpler systems, e.g., Arduino or other embedded systems, provide only raw sensor data, and the orientation is calculated by a processing unit. Other, more complex systems such as smartphones or VR headsets already calculate their position from the raw data using complex sensor integration algorithms. In this paper, a measurement procedure for measuring the precision of IMU sensors is presented. The need of absolute orientation calibration is addressed, and challenges of sensor data drift are discussed. Measurement results for simple embedded systems and complex systems found in smartphones are presented, and an estimation of IMU sensor quality for binaural synthesis is made.