Dynamic Binaural Synthesis Research Articles

Capturing differences in perception and aesthetic judgment of live or medially presented music: development of a self-report instrument.

Nowadays there are multiple ways to perceive music, from attending concerts (live) to listening to recorded music through headphones (medial). In between there are many mixed modes, such as playback performances. In empirical music research, this plurality of performance forms has so far found little recognition. Until now no measuring instrument has existed that could adequately capture the differences in perception and aesthetic judgment. The purpose of our empirical investigation was to capture all dimensions relevant to such an assessment. Using 3D-simulations and dynamic binaural synthesis, various live and medial situations were simulated. A qualitative survey was conducted at the Department of Audio Communication of the Technical University of Berlin (TU Berlin). With the help of the repertory grid technique, a data pool of approximately 400 attribute pairs was created and individual rating data were collected. Our first study served to create a semantic differential. In a second study, this semantic differential was evaluated. The development of the semantic differential was carried out by first using a mixed-method approach to qualitative analysis according to grounded theory. Thereafter, a principal component analysis reduced the attribute pairs to 67 items in four components. The semantic differential consists of items concerning acoustic, visual and audio-visual interaction as well as items with an overarching assessment of the stimuli. The evaluation study, comprising 45 participants (23 male and 22 female, M = 42.56 years, SD = 17.16) who rated 12 stimuli each, reduced the items to 61 and resulted in 18 subscales and nine single items. Because the survey used simulations, the social component may be underrepresented. Nevertheless, the questionnaire we created enables the evaluation of music performances (especially for classical concerts) in a new scope, thus opening many opportunities for further research. For example, in a live concert context, we observed not only that seating position influences the judgment of sound quality but also that visual elements influence immersion and felt affect. In the future, the differential could be reviewed for a larger stimulus pool, extended or used modularly for different research questions.

Human Head Yaw Estimation Based on Two 3-Axis Accelerometers

The estimation of the orientation of an object, and a human head in particular, can be defined by the Euler angles: the yaw, pitch and roll. The robust and drift-free estimation of those angles is usually achieved with the data from several sensors such as accelerometers, gyroscopes and magnetometers, processed with sensor fusion algorithms. However, wearable devices such as hearing instruments are rarely equipped with all those sensors, and might usually only have a single accelerometer embedded per device. While it is possible to retrieve a correct estimation of the roll and pitch using only a single accelerometer, estimating the yaw is a more challenging task as accelerations around the gravity vector cannot be detected by an accelerometer. In the context of binaural communication devices and spatial hearing, the yaw of the head is a key information that helps achieving dynamic binaural synthesis. This work proposes an algorithm that aims at estimating the yaw of a human head by using only two 3-axis accelerometers that are placed at each side of the head. The algorithm is evaluated with acceleration measurements of human subjects achieving a realistic task. The results suggest that the strategy used is promising as a good performance can be achieved for a majority of the measures as long as an individualized set of parameters can be selected. The results also suggest that the performance of the current implementation is sensitive to sensor displacement after calibration or a wrong estimation during initialization.

Spatial auditory presentation of a partner\u2019s presence induces the social Simon effect

Social presence is crucial for smooth communications in virtual reality (VR). Current telecommunication systems rarely submit spatial auditory information originating from remote people. However, such information may enhance social presence in VR. In this study, we constructed a dynamic binaural synthesis system and investigated the effect of spatial auditory information of a remote partner on a participant’s behavior using the social Simon effect (SSE). The SSE is a spatial stimulus–response compatibility effect between two persons. The SSE occurs when one perceives that their partner is present. Several studies have confirmed the SSE in actual environments. We presented partner sounds diotically (i.e., without spatial information) to one group or binaurally (i.e., with spatial information) to another group through headphones without providing visual information about the partner. The results showed that the SSE was induced only in the binaural group in the current auditory VR (Experiment 1), whereas both groups exhibited the SSE in an actual environment (Experiment 2). These results suggest that the auditory spatial information of remote people is sufficient to induce the SSE and has a potential to enhance social presence.

Binaural reproduction of dummy head and spherical microphone array data-A perceptual study on the minimum required spatial resolution.

Dynamic binaural synthesis requires binaural room impulse responses (BRIRs) for each head orientation of the listener. Such BRIRs can either be measured with a dummy head or calculated from the spherical microphone array (SMA) data. Because the dense dummy head measurements require enormous effort, alternatively sparse measurements can be performed and then interpolated in the spherical harmonics domain. The real-world SMAs, on the other hand, have a limited number of microphones, resulting in spatial undersampling artifacts. For both of the methods, the spatial order N of the underlying sampling grid influences the reproduction quality. This paper presents two listening experiments to determine the minimum spatial order for the direct sound, early reflections, and reverberation of the dummy head or SMA measurements required to generate the horizontally head-tracked binaural synthesis perceptually indistinguishable from a high-resolution reference. The results indicate that for direct sound, N = 9-13 is required for the dummy head BRIRs, but significantly higher orders of N = 17-20 are required for the SMA BRIRs. Furthermore, significantly lower orders are required for the late parts with N = 4-5 for the early reflections and reverberation of the dummy head BRIRs but N = 12-13 for the early reflections and N = 6-9 for the reverberation of the SMA BRIRs.

Binaural Auralization of Microphone Array Room Impulse Responses Using Causal Wiener Filtering

Binaural room auralization involves Binaural Room Impulse Responses (BRIRs). Dynamic binaural synthesis (i.e., head-tracked presentation) requires BRIRs for multiple head poses. Artificial heads can be used to measure BRIRs, but BRIR modeling from microphone array room impulse responses (RIRs) is becoming popular since personalized BRIRs can be obtained for any head pose with low extra effort. We present a novel framework for estimating a binaural signal from microphone array signals, using causal Wiener filtering and polynomial matrix formalism. The formulation places no explicit constraints on the geometry of the microphone array and enables directional weighting of the estimation error. A microphone noise model is used for regularization and to balance filter performance and noise gain. A complete procedure for BRIR modeling from microphone array RIRs is also presented, employing the proposed Wiener filtering framework. An application example illustrates the modeling procedure using a 19-channel spherical microphone array. Direct and reflected sound segments are modeled separately. The modeled BRIRs are compared to measured BRIRs and are shown to be waveform-accurate up to at least 1.5 kHz. At higher frequencies, correct statistical properties of diffuse sound field components are aimed for. A listening test indicates small perceptual differences to measured BRIRs. The presented method facilitates fast BRIR data set acquisition for use in dynamic binaural synthesis and is a viable alternative to Ambisonics-based binaural room auralization.

Localisation Properties of Wave Field Synthesis in a Listening Room

Theoretic assumptions for sound field synthesis include an anechoic environment for reproduction. As in practice loudspeaker arrays are usually installed inside listening rooms, the synthesised sound field is altered by reflections. These physical changes potentially influence human perception of the sound field. This paper analyses the localisation properties of 2.5D Wave Field Synthesis in free field and in listening rooms with different reflection factors for varying spacings between the secondary sources, in comparison to the localisation of real sources in a listening experiment. Only the directional aspect of localisation in the horizontal plane, not distance or elevation, is investigated. Stimuli were presented using dynamic binaural synthesis. Binaural room impulse responses of the listening rooms were simulated with the image source method. The results show that accurate localisation in Wave Field Synthesis in a reflective environment is only possible for small secondary source spacings as has already been known for free field conditions from the literature. Furthermore, the localisation ability of listeners for both real and virtual sources is slightly degraded by reflections leading to a larger localisation error with a greater localisation blur.

On the authenticity of individual dynamic binaural synthesis.

A simulation that is perceptually indistinguishable from the corresponding real sound field could be termed authentic. Using binaural technology, such a simulation would theoretically be achieved by reconstructing the sound pressure at a listener's ears. However, inevitable errors in the measurement, rendering, and reproduction introduce audible degradations, as it has been demonstrated in previous studies for anechoic environments and static binaural simulations (fixed head orientation). The current study investigated the authenticity of individual dynamic binaural simulations for three different acoustic environments (anechoic, dry, wet) using a highly sensitive listening test design. The results show that about half of the participants failed to reliably detect any differences for a speech stimulus, whereas all participants were able to do so for pulsed pink noise. Higher detection rates were observed in the anechoic condition, compared to the reverberant spaces, while the source position had no significant effect. It is concluded that the authenticity mainly depends on how comprehensive the spectral cues are provided by the audio content, and the amount of reverberation, whereas the source position plays a minor role. This is confirmed by a broad qualitative evaluation, suggesting that remaining differences mainly affect the tone color rather than the spatial, temporal or dynamical qualities.

Binaural auralization of proposed room modifications based on measured omnidirectional room impulse responses

The auralization of rooms with dynamic binaural synthesis using binaural room impulse responses (BRIRs) is an established approach in virtual audio. The BRIRs can either be obtained by simulations or by measurements. Up to now changed acoustical properties, as they occur when a room is altered in a renovation, cannot easily be considered in a measurement-based approach. This paper presents a new method to auralize modifications of existing rooms. The authors already have shown in a previous publication that such an auralization can be done by appropriately shaping the reverberation tail of an impulse response. Furthermore, the authors have presented an approach to synthesize BRIRs based on one omnidirectional room impulse response (RIR). In this paper, both methods are combined: A single measured omnidirectional RIR is enhanced and adapted to create a binaural representation of a modified room. A listening experiment has been performed to evaluate the procedure and to investigate differences between synthesized and measured BRIRs. The advantages of this method are obvious: Planned room modifications can be made audible without complex measurements or simulations; just one omnidirectional RIR is required to provide a binaural representation of the desired acoustic treatment.

Audio-visual room perception: Cross-modal and interaction effects put to test

The perception of rooms involves different modalities, particularly hearing and sight. Fundamental issues such as the acoustical and optical shares in certain perceptual features have, however, not been experimentally addressed yet. We investigated to what extent the acoustical and optical properties of performance rooms influenced auditory and visual features. Specifically, cross-modal effects and interaction effects were a matter of particular interest. We also quantified the respective proportion of acoustical and optical information accounting for the perceptual features. The main preconditions for such an undertaking are the dissociation of the acoustical and optical components of the stimuli, the commensurability of these components, and rich cue conditions. We acquired binaural room impulse responses and panoramic stereoscopic images of six rooms in order to recreate these rooms virtually, and added recordings of both a music and a speech performance by applying dynamic binaural synthesis and chroma-key compositing. By the use of a linearized extraaural headset and a semi-panoramic stereoscopic projection system we presented the scenes to test participants and asked them to rate unimodal features such as loudness, highs, lows, clarity, reverberance, and envelopment as well as brightness, contrast, color intensity, and hue. The statistical analyses indicate a straight-forward processing of low-level features.

Assessing localization accuracy in sound field synthesis

Sound field synthesis methods like Wave Field Synthesis (WFS) and Near-Field Compensated Higher Order Ambisonics synthesize a sound field in an extended area surrounded by loudspeakers. Because of the limited number of applicable loudspeakers the synthesized sound field includes artifacts. This paper investigates the influence of these artifacts on the accuracy with which a listener can localize a synthesized source. This was performed with listening tests using dynamic binaural synthesis to simulate different sound field synthesis methods and incorporated several listening positions. The results show that WFS is able to provide good localization accuracy in the whole listening area even for a low number of loudspeakers. For Near-Field Compensated Higher Order Ambisonics the achievable localization accuracy of the listener depends highly on the Ambisonics order and shows large localization deviations for low orders, where splitting of the perceived sound source was sometimes reported.

The influence of hearing and seeing on aesthetic judgments of performance spaces

The perception of rooms involves different modalities, particularly hearing and seeing. Fundamental questions of audio-visual room perception have, however, not been answered yet. We investigated to what extent aesthetic judgments on music and speech performances in different spaces are based on hearing, seeing, and their interaction. Meeting methodological criteria such as optoacoustic commensurability and optoacoustic dissociation, and applying a data-based, three-dimensional, high-resolution optoacoustic simulation providing nearly all perceptually relevant physical cues allowed for a valid proportionate quantification of these shares for the first time. BRIRs and panoramic stereoscopic images were acquired in six rooms. Recordings of both a music and a speech performance were put in the rooms by applying dynamic binaural synthesis and chroma-key compositing. The scenes were played back by the use of a linearized extraaural headset and a semi-panoramic stereoscopic projection system. Test participants were asked to rate the pleasantness, powerfulness, and excitingness of the performance, their spatial presence in the virtual scene, and the matching of the acoustical and the optical room. Statistical analyses indicate among others, that the aesthetic judgments mainly relied on acoustic information, whereas the presence and the matching judgment was mainly influenced by the interaction of optical and acoustical room properties.

Efficient algorithm and localization experiment on spherical microphone array recording and binaural rendering

Spherical microphone array recording and binaural rendering (SMABR) is a novel spatial sound technique, which records spatial information of sound field, transfers to dynamic binaural signals, and then renders via headphone. To improve the computational efficiency of SMABR, the present work proposes a PCA-based (principal component analysis) algorithm for dynamic binaural synthesis from the beamforming outputs of spherical microphone array, in which only 33 pairs of sharable filters are required. Incorporated the proposed algorithm, a PC-based SMABR system consisting of a 64-channel spherical microphone array, an electromagnetic head-tracker, and a headphone is established. A virtual source localization experiment is carried out to evaluate the system and algorithm. Results indicate that the system yields reasonable localization performance within a target virtual source region near the horizontal plane. However, larger localization error is also observed for high elevation above 30 degrees or low elevation below -30 degrees. Localization error at high or low elevation is caused by the spatial aliasing error of spherical microphone array at high frequency, which spoils the spectral cue for elevation localization. Therefore, further improvement on spherical microphone array recording is needed. [Supported by the National Natural Science Foundation of China, Grant No.11174087.]

Auditory and Visual Contribution to Egocentric Distance and Room Size Perception

The perception of rooms involves various unimodal and multimodal aspects on different perceptual levels. Rather abstract yet self-evident aspects are the source distance and the room size. We investigated to what extent the perceived room size and egocentric source distance as supramodal aspects are based on the auditory and the visual modality, i.e. experimentally influenced by the acoustic and optical stimulus. The statistical determination of the respective contributions demands the mutually independent variation of optical and acoustical room properties, usually referred to as conflicting stimulus paradigm. Simulation data of four rooms were collected acoustically by acquiring binaural room impulse responses for different head orientations and optically by acquiring stereoscopic images of the rooms including the electroacoustic sound source. In the laboratory, respective acoustic scenes were played back applying dynamic binaural synthesis, whereas the optical scenes were presented by the use of a stereoscopic display. Test participants were asked to assess the source distance and the room size. Results show main effects of the modalities rather than an interaction effect. It was found that distance perception in rooms is predominantly based on the acoustic stimulus characteristics whereas room size perception predominantly relies on optical information. There is no evidence in favour of the proximity image effect hypothesis. Results do also not confirm an auditory or audio-visual underestimation of the source distance but show a general underestimation of room size and indicate an audio-visual asymmetry regarding the accuracy of room size perception. Maximum accuracy and cross-modal consistency of judgements were distinctively observed in low-absorbent rooms.

The relative contribution of dynamic and spectral cues in virtual sound source localization

It is well known that the dynamic cues caused by head turning and pinna-based spectral cue are vital to sound source localization, especially for front-back and vertical localization. A series of localization experiment is carried out via virtual auditory display to explore the relative contribution of dynamic and spectral cues to localization. Virtual sources at different intended spatial locations are recreated using various combinations with noise stimuli at full audio or 4 kHz-lowpass frequency ranges, individualized HRTFs or non-individual HRTFs derived from KEMAR and spherical-head model, and static or dynamic binaural synthesis. Furthermore, statistical analyses are performed on localization performance in terms of the percentage of front-back and up-down confusion as well as the mean angle error in virtual source localization. Primary results indicate that both dynamic and spectral cues contribute to front-back and vertical localization; whereas dynamic cue contributes more to front-back localization, while individualized spectral cue at high frequency above 4 kHz contributes more to vertical localization. Dynamic and high-frequency individualized spectral cues are also helpful to reduce the angle error in localization. [Work supported by the National Natural Science Foundation of China,11174087, 50938003, and State Key Lab of Subtropical Building Science, South China University of Technology.]

Assessing the Plausibility of Virtual Acoustic Environments

Aiming at the perceptual evaluation of virtual acoustic environments (VAEs), ‘plausibility’ is introduced as a quality criterion that can be of value for many applications of virtual realities. We suggest a definition as well as an experimental operationalization for plausibility, referring to the perceived agreement with the listener’s expectation towards an equivalent real acoustic event. A listening test methodology for the criterion-free assessment of the deviation from this non-explicit, inner reference is proposed. It requires the rating of corresponding real and simulated stimuli in a Yes/No test paradigm, and the analysis of the results according to signal detection theory. The specification of minimum effect hypotheses allows the testing of plausibility with any desired strictness. The approach is demonstrated with the perceptual evaluation of a system for dynamic binaural synthesis in two different development stages.

Analysis on the audibility in directional differences of head-related transfer function magnitudes

A head-related transfer function (HRTF) varies as a function of sound source directions. In a dynamic virtual auditory display (VAD) based on HRTF filtering, dynamic binaural synthesis requires HRTFs to be updated constantly to accommodate the transient virtual source direction relative to listener. A HRTFs set with appropriate directional resolution can generate smooth transition when switching between HRTFs, and at the same time, simplifies dynamic binaural synthesis. Based on the measured HRTFs from KEMAR mannequin and human subjects with supper high directional resolution, the directional difference in HRTF magnitude spctra, binaural loudness level spectra and interaural localization cues are analyzed in present work. Combined with the statistical results of psychoacoustic experiments, psychometric functions for directional discrimination of HRTF magnitudes are derived. Finally, in terms of resultant audibility in directional differences of HRTF magnitudes, the directional resolutions of HRTFs for dynamically synthesizing virtual source at various directions are suggested.

Efficiency of two‐dimensional interpolation algorithms for high‐quality dynamic binaural synthesis

Binaural synthesis of acoustical environments is based on binaural room impulse responses (BRIRs) measured with an angular resolution of typically between 1° and 15°. Considering the size of the resulting BRIR database used for auralization and the long measurement duration for its acquisition, it is reasonable to use interpolation from a lower resolution BRIR grid. Based on a mathematical formulation of the interpolation problem for BRIRs, a set of different solutions for two‐dimensional spaces is described and compared with regard to efficiency and real time performance. In order to evaluate the degradation introduced by interpolation, different methods have been implemented and applied to a HRTF and a BRIR database measured in two degrees of freedom with 1°/1° horizontal/vertical resolution. These are bilinear interpolation (i.e. nearest neighbour, inverse distance weighting), spherical spline interpolation, wavelet interpolation, and interpolation based on principal component analysis (PCA). A listening test following an ABX procedure has been performed to evaluate the efficiency of the different interpolation methods according to the detection rates of interpolated versus measured BRIR databases in a dynamically auralized acoustical environment.

Minimum BRIR grid resolution for dynamic binaural synthesis

Binaural synthesis of acoustical environments is based on binaural room impulse responses (BRIRs) measured with a certain grid of spherical coordinates separated by angles of typically between 1° and 15°. The resulting spatial resolution defines the size of the BRIR database as well as the duration of its measurement. Perceptual evaluations of datasets with a different spatial resolution using HRTFs (anechoic case) have been reported from. Most of these studies use the localization performance of listeners as a criterion; a more sensitive measurement for slight degradations in audio quality can be expected from a criterion-free comparison of datasets with different angular grids. Thus, to determine the minimum grid resolution required for dynamic binaural synthesis a listening test was performed. Following an adaptive 3AFC procedure, the spatial resolution of a recorded BRIR dataset was gradually lowered from a maximum of 1°/1° resolution until audible artefacts were introduced. This was done for a sound source located at 0°/0° presented with dynamical auralization in two degrees of freedom. To test for interaction effects the thresholds were derived independently for azimuth and elevation. The datasets used were acquired in an anechoic environment and in two rooms of different size and reverberation time.