Spatial Audio Reproduction Research Articles

Spatial audio reproduction for studying second language speech perception in varying acoustic environments

This paper introduces recent attempts of utilising spatial audio reproduction in linguistic studies and discusses challenges of the new methodology. Thanks to its high reproducibility of arbitrary acoustic environments in a controlled laboratory setting, spatial audio reproduction has a potential to address the difficulties of examining participants with different language background that have to be recruited in geographically separated locations. A case study that investigated the difference in speech intelligibility in noise between first and second language listeners in varying acoustic environments is reported with its experimental results. The paper then attempts to verify the validity of the methodology using spatial audio reproduction by comparing experimental results conducted in real and virtual rooms.

Spatial audio reproduction of floor impact-borne sounds for auditory demonstrations in living environment

Floor impact-borne sound in the multistory residential buildings has been a serious social issue in Korea for a long time. The housing and land authority in a ministry of Korea recently amended the floor impact-borne sound evaluation method and criteria in response to the acoustic characteristics of the actual impact sources perceived in our daily lives. However, many people have not yet been convinced by the effectiveness of those evaluation method and criteria. This study discusses spatial audito reproduction of the various floor impact-borne sounds for auditory demonstrations to increase the laypersons’ understanding of the acoustic features of impact sources. The impact-borne sounds generated both by the standard impact sources used in evaluation and by some actual ones were recorded at a test-bed built in a typical apartment type. Their sound levels and reverberation times were also obtained across 1/3rd octave bands for reference. A multi-channel audio playback system engaging front/rear/ceiling speakers with subwoofers was integrated in a test-bed with a typical setting of living environment to reproduce a realistic sound field. It was optimized to create the sounds close to those actually heard in the room. The impact-borne sounds of the different damping floor systems will also be presented.

Spatial audio reproduction over ad hoc loudspeaker array using near-field compensation in spherical harmonics domain

Spatial audio reproduction using the loudspeaker array introduces the curvature effect leading to a distorted listening experience when the listener is in the near field. In the near-field, the loudspeakers are approximated as point sources (spherical wave) and amplify the mode vectors. Further, the problem becomes more challenging for the irregular loudspeaker arrangement, which causes uneven energy distribution in the reproduction region. In this context, a near-field compensation is applied to the encoded ambisonics coefficients. An optimization problem is formulated, such as the loudspeaker gains encoded with spherical harmonics basis coefficients should match the target ambisonics coefficients. Further, the in-phase and quadrature components of the energy localization vector are imposed as the constraints to direct maximum energy in the reproduction region. The solution to the optimization problem is obtained using a derivative-free optimization solver. The performance of the proposed methods is evaluated for ITU-R recommended loudspeaker layouts using the technical and perceptual evaluation attributes.

Direct acoustic velocity vector measurements for evaluation of multichannel sound reproduction systems with signals of varied partial coherence

In multichannel sound reproduction systems, the perceived virtual auditory scene is strongly influenced by inter-channel phase-shifts and the degree of mutual coherence among channel signals. Such degrees of freedom may be employed to effect an expanded virtual sound stage or a sense of spaciousness, but they also may lead to unwanted spectral coloration, diffuse or ambiguous virtual source locations, or startling and disorienting spatial image perceptions often described as “phaseyness.” Such effects may be described and understood in terms of the acoustic velocity in the vicinity of the listener. In this work we present theoretical predictions of the acoustic velocity vector for a stereo reproduction system with channel signals of varying phase and levels of mutual coherence, and we report direct measurements of the acoustic velocity vector employing a 3D sound intensity probe for several specific cases. Acoustic velocity vector predictions and measurements also are correlated with listener perceptions. The presented measurements are also compared to subjective localization of sound sources. The direct measurement of the acoustic velocity vector in spatial sound reproduction systems is a useful tool to quantitatively evaluate the performance of multi-channel spatial audio reproduction systems, to predict listener perceptions of the acoustic field reproduced by such systems, and to investigate the impact of early reflections in a room on localization.

The Soundlapse Project: Exploring Spatiotemporal Features of Wetland Soundscapes

Abstract This article discusses an interdisciplinary project aimed at highlighting the acoustical heritage of wetlands by means of field recordings and a novel time-lapse montage method. The authors discuss a site-specific sound installation that was designed using original wetlands field recordings, live processing and spatial audio multichannel reproduction. The discussion focuses on spatial and temporal features of different types of recorded wetlands soundscapes. Future developments of this project will consider the implementation of a standalone spatiotemporal application, to be used in the context of virtual reality applications, game audio and interactive dance performance.

On the selection of the number of beamformers in beamforming-based binaural reproduction

In recent years, spatial audio reproduction has been widely researched with many studies focusing on headphone-based spatial reproduction. A popular format for spatial audio is higher order Ambisonics (HOA), where a spherical microphone array is typically used to obtain the HOA signals. When a spherical array is not available, beamforming-based binaural reproduction (BFBR) can be used, where signals are captured with arrays of a general configuration. While shown to be useful, no comprehensive studies of BFBR have been presented and so its limitations and other design aspects are not well understood. This paper takes an initial step towards developing a theory for BFBR and develops guidelines for selecting the number of beamformers. In particular, the average directivity factor of the microphone array is proposed as a measure for supporting this selection. The effect of head-related transfer function (HRTF) order truncation that occurs when using too many beamformer directions is presented and studied. In addition, the relation between HOA-based binaural reproduction and BFBR is discussed through analysis based on a spherical array. A simulation study is then presented, based on both a spherical and a planar array, demonstrating the proposed guidelines. A listening test verifies the perceptual attributes of the methods presented in this study. These results can be used for more informed beamformer design for BFBR.

Interpolating the Directional Room Impulse Response for Dynamic Spatial Audio Reproduction

Virtual reality (VR) is increasingly important for exploring the real world, which has partially moved to virtual workplaces. In order to create immersive presence in a simulated scene for humans, VR needs to reproduce spatial audio that describes three-dimensional acoustic characteristics in the counterpart physical environment. When the user moves, this reproduction should be dynamically updated, which provides practical challenges because the bandwidth for continuously transmitting audio and video scene data may be limited. This paper proposes an interpolation approach for dynamic spatial audio reproduction using acoustic characteristics of direction and reverberation at limited numbers of positions, which are represented using a first order Ambisonics encoding of the room impulse response (RIR), called the directional RIR (DRIR). We decompose two known DRIRs into reflection components, before interpolating early dominant components for DRIR synthesis and utilizing DRIR recordings for accuracy evaluation. Results indicate that the most accurate interpolation is obtained by the proposed method over two comparative approaches, particularly in a simulated small room where most direction of arrival estimation errors of early components are below five degrees. These findings suggest precise interpolated DRIRs with limited data using the proposed approach, which is vital for dynamic spatial audio reproduction for VR applications.

Spatial Audio Reproduction Over Ad Hoc Loudspeaker Array Using Near-Field Compensation in Spherical Harmonics Domain

Spatial Audio Reproduction Over Ad Hoc Loudspeaker Array Using Near-Field Compensation in Spherical Harmonics Domain

Immersive audio-visual scene reproduction using semantic scene reconstruction from 360 cameras

As personalised immersive display systems have been intensely explored in virtual reality (VR), plausible 3D audio corresponding to the visual content is required to provide more realistic experiences to users. It is well known that spatial audio synchronised with visual information improves a sense of immersion but limited research progress has been achieved in immersive audio-visual content production and reproduction. In this paper, we propose an end-to-end pipeline to simultaneously reconstruct 3D geometry and acoustic properties of the environment from a pair of omnidirectional panoramic images. A semantic scene reconstruction and completion method using a deep convolutional neural network is proposed to estimate the complete semantic scene geometry in order to adapt spatial audio reproduction to the scene. Experiments provide objective and subjective evaluations of the proposed pipeline for plausible audio-visual VR reproduction of real scenes.

Creation of Auditory Augmented Reality Using a Position-Dynamic Binaural Synthesis System—Technical Components, Psychoacoustic Needs, and Perceptual Evaluation

For a spatial audio reproduction in the context of augmented reality, a position-dynamic binaural synthesis system can be used to synthesize the ear signals for a moving listener. The goal is the fusion of the auditory perception of the virtual audio objects with the real listening environment. Such a system has several components, each of which help to enable a plausible auditory simulation. For each possible position of the listener in the room, a set of binaural room impulse responses (BRIRs) congruent with the expected auditory environment is required to avoid room divergence effects. Adequate and efficient approaches are methods to synthesize new BRIRs using very few measurements of the listening room. The required spatial resolution of the BRIR positions can be estimated by spatial auditory perception thresholds. Retrieving and processing the tracking data of the listener’s head-pose and position as well as convolving BRIRs with an audio signal needs to be done in real-time. This contribution presents work done by the authors including several technical components of such a system in detail. It shows how the single components are affected by psychoacoustics. Furthermore, the paper also discusses the perceptive effect by means of listening tests demonstrating the appropriateness of the approaches.

Comparison of 3D Audio Reproduction Methods Using Hearing Devices

In order to study 3D auditory perception for hearing aid users, spatial audio reproduction systems have been considered. Numerous solutions were designed in the past, but few studies evaluated the effect of 3D audio reproduction artifacts on hearing devices. This study focuses on loudspeaker-based 3D audio reproduction and compares the effects of distance-based amplitude panning (DBAP), third and fifth order Ambisonics (HOA), multiple-direction amplitude panning (MDAP), and vector-base amplitude panning (VBAP) on the 3D directivity index (DI) of hearing device beamformers, on interaural time differences (ITD), and on interaural level differences (ILD). Measurements were conducted at the sweet spot and for two off-center positions in a non-anechoic room using 32 loudspeakers. Results show that the effect of the reproduction system on the DI at the sweet spot was larger for third order Ambisonics and MDAP and smaller for fifth order Ambisonics. They also show that DBAP cannot reproduce ITDs correctly and that the ILDs obtained with HOA reproduction contain larger errors than with the other reproduction systems. Finally the results show that the DI, ITD, and ILD errors increase with the distance to the sweet spot and that this increase is larger for HOA than for the other reproduction methods

Pre-Trained-Based Individualization Model for Real-Time Spatial Audio Rendering System

Spatial audio has attracted more and more attention in the fields of virtual reality (VR), blind navigation and so on. The individualized head-related transfer functions (HRTFs) play an important role in generating spatial audio with accurate localization perception. Existing methods only focus on one database, and do not fully utilize the information from multiple databases. In light of this, a pre-trained-based individualization model is proposed to predict HRTFs for any target user in this paper, and a real-time spatial audio rendering system built on a wearable device is implemented to produce an immersive virtual auditory display. The proposed method first builds a pre-trained model based on multiple databases using a DNN-based model combined with an autoencoder-based dimensional reduction method. This model can capture the nonlinear relationship between user-independent HRTFs and position-dependent features. Then, fine tuning is done using a transfer learning technique at a limit number of layers based on the pre-trained model. The key idea behind fine tuning is to transfer the pre-trained user-independent model to the user-dependent one based on anthropometric features. Finally, real-time issues are discussed to guarantee a fluent auditory experience during dynamic scene update, including fine-grained head-related impulse response (HRIR) acquisition, efficient spatial audio reproduction, and parallel synthesis and playback. These techniques ensure that the system is implemented with little computational cost, thus minimizing processing delay. The experimental results show that the proposed model outperforms other methods in terms of subjective and objective metrics. Additionally, our rendering system runs on HTC Vive, with almost unnoticeable delay.

Effects and applications of spatial acuity in advanced spatial audio reproduction systems with loudspeakers

Spatial audio reproduction systems using loudspeakers produce coloration effects at high frequencies due to spatial interference between loudspeakers, in both those based on panning and those based on field synthesis. As a response to this problem and in order to reduce coloration, this paper studies the feasibility of an alternative approach where high-frequencies are reproduced from a single loudspeaker, with a different direction from that of its panned low-frequency counterpart. Listening tests are conducted to investigate the localization and quality of the source in the case that frequencies higher than 1.5 kHz are reproduced from a different direction than the low frequencies. In this context, the human ability to discriminate the spatial direction of low/high frequency bands and the error in the perceived direction of arrival for different separation angles is evaluated and quantified. The resulting data has been analyzed with ANOVA, providing significant results that allow us to establish a threshold in the angular separation of the high and low frequency parts where subjects do not perceive source location artifacts. The term just noticeable band splitting angle (JNBSA) is defined and introduced. It represents the minimum angle of separation between high and low frequencies from which the listener starts to perceive artifacts in the reproduction of a sound source using loudspeakers.

SCaLAr – A surrounding spherical cap loudspeaker array for flexible generation and evaluation of virtual acoustic environments

Introduction: Surrounding spherical loudspeaker arrays facilitate the application of various spatial audio reproduction methods and can be used for a broad range of acoustic measurements and perceptual evaluations. Methods: Installed in an anechoic chamber, the design and implementation of such an array of 68 coaxial loudspeakers, sampling a spherical cap with a radius of 1.35 m on an equal-area grid, is presented. A network-based audio backbone enables low-latency signal transmission with low-noise amplifiers providing a high signal-to-noise ratio. To address batch-to-batch variations, the loudspeaker transfer functions were equalised by individually designed 512-taps finite impulse response filters. Time delays and corresponding level adjustments further helped to minimise radial mounting imperfections. Results: The equalised loudspeaker transfer functions measured under ideal conditions and when mounted, their directivity patterns, and in-situ background noise levels satisfy key criteria towards applicability. Advantages and shortcomings of the selected decoders for panning-based techniques, as well as the influence of loudspeaker positioning errors, are analysed in terms of simulated performance metrics. An evaluation of the achievable channel separation allows deriving recommendations of feasible subset layouts for loudspeaker-based binaural reproduction. Conclusion: The combination of electroacoustic properties, simulated sound field synthesis performance and measured channel separation classifies the system as suitable for its target applications.

Efficient Representation and Sparse Sampling of Head-Related Transfer Functions Using Phase-Correction Based on Ear Alignment

With the proliferation of high quality virtual reality systems, the demand for high fidelity spatial audio reproduction has grown. This requires individual head-related transfer functions (HRTFs) with high spatial resolution. Acquiring such HRTFs is not always possible, which motivates the need for sparsely sampled HRTFs. Additionally, real-time applications require compact representation of HRTFs. Recently, spherical-harmonics (SH) has been suggested for efficient interpolation and representation of HRTFs. However, representation of sparse HRTFs with a limited SH order may introduce spatial aliasing and truncation errors, which have a detrimental effect on the reproduced spatial audio. This is because the HRTF is inherently of a high spatial order. One approach to overcome this limitation is to pre-process the HRTF, with the aim of reducing its effective SH order. A recent study showed that order-reduction can be achieved by time-alignment of HRTFs, through numerical estimation of the time delays of the HRTFs. In this paper, a new method for pre-processing HRTFs in order to reduce their effective order is presented. The method uses phase-correction based on ear alignment, by exploiting the dual-centering nature of HRTF measurements. In contrast to time-alignment, the phase-correction is performed parametrically, making it more robust to measurement noise. The SH order reduction and ensuing interpolation errors due to sparse sampling were analyzed for these two methods. Results indicate significant reduction in the effective SH order, where only 100 measurements and order 6 are required to achieve a normalized mean square error below $-$ 10 dB compared to a fully-sampled, high-order HRTF.

Ambisonic Decoding for Compensated Amplitude Panning

Compensated amplitude panning (CAP) is a spatial audio reproduction method for loudspeakers that takes the listener head orientation into account. It can produce stable images in any direction using only two loudspeakers. In its original form, CAP is inherently an object-based method, with each image produced separately. An exact and efficient method is presented here for dynamically decoding a first order Ambisonic encoding, which is equivalent to using CAP to separately reproduce the constituents of the encoding. Both the stereo and multichannel cases are considered.

Quality assessment of acoustic environment reproduction methods for cinematic virtual reality in soundscape applications

With the advent of virtual reality (VR) technology, spatial audio has been increasingly adopted to evaluate the acoustic environment in soundscape research. It is therefore imperative to assess the quality of commonly used spatial audio reproduction methods to determine their ecological validity. Through subjective evaluations with 30 participants, the same participant evaluated four outdoor in situ locations vis-à-vis its corresponding audio-visual recording in VR on a separate day. A total of three spatial audio reproduction methods were assessed in VR, and they were all down-mixed from the first-order ambisonics (FOA) recordings to headphone-based FOA-static binaural, FOA-tracked binaural; and FOA 2-dimensional (2D) octagonal speaker array. The participants evaluated the acoustic environment in terms of the overall soundscape quality and perceived spatial qualities at each location. Regarding overall soundscape quality, there were no significant differences in evaluating the sound-source dominance and affective soundscape qualities between in situ and all VR methods. However, significant differences were found in the perceived spatial qualities between three reproduction methods and in situ. Among the source-related spatial attributes, the perceived distance of the dominating sounds was farther in the virtual than in the in situ evaluations. In the localization of sound sources, both the FOA-tracked binaural and the FOA-2D speaker array exhibited higher spatial acoustic fidelity than FOA-static binaural. Regarding the environment-related spatial quality attributes, the 2D speaker array reproduction was perceived as more immersive and realistic than other reproduction methods. Overall, the FOA-tracked binaural appears to exhibit sufficient fidelity for cinematic VR evaluation of soundscapes.

Adapting masking thresholds for spatially separated sounds in two dimensional ambisonics

Spatial audio reproduction has gained renewed interest in recent years with the increasing presence of virtual reality applications. Ambisonics is currently the most widely adopted format for spatial audio content distributed via internet streaming services, and this has raised the need for audio compression relevant to the format. Current perceptual audio coders used for stereo audio content rely heavily on masking thresholds to reduce data rates, but these thresholds do not take into consideration spatial release from masking. This study begins an effort to update these thresholds for spatially separated sources in ambisonics. The listening tests were performed with sounds encoded in ambisonics to allow for tests to integrate whatever inherent limitations exist in the format. Initial listening tests were carried out for a subset of possible conditions—sound sources were separated along the horizontal plane, with a specific set of separation angles between the masker and maskee. Suggestions are given for continuing the work for the full range of possible conditions.

Qualitative Evaluation of Media Device Orchestration for Immersive Spatial Audio Reproduction

The challenge of installing and setting up dedicated spatial audio systems can make it difficult to deliver immersive listening experiences to the general public. However, the proliferation of smart mobile devices and the rise of the Internet of Things mean that there are increasing numbers of connected devices capable of producing audio in the home. \Media device orchestration (MDO) is the concept of utilizing an ad hoc set of devices to deliver or augment a media experience. In this paper, the concept is evaluated by implementing MDO for augmented spatial audio reproduction using objectbased audio with semantic metadata. A thematic analysis of positive and negative listener comments about the system revealed three main categories of response: perceptual, technical, and content-dependent aspects. MDO performed particularly well in terms of immersion/envelopment, but the quality of listening experience was partly dependent on loudspeaker quality and listener position. Suggestions for further development based on these categories are given.

Determination and Validation of Mix Parameters for Modifying Envelopment in Object-Based Audio

Envelopment is an important attribute of listener preference for spatial audio reproduction. Object-based audio offers the possibility of altering the rendering of an audio scene in order to modify or maintain perceptual attributes - including envelopment - if the relationships between attributes and mix parameters are known. In a method of adjustment experiment, mixing engineers were asked to produce mixes of four program items at low, medium, and high levels of envelopment, in 2-channel, 5-channel, and 22-channel reproduction systems. The participants could vary a range of level, position, and equalization parameters that can be modified in object-based audio systems. The parameters could be varied separately for different semantic object categories. Nine parameters were found to have significant relationships with envelopment; parameters relating to the horizontal and vertical spread of sources were shown to be most important. A follow-on experiment demonstrated that these parameters can be adjusted to produce a range of envelopment levels in other program items.