Surrounding Research Articles

A Review on Head-Related Transfer Function Generation for Spatial Audio

A head-related transfer function (HRTF) is a mathematical model that describes the acoustic path between a sound source and a listener’s ear. Using binaural synthesis techniques, HRTFs play a crucial role in creating immersive audio experiences through headphones or loudspeakers, using binaural synthesis techniques. HRTF measurements can be conducted either with standardised mannequins or with in-ear microphones on real subjects. However, various challenges arise in, for example, individual differences in head shape, pinnae geometry, and torso dimensions, as well as in the extensive number of measurements required for optimal audio immersion. To address these issues, numerous methods have been developed to generate new HRTFs from existing data or through computer simulations. This review paper provides an overview of the current approaches and technologies for generating, adapting, and optimising HRTFs, with a focus on physical modelling, anthropometric techniques, machine learning methods, interpolation strategies, and their practical applications.

Driving function in wave field synthesis with integral approximation considering uneven contribution of loudspeaker units.

Wave field synthesis is a spatial audio technique for reproducing the sound field of a target source. This process is implemented by applying driving functions to loudspeakers enclosing a listening space. To reduce computational and hardware costs, several studies assume that the target source is sufficiently far from the loudspeakers, but this assumption depends on source frequency and can lead to errors. A recent method attempted to address this issue using oversampling and averaging; however, this approach increases computational demands, particularly for multiple target sources. Such steps result in high computational costs when there are numerous target sources. In this study, we propose an alternative method using an integral approximation that imitates the linearity of integral operations, taking into account the uneven contributions of loudspeakers. This eliminates the need for oversampling and averaging. Simulation results showed that our method improved sound reproduction compared to the recent method at high frequencies and conventional methods at low frequencies, especially when the target source was close to loudspeakers and not aligned with them. Our approach could potentially be applied to tasks that use the same integrand, widely used to model sources and fields.

Realistic dual-task listening-while-balancing in older adults with normal hearing and hearing loss with and without hearing aids

Age-related hearing loss is a risk factor for mobility problems and falls, possibly due to poor access to spatial sounds or the higher allocation of attention required to listen, thereby reducing cognitive resources to support mobility. Introducing stabilizing spatial sounds or reducing cognitive load through hearing aids could possibly improve balance performance; however, evidence is mixed. Few studies have evaluated the effects of hearing aids and spatial sounds on balance during realistic, multisensory, dual-tasking conditions. This study used virtual reality to simulate a listening-while-balancing task in 22 older adults with normal hearing and 22 hearing aid users, tested with their aids on versus off. Participants performed a competing digits listening task (two, four digits) and a standing postural task, alone and in combination (dual task) under different visual, postural, and acoustical loads. Listening accuracy and postural outcomes (centre of pressure mean velocity, anterior–posterior standard deviation, medial–lateral standard deviation) were collected. With respect to listening accuracy, as expected, normal hearing adults had higher listening accuracy than those with hearing loss (aided better than unaided) and both groups performed better with eyes closed (vs. open) and under lower postural loads (firm vs. compliant). With respect to postural performance, hearing aids did not remarkably improve balance overall, with no effects on dual-task costs to posture. Other factors related to the complexity of the conditions (i.e., listening, visual, postural demands) differently influenced dual-task costs to posture in individuals with and without hearing loss. Overall, these results contribute to our understanding of how age-related hearing loss and hearing aids affect balance-related outcomes under realistic, complex, multisensory, multitasking conditions.

Limitations and Performance Analysis of Spherical Sector Harmonics for Sound Field Processing

Developing spherical sector harmonics (SSHs) benefits sound field decomposition and analysis over spherical sector regions. Although SSHs demonstrate potential in the field of spatial audio, a comprehensive investigation into their properties and performance is absent. This paper seeks to close this gap by revealing three key limitations of SSHs and exploring their performance in two aspects: sector sound field radial extrapolation and sector sound field decomposition and reconstruction. First, SSHs are not solutions to the Helmholtz equation, which is their main limitation. Then, due to the violation of the Helmholtz equation, SSHs lack the ability to conduct sound field radial extrapolation, especially for interior cases. Third, when using SSHs to decompose and reconstruct a sound field, the shifted associated Legendre polynomials and scaled exponential function in SSHs result in severe distortion around the edge of the sector region. In light of these three limitations, the future implementation of SSHs should focus on processing and analyzing the measurement sector region without any extrapolation process, and the measurement region should be larger than the target sector region.

Networked microcontrollers for accessible, distributed spatial audio

State-of-the-art systems for spatial and immersive audio are typically very costly, being reliant on specialist audio hardware capable of performing computationally intensive signal processing and delivering output to many tens, if not hundreds, of loudspeakers. Centralised systems of this sort suffer from limited accessibility due to their inflexibility and expense. Building on the research of the past few decades in the transmission of audio data over computer networks, and the emergence in recent years of increasingly capable, low-cost microcontroller-based development platforms with support for both networking and audio functionality, we present a prototype decentralised, modular alternative. Having previously explored the feasibility of running a microcontroller device as a networked audio client, here we describe the development of a client-server system with improved scalability via multicast data transmission. The system operates on ubiquitous, commonplace computing and networking equipment, with a view to it being a simple, versatile, and highly-accessible platform, capable of granting users the freedom to explore audio spatialisation approaches at vastly reduced expense. Though faced by significant technical challenges, particularly with regard to maintaining synchronicity between distributed audio processors, the system produces perceptually plausible results. Findings are commensurate with a capability, with further development and research, to disrupt and democratise the fields of spatial and immersive audio.

Ambisonics neural speech extraction with directional feature and rotary steering

In scenes with noise and overlapping speakers, directionally extracting audio tracks corresponding to individual speakers is crucial for immersive and interactive spatial audio systems. Although neural networks have been successful in this task, existing steering approaches for adjusting the direction of neural speech extraction mainly target spatial audio directly collected by microphone arrays, while directional speech extraction with Ambisonics spatial audio is less well studied. Therefore, to encode the target directional information as input for the neural network, this paper proposes two Ambisonics directional features based on the spatial feature difference and beamforming principle: the relative harmonic difference and the directional signal enhancement ratio. Using the special property of Ambisonics' rotation transform, a rotary steering pre-processing is also proposed to align the target speaker's direction with a fixed reference by inversely rotating the sound field, thereby simplifying multi-directional extraction to fixed-directional extraction. Finally, we integrate these proposed approaches with the existing temporal-spectral-spatial filtering neural networks to establish a generalized framework for steerable speech extraction and conduct experiments on a simulated Ambisonics dataset containing multiple speakers and noise sources. The experiments show that the proposed approaches outperform existing conditional steering and can be applied to various existing neural network architectures.

There Is More to Avatars Than Visuals: Investigating Combinations of Visual and Auditory User Representations for Remote Collaboration in Augmented Reality

Supporting remote collaboration through augmented reality facilitates the experience of co-presence by presenting the collaborator’s avatar in the user’s physical environment. While visual user representations are continuously researched and advanced, the audio configuration – especially in combination with different visualizations – is rarely considered. In a user study (n = 48, 24 dyads), we evaluate the combination of two visual (Simple vs. Rich Avatar) with two auditory (Mono vs. Spatial Audio) user representations, to investigate their impact on user’s overall experience, performance, and perceived social presence during collaboration. Our results show a preference for rich auditory and visual user representations, as Spatial Audio supports completion of parallel tasks and the Rich Avatar positively influence user experience. However, the Simple Avatar draws less attention, which potentially benefits task efficiency, advocating for simpler visualizations in performance-oriented settings. Our findings contribute to a deeper understanding of how visual and auditory user representation combinations impact remote collaboration in augmented reality.

Ensemble width estimation in HRTF-convolved binaural music recordings using an auditory model and a gradient-boosted decision trees regressor

Binaural audio recordings become increasingly popular in multimedia repositories, posing new challenges in indexing, searching, and retrieval of such excerpts in terms of their spatial audio scene characteristics. This paper presents a new method for the automatic estimation of one of the most important spatial attributes of binaural recordings of music, namely “ensemble width.” The method has been developed using a repository of 23,040 binaural excerpts synthesized by convolving 192 multi-track music recordings with 30 sets of head-related transfer functions (HRTF). The synthesized excerpts represented various spatial distributions of music sound sources along a frontal semicircle in the horizontal plane. A binaural auditory model was exploited to derive the standard binaural cues from the synthesized excerpts, yielding a dataset representing interaural level and time differences, complemented by interaural cross-correlation coefficients. Subsequently, a regression method, based on gradient-boosted decision trees, was applied to the formerly calculated dataset to estimate ensemble width values. According to the obtained results, the mean absolute error of the ensemble width estimation averaged across experimental conditions amounts to 6.63° (SD 0.12°). The accuracy of the method is the highest for the recordings with ensembles narrower than 30°, yielding the mean absolute error ranging between 0.8° and 10.2°. The performance of the proposed algorithm is relatively uniform regardless of the horizontal position of an ensemble. However, its accuracy deteriorates for wider ensembles, with the error reaching 25.2° for the music ensembles spanning 90°. The developed method exhibits satisfactory generalization properties when evaluated both under music-independent and HRTF-independent conditions. The proposed method outperforms the technique based on “spatiograms” recently introduced in the literature.

Spatial versus normal audio guides in exhibitions: Cognitive mechanisms and effects on learning

AbstractThe present paper examines possible benefits of spatial audio guides on learning outcomes in the spatial learning setting of an experimental exhibition and attempts to differentiate between different mechanisms underlying such an effect. In Experiment 1, we examined whether the spatial contiguity principle may be such a mechanism. A spatial audio guide was used that allows for perception of audio explanations either at the location of the exhibit (audio-visuo-spatially close) or within the headphones (audio-visuo-spatially distant). The results showed that with the external, exhibit-located compared to headphone-located presentation, memory of audio explanations was significantly better. No significant differences were found regarding text-picture-integration. However, these results cannot only be explained by audio-visuo-spatial contiguity but also by externalization and audio-spatial cueing. In Experiment 2, we therefore tried to disentangle these possible mechanisms and compared three conditions (audio explanations perceptually located within the exhibits, behind the visitors at the opposite wall, always at the same location in the room), thereby keeping externalization constant while varying audio-visuo-spatial contiguity and audio-spatial cueing. No significant differences between the conditions were found regarding memory of the audio explanations and text-picture integration, whereby memory of audio explanations in all conditions was at the same level as in the picture-located audio condition in the first experiment. This excludes spatial contiguity effects and audio-spatial cueing as explanations for the results in Experiment 1 and indicates a general externalization effect needing closer examination. The results are discussed regarding future studies and the use of spatial audio guides in museums and exhibitions.

Effect of discretization on off-axis listeners in binaural synthesis based on the optimal source distribution

3D sound reproduction based on binaural synthesis over loudspeakers is one of the most important techniques in immersive spatial audio. The system inversion involved in the binaural synthesis gives rise to several problems such as dynamic range loss, distortion of the reproduction transfer function, and a lack of robustness to room reflections. The principle of Optimal Source Distribution was proposed to overcome these problems. This utilizes a pair of conceptual monopole transducers whose azimuthal location varies continuously as a function of frequency. Another potential of the principle is that it can provide listening positions for multiple off-axis listeners, in addition to the on-axis listener whose inverse matrix is given, since the phase interference state is aligned at all frequencies. However, these continuously varying transducers are too conceptual to realize in practice. It is necessary to provide discretized sound sources with appropriate frequency ranges according to the principle. In this paper, the effect of discretization of the principle on off-axis listeners' performance is numerically investigated to reveal how those off-axis sweet spots are formed as the discretization becomes finer. An experiment is also carried out using an actual discretized system and the effect of the head-related transfer functions is considered.

Validation of sound source localization quality in binaural audio synthesis with head-tracking

Binaural synthesis with head tracking has become a highly important tool in creating an auditory experience for users that is as similar as possible to their perception of sound in the real world. This is crucial in many different applications that rely on accurate representation of real or virtual sound environments, ranging from scientific research to gaming applications. The quality of the binaural synthesis process and the accuracy and naturalness of the produced result depend on numerous factors, ranging from the selection of suitable IMU sensor technology used for head tracking to the choice of methods for implementing binaural synthesis using generic or individual sets of Head-Related Transfer Functions (HRTF). This is especially significant in Virtual Reality (VR) systems with three or six degrees of freedom that fully immerse the users in the virtual world through visual and auditory stimuli. This paper presents the authors' experiences in addressing a variety of challenges that arose during the design of such spatial audio systems and the testing of sound source localization quality. Strong emphasis is put on the testing of sound source localization quality, where the position of the source dynamically changes in relation to the listener.

AudioMove: Applying the Spatial Audio to Multi-Directional Limb Exercise Guidance

Guiding users with limb exercise can assist in muscle training or physical recovery. However, traditional vision-based methods often require multiple camera angles to help users understand the motions and require them to be within the range of the screen. Therefore, we propose a non-visual system that can guide users with multiple-directional limb motions utilizing spatial audio, AudioMove , with commercial-off-the-shelf (COTS) devices (i.e., smartphones and earphones). The proposed system addresses the challenge of conveying directional information encompassing multiple planes in real-time. We conduct a mixed-method user study to evaluate the effectiveness of the system with three methods combining motion data with spatial audio perception. Additionally, a user interface is built to collect users' comments. The results conclude that spatial audio guidance could create a natural, pervasive, and non-visual exercise training solution in daily life.

A Quality of Experience and Visual Attention Evaluation for 360° Videos with Non-spatial and Spatial Audio

This article presents the results of an empirical study that aimed to investigate the influence of various types of audio (spatial and non-spatial) on the user quality of experience (QoE) of and visual attention in 360° videos. The study compared the head pose, eye gaze, pupil dilations, heart rate, and subjective responses of 73 users who watched ten 360° videos with different sound configurations. The configurations evaluated were no sound; non-spatial (stereo) audio; and two spatial sound conditions (first- and third-order ambisonics). The videos covered various categories and presented both indoor and outdoor scenarios. The subjective responses were analyzed using an ANOVA (Analysis of Variance) to assess mean differences between sound conditions. Data visualization was also employed to enhance the interpretability of the results. The findings reveal diverse viewing patterns, physiological responses, and subjective experiences among users watching 360° videos with different sound conditions. Spatial audio, in particular third-order ambisonics, garnered heightened attention. This is evident in increased pupil dilation and heart rate. Furthermore, the presence of spatial audio led to more diverse head poses when sound sources were distributed across the scene. These findings have important implications for the development of effective techniques for optimizing processing, encoding, distributing, and rendering content in virtual reality (VR) and 360° videos with spatialized audio. These insights are also relevant in the creative realms of content design and enhancement. They provide valuable guidance on how spatial audio influences user attention, physiological responses, and overall subjective experiences. Understanding these dynamics can assist content creators and designers in crafting immersive experiences that leverage spatialized audio to captivate users, enhance engagement, and optimize the overall quality of VR and 360° video content. The dataset, scripts used for data collection, ffmpeg commands used for processing the videos, and the subjective questionnaire and its statistical analysis are publicly available.

A Study of Improved Two-Stage Dual-Conv Coordinate Attention Model for Sound Event Detection and Localization.

Sound Event Detection and Localization (SELD) is a comprehensive task that aims to solve the subtasks of Sound Event Detection (SED) and Sound Source Localization (SSL) simultaneously. The task of SELD lies in the need to solve both sound recognition and spatial localization problems, and different categories of sound events may overlap in time and space, making it more difficult for the model to distinguish between different events occurring at the same time and to locate the sound source. In this study, the Dual-conv Coordinate Attention Module (DCAM) combines dual convolutional blocks and Coordinate Attention, and based on this, the network architecture based on the two-stage strategy is improved to form the SELD-oriented Two-Stage Dual-conv Coordinate Attention Model (TDCAM) for SELD. TDCAM draws on the concepts of Visual Geometry Group (VGG) networks and Coordinate Attention to effectively capture critical local information by focusing on the coordinate space information of the feature map and dealing with the relationship between the feature map channels to enhance the feature selection capability of the model. To address the limitation of a single-layer Bi-directional Gated Recurrent Unit (Bi-GRU) in the two-stage network in terms of timing processing, we add to the structure of the two-layer Bi-GRU and introduce the data enhancement techniques of the frequency mask and time mask to improve the modeling and generalization ability of the model for timing features. Through experimental validation on the TAU Spatial Sound Events 2019 development dataset, our approach significantly improves the performance of SELD compared to the two-stage network baseline model. Furthermore, the effectiveness of DCAM and the two-layer Bi-GRU structure is confirmed by performing ablation experiments.

Optimal sensor placement for the spatial reconstruction of sound fields

The estimation sound fields over space is of interest in sound field control and analysis, spatial audio, room acoustics and virtual reality. Sound fields can be estimated from a number of measurements distributed over space yet this remains a challenging problem due to the large experimental effort required. In this work we investigate sensor distributions that are optimal to estimate sound fields. Such optimization is valuable as it can greatly reduce the number of measurements required. The sensor positions are optimized with respect to the parameters describing a sound field, or the pressure reconstructed at the area of interest, by finding the positions that minimize the Bayesian Cramér-Rao bound (BCRB). The optimized distributions are investigated in a numerical study as well as with measured room impulse responses. We observe a reduction in the number of measurements of approximately 50% when the sensor positions are optimized for reconstructing the sound field when compared with random distributions. The results indicate that optimizing the sensors positions is also valuable when the vector of parameters is sparse, specially compared with random sensor distributions, which are often adopted in sparse array processing in acoustics.

EchoSee: An Assistive Mobile Application for Real-Time 3D Environment Reconstruction and Sonification Supporting Enhanced Navigation for People with Vision Impairments.

Improving the quality of life for people with vision impairments has been an important goal in the research and design of assistive devices for several decades. This paper seeks to further that goal by introducing a novel assistive technology platform that leverages real-time 3D spatial audio to promote safe and efficient navigation for people who are blind or visually impaired (PVI). The presented platform, EchoSee, uses modern 3D scanning technology on a mobile device to construct a live, digital 3D map of a user's environment as they move about their surroundings. Spatialized, virtual audio sources (i.e., virtual speakers) are dynamically placed within the digital 3D scan of the world, providing the navigator with a real-time 3D stereo audio "soundscape." The digital 3D map, and its resultant soundscape, are continuously updated as the user moves about their environment. The generated soundscape is played back through headphones connected to the navigator's device. This paper details (1) the underlying technical components and how they were integrated to produce the mobile application that generates a dynamic soundscape on a consumer mobile device, (2) a methodology for analyzing navigation performance with the application, (3) the design and execution of a user study investigating the effectiveness of the presented system, and (4) a discussion of the results of that study along with a proposed future study and possible improvements. Altogether, this paper presents a novel software platform aimed at assisting individuals with vision impairments to navigate and understand spaces safely, efficiently, and independently and the results of a feasibility study analyzing the viability of the approach.

A sense of space in the core of the bore: Enhancing the MRI experience through use of spatial audio

BackgroundMagnetic Resonance Imaging (MRI) continues to be an imaging technique that causes anxiety and concern for many patients. One longstanding approach that helps many patients is listening to music as a distraction technique during a scan. However, despite improvements in scanner technology itself, the means of providing music in this way has hardly changed, and so there may be opportunity to utilise emerging approaches in audio technology to further enhance the positive effect beyond simple distraction alone. ObjectivesThe aim of this technical note is to introduce spatial audio and its potential application in the MRI setting to create a sense of space within the confined environment of a scanner. FindingsInitial feedback from a few patients indicate that spatial audio music may indeed enhance the perception of space within the MRI scanner, potentially mitigating or lowering feelings of claustrophobia and anxiety. ConclusionSpatial audio could enhance the sense of space felt within an MRI scanner, but research is needed to evidence whether this would indeed have an improved effect compared to rot conventional audio. Implications for practiceThis technical note sets the stage for further exploration of spatial audio technology within the setting of MRI to help adjust cognitive appraisal and concern over its confined nature. Providing music through this means could be another approach to help patients manage a scan and improve their experience of it.

Practitioners' Perspectives on Spatial Audio: Insights into Dolby Atmos and Binaural Mixes in Popular Music

Practitioners' Perspectives on Spatial Audio: Insights into Dolby Atmos and Binaural Mixes in Popular Music

Social connectedness in spatial audio calling contexts

People often use audio-only communication to connect with others. Spatialization of audio has been previously found to improve immersion, presence, and social presence during conversations. We propose that spatial audio improves social connectedness between dyads. Participants engaged in three 8-min semi-structured conversations with an acquainted partner in three conditions: in-person communication, monaural audio communication, and spatial audio communication. Using Media Naturalness Theory as our theoretical framework, we found that the use of spatial audio benefited aspects of social connectedness. While in-person communication yielded the greatest social connectedness, spatial audio better facilitated social connectedness than traditional monaural communication. Spatial audio improved feelings of being physically in the same room and being on the same wavelength and produced more nonverbal behaviors associated with rapport building than monaural communication.

The Role of Audio in Immersive Storytelling: a Systematic Review in Cultural Heritage

AbstractNowadays, Virtual and Augmented Reality technologies play a supportive role in many research fields. In cultural heritage, various examples are available, including storytelling and narratives, where they can provide immersive and enhanced experiences to visitors and tourists, especially for entertainment and educational purposes. This review aims to investigate the opportunities that soundscape design and advanced sonic interactions in virtual and augmented environments can bring to cultural heritage sites and museums in terms of presence, emotional content, and cultural dissemination. Nineteen-two papers have been identified through the PRISMA methodology, and a promising positive effect of sonic interaction on user experience in a virtual environment can be observed in various studies, notwithstanding a general lack of specific contributions on the use of sound rendering and audio spatialisation for improving such experiences. Moreover, this work identifies the main involved research areas and discusses the state-of-the-art best practices and case studies where sonic interactions may assume a central role. The final part suggests possible future directions and applications for more engaging and immersive storytelling in the cultural heritage domain.