Surrounding Research Articles

Physics-informed neural network assisted spherical microphone array signal processing

Thanks to their rotational symmetry that facilitates three-dimensional signal processing, spherical microphone arrays are the common array apertures used for spatial audio and acoustic applications. However, practical implementations of spherical microphone arrays suffer from two issues. First, at high frequency range, a large number of sensors are needed to accurately capture a sound field. Second, the accompanying signal processing algorithm, i.e., the spherical harmonic decomposition method, requires a variable radius array or a rigid surface array to circumvent the spherical Bessel function nulls. Such arrays are hard to design and introduce a scattering field. To address these issues, this paper proposes to assist a spherical microphone array with a physics-informed neural network (PINN) for three-dimensional signal processing. The PINN models the sound field around the array based on the sensor measurements and the acoustic wave equation, augmenting the sound field information captured by the array through prediction. This makes it possible to analyze a high frequency sound field with a reduced number of sensors and avoid the spherical Bessel function nulls with a simple single radius open-sphere microphone array.

A binaural room impulse response dataset and Shorelining psychophysical task for the evaluation of auditory sensory augmentation

Sensory augmentation using spatial sound presented in augmented-reality (AR) can assist people with low vision or blindness in navigating their environment (Katz et al., 2012). Nonetheless, in many situations, the poor quality of the binaural sound rendered using current tool sets limits the potential capability of the assistive technology. In particular, acoustic environments with near-field sources and reflections pose significant challenges. In this work, we provide a reference binaural room impulse response (BRIR) dataset with near-field sources and reflections and an associated shorelining psychophysical task that is useful for the evaluation of AR spatial audio. The dataset consists of 12∼small loudspeakers arranged on a 3-by-4 grid in a complex acoustic environment. BRIR measurements are recorded using the Head and Torso Simulator (HATS) for 17∼different receiver positions with a 5o angular resolution. Room impulse response measurements are also recorded using the Eigenmike for each of the 17 receiver positions. Using the Razer Anzu smart glasses to render the binaural AR spatial audio, we compare psychophysical performance on the shorelining navigation task using the recorded dataset and various existing binaural AR tool sets.

Head-related transfer functions upsampling with physics-informed spherical convolutional neural network

Head-related transfer functions (HRTFs) play a crucial role in virtual acoustics and spatial audio applications. However, obtaining personalized, high-resolution HRTFs remains challenging due to the time-consuming and expensive measurements. Recently, deep learning-based methods have shown promise in predicting high-resolution HRTFs from sparse measurements. Nevertheless, most of these methods often treat HRTF upsampling as an image super-resolution task, which overlooks critical spatial information and acoustic principles, leading to overfitting on datasets. This paper proposes a physics-informed spherical convolutional neural network for HRTF upsampling. First, spherical convolutional layers are used to capture spatial features on the sphere, allowing efficient handling of spherical sampled HRTF data. Second, in the upsampling process, the proposed method incorporates the Helmholtz equation as a constraint, adhering tothe physics of the acoustic system. This method ensures the generation of physically feasible HRTF interpolations, thus promoting better generalization.

Three-dimensional sound recording using directional beams

The recording of three-dimensional spatial audio is typically carried out using microphone arrays that produce higher-order B-format responses, which are defined by spherical harmonics. Spherical harmonics provide a compact, orthonormal representation of the sound field, but can be intuitively challenging for general users because they do not have a direction, as opposed to typical microphones used in the audio industry. Furthermore, spherical microphone arrays typically use microphone angles based on Platonic solids which do not have intuitive directions such as front, rear, left, right, up, and down. This paper considers the use of sets of directional beams that have orthonormal properties in a similar manner to spherical harmonics. Sets of angles that have intuitive directions are used, and a method of approximating orthonormal beams is developed. The approach can also be applied to angles obtained from Platonic solids. Finally, a prototype microphone array that implements the directional recording format is described.

Impact of direct-to-reverberation ratio and interaural cross-correlation on externalization in binaural audio rendering

To achieve realistic binaural reproduction of spatial audio in a headphone scenario, addressing inside-the-head-locatedness is crucial, which can be accomplished through appropriate externalization techniques. Two acoustic cues, the direct-to-reverberation ratio (DRR) and interaural cross-correlation (IACC), are known to influence perceived externalization. However, the precise relationship between these cues and their combined effect has not been extensively studied. In this investigation, we aim to explore the relative contribution of DRR and IACC to externalization by examining various combinations of these two acoustic cues. To this end, participants were exposed to various binaural room impulse responses (BRIRs) synthesized to produce three distinct levels of DRRs and IACCs for two different source directions. The listening test was conducted in an anechoic chamber with participants’ eyes blindfolded to minimize the influence of room divergence effect. We also investigated the influence of head tracking by conducting experiments with and without head trackers. The correlation maps between externalization and the two acoustic cues were generated for each source direction. The study reveals that the contribution of the two acoustic cues differs with and without head tracking, and the joint contribution also varies depending on the source direction and amount of reflections included in the BRIR.

The effect of Surround sound on embodiment and sense of presence in cinematic experience: a behavioral and HD-EEG study.

Although many studies have investigated spectators' cinematic experience, only a few of them explored the neurophysiological correlates of the sense of presence evoked by the spatial characteristics of audio delivery devices. Nevertheless, nowadays both the industrial and the consumer markets have been saturated by some forms of spatial audio format that enrich the audio-visual cinematic experience, reducing the gap between the real and the digitally mediated world. The increase in the immersive capabilities corresponds to the instauration of both the sense of presence and the psychological sense of being in the virtual environment and also embodied simulation mechanisms. While it is well-known that these mechanisms can be activated in the real world, it is hypothesized that they may be elicited even in a virtual acoustic spatial environment and could be modulated by the acoustic spatialization cues reproduced by sound systems. Hence, the present study aims to investigate the neural basis of the sense of presence evoked by different forms of mediation by testing different acoustic space sound delivery (Presentation modes: Monophonic, Stereo, and Surround). To these aims, a behavioral investigation and a high-density electroencephalographic (HD-EEG) study have been developed. A large set of ecological and heterogeneous stimuli extracted from feature films were used. Furthermore, participants were selected following the generalized listener selection procedure. We found a significantly higher event-related desynchronization (ERD) in the Surround Presentation mode when compared to the Monophonic Presentation mode both in Alpha and Low-Beta centro-parietal clusters. We discuss this result as an index of embodied simulation mechanisms that could be considered as a possible neurophysiological correlation of the instauration of the sense of presence.

Navigating Hierarchical Menus Through Sound

Much of the current user experience navigating digital information relies on visual displays. Providing auditory alternatives enables access for visually impaired and blind users. The current study evaluated hierarchical navigation in a novel spatialized auditory interface compared to a screen reader and visual navigation mode. Past studies suggest that spatialized audio may provide performance improvements. Sixteen participants navigated menu structures of varying depth / breadth to select a target item with three different interface styles (spatial audio, screen reader, and visual). Time-to-completion, errors, and SUS scores were compared across interfaces. Results showed that spatial audio was significantly slower, more error prone, and less usable than the other conditions. However, the problems of spatial audio might be overcome with simple changes in the interaction mode and optimization of display space. Our experience shows that designers must solve the problem of auditory clutter and spatial selection to achieve usable auditory navigation.

Sonification of a juggling performance using spatial audio

We developed a new musical instrument by equipping juggling balls with accelerometers, gyroscopes, and WiFi sensors. The system measures acceleration and rotation, allowing us to distinguish between ball movement and static states. We can determine the direction of each throw based on the acceleration data. A movement-driven instrument animates the virtual audio source position based on the throw direction. A different synthesizer is assigned to each ball, resulting in a synesthetic experience. The system integrates the juggling patterns simulator with OSC and MIDI API, allowing real-time rhythm generation and composition.

Un Bon Voyage Sonore: Avoiding Listener Discomfort in Immersive Audio Documentary

ABSTRACT This article interrogates claims made in the emerging discourse of immersive audio documentary that spatial sound is more real, allowing the listener to step into another space, and understand the world better. However, the analysis shows makers are failing to make good on these claims. Use of the technical affordances of spatial audio is limited and producers enroll concepts of the real and of transportation in a colonial discourse of exploration and adventure, reproducing a disengaged mode of listening, while avoiding discomfort at all costs.

AudioEar: Single-View Ear Reconstruction for Personalized Spatial Audio

Spatial audio, which focuses on immersive 3D sound rendering, is widely applied in the acoustic industry. One of the key problems of current spatial audio rendering methods is the lack of personalization based on different anatomies of individuals, which is essential to produce accurate sound source positions. In this work, we address this problem from an interdisciplinary perspective. The rendering of spatial audio is strongly correlated with the 3D shape of human bodies, particularly ears. To this end, we propose to achieve personalized spatial audio by reconstructing 3D human ears with single-view images. First, to benchmark the ear reconstruction task, we introduce AudioEar3D, a high-quality 3D ear dataset consisting of 112 point cloud ear scans with RGB images. To self-supervisedly train a reconstruction model, we further collect a 2D ear dataset composed of 2,000 images, each one with manual annotation of occlusion and 55 landmarks, named AudioEar2D. To our knowledge, both datasets have the largest scale and best quality of their kinds for public use. Further, we propose AudioEarM, a reconstruction method guided by a depth estimation network that is trained on synthetic data, with two loss functions tailored for ear data. Lastly, to fill the gap between the vision and acoustics community, we develop a pipeline to integrate the reconstructed ear mesh with an off-the-shelf 3D human body and simulate a personalized Head-Related Transfer Function (HRTF), which is the core of spatial audio rendering. Code and data are publicly available in https://github.com/seanywang0408/AudioEar.

Validation of collaborative cyberspace virtual reality oculometry enhanced with near real-time spatial audio

Currently, most medical image data, such as optical coherence tomography (OCT) images, are displayed in two dimensions on a computer screen. Advances in computer information technology have contributed to the growing storage of these data in electronic form. However, the data are usually processed only locally on site. To overcome such hurdles, a cyberspace virtual reality (csVR) application was validated, in which interactive OCT data were presented simultaneously to geographically distant sites (Lucerne, London, and Barcelona) where three graders independently measured the ocular csVR OCT diameters. A total of 109 objects were measured, each three times, resulting in a total of 327 csVR measurements. A minor mean absolute difference of 5.3 µm was found among the 3 measurements of an object (standard deviation 4.2 µm, coefficient of variation 0.3% with respect to the mean object size). Despite the 5 h of online work, csVR was well tolerated and safe. Digital high-resolution OCT data can be remotely and collaboratively processed in csVR. With csVR, measurements and actions enhanced with spatial audio communication can be made consistently in near real time, even if the users are situated geographically far apart. The proposed visuo-auditory framework has the potential to further boost the convenience of digital medicine toward csVR precision and collaborative medicine.

Visually-Guided Audio Spatialization in Video with Geometry-Aware Multi-task Learning

Visually-Guided Audio Spatialization in Video with Geometry-Aware Multi-task Learning

Audio for extended realities: A case study informed exposition

An area of immersive storytelling in rapid evolution is that of extended reality. This emergent mode of experience employs spatial mapping, and both plane and object detection to superimpose computer-generated images in the volumetric context of a physical space via a head-mounted display. This in turn produces a unique set of challenges and opportunities for associated audio implementation and aesthetics. Creative development of this audio is often a function of evolving toolsets, and the associated workflow is far from standardized. This paper forms a context for such audio workflow, one that draws from precursor technologies such as audio for games and virtual reality and develops this into an outline taxonomy that is both representative of the state of the art, and forward facing towards evolution of the technology stack. The context is framed through a series of case studies. Between 2019 and 2021, BBC and Oculus TV commissioned Alchemy Immersive and Atlantic Productions to produce virtual reality and mixed reality experiences of several classic documentary series by Sir David Attenborough: Museum Alive, Micro Monsters, First Life VR, Museum Alive AR and Kingdom of Plants. This portfolio received numerous award nominations and prizes, including from the Raindance Festival and a double Emmy. The sound design, audio postproduction and spatial audio for these experiences were implemented by the company 1.618 Digital, and drawing from first-hand-creator involvement, the workflows are deconstructed and explored with reference to tools, technologies, techniques and perception. Such an exposition forms the basis for an analysis of both this and broader creative practice in the field of audio for extended reality, and this is subsequently used to present a speculative vision of audio in the future of immersive storytelling.

Pengembangan Aplikasi Augmented Reality Pada Mata Pelajaran Bahasa Inggris Untuk Kelas 1 Sekolah Dasar

This research was developed to overcome problems in students who tend to get bored in English subjects with book media. This research aims to develop Augmented Reality in English subject for grade 1 elementary school. The research method used in this study namely research and development (R&D) with the ADDIE application development model (Analysis, design, development, implementation, and evaluation), in this study is limited to stage implementation. Products are developed using game engine Unity 3D and Vuforia. Data analysis techniques in this study will use descriptive and quantitative analysis techniques. The data collection instrument used was a questionnaire with 8 respondents. The end result of the product is an educational game-based application Augmented Reality learning English which contains 3D objects, descriptions, sentences and sounds from objects that are run. The results of the feasibility test obtained an average score of 81.90% in the very good category. With the application Augmented Reality this is expected to help 1st grade elementary school students in learning English.

SoundFields: A Virtual Reality Home-Based Intervention for Auditory Hypersensitivity Experienced by Autistic Children

Previous studies have shown that autistic people often display atypical responses when processing sensory information, with particular prevalence within the auditory domain. Often provoked by common everyday sounds, auditory hypersensitivity can result in self-regulatory fear responses. This can be potentially harmful to autistic individuals and the people around them and is associated with greater occurrence of anxiety, depression, and poorer overall quality of life in the autistic population. Rather than a physiological causation, the literature suggests that hypersensitivity to sound is likely to be caused by how auditory stimuli are processed in the brain. This paper reports a home-based digital intervention aimed to address auditory hypersensitivity in autistic children. Developed as an interactive virtual reality game, the system integrates exposure-based therapy techniques into game mechanics and delivers target auditory stimuli to the player rendered via binaural-based spatial audio. The performance of the platform was evaluated in a 10-week feasibility study, during which children (n = 7) engaged weekly with the game during a 30 min session. Following this period, a comparison of pre- and post-study measurements showed a decrease in sensitivity for five participants, with qualitative feedback highlighting an increase in tolerance towards real-world stimuli and challenging environments. These results provide initial support for SoundFields as a home-based intervention targeting auditory hypersensitivity experienced by autistic children.

Effect of Perceptual Training with Sound-Guided and Kinesthetic Feedback on Human 3D Sound Localization Capabilities.

In this paper, we experimentally investigate how the 3D sound localization capabilities of the blind can improve through perceptual training. To this end, we develop a novel perceptual training method with sound-guided feedback and kinesthetic assistance to evaluate its effectiveness compared to conventional training methods. In perceptual training, we exclude visual perception by blindfolding the subjects to apply the proposed method to the visually impaired. Subjects used a specially designed pointing stick to generate a sound at the tip, indicating localization error and tip position. The proposed perceptual training aims to evaluate the training effect on 3D sound localization, including variations in azimuth, elevation, and distance. The six days of training based on six subjects resulted in the following outcomes: (1) In general, accuracy in full 3D sound localization can be improved based on training. (2) Training based on relative error feedback is more effective than absolute error feedback. (3) Subjects tend to underestimate distance when the sound source is near, less than 1000 mm, or larger than 15° to the left, and overestimate the elevation when the sound source is near or in the center, and within ±15° in azimuth estimations.

Sound event localization and detection using element-wise attention gate and asymmetric convolutional recurrent neural networks

There are problems that standard square convolution kernel has insufficient representation ability and recurrent neural network usually ignores the importance of different elements within an input vector in sound event localization and detection. This paper proposes an element-wise attention gate-asymmetric convolutional recurrent neural network (EleAttG-ACRNN), to improve the performance of sound event localization and detection. First, a convolutional neural network with context gating and asymmetric squeeze excitation residual is constructed, where asymmetric convolution enhances the capability of the square convolution kernel; squeeze excitation can improve the interdependence between channels; context gating can weight the important features and suppress the irrelevant features. Next, in order to improve the expressiveness of the model, we integrate the element-wise attention gate into the bidirectional gated recurrent network, which is to highlight the importance of different elements within an input vector, and further learn the temporal context information. Evaluation results using the TAU Spatial Sound Events 2019-Ambisonic dataset show the effectiveness of the proposed method, and it improves SELD performance up to 0.05 in error rate, 1.7% in F-score, 0.7° in DOA error, and 4.5% in Frame recall compared to a CRNN method.

Future of virtual education and telementoring

To summarize recent technological advancements in medical and surgical education and explore what the future of medicine might be as it relates to blockchain technology, the metaverse, and web3. Through the use of digitally assisted ophthalmic surgery and high dynamic range 3D cameras, it is now possible to record and live stream 3D video content. Although the 'metaverse' is still in its early stages, there are a variety of proto-metaverse technologies that exist to facilitate user interactions that can mimic the real world through the use of shared digital environments and 3D spatial audio. Advanced blockchain technologies can allow for further development of interoperable virtual worlds where a user has an on-chain identity, credentials, data, assets, and much more that they can carry across platforms seamlessly. As remote real-time communication becomes an integral part of human interaction, 3D live streaming has the potential to revolutionize ophthalmic education by removing traditional geographic and physical constraints of in-person surgical viewing. The incorporation of metaverse and web3 technologies has created new outlets for knowledge sharing that may improve how we operate, teach, learn, and transfer knowledge.

Feasibility of Virtual Reality-Based Auditory Localization Training With Binaurally Recorded Auditory Stimuli for Patients With Single-Sided Deafness.

To train participants to localize sound using virtual reality (VR) technology, appropriate auditory stimuli that contain accurate spatial cues are essential. The generic head-related transfer function that grounds the programmed spatial audio in VR does not reflect individual variation in monaural spatial cues, which is critical for auditory spatial perception in patients with single-sided deafness (SSD). As binaural difference cues are unavailable, auditory spatial perception is a typical problem in the SSD population and warrants intervention. This study assessed the applicability of binaurally recorded auditory stimuli in VR-based training for sound localization in SSD patients. Sixteen subjects with SSD and 38 normal-hearing (NH) controls underwent VR-based training for sound localization and were assessed 3 weeks after completing training. The VR program incorporated prerecorded auditory stimuli created individually in the SSD group and over an anthropometric model in the NH group. Sound localization performance revealed significant improvements in both groups after training, with retained benefits lasting for an additional 3 weeks. Subjective improvements in spatial hearing were confirmed in the SSD group. By examining individuals with SSD and NH, VR-based training for sound localization that used binaurally recorded stimuli, measured individually, was found to be effective and beneficial. Furthermore, VR-based training does not require sophisticated instruments or setups. These. suggest that this technique represents a new therapeutic treatment for impaired sound localization.

성덕대왕신종(聖德大王神鍾)의 음향학적 에뮬레이션 연구

The purpose of this study is to analyze the possibility of using the “Sacred Bell of Great King Seongdeok” which is designated as National Treasure No. 29, as a musical resource through acoustic analysis and sound synthesis method. Based on the acoustically analyzed data of the Bell, it was tried to synthesized with frequency and fluctuation of partial sound, the beat cycle and envelop application by the additive synthesis among the method of sound synthesis The spatial sounds were modelled by an artificial neural network to estimate the exact reverberation of the source. The final result was produced with Native Instruments Reactor6, which can be easily used not only by sound designers and multimedia artists but also by many musicians, to preserve the history and identity of Sacred Bell of Great King Seongdeok, and to confirm its potential as a medium to be fused to new media.