Immersive Audio Research Articles

3D audio recording of complex underwater sound fields

In this study, we developed a method to capture underwater sound fields and reproduce them for human listeners, providing an immersive 3D auditory experience. We recorded the sound field with an array of four hydrophones arranged in a tetrahedral structure. The side length of the tetrahedron (around 70 cm) was a compromise between capturing accurately the low frequency pressure gradients and spatial aliasing effects at higher frequencies. Subsequently, we transformed the signals into the domain of spherical harmonics facilitating a representation as first-order Ambisonics that is independent of the recording. For playback on devices such as headphones or loudspeaker systems, we used established decoding routines that enhance the immersive audio experience with techniques such as parametric decoding and diffuse sound field equalization. During an expedition to northern Norway, we recorded the underwater soundscape in deep fjords with vocalizing killer whales and singing humpback whales. The resulting binaural renderings for headphones of these sound scenes are astonishing and sparked interest among scientist, artists and general public. This research also helps to enhance our understanding and perception of these intelligent marine mammals and their unique acoustic underwater world, ultimately aiding in better protecting them from noise pollution.

Singer and Audience Evaluations of a Networked Immersive Audio Concert

Singer and Audience Evaluations of a Networked Immersive Audio Concert

Sonic Gathering Place: implementation of a biophilic soundscape design and its evaluation

The Sonic Gathering Place (SGP) is a prototype urban furniture installation that uses sound design to enhance the biophilic effects of plants. It uses field recordings from four different biomes in South-Eastern Australia mixed with plants that are representative of those biomes. The SGP has a circular seating structure comprising four integrated planter boxes, each with a single speaker. Nature-based field recordings play through the four-speaker array to create an immersive audio playback experience, though without masking the existing acoustic environment of the installation site. This paper focuses, as practice-led research, on the design process of the installation that encompasses four disciplinary approaches: soundscape design, biophilic design, field recording and urban furniture. Innovatively, it then reports a quantitative survey of the user’s experience of the SGP that hones understanding of human response to the design. Measured user-responses to the SGP, collected through a smartphone-based questionnaire, were overwhelmingly affirmative of the design, including a positive change in self-reported mood resulting from the visit to the SGP. These responses were observed with introduced sound levels set only marginally louder than sound levels that existed before the installation. The results signal a wider role that biophilic sound installations might play in urban landscape design.

Immersive technologies in the formation of musical audio space

Immersive audio technology is increasingly used in music, with devices like cell phones and televisions replicating formats. The increasing adoption of immersive audio technologies by both consumers and professionals in the music industry has made this topic more relevant. The purpose of the research is to analyse immersive technologies that contribute to the creation of the musical audio environment. The research employed methods: literature and historical review, technical analysis, practical cases, statistical method, and the author’s personal empirical experience. The article examines the function of audio space in relation to the perception of musical art and the characteristics of musical audio space. The proposed classification categories musical audio spaces into two distinct types: natural and artificial, as well as offline and online. The article’s statistical data illustrates the changing patterns of growth in online audio consumers and the rivalry between radio stations and streaming platforms to enhance their impact on shaping listeners’ audio spaces. The research demonstrates that streaming platforms actively encourage the incorporation of musical content in immersive formats as a means to expand their market share and evoke heightened emotional and perceptual responses in customers. The article outlines techniques used in immersive music spaces to provide a distinct aural experience for the listener. The article’s practical analysis of immersive audio technologies encompasses the examination of requisite microphones and digital audio workstations that facilitate the transition from two-dimensional audio space formation to threedimensional. In contemporary circumstances, the distribution of musical items is progressively employing threedimensional immersive technologies. The article enumerates audio technologies that guarantee the creation of an immersive auditory environment for both listeners and musicians: Ambisonics, KLANG: technologies, and DPA 5100. The practical significance of this work rests in examining the perceptual attributes of users of immersive technologies and exploring the potential for developing immersive technologies, such as immersive musical instruments or immersive radio stations

Optimizing Sound Quality and Immersion of a Proposed Cinema in Victoria Island, Nigeria

Enhancing sound quality and immersion is vital for an enriched cinematic journey. This study is centered on the proposed cinema in Victoria Island, Nigeria, aiming to tackle challenges and opportunities in this realm. It explores the current landscape of cinema sound technology, focusing on innovations like Dolby Atmos and DTS:X, renowned for transforming global audio experiences. Through thorough analysis, the research aims to offer comprehensive recommendations for augmenting sound quality and immersion in the proposed cinema. It scrutinizes acoustic aspects, employing acoustic treatments and advanced audio processing technologies, alongside immersive audio solutions. Cultural nuances specific to Nigeria are factored in to ensure resonance with local audiences. Ultimately, the research seeks to establish new benchmarks for cinematic audio experiences in Nigeria.

Simulation of acoustical parameters of churches in a virtual acoustics laboratory

Current ISO standards for acoustical assessment limit the use of traditional mono-dimensional microphones for measuring equipment. However, microphone arrays offer more accurate spatial information compared to traditional microphones. This presents an opportunity to enhance research on architectural acoustics and preserve the acoustics of cultural heritage more effectively. Furthermore, modern recording studios equipped with virtual acoustics systems allow for the integration of spatial room impulse responses. This enables real-time auralization in controlled environments and enhances the overall immersive audio experience for users and musicians. This paper analyzes the main acoustical parameters obtained from spatial impulse responses captured in churches using various ambisonic microphones and inserted into a controlled, interactive, and immersive virtual acoustics system. The captured spatial acoustic measurements are then reproduced and evaluated in a virtual acoustics laboratory to identify any discrepancies between real and virtual spaces.

Quantifying active brain areas at spatial hearing process using Electroencephalography (EEG) source localization approach

The human ability for spatial hearing encourages the development of spatial audio technology to support many human activities. Spatial audio mimics the real-life sound and provides a more immersive hearing experience as if the listener were present in the environment where the sound source was recorded. Recently, research on how spatial hearing processes are encoded in the brain has begun to be developed. As a modality for brain activity measurement that is non-invasive and has a high temporal resolution, Electroencephalography (EEG) is suitable for studying brain responses to moving sound stimuli. This study compared brain activity in hearing stereo and spatial audio stimulus. Through measurements using 21 EEG electrodes on the scalp, the results showed an increase in the mean PSD for all Theta, Alpha, and Beta waves in the spatial audio stimulus compared to stereo audio. The most significant increase in the mean PSD occurred in the Beta wave of 93.8008 µV2/Hz. Furthermore, the source localization approach with Independent Component Analysis (ICA) and Low-resolution Brain Electromagnetic Tomography (LORETA) method was used to estimate and quantify the active brain area of this process. The results showed an activation of the non-auditory cortex when the subject was hearing spatial audio stimulus. By using Paired T-Test of current density for both hearing processes, the results showed there was no significant difference (p > 0.05) in Brodmann area (BA) 41 (Primary Auditory Cortex) and BA 42 (Secondary Auditory Cortex). Meanwhile, for the spatial audio hearing process, there was a significant difference (p < 0.05) in BA 6 (Premotor Cortex), which is related to spatial orientation, and BA 9 (Dorsolateral Prefrontal Cortex), which is associated with executive functions, including working memory and selective attention. This study offers potential insights into spatial hearing research and immersive audio production.

Co-immersion in Audio Augmented Virtuality: the Case Study of a Static and Approximated Late Reverberation Algorithm.

In immersive Audio Augmented Reality, a virtual sound source should be indistinguishable from the existing real ones. This property can be evaluated with the co-immersion criterion, which encompasses scenes constituted by arbitrary configurations of real and virtual objects. Thus, we introduce the term Audio Augmented Virtuality (AAV) to describe a fully virtual environment consisting of auditory content captured from the real world, augmented by synthetic sound generation. We propose an experimental design in AAV investigating how simplified late reverberation (LR) affects the co-immersion of a sound source. Participants listened to simultaneous virtual speakers dynamically rendered through spatial Room Impulse Responses, and were asked to detect the presence of an impostor, i.e., a speaker rendered with one of two simplified LR conditions. Detection rates were found to be close to chance level, especially for one condition, suggesting a limited influence on co-immersion of the simplified LR in the evaluated AAV scenes. This methodology can be straightforwardly extended and applied to different acoustics scenes, complexities, i.e., the number of simultaneous speakers, and rendering parameters in order to further investigate the requirements for immersive audio technologies in AAR and AAV applications.

Looking for the Edge of the World: How 3D Immersive Audio Produces a Shift from an Internalised Inner Voice to Unsymbolised Affect-Driven Ways of Thinking and Heightened Sensory Awareness.

In this practice-based case study, we investigate the subjective aesthetic and affective responses to a shift from 2D stereo-based modelling to 3D object-based Dolby Atmos in an audio installation artwork. Dolby Atmos is an infinite object-based audio format released in 2012 but only recently incorporated into more public-facing formats. Our analysis focuses on the artist Sadia Sadia's 30-channel audio installation 'Notes to an Unknown Lover', based on her book of free verse poetry of the same title, which was rebuilt and reformatted in a Dolby Atmos specified studio. We examine what effect altered spatiality with an infinite number of 'placements' has on the psychoacoustic and neuroaesthetic response to the text. The effectiveness of three-dimensional (3D) object-based audio is interrogated against more traditional stereo and two-dimensional (2D) formats regarding the expression and communication of emotion and what effect altered spatiality with an infinite number of placements has on the psychoacoustic and neuroaesthetic response to the text. We provide a unique examination of the consequences of a shift from 2D to wholly encompassing object-based audio in a text-based artist's audio installation work. These findings may also have promising applications for health and well-being issues.

Greater artistic and technological performance through the converged technologies of architectural acoustics, electroacoustic enhancement, and immersive audio technologies

Producing meaningful, creative, and engaging experiences for audiences requires highly integrated acoustics and audio technologies. The fields of architectural acoustics, electro acoustic enhancement and immersive audio technologies have converged, altering the way we approach entertainment venue design. Globally, demand has increased for immersive audio in live and reproduced sound, and electro acoustic enhancement technologies are becoming more widespread. We will discuss how architecturally integrated end devices, multi-function signal processing, and architectural acoustical design should be optimized to create next-generation performance venues. We will use a project example that integrates sound system, acoustic enhancement, and architectural acoustics, illustrating how increasing clarity in audience with acoustic enhancement and supporting the immersive audio increases audience engagement. We will show what can be improved to elevate the experience in a converged approach. The paper will illustrate how a convergence of audio technologies, infrastructure, and a unified design and engineering processes is creating far greater opportunities for artistic experimentation and expression.

Sound environment control using an immersive audio system—Advancing sound experience for creators and listeners using Active Field Control

Spatial and immersive systems have gained popularity in recent years and are broadly used in various types of applications to realize natural or augmented sound environment. To create a variety of sound environment freely and flexibly, Yamaha immersive audio system, known as Active Filed Control (AFC), has two functions. One is “sound source control” based on object-based audio rendering technique and another is “sound field control” based on acoustic enhancement technology. This paper presents several unique solutions applying immersive audio systems that use either or both sound source control and sound field control. We conducted post-event acoustic measurements or survey of questionnaire supporting that intended acoustic controls had been successfully implemented.

High quality low complexity binaural rendering for headphones

Playing back immersive audio via headphones has been a research topic for a long time. Although there have been advances, bigger breakthroughs are still missing. The basic idea of binaural rendering systems can be summarized with the following elements: The emission characteristics of the audio source, the transmission path from audio source to listener as well as suitable head related transfer functions. Systems following this basic paradigm range from dummy head stereophony over purely simulation based approaches to current parametric algorithms including the usage of room impulse responses. According to research in audio cognition, it is important to incorporate positional and rotational tracking data of the head close to real time as well as acoustic information about the listening room. Based on the chosen approach, the effort to acquire the measurements, to prepare the rendering or the computational demand of the rendering itself might be high. In this paper, we report on a low complexity rendering system which allows high quality binaural reproduction. A recent listening test and many reviews from listeners have proven, that the perception of the rendered audio signals is equivalent to listening to real audio sources in a room.

Practice-Based Research as Reciprocal Protocol: Constellations of Care in The Seventh Fire

The Seventh Fire is an immersive audio performance that sources traditional, oral Anishinaabe stories and societal roles as a way to evoke ceremony in the everyday. I use the experience of creating this new form and premiering this work as an opportunity to reflect on principles of reciprocity that underscore and infuse any practice-based inquiry, as well as propose some necessary structures of care to support these principles.

Un Bon Voyage Sonore: Avoiding Listener Discomfort in Immersive Audio Documentary

ABSTRACT This article interrogates claims made in the emerging discourse of immersive audio documentary that spatial sound is more real, allowing the listener to step into another space, and understand the world better. However, the analysis shows makers are failing to make good on these claims. Use of the technical affordances of spatial audio is limited and producers enroll concepts of the real and of transportation in a colonial discourse of exploration and adventure, reproducing a disengaged mode of listening, while avoiding discomfort at all costs.

Exploring the relationships between teacher noticing, ambisonic audio, and variance in focus when viewing 360 video.

A growing body of research has supported the implementation of innovative and immersive video for teaching and learning across the lifespan. Immersive video, delivered through eXtended Reality (XR) tools like 360 video, provides users with new ways to see real or created environments. Unfortunately, most of the existing research has highlighted immersive video without accompanying immersive audio. This use of monophonic audio can create a disconnect for viewers as they experience close to real world video with sounds that do not match a real-world environment. The purpose of this study was to respond to this gap in the literature by exploring the use of ambisonic audio and its impact on preservice teacher noticing and variability of viewing focus when watching 360 video. Data were collected from undergraduate teacher education students who participated in a self-paced online activity that included watching 360 videos and responding to a questionnaire. A convergent mixed methods design was employed to compare participants' professional noticing and observed viewing behavior in the context of ambisonic and monophonic audio. Results showed that users in ambisonic audio conditions in 360 video environments were more likely to have higher focus. Moreover, for users who had specific professional knowledge, monophonic audio with immersive video had a negative impact on their variance in focus. The paper concludes with recommendations for future research on the use of audio in virtual and augmented reality environments.

Low-Voltage Stretchable Electroluminescent Loudspeakers with Synchronous Sound and Light Generation.

Stretchable sound-in-displays, which can generate synchronous sound and light directly from the display without a separate speaker, allow immersive audio and visual perception even on curved surfaces. In stretchable sound-in-displays, alternating current electroluminescent (ACEL) devices have been used as light-emitting sources owing to their high brightness and stability. However, stretchable ACEL devices that use low dielectric constant (κ) materials require a high operating voltage for generating light and sound. Herein, we demonstrate a stretchable ACEL loudspeaker with a low operating voltage using stretchable high-κ dielectrics and strain-insensitive electrodes. Our device exhibits 87.7 cd/m2 of luminance and 79.70 dB of sound pressure level at an operating voltage of 120 V and 10 kHz. As the next platform of wearable devices, the suggested ACEL loudspeaker exhibits high-quality synchronous light and sound generation performance even under various types of mechanical deformation, such as finger flexion and wrist bending.

Smart Multi-Agent Framework for Automated Audio Testing

With the widespread use of embedded software in consumer electronics, automotive industry, medical devices, and industrial environments, embedded software testing is gaining significance as an indispensable part of development and deployment of embedded products. With more than 20 years of research, development, and testing of various consumer technologies and products based on digital signal processors (DSPs) and advanced reduced instruction set computers (ARMs), we obtained insight into typical embedded development process and testing, and the pros and cons of various testing approaches and environments. In this paper, we propose the Smart Multi-Agent Framework based on IoT and Jenkins agents, customised for audio technologies in the Home Audio domain. We evaluated our solution on several complex immersive audio technologies implemented on a multicore DSP. Our uniform, customised, fully automated approach proved to be time efficient, error resilient, easy to replicate and use across all development, certification, and deployment phases of the product life cycle.

Design and Implementation of Two Immersive Audio and Video Communication Systems Based on Virtual Reality

Due to the impact of the COVID-19 pandemic in recent years, remote communication has become increasingly common, which has also spawned many online solutions. Compared with an in-person scenario, the feeling of immersion and participation is lacking in these solutions, and the effect is thus not ideal. In this study, we focus on two typical virtual reality (VR) application scenarios with immersive audio and video experience: VR conferencing and panoramic live broadcast. We begin by introducing the core principles of traditional video conferencing, followed by the existing research results of VR conferencing along with the similarities, differences, pros, and cons of each solution. Then, we outline our view about what elements a virtual conferencing room should have. After that, a simple implementation scheme for VR conferencing is provided. Regarding panoramic video, we introduce the steps to produce and transmit a panoramic live broadcast and analyze several current mainstream encoding optimization schemes. By comparing traditional video streams, the various development bottlenecks of panoramic live broadcast are identified and summarized. A simple implementation of a panoramic live broadcast is presented in this paper. To conclude, the main points are illustrated along with the possible future directions of the two systems. The simple implementation of two immersive systems provides a research and application reference for VR audio and video transmission, which can guide subsequent relevant research studies.

Dual quaternion ambisonics array for six-degree-of-freedom acoustic representation

Spatial audio methods are gaining a growing interest due to the spread of immersive audio experiences and applications, such as virtual and augmented reality. For these purposes, 3D audio signals are often acquired through arrays of Ambisonics microphones, each comprising four capsules that decompose the sound field in spherical harmonics. In this paper, we propose a dual quaternion representation of the spatial sound field acquired through an array of two First Order Ambisonics (FOA) microphones. The audio signals are encapsulated in a dual quaternion that leverages quaternion algebra properties to exploit correlations among them. This augmented representation with 6 degrees of freedom (6DOF) involves a more accurate coverage of the sound field, resulting in a more precise sound localization and a more immersive audio experience. We evaluate our approach on a sound event localization and detection (SELD) benchmark. We show that our dual quaternion SELD model with temporal convolution blocks (DualQSELD-TCN) achieves better results with respect to real and quaternion-valued baselines thanks to our augmented representation of the sound field. Full code is available at: https://github.com/ispamm/DualQSELD-TCN.

First Live Production and Transmission of Immersive Audio on Commercial OTA Digital Television in Latin America

Since 2017, Globo Television has been testing and evaluating immersive audio systems to use on over-the-air (OTA), pay TV, and streaming services to increase the user experience of our viewers and listeners, giving a sense of “being there” and exploring off-screen sound information, especially on live events. From that point until the present experimental content has been produced, some experimental transmissions made via streaming services, over-the-top (OTT), and OTA. This was done in collaboration with representatives from industrial and academic sectors in a regulatory committee. In 2020, a reviewed standard NBR 15602–2—the audio coding part of Brazil’s Digital TV standards—was published, which included immersive audio systems as optional. In September 2020, immersive audio was delivered on an OTA commercial channel in São Paulo, reaching more than 17 million people. Dolby Atmos audio service was used in the transmission with no negative impact on legacy receivers. This innovation presents live immersive audio transmissions across an entire workflow, from venue microphone audio capture, through an audio production chain, and extending from the exhibition center to TV transmission.