Audio Augmented Reality Research Articles

Audio augmented reality using sonification to enhance visual art experiences: Lessons learned

Augmenting visual art in art galleries can be an effective Audio Augmented Reality (AAR) application for indoor exploration. In the current study, eight paintings from four genres were augmented with audio through their sonification. Basic Audio was generated using a sonification algorithm by identifying the major colors of the paintings, and Enhanced Audio was generated by a musician enhancing the Basic Audio; these were presented with the paintings to compare against No Audio. Twenty-six participants viewed each painting in all three conditions; eye gaze metrics, and qualitative data were collected. Results showed that Enhanced Audio led to significantly greater engagement and positive sentiments, compared to Basic Audio. Thematic analysis showed semantic and syntactic relationships of the audio with the paintings, and a tendency to guide users’ gaze over time. Findings from this study can guide future AAR developments to improve auditory display designs to enhance visual experiences.

An end-to-end approach for blindly rendering a virtual sound source in an audio augmented reality environment

Audio augmented reality (AAR), a prominent topic in the field of audio, requires understanding the listening environment of the user for rendering an authentic virtual auditory object. Reverberation time (RT60\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$RT_{60}$$\\end{document}) is a predominant metric for the characterization of room acoustics and numerous approaches have been proposed to estimate it blindly from a reverberant speech signal. However, a single RT60\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$RT_{60}$$\\end{document} value may not be sufficient to correctly describe and render the acoustics of a room. This contribution presents a method for the estimation of multiple room acoustic parameters required to render close-to-accurate room acoustics in an unknown environment. It is shown how these parameters can be estimated blindly using an audio transformer that can be deployed on a mobile device. Furthermore, the paper also discusses the use of the estimated room acoustic parameters to find a similar room from a dataset of real BRIRs that can be further used for rendering the virtual audio source. Additionally, a novel binaural room impulse response (BRIR) augmentation technique to overcome the limitation of inadequate data is proposed. Finally, the proposed method is validated perceptually by means of a listening test.

Co-immersion in Audio Augmented Virtuality: the Case Study of a Static and Approximated Late Reverberation Algorithm.

In immersive Audio Augmented Reality, a virtual sound source should be indistinguishable from the existing real ones. This property can be evaluated with the co-immersion criterion, which encompasses scenes constituted by arbitrary configurations of real and virtual objects. Thus, we introduce the term Audio Augmented Virtuality (AAV) to describe a fully virtual environment consisting of auditory content captured from the real world, augmented by synthetic sound generation. We propose an experimental design in AAV investigating how simplified late reverberation (LR) affects the co-immersion of a sound source. Participants listened to simultaneous virtual speakers dynamically rendered through spatial Room Impulse Responses, and were asked to detect the presence of an impostor, i.e., a speaker rendered with one of two simplified LR conditions. Detection rates were found to be close to chance level, especially for one condition, suggesting a limited influence on co-immersion of the simplified LR in the evaluated AAV scenes. This methodology can be straightforwardly extended and applied to different acoustics scenes, complexities, i.e., the number of simultaneous speakers, and rendering parameters in order to further investigate the requirements for immersive audio technologies in AAR and AAV applications.

Taxonomy and definition of audio augmented reality (AAR): A grounded theory study

AR applications have mostly considered visual augmentations while excluding other modalities. Recent developments in audio augmented reality (AAR) applications have been based on the definitions of visual AR or mixed reality (MR), and thus, AAR technology development has lacked systematic efforts. We investigated the concept of augmented reality through audio to provide a systematic understanding and generate a taxonomy and a definition for AAR. A conference workshop (N = 28), focus groups (N = 18), and expert interviews (N = 6) generated qualitative data regarding the concept of AAR. Grounded Theory (GT) was used to analyze the data and produce a new taxonomy and a definition. The AAR taxonomy consists of three categories – Environment Connected, Goal Directed, and Context Adapted, with three subcategories respectively. The need for a separate taxonomy for AAR is highlighted to aid in the development of AAR applications in a systematic manner. The taxonomy is expected to be used as a heuristic tool that can guide developers to build AAR applications and can be used in evaluating user experience with AAR applications.

Enhancing Art Gallery Visitors' Experiences through Audio Augmented Reality Technology

Audio Augmented Reality (AAR) applications are gaining traction, especially for entertainment purposes. To that extent, the current study explored its use and effectiveness in enhancing art gallery visitors’ experiences. Four paintings were selected and sonified using the Jython algorithm to produce computer generated music (Basic AAR); the audio was then further enhanced with traditional music by a musician (Enhanced AAR). Twenty-six participants experienced each painting in Basic, Enhanced, and No AAR condition. Results show that AAR cues had a significant effect on participants’ subjective feedback towards the paintings. Sentiment Analysis shows that participants mentioned significantly more positive words from Enhanced AAR than the others. Enhanced AAR also made participants express a sense of immersion, whereas Basic AAR made them concentrate more on forlorn aspects of the paintings. Findings from this study suggest ways to improve and customize AAR cues for different painting styles, and indicate the need for multi-modal augmentations.

Are you talking to me? An Audio Augmented Reality conversational guide for cultural heritage

Augmented Reality (AR) technologies are increasingly utilized as a means of stimulating immersive experiences to cultural site visitors, mainly through visual superimposition of interactive digital elements onto the physical world. Recent research has investigated the use of Audio AR (AAR) in heritage sites, wherein visitors listen to spatially registered sound which could be attributed to ‘talking’ physical artefacts. A parallel trend in the audience engagement programs of cultural institutions involves the employment of AI chatbots which are engaged in dialogues with followers or visitors to provide meaningful responses to a number of user questions. Herein, we present Exhibot, an intelligent audio guide system aiming at enhancing the user experience of cultural site visitors. Exhibot involves the combination of AAR and chatbot technologies to enable natural visitor-exhibit interaction, while also leveraging IoT devices to contextualize the delivered information. The key contribution of the proposed system lies in the interplay of AAR, chatbot and IoT technologies to create immersive learning experiences in the context of an integrated cultural guide system. Exhibot has undergone field trials to validate its usability and utility in realistic operational conditions. As a case study, we have chosen the statue of a prominent politician situated at a central square in Heraklion, Greece. The evaluation results indicated a very positive attitude of users, which is attributed both to the sense of immersion evoked by the AAR-powered storytelling and the natural human-like conversation enabled by the chatbot.

Audio Augmented Reality in Smart Learning Environments

Image processing-based augmented reality (AR) is widely used in several fields. However, audio content is also crucial in certain cases, for example, focusing on appreciating artwork in a museum, rather than a virtually synthesized image. In this study, an attempt was made to provide ubiquitous learning (u-learning) services by using audio AR for smart cities where people do not need smartphones, tablets, or wearable devices for learning. The concept of “unmanned stores” was introduced in the proposed learning environments. A 360° camera was mounted on the ceiling of an exhibition room. An algorithm was used to detect moving objects and locate a visitor, from whose position an information-processing platform searched the showpiece nearest to the visitor and selected the corresponding audio guide. For multiple visitors, infrared (IR) sensors were used to count the visitors inside the room. A loudspeaker array was built to emit directional sounds for eliminating interference between multiple sound sources. Along with the proposed algorithm for increasing the accuracy while sensing multiple movements, the conventional sound field synthesis method was modified to improve the listening experience and learning quality. Image-based sensor technology, IR detection, and an audio signal-processing platform were integrated into a smart environment in which people can conduct u-learning without personal devices. The method can provide immersive audio tours with virtual narrators and peer-helpers in schools, museums, and art galleries. Qualitative and quantitative analyses indicated that effective hands-free and eyes-free learning was achieved.

Audio Augmented Reality: A Systematic Review of Technologies, Applications, and Future Research Directions

Audio Augmented Reality: A Systematic Review of Technologies, Applications, and Future Research Directions

Investigating the use of Spatialized Audio Augmented Reality to enhance the outdoor running experience

Recent years have seen increasing interest in the use of Virtual Reality technology to enhance the experience of physical exercise activities. When such technologies are used, they are traditionally employed to an indoor setting. In this study, we investigate the use of Spatialized Audio Augmented Reality (AAR) to allow users to compete against their past running records in real time. An experiment study was carried out where 84 participants used the AAR system to compete against a virtual opponent generated using (1) their previous best record, (2) an enhanced version of their previous best record, (3) the previous best record of another user in comparison to (4) a control group. The results showed that while the Augmented Reality conditions did significantly enhance the running performance of users, it did not outperform the traditional training experience to a significant degree. In terms of user experience, participants competing in AAR against their past best record and an enhanced version of their previous best record reported a higher level of tension but did not feel more challenge or competence. After reflecting upon the design of our system, we then highlight several insights regarding the use and design of AAR to promote running related exercise activities.

Extracting a Definition and Taxonomy for Audio Augmented Reality (AAR) Using Grounded Theory

The concept of Augmented Reality (AR) has evolved over the years since its inception in 1957. However, as of today, AR is mostly realized using a visual medium. Almost all applications of AR rely on the user’s ability to see. A few recent works have attempted to explore a new modality for AR–e.g., audio. However, the concept of using audio to augment reality has been considered debatable and there is no specific definition of the concept. To better understand this new concept, we launched a study using Grounded Theory to develop a definition and taxonomy for the concept of Audio Augmented Reality (AAR). This paper shares the preliminary results based on the activities conducted thus far, and hopes to generate discussion on using audio to augment reality within the Human Factors community.

Design Dimensions of Co-Located Multi-Device Audio Experiences

The widespread distribution of mobile computing presents new opportunities for the consumption of interactive and immersive media experiences using multiple connected devices. Tools now exist for the creation of these experiences; however, there is still limited understanding of the best design practices and use cases for the technology, especially in the context of audio experiences. In this study, the application space of co-located multi-device audio experiences is explored and documented through a review of the literature and a survey. Using the obtained information, a set of seven design dimensions that can be used to characterise and compare experiences of this type is proposed; these are synchronisation, context, position, relationship, interactivity, organisation, and distribution. A mapping of the current application space is presented where four categories are identified using the design dimensions, these are public performances, interactive music, augmented broadcasting, and social games. Finally, the overlap between co-located multi-device audio and audio-augmented reality (AAR) experiences is highlighted and discussed. This work will contribute to the wider discussion about the role of multiple devices in audio experiences and provide a source of reference for the design of future multi-device audio experiences.

Investigating the Use of Spatialized Audio Augmented Reality to Enhance the Outdoor Running Experience

Investigating the Use of Spatialized Audio Augmented Reality to Enhance the Outdoor Running Experience

Prediction and Controlling of Auditory Perception in Augmented Environments. A Loudness-Based Dynamic Mixing Technique

At the core of augmented reality audio (ARA) technology lies the ARA mix, a process responsible for the assignment of a virtual environment to a real one. Legacy ARA mix models have focused on the natural reproduction of the real environment, whereas the virtual environment is simply mixed through fixed gain methods. This study presents a novel approach of a dynamic ARA mix that facilitates a smooth adaptation of the virtual environment to the real one, as well as dynamic control of the virtual audio engine, by taking into account the inherent characteristics of both ARA technology and binaural auditory perception. A prototype feature extraction technique of auditory perception characteristics through a real-time binaural loudness prediction method was used to upgrade the legacy ARA mix model into a dynamic model, which was evaluated through benchmarks and subjective tests and showed encouraging results in terms of functionality and acceptance.

A constrained optimal hear-through filter design approach for earphones

The sound from environment will be altered after it transmits through headphones to the ear canal. A hear-though filter can be designed and implemented in the headphones to create a more natural hearing experience, i.e., offer a transparent mode for headphones. The design of hear-through filter is also required in some other applications, e.g., augmented reality audio. In this paper, a constrained hear-through filter design approach is proposed. It is firstly shown that the hear-through filter design problem can be formulated in a similar form to active noise control filter design in the frequency domain. One advantage of this design approach is that multiple practical constraints can be applied conveniently by formulating a constrained optimization problem. Then the constrained optimization problem for hear-through filter design is reformulated as cone programming problem which can be solved efficiently. The proposed design approach can also specify the desired delay of reproduced sound. The designed filter can be directly implemented in an active noise control system in the headphone such that the requirement for extra electronic components can be minimal.

Interactive Audio Augmented Reality in Participatory Performance

Interactive Audio Augmented Reality (AAR) facilitates collaborative storytelling and human interaction in participatory performance. Spatial audio enhances the auditory environment and supports real-time control of media content and the experience. Nevertheless, AAR applied to interactive performance practices remains under-explored. This study examines how audio human-computer interaction can prompt and support actions, and how AAR can contribute to developing new kinds of interactions in participatory performance.This study investigates an AAR participatory performance based on the theater and performance practice by theater maker Augusto Boal. It draws from aspects of multi-player audio-only games and interactive storytelling. A user experience study of the performance shows that people are engaged with interactive content and interact and navigate within the spatial audio content using their whole body. Asymmetric audio cues, playing distinctive content for each participant, prompt verbal and non-verbal communication. The performative aspect was well-received and participants took on roles and responsibilities within their group during the experience.

Materialising contexts: virtual soundscapes for real-world exploration

This article presents the results of a study based on a group of participants’ interactions with an experimental sound installation at the National Science and Media Museum in Bradford, UK. The installation used audio augmented reality to attach virtual sound sources to a vintage radio receiver from the museum’s collection, with a view to understanding the potentials of this technology for promoting exploration and engagement within museums and galleries. We employ a practice-based design ethnography, including a thematic analysis of our participants’ interactions with spatialised interactive audio, and present an identified sequence of interactional phases. We discuss how audio augmented artefacts can communicate and engage visitors beyond their traditional confines of line-of-sight, and how visitors can be drawn to engage further, beyond the realm of their original encounter. Finally, we provide evidence of how contextualised and embodied interactions, along with authentic audio reproduction, evoked personal memories associated with our museum artefact, and how this can promote interest in the acquisition of declarative knowledge. Additionally, through the adoption of a functional and theoretical aura-based model, we present ways in which this could be achieved, and, overall, we demonstrate a material object’s potential role as an interface for engaging users with, and contextualising, immaterial digital audio archival content.

A new audio augmented reality interaction and adaptation model for museum visits

Augmented Reality is a recent impressive technology applied in many areas. However, in tourism and especially in museums, where visual contact with exhibitions is fundamental, augmenting reality with graphical objects, or using graphical applications, can distract visitors’ attention from the main subject. Also, having a classical experience with no personalization or adaptation can make a negative sensation for the whole experience. In this paper, we introduce an adaptive gesture-based audio augmented reality system, in which content personalization is a key design for the interaction experience. The evaluation has been conducted in a real environment which was the Musée des arts et Métiers in Paris. This evaluation has confirmed the ease of use, as well as the usefulness and enjoyment impact of our model.

An Efficient Implementation of Parallel Parametric HRTF Models for Binaural Sound Synthesis in Mobile Multimedia

The extended use of mobile multimedia devices in applications like gaming, 3D video and audio reproduction, immersive teleconferencing, or virtual and augmented reality, is demanding efficient algorithms and methodologies. All these applications require real-time spatial audio engines with the capability of dealing with intensive signal processing operations while facing a number of constraints related to computational cost, latency and energy consumption. Most mobile multimedia devices include a Graphics Processing Unit (GPU) that is primarily used to accelerate video processing tasks, providing high computational capabilities due to its inherent parallel architecture. This paper describes a scalable parallel implementation of a real-time binaural audio engine for GPU-equipped mobile devices. The engine is based on a set of head-related transfer functions (HRTFs) modelled with a parametric parallel structure, allowing efficient synthesis and interpolation while reducing the size required for HRTF data storage. Several strategies to optimize the GPU implementation are evaluated over a well-known kind of processor present in a wide range of mobile devices. In this context, we analyze both the energy consumption and real-time capabilities of the system by exploring different GPU and CPU configuration alternatives. Moreover, the implementation has been conducted using the OpenCL framework, guarantying the portability of the code.

Audio Legends: Investigating Sonic Interaction in an Augmented Reality Audio Game

Augmented Reality Audio Games (ARAG) enrich the physical world with virtual sounds to express their content and mechanics. Existing ARAG implementations have focused on exploring the surroundings and navigating to virtual sound sources as the main mode of interaction. This paper suggests that gestural activity with a handheld device can realize complex modes of sonic interaction in the augmented environment, resulting in an enhanced immersive game experience. The ARAG “Audio Legends” was designed and tested to evaluate the usability and immersion of a system featuring an exploration phase based on auditory navigation, as well as an action phase, in which players aim at virtual sonic targets and wave the device to hit them or hold the device to block them. The results of the experiment provide evidence that players are easily accustomed to auditory navigation and that gestural sonic interaction is perceived as difficult, yet this does not affect negatively the system’s usability and players’ immersion. Findings also include indications that elements, such as sound design, the synchronization of sound and gesture, the fidelity of audio augmentation, and environmental conditions, also affect significantly the game experience, whereas background factors, such as age, sex, and game or music experience, do not have any critical impact.

Bridging Audio and Augmented Reality towards a new Generation of Serious Audio‑only Games

Abstract: Educational practices are constantly adjusting to technological advances, in order to improve their effectiveness in delivering knowledge and preparing students for the challenges of modern digital society. Electronic games and augmented reality environments are two such media that can shape powerful modes of interactive and immersive experiences. Audio games in particular, i.e. electronic games that utilise data sonification and audio interaction techniques to express all narrative and gameplay content, have been proven to enhance players’ skills, such as concentration, memory, fantasy, emotional response, perception, data management and cooperation. However, despite the promising research findings reported in literature, there has been no systematic approach in integrating systems based on audio interaction into formal education. This article suggests that audio games can be designed and implemented for the delivery of targeted curricula through an engaging learning experience, whereas fusing audio game mechanics into new interactive technologies, such as augmented reality environments, will further enhance the students’ immersiveness in the learning process. Towards establishing the theoretical framework for the design of educational audio games, as well as educational augmented reality audio games, the authors review existing approaches and argue that the inherent features of both audio interaction and augmented reality systems agree with the official educational goals formulated by the Greek Ministry of Education and the International Baccalaureate Institution. Then, they discuss four original projects in terms of sound and mechanics design to demonstrate that audio games can address a wide range of subjects, adopt cross‑curricular strategies, facilitate complex modes of interaction, and realise creative, collaborative and inclusive learning activities.