Room-aware portable Auditory Augmented Reality: Real-time spatial audio generation with geometric data analysis

Abstract

This study proposes an innovative and practical approach to spatial acoustic rendering for Auditory Augmented Reality (AAR) in small room usage. AAR aims to enhance user experiences by overlaying spatially appropriate sounds in the real environment, creating the illusion that virtual sounds coexist in the physical world. However, current technological constraints in hardware and AAR rendering present challenges in creating a truly immersive sensation of “being there.” To address these challenges, we present a dynamic processing framework for a typical mobile AAR setting. This framework combines a game-audio engine with 3-D visual computation both specialized for real-time processing, providing a plausible spatial impression with lightweight computation. The process involves capturing room information by the Light Detection And Ranging (LiDAR) sensor mounted on the device, followed by estimating its dominant acoustic materials. The generated data will be used to compute dynamic early reflections and late reverberation. Early reflections are calculated up to the fourth reflection using the image source method, considering the acoustic materials. Late reverberation is generated using the Feedback Delay Network (FDN), with room frequency characteristics and spatial directivity varying relative to the listener's position. All components will be represented in fifth-order Ambisonics and binauralized with head-tracking. Notably, this dynamic rendering method could be light enough to handle all processing on a single mobile device.