Sonic interaction design for virtual and augmented reality environments

Abstract

Sonic Interaction Design (SID) primarily focuses on enhancing the quality of an interactive experience for the user by utilizing explicit techniques and defining advanced computational objects. While the diverse methods in SID are being documented, we argue for a need to refocus on the experiential, embodied, contextual, playful, and holistic qualities of sonic interactions. In this paper, we discuss specific SID techniques that can be used to advance story and gameplay in virtual and augmented environments. We shortly introduce the audio technologies used in virtual reality, and complement these with the practical examples about their (mis)use. We then illustrate the SID in the augmented environment of Leviathan, featured in Intel's 2014 CES keynote. We discuss methods such as defining characters through sound, embracing and critically using ambiguity, and coordinating sound and music. Through the notion of multimodal listening we provide guidelines to sonic interaction designers, so that they not only make their audience to listen in to digital content, but also listen up, out, through, and around.