Can you Hear the Colour? Designing Virtual Worlds for Synaesthetic and Multimodal Experiences

Abstract

Abstract Synaesthesia is a phenomenon where sensory ‘fusion’ occurs resulting in, for example, ‘seeing’ music or ‘hearing’ colours, making it of great interest in human–computer interaction for creating new or enhanced experiences and interactions in virtual worlds. In virtual reality, research has mainly focused on evaluating advanced graphics and capturing immersion levels and user experience within ‘typical’ and ‘expected’ interactions. This paper investigates how multimodal design characteristics can lay the foundations to a more ‘synaesthetic’ design approach in mixed reality to identify how ‘atypical’ interactions can also affect user experience. Twenty participants completed a maze activity, emotion and immersion surveys and interviews. Analysis on expressed sentiments and correlations suggest that space, timing and user location qualitatively interplay in determining these. Design approaches need to take into consideration a more holistic design model to enrich mixed reality with ‘fused’ user experience accounting for space, time, artefacts, location, physicality, virtuality, cognition and sentiments.