Quantifying User Behaviour in Multisensory Immersive Experiences

Abstract

With the advent of standalone virtual reality (VR) headsets, multisensory VR experiences are emerging with the hopes of better simulating real-world experiences. For example, innovations in haptic suits and scent diffusion devices are burgeoning. While stimulating multiple senses re-orientates the perceived quality of experience (QoE) by increasing factors such as realism, presence and immersion, the impact it has on user behaviour has yet to be fully quantified. Advances in wearables and instrumented VR headsets, however, have allowed for such user behaviours to be easily measured in real-time. In this paper, we describe a pilot experiment in which participants play a custom-developed VR game under two conditions: (1) conventional audio-visual and (2) multisensory, where haptic feedback is provided via a haptic sleeve and olfaction is enabled via a scent diffusion add-on for the VR headset. We describe a developed instrumented VR headset capable of measuring electroencephalography (EEG), electrocardiography (ECG), and electro-oculography (EOG) signals in real-time. From these signals, metrics of mental workload, engagement, and attention are extracted. We report changes seen in these behavioural measures between the two conditions, thus providing insights on factors driving the improved QoE seen with multisensory experiences.