Influencing factors on multimodal interaction during selection tasks

Abstract

When developing multimodal interactive systems it is not clear which importance should be given to which modality. In order to study influencing factors on multimodal interaction, we conducted a Wizard of Oz study on a basic recurrent task: 53 subjects performed diverse selections of objects on a screen. The way and modality of interaction was not specified nor predefined by the system, and the users were free in how and what to select. Natural input modalities like speech, gestures, touch, and arbitrary multimodal combinations of these were recorded as dependent variables. As independent variables, subjects’ gender, personality traits, and affinity towards technical devices were surveyed, as well as the system’s varying presentation styles of the selection. Our statistical analyses reveal gender as a momentous influencing factor and point out the role of individuality for the way of interaction, while the influence of the system output seems to be quite limited. This knowledge about the prevalent task of selection will be useful for designing effective and efficient multimodal interactive systems across a wide range of applications and domains.