The Observer Experiment: A View of the Dynamics of Multimodal Interaction

Abstract

Here we present and also apply the “observer gaze” methodology in order to study the structural organization of multimodal, face-to-face conversations. Tracking the gaze of a third party observer can provide an online and implicit method for investigating the dynamics of interactions and the relevant features that capture the observer’s attention according to the amount of information the observer has available. The way an outsider visually perceives the interactional exchanges may also suggest a better modeling of human-human communication for human-machine communication system. With these goals, ten observers participated in the present study, all being native speakers of Persian. The observers watched a short clip of a dyadic, spontaneous conversation in Persian. Using an eye tracker, the observers’ gaze was recorded in two different watching situations, namely: audio-visual, and visual-only. Afterwards, using ELAN, the observers’ gaze data were temporally aligned with verbal and nonverbal behavior of the interactants (such as head movements, hand gestures, body shift, facial expressions and turn-taking). First, the observers’ visual attention, measured as the amount of fixation time on the speaker and listener, was described across conditions. Second, the observers’ gaze was analyzed in presence of turn-transitions and we showed that they were able to a large extent to anticipate turns and also follow them depending on the conditions. In summary, the observers’ gazing behavior was affected by the conditions.