An agent-based architecture for multimodal interaction

Abstract

In this paper, an executable generic process model is proposed for combined verbal and non-verbal communication processes and their interaction. The agent-based architecture can be used to create multimodal interaction. The generic process model has been designed, implemented and used to simulate different types of interaction between verbal and non-verbal communication processes: a non-verbal communication process can add and modify content to a verbal communication process, but can also provide a protocol for the (control of the) verbal communication process. With respect to the communication protocol both stimulus–response behaviour and deliberative behaviour have been modelled and simulated. The semantics of the model has been formalized by three-levelled partial temporal models, covering both the material and mental proceses and their relations.