Multimodal cooperation with the DenK system

Abstract

In this chapter we present the DenK project, a long-term effort where the aim is to build a generic cooperative human-computer interface combining multiple input and output modalities. We discuss the view on human-computer interaction that underlies the project and the emerging DenK system. The project integrates results from fundamental research in knowledge representation, communication, natural language semantics and pragmatics, and object-oriented animation. Central stage in the project is occupied by the design of a cooperative and knowledge-able electronic assistant that communicates in natural language and that has internal access to an application domain which is presented visually to the user. The assistant, that we call the Cooperative Assistant, has an information state that is represented in a rich form of type theory, a formalism that enables us to model the inherent cognitive dynamics of a dialogue participant. This formalism is used both for modeling domain knowledge, for representing the current dialogue context, and for implementing a context-change theory of communication.