Multimodal Dialogue System Research Articles

SIMONA – the Slovak embodied conversational agent

The proposed paper deals with the design and development of SIMONA – the Slovak embodied conversational agent. It was created as an extension of a multimodal dialogue system with the BML realizer Elckerlyc as an embodied agent. Elckerlyc is a BML com

User Localization During Human-Robot Interaction

This paper presents a user localization system based on the fusion of visual information and sound source localization, implemented on a social robot called Maggie. One of the main requisites to obtain a natural interaction between human-human and human-robot is an adequate spatial situation between the interlocutors, that is, to be orientated and situated at the right distance during the conversation in order to have a satisfactory communicative process. Our social robot uses a complete multimodal dialog system which manages the user-robot interaction during the communicative process. One of its main components is the presented user localization system. To determine the most suitable allocation of the robot in relation to the user, a proxemic study of the human-robot interaction is required, which is described in this paper. The study has been made with two groups of users: children, aged between 8 and 17, and adults. Finally, at the end of the paper, experimental results with the proposed multimodal dialog system are presented.

Building Autonomous Sensitive Artificial Listeners

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

User preferences can drive facial expressions: evaluating an embodied conversational agent in a recommender dialogue system

Tailoring the linguistic content of automatically generated descriptions to the preferences of a target user has been well demonstrated to be an effective way to produce higher-quality output that may even have a greater impact on user behaviour. It is known that the non-verbal behaviour of an embodied agent can have a significant effect on users' responses to content presented by that agent. However, to date no-one has examined the contribution of non-verbal behaviour to the effectiveness of user tailoring in automatically generated embodied output. We describe a series of experiments designed to address this question. We begin by introducing a multimodal dialogue system designed to generate descriptions and comparisons tailored to user preferences, and demonstrate that the user-preference tailoring is detectable to an overhearer when the output is presented as synthesised speech. We then present a multimodal corpus consisting of the annotated facial expressions used by a speaker to accompany the generated tailored descriptions, and verify that the most characteristic positive and negative expressions used by that speaker are identifiable when resynthesised on an artificial talking head. Finally, we combine the corpus-derived facial displays with the tailored descriptions to test whether the addition of the non-verbal channel improves users' ability to detect the intended tailoring, comparing two strategies for selecting the displays: one based on a simple corpus-derived rule, and one making direct use of the full corpus data. The performance of the subjects who saw displays selected by the rule-based strategy was not significantly different than that of the subjects who got only the linguistic content, while the subjects who saw the data-driven displays were significantly worse at detecting the correctly tailored output. We propose a possible explanation for this result, and also make recommendations for developers of future systems that may make use of an embodied agent to present user-tailored content.

Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation

This paper argues that the problems of dialogue management (DM) and Natural Language Generation (NLG) in dialogue systems are closely related and can be fruitfully treated statistically, in a joint optimisation framework such as that provided by Reinforcement Learning (RL). We first review recent results and methods in automatic learning of dialogue management strategies for spoken and multimodal dialogue systems, and then show how these techniques can also be used for the related problem of Natural Language Generation. This approach promises a number of theoretical and practical benefits such as fine-grained adaptation, generalisation, and automatic (global) optimisation, and we compare it to related work in statistical/trainable NLG. A demonstration of the proposed approach is then developed, showing combined DM and NLG policy learning for adaptive information presentation decisions. A joint DM and NLG policy learned in the framework shows a statistically significant 27% relative increase in reward over a baseline policy, which is learned in the same way only without the joint optimisation. We thereby show that that NLG problems can be approached statistically, in combination with dialogue management decisions, and we show how to jointly optimise NLG and DM using Reinforcement Learning.

Natural interaction with a virtual guide in a virtual environment

This paper describes the Virtual Guide, a multimodal dialogue system represented by an embodied conversational agent that can help users to find their way in a virtual environment, while adapting its affective linguistic style to that of the user. We discuss the modular architecture of the system, and describe the entire loop from multimodal input analysis to multimodal output generation. We also describe how the Virtual Guide detects the level of politeness of the user’s utterances in real-time during the dialogue and aligns its own language to that of the user, using different politeness strategies. Finally we report on our first user tests, and discuss some potential extensions to improve the system.

Modeling Multimodal Multitasking in a Smart House

This paper belongs to an ongoing series of papers presented in different conferences illustrating the results obtained from the analysis of the MIMUS corpus. This corpus is the result of a number of WoZ experiments conducted at the University of Seville as part of the TALK Project. The main objective of the MIMUS corpus was to gather information about different users and their performance, preferences and usage of a multimodal multilingual natural dialogue system in the Smart Home scenario. The focus group is composed by wheel-chair- bound users. In previous papers the corpus and all relevant information related to it has been analyzed in depth. In this paper, we will focus on multimodal multitasking during the experiments, that is, modeling how users may perform more than one task in parallel. These results may help us envision the importance of discriminating complementary vs. independent simultaneous events in multimodal systems. This gains more relevance when we take into account the likelihood of the co- occurrence of these events, and the fact that humans tend to multitask when they are sufficiently comfortable with the tools they are handling.

MushyPeek: A Framework for Online Investigation of Audiovisual Dialogue Phenomena

Evaluation of methods and techniques for conversational and multimodal spoken dialogue systems is complex, as is gathering data for the modeling and tuning of such techniques. This article describes MushyPeek, an experiment framework that allows us to manipulate the audiovisual behavior of interlocutors in a setting similar to face-to-face human-human dialogue. The setup connects two subjects to each other over a Voice over Internet Protocol (VoIP) telephone connection and simultaneously provides each of them with an avatar representing the other. We present a first experiment which inaugurates, exemplifies, and validates the framework. The experiment corroborates earlier findings on the use of gaze and head pose gestures in turn-taking.

A Study in Efficiency and Modality Usage in Multimodal Form Filling Systems

The usage patterns of speech and visual input modes are investigated as a function of relative input mode efficiency for both desktop and personal digital assistant (PDA) working environments. For this purpose the form-filling part of a multimodal dialogue system is implemented and evaluated; three multimodal modes of interaction are implemented: ldquoClick-to-Talk,rdquo ldquoOpen-Mike,rdquo and ldquoModality-Selection.rdquo ldquoModality-Selectionrdquo implements an adaptive interface where the system selects the most efficient input mode at each turn, effectively alternating between a ldquoClick-to-Talkrdquo and ldquoOpen-Mikerdquo interaction style as proposed in ldquoModality tracking in the multimodal Bell Labs Communicator,rdquo in Proceedings of the Automatic Speech Recognition and Understanding Workshop, by A. Potamianos, , 2003. The multimodal systems are evaluated and compared with the unimodal systems. Objective and subjective measures used include task completion, task duration, turn duration, and overall user satisfaction. Turn duration is broken down into interaction time and inactivity time to better measure the efficiency of each input mode. Duration statistics and empirical probability density functions are computed as a function of interaction context and user. Results show that the multimodal systems outperform the unimodal systems in terms of objective and subjective criteria. Also, users tend to use the most efficient input mode at each turn; however, biases towards the default input modality and a general bias towards the speech modality also exists. Results demonstrate that although users exploit some of the available synergies in multimodal dialogue interaction, further efficiency gains can be achieved by designing adaptive interfaces that fully exploit these synergies.

Ontology-based information extraction and integration from heterogeneous data sources

In this paper we present the design, implementation and evaluation of SOBA, a system for ontology-based information extraction from heterogeneous data resources, including plain text, tables and image captions. SOBA is capable of processing structured information, text and image captions to extract information and integrate it into a coherent knowledge base. To establish coherence, SOBA interlinks the information extracted from different sources and detects duplicate information. The knowledge base produced by SOBA can then be used to query for information contained in the different sources in an integrated and seamless manner. Overall, this allows for advanced retrieval functionality by which questions can be answered precisely. A further distinguishing feature of the SOBA system is that it straightforwardly integrates deep and shallow natural language processing to increase robustness and accuracy. We discuss the implementation and application of the SOBA system within the SmartWeb multimodal dialog system. In addition, we present a thorough evaluation of the different components of the system. However, an end-to-end evaluation of the whole SmartWeb system is out of the scope of this paper and has been presented elsewhere by the SmartWeb consortium.

Middleware mechanisms for supporting multimodal interactions in smart environments

Smart and intelligent environments can proficiently benefit a new generation of multimodal dialog systems, which enable users to interact in natural ways like voice, gesture, facial motions, and so on. However, in order to make this possible, it is necessary to integrate such systems and services, possibly in a transparent way. In the present paper, we propose an architectural model for multimodal smart environments; then, an agent-based middleware infrastructure that provides basic components for the integration of multimodal interaction systems; and, a set of ontologies and rules for semantically handling interacting properties of devices and resources available in smart environments.

Information Seeking Spoken Dialogue Systems— Part II: Multimodal Dialogue

For pt.1see ibid., vol. 9, p. 3 (2007). In this paper, the task and user interface modules of a multimodal dialogue system development platform are presented. The main goal of this work is to provide a simple, application-independent solution to the problem of multimodal dialogue design for information seeking applications. The proposed system architecture clearly separates the task and interface components of the system. A task manager is designed and implemented that consists of two main submodules: the electronic form module that handles the list of attributes that have to be instantiated by the user, and the agenda module that contains the sequence of user and system tasks. Both the electronic forms and the agenda can be dynamically updated by the user. Next a spoken dialogue module is designed that implements the speech interface for the task manager. The dialogue manager can handle complex error correction and clarification user input, building on the semantics and pragmatic modules presented in Part I of this paper. The spoken dialogue system is evaluated for a travel reservation task of the DARPA Communicator research program and shown to yield over 90% task completion and good performance for both objective and subjective evaluation metrics. Finally, a multimodal dialogue system which combines graphical and speech interfaces, is designed, implemented and evaluated. Minor modifications to the unimodal semantic and pragmatic modules were required to build the multimodal system. It is shown that the multimodal system significantly outperforms the unimodal speech-only system both in terms of efficiency (task success and time to completion) and user satisfaction for a travel reservation task

Perspectives on Dialogue in the New Millennium (review)

Reviewed by: Perspectives on dialogue in the new millennium ed. by Peter Kühnlein, Hannes Rieser, and Henk Zeevat Eric McCready Perspectives on dialogue in the new millennium. Ed. by Peter Kühnlein, Hannes Rieser, and Henk Zeevat. (Pragmatics & beyond new series 114.) Amsterdam: John Benjamins, 2003. Pp. xii, 395. ISBN 0922842X. $150 (Hb). This volume collects nineteen papers selected from the BI-DIALOG workshop on dialogue held at the University of Bielefeld in 2001, along with an introduction by the editors. Most of the papers are concerned with issues relating to the semantics and pragmatics of dialogue, though many of the problems and solutions raised are also relevant to other contexts, such as single-author discourse and individual sentences. A wide range of approaches to dialogue is represented, but the primary focus is on semantics, pragmatics, and computational linguistics. Alex Lascarides and Nicholas Asher are concerned with the interpretation of imperatives and their interaction with discourse context. Jonathan Ginzburg, Ivan Sag, and Matthew Purver provide a means of integrating conversational moves into a grammar for dialogue within the type hierarchy of an HPSG framework, within which they propose a semantics for interjections. Along somewhat similar lines, Claudia Sassen proposes an HPSG treatment of illocutionary acts based on the propositional logic for speech acts proposed in John Searle and Daniel Vanderveken’s Foundations of illocutionary logic (Cambridge: Cambridge University Press, 1985). Continuing the discussion of speech acts, Rob van der Sandt analyzes denial acts, taken to be presuppositional, within a logical system based on his DRT-based system for presupposition projection. Jennifer Spenader also discusses presupposition; she examines the literature on bridging definite descriptions and argues that linguistic context should be given a larger role in their analysis. The last paper on presupposition, by Alessandro Capone, presents a critical discussion of existing analyses of presupposition as a prologue to an analysis of Italian presuppositional clitics. Next, Etsuko Oishi uses speech-act related distinctions made by J. L. Austin to analyze generics, referential and attributive uses of definite descriptions, and ‘aboutness’. Marina Terkourafi’s paper is also about pragmatic issues, analyzing politeness as stemming from particularized implicature and properties of the discourse context. William Mann’s contribution is a survey of different models of intention, focusing on attributes of intentions that have been proposed in the literature. Jörn Kreutel and Colin Matheson’s contribution begins the computationally oriented section of the book. They consider dialogue acts in terms of update of information states. Kersten Fischer uses an experiment in human-robot communication to determine what elements of the common ground are drawn on most by speakers in communication with unfamiliar partners. Alois Knoll discusses a multimodal dialogue system for communicating with robot systems. Oliver Lemon, Anne Bracy, Alexander Gruenstein, and Stanley Peters discuss a multimodal system for interaction between humans and mobile robots that also makes use of dynamic information state models. Berne Ludwig presents a dialogue modeling system that incorporates information about participant beliefs and intentions and supports [End Page 687] belief update of participants and inferences about discourse relations. Negotiative dialogue is the topic of Robin Cooper, Stina Ericsson, Staffan Larsson, and Ian Lewin’s contribution, which points out difficulties with earlier analyses of such dialogue and uses the Questions Under Discussion model of information states to remedy them. The implementation-oriented part of the book closes with the contribution of David Schlangen, Alex Lascarides, and Ann Copestake, which describes a component of dialogue systems that resolves underspecification in speech-act types, goals, and the denotation of temporal expressions. The last part of the book returns to more purely theoretical concerns. Anton Benz’s paper uses a formulation of bidirectional optimality theory to model epistemic contexts and expectations, which are shown to play a role in anaphora resolution. Elena Karagjosova analyzes German modal particles as marking a speaker...

Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

The way we see the objects around us determines speech and gestures we use to refer to them. The gestures we produce structure our visual perception. The words we use have an influence on the way we see. In this manner, visual perception, language and gesture present multiple interactions between each other. The problem is global and has to be tackled as a whole in order to understand the complexity of reference phenomena and to deduce a formal model. This model may be useful for any kind of human–machine dialogue system that focuses on deep comprehension. We show how a referring act takes place in a contextual subset of objects. This subset is called ‘reference domain’ and is implicit. It can be deduced from a lot of clues. Among these clues are those which come from the visual context and those which come from the multimodal utterance. We present the ‘multimodal reference domain’ model that takes these clues into account and that can be exploited in a multimodal dialogue system when interpreting.

Generation and evaluation of user tailored responses in multimodal dialogue

When people engage in conversation, they tailor their utterances to their conversational partners, whether these partners are other humans or computational systems. This tailoring, or adaptation to the partner takes place in all facets of human language use, and is based on a mental model or a user model of the conversational partner. Such adaptation has been shown to improve listeners’ comprehension, their satisfaction with an interactive system, the efficiency with which they execute conversational tasks, and the likelihood of achieving higher level goals such as changing the listener’s beliefs and attitudes. We focus on one aspect of adaptation, namely the tailoring of the content of dialogue system utterances for the higher level processes of persuasion, argumentation and advice-giving. Our hypothesis is that algorithms that adapt content for these processes, according to a user model, will improve the usability, efficiency, and effectiveness of dialogue systems. We describe a multimodal dialogue system and algorithms for adaptive content selection based on multi-attribute decision theory. We demonstrate experimentally the improved efficacy of system responses through the use of user models to both tailor the content of system utterances and to manipulate their conciseness.

Large-scale software integration for spoken language and multimodal dialog systems

The development of large-scale dialog systems requires a flexible architecture model and adequate software support to cope with the challenge of presents a general framework for building integrated natural-language and multimodal dialog systems. Our approach relies on a distributed component model that enables flexible re-use and extension of existing software modules and is able to deal with a heterogeneous software environment. A practical result of our research is the development of a sophisticated integration platform, called MULTIPLATFORM, which is based on the proposed framework. This MULTIPLATFORM testbed has been used in various large and mid-size research projects to develop integrated system prototypes.

Generation and evaluation of user tailored responses in multimodal dialogue

AbstractWhen people engage in conversation, they tailor their utterances to their conversational partners, whether these partners are other humans or computational systems. This tailoring, or adaptation to the partner takes place in all facets of human language use, and is based on a mental model or a user model of the conversational partner. Such adaptation has been shown to improve listeners' comprehension, their satisfaction with an interactive system, the efficiency with which they execute conversational tasks, and the likelihood of achieving higher level goals such as changing the listener's beliefs and attitudes. We focus on one aspect of adaptation, namely the tailoring of the content of dialogue system utterances for the higher level processes of persuasion, argumentation and advice‐giving. Our hypothesis is that algorithms that adapt content for these processes, according to a user model, will improve the usability, efficiency, and effectiveness of dialogue systems. We describe a multimodal dialogue system and algorithms for adaptive content selection based on multi‐attribute decision theory. We demonstrate experimentally the improved efficacy of system responses through the use of user models to both tailor the content of system utterances and to manipulate their conciseness.

Evaluation and usability of multimodal spoken language dialogue systems

With the technical advances and market growth in the field, the issues of evaluation and usability of spoken language dialogue systems, unimodal as well as multimodal, are as crucial as ever. This paper discusses those issues by reviewing a series of European and US projects which have produced major results on evaluation and usability. Whereas significant progress has been made on unimodal spoken language dialogue systems evaluation and usability, the emergence of, among others, multimodal, mobile, and domain-oriented systems continues to pose entirely new challenges to research in evaluation and usability.

Resolution of referring expressions in a Korean multimodal dialogue system

Referring expressions in multimodal dialogues have different aspects compared to those in language-only dialogues. They often refer to the items signified by either a gesture or visual means. In this article we classify referring expressions into two types (i.e., a deictic reference and an anaphoric reference), and propose two general methods to resolve these referring expressions. One method is a simple mapping algorithm that can find items referred with/without pointing gestures on a screen. The other is the centering algorithm with a dual cache model, to which Walker's centering algorithm is extended for a multimodal dialogue system. The extended algorithm is appropriate for resolving various anaphoric references in a multimodal dialogue. In the experiments, the proposed system correctly resolved 376 out of 405 referring expressions in 40 dialogues (0.54 referring expressions per utterance) showing 92.84 percent correctness.

Situated multimodal interaction in S martK om

In this paper, we give a short overview of the S martK om system, a flexible and adaptive multimodal dialog system. The system uses a common dialog backbone to realize three interaction scenarios. Through the use of generic methods and knowledge sources for multimodal fusion and fission, we were able to develop a generic dialog system for flexible multimodal interaction.