Multimodal Dialogue Research Articles

How to communicate smartly with your house?

We are involved in conceiving and implementing man machine interfaces allowing multimodal and situated dialogue in order to interact with smart digital environments. Several methods are presented depending on how and what the user can say. We describe our approach and some tracks to permit to end-users to communicate more naturally with their house, by speech or by gesture, for instance. Some applications based on workflow description are presented to show how designers are able to model their own man machine dialogue. Finally, this work is complemented with a Model Driven Engineering approach using Statecharts for a smart digital home.

New technologies, new multimodal literacy practices and young children's metacognitive development

This paper discusses concepts of learning through ‘collaborative multimodal dialogue’. It draws on an ESRC‐funded study (RES‐000‐22‐2451) investigating 3‐ and 4‐year‐old children's encounters with literacy as they engage with a range of printed and digital technologies at home and in a nursery. The study goes beyond analysis of spoken language, giving a more complete understanding of literacy learning processes through detailed analysis of how children use multiple communicative modes as they experience literacy in different media. These experiences underpin metacognitive development and are crucial to children's abilities to act strategically in future situations. Drawing on notions of literacy as social practice, this paper discusses how the advent of new technologies has introduced new dimensions into young children's literacy learning, the implications of which have not yet been fully recognised in early years policy guidance, training or practice.

User preferences can drive facial expressions: evaluating an embodied conversational agent in a recommender dialogue system

Tailoring the linguistic content of automatically generated descriptions to the preferences of a target user has been well demonstrated to be an effective way to produce higher-quality output that may even have a greater impact on user behaviour. It is known that the non-verbal behaviour of an embodied agent can have a significant effect on users' responses to content presented by that agent. However, to date no-one has examined the contribution of non-verbal behaviour to the effectiveness of user tailoring in automatically generated embodied output. We describe a series of experiments designed to address this question. We begin by introducing a multimodal dialogue system designed to generate descriptions and comparisons tailored to user preferences, and demonstrate that the user-preference tailoring is detectable to an overhearer when the output is presented as synthesised speech. We then present a multimodal corpus consisting of the annotated facial expressions used by a speaker to accompany the generated tailored descriptions, and verify that the most characteristic positive and negative expressions used by that speaker are identifiable when resynthesised on an artificial talking head. Finally, we combine the corpus-derived facial displays with the tailored descriptions to test whether the addition of the non-verbal channel improves users' ability to detect the intended tailoring, comparing two strategies for selecting the displays: one based on a simple corpus-derived rule, and one making direct use of the full corpus data. The performance of the subjects who saw displays selected by the rule-based strategy was not significantly different than that of the subjects who got only the linguistic content, while the subjects who saw the data-driven displays were significantly worse at detecting the correctly tailored output. We propose a possible explanation for this result, and also make recommendations for developers of future systems that may make use of an embodied agent to present user-tailored content.

Multimodal and mobile conversational Health and Fitness Companions

Multimodal conversational spoken dialogues using physical and virtual agents provide a potential interface to motivate and support users in the domain of health and fitness. This paper describes how such multimodal conversational Companions can be implemented to support their owners in various pervasive and mobile settings. We present concrete system architectures, virtual, physical and mobile multimodal interfaces, and interaction management techniques for such Companions. In particular how knowledge representation and separation of low-level interaction modelling from high-level reasoning at the domain level makes it possible to implement distributed, but still coherent, interaction with Companions. The distribution is enabled by using a dialogue plan to communicate information from domain level planner to dialogue management and from there to a separate mobile interface. The model enables each part of the system to handle the same information from its own perspective without containing overlapping logic, and makes it possible to separate task-specific and conversational dialogue management from each other. In addition to technical descriptions, results from the first evaluations of the Companions interfaces are presented.

Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation

This paper argues that the problems of dialogue management (DM) and Natural Language Generation (NLG) in dialogue systems are closely related and can be fruitfully treated statistically, in a joint optimisation framework such as that provided by Reinforcement Learning (RL). We first review recent results and methods in automatic learning of dialogue management strategies for spoken and multimodal dialogue systems, and then show how these techniques can also be used for the related problem of Natural Language Generation. This approach promises a number of theoretical and practical benefits such as fine-grained adaptation, generalisation, and automatic (global) optimisation, and we compare it to related work in statistical/trainable NLG. A demonstration of the proposed approach is then developed, showing combined DM and NLG policy learning for adaptive information presentation decisions. A joint DM and NLG policy learned in the framework shows a statistically significant 27% relative increase in reward over a baseline policy, which is learned in the same way only without the joint optimisation. We thereby show that that NLG problems can be approached statistically, in combination with dialogue management decisions, and we show how to jointly optimise NLG and DM using Reinforcement Learning.

A multi-modal dialogue analysis method for medical interviews based on design of interaction corpus

We propose a multi-modal dialogue analysis method for medical interviews that hierarchically interprets nonverbal interaction patterns in a bottom-up manner and simultaneously visualizes the topic structure. Our method aims to provide physicians with the clues generally overlooked by conventional dialogue analysis to form a cycle of dialogue practice and analysis. We introduce a motif and a pattern cluster in the designs of the hierarchical indices of interaction and exploit the Jensen---Shannon divergence (JSD) metric to reduce the number of usable indices. We applied the proposed interpretation method of interaction patterns to develop a corpus of interviews. The results of a summary reading experiment confirmed the validity of the developed indices. Finally, we discussed the integrated analysis of the topic structure and a nonverbal summary.

Reading multimodal translational webs Dialogue of the visual and the verbal in Vanity Fair's publication of Chronicle of a Death Foretold

This paper focuses on Gregory Rabassa's translation of García Márquez's Crónica de una muerte anunciada, published with Fernando Botero's illustrations in Vanity Fair in 1983. I analyse its emblematic title pages, showing how they create a remarkable continuum: an integrated whole of visual and verbal elements, strongly dependent on both semiotic codes but not reducible to the components of either. This is a special type of multimodal target text, since Rabassa's translation and Botero's images develop direct relationships with García Márquez's novela. Both contributions are second-order derivative texts, which deploy similar interpretants and adopt equally “displacing” or foreignizing strategies of translation vis-à-vis the context of publication. Their connection with the source text prepares the development of a subtle intersemiotic translational web, favouring internal coherence within the target text itself. To investigate an oggetto traduttivo of this kind, a broad notion of translation criticism is needed, open to external contributions from other research fields such as visual semiotics and rhetoric.

Natural interaction with a virtual guide in a virtual environment

This paper describes the Virtual Guide, a multimodal dialogue system represented by an embodied conversational agent that can help users to find their way in a virtual environment, while adapting its affective linguistic style to that of the user. We discuss the modular architecture of the system, and describe the entire loop from multimodal input analysis to multimodal output generation. We also describe how the Virtual Guide detects the level of politeness of the user’s utterances in real-time during the dialogue and aligns its own language to that of the user, using different politeness strategies. Finally we report on our first user tests, and discuss some potential extensions to improve the system.

Information enquiry kiosk with multimodal user interface

A multimodal interactive dialogue automaton (kiosk) for self-service is presented in the paper. Multimodal user interface allow people to interact with the kiosk by natural speech, gestures additionally to the standard input and output devices. Architecture of the kiosk contains key modules of speech processing and computer vision. An array of four microphones is applied for far-field capturing and recording of user’s speech commands, it allows the kiosk to detect voice activity, to localize sources of desired speech signals, and to eliminate environmental acoustical noises. A noise robust speaker-independent recognition system is applied to automatic interpretation and understanding of continuous Russian speech. The distant speech recognizer uses grammar of voice queries as well as garbage and silence models to improve recognition accuracy. Pair of portable video-cameras are applied for vision-based detection and tracking of user’s head and body position inside of the working area. Russian-speaking talking head serves both for bimodal audio-visual speech synthesis and for improvement of communication intelligibility by turning the head to an approaching client. Dialogue manager controls the flow of dialogue and synchronizes sub-modules for input modalities fusion and output modalities fission. The experiments made with the multimodal kiosk were directed to cognitive and usability studies of human-computer interaction by different communication means

Robust speech interaction in motorcycle environment

Aiming at robust spoken dialogue interaction in motorcycle environment, we investigate various configurations for a speech front-end, which consists of speech pre-processing, speech enhancement and speech recognition components. These components are implemented as agents in the Olympus/RavenClaw framework, which is the core of a multimodal dialogue interaction interface of a wearable solution for information support of the motorcycle police force on the move. In the present effort, aiming at optimizing the speech recognition performance, different experimental setups are considered for the speech front-end. The practical value of various speech enhancement techniques is assessed and, after analysis of their performances, a collaborative scheme is proposed. In this collaborative scheme independent speech enhancement channels operate in parallel on a common input and their outputs are fed to the multithread speech recognition component. The outcome of the speech recognition process is post-processed by an appropriate fusion technique, which contributes for a more accurate interpretation of the input. Investigating various fusion algorithms, we identified the Adaboost.M1 algorithm as the one performing best. Utilizing the fusion collaborative scheme based on the Adaboost.M1 algorithm, significant improvement of the overall speech recognition performance was achieved. This is expressed in terms of word recognition rate and correctly recognized words, as accuracy gain of 8.0% and 5.48%, respectively, when compared to the performance of the best speech enhancement channel, alone. The advance offered in the present work reaches beyond the specifics of the present application, and can be beneficial to spoken interfaces operating in non-stationary noise environments.

Modeling Multimodal Multitasking in a Smart House

This paper belongs to an ongoing series of papers presented in different conferences illustrating the results obtained from the analysis of the MIMUS corpus. This corpus is the result of a number of WoZ experiments conducted at the University of Seville as part of the TALK Project. The main objective of the MIMUS corpus was to gather information about different users and their performance, preferences and usage of a multimodal multilingual natural dialogue system in the Smart Home scenario. The focus group is composed by wheel-chair- bound users. In previous papers the corpus and all relevant information related to it has been analyzed in depth. In this paper, we will focus on multimodal multitasking during the experiments, that is, modeling how users may perform more than one task in parallel. These results may help us envision the importance of discriminating complementary vs. independent simultaneous events in multimodal systems. This gains more relevance when we take into account the likelihood of the co- occurrence of these events, and the fact that humans tend to multitask when they are sufficiently comfortable with the tools they are handling.

MushyPeek: A Framework for Online Investigation of Audiovisual Dialogue Phenomena

Evaluation of methods and techniques for conversational and multimodal spoken dialogue systems is complex, as is gathering data for the modeling and tuning of such techniques. This article describes MushyPeek, an experiment framework that allows us to manipulate the audiovisual behavior of interlocutors in a setting similar to face-to-face human-human dialogue. The setup connects two subjects to each other over a Voice over Internet Protocol (VoIP) telephone connection and simultaneously provides each of them with an avatar representing the other. We present a first experiment which inaugurates, exemplifies, and validates the framework. The experiment corroborates earlier findings on the use of gaze and head pose gestures in turn-taking.

Learning human multimodal dialogue strategies

AbstractWe investigate the use of different machine learning methods in combination with feature selection techniques to explore human multimodal dialogue strategies and the use of those strategies for automated dialogue systems. We learn policies from data collected in a Wizard-of-Oz study where different human ‘wizards’ decide whether to ask a clarification request in a multimodal manner or else to use speech alone. We first describe the data collection, the coding scheme and annotated corpus, and the validation of the multimodal annotations. We then show that there is a uniform multimodal dialogue strategy across wizards, which is based on multiple features in the dialogue context. These are generic features, available at runtime, which can be implemented in dialogue systems. Our prediction models (for human wizard behaviour) achieve a weighted f-score of 88.6 per cent (which is a 25.6 per cent improvement over the majority baseline). We interpret and discuss the learned strategy. We conclude that human wizard behaviour is not optimal for automatic dialogue systems, and argue for the use of automatic optimization methods, such as Reinforcement Learning. Throughout the investigation we also discuss the issues arising from using small initial Wizard-of-Oz data sets, and we show that feature engineering is an essential step when learning dialogue strategies from such limited data.

Automatic Detection of Disfluency Boundaries in Spontaneous Speech of Children Using Audio–Visual Information

The presence of disfluencies in spontaneous speech, while poses a challenge for robust automatic recognition, also offers means for gaining additional insights into understanding a speaker's communicative and cognitive state. This paper analyzes disfluencies in children's spontaneous speech, in the context of spoken dialog based computer game play, and addresses the automatic detection of disfluency boundaries. Although several approaches have been proposed to detect disfluencies in speech, relatively little work has been done to utilize visual information to improve the performance and robustness of the disfluency detection system. This paper describes the use of visual information along with prosodic and language information to detect the presence of disfluencies in a child's computer-directed speech and shows how these information sources can be integrated to increase the overall information available for disfluency detection. The experimental results on our children's multimodal dialog corpus indicate that disfluency detection accuracy of over 80% can be obtained by utilizing audio-visual information. Specifically, results showed that the addition of visual information to prosody and language features yield relative improvements in disfluency detection error rates of 3.6% and 6.3%, respectively, for information fusion at the feature level and decision level.

A Study in Efficiency and Modality Usage in Multimodal Form Filling Systems

The usage patterns of speech and visual input modes are investigated as a function of relative input mode efficiency for both desktop and personal digital assistant (PDA) working environments. For this purpose the form-filling part of a multimodal dialogue system is implemented and evaluated; three multimodal modes of interaction are implemented: ldquoClick-to-Talk,rdquo ldquoOpen-Mike,rdquo and ldquoModality-Selection.rdquo ldquoModality-Selectionrdquo implements an adaptive interface where the system selects the most efficient input mode at each turn, effectively alternating between a ldquoClick-to-Talkrdquo and ldquoOpen-Mikerdquo interaction style as proposed in ldquoModality tracking in the multimodal Bell Labs Communicator,rdquo in Proceedings of the Automatic Speech Recognition and Understanding Workshop, by A. Potamianos, , 2003. The multimodal systems are evaluated and compared with the unimodal systems. Objective and subjective measures used include task completion, task duration, turn duration, and overall user satisfaction. Turn duration is broken down into interaction time and inactivity time to better measure the efficiency of each input mode. Duration statistics and empirical probability density functions are computed as a function of interaction context and user. Results show that the multimodal systems outperform the unimodal systems in terms of objective and subjective criteria. Also, users tend to use the most efficient input mode at each turn; however, biases towards the default input modality and a general bias towards the speech modality also exists. Results demonstrate that although users exploit some of the available synergies in multimodal dialogue interaction, further efficiency gains can be achieved by designing adaptive interfaces that fully exploit these synergies.

Ontology-based information extraction and integration from heterogeneous data sources

In this paper we present the design, implementation and evaluation of SOBA, a system for ontology-based information extraction from heterogeneous data resources, including plain text, tables and image captions. SOBA is capable of processing structured information, text and image captions to extract information and integrate it into a coherent knowledge base. To establish coherence, SOBA interlinks the information extracted from different sources and detects duplicate information. The knowledge base produced by SOBA can then be used to query for information contained in the different sources in an integrated and seamless manner. Overall, this allows for advanced retrieval functionality by which questions can be answered precisely. A further distinguishing feature of the SOBA system is that it straightforwardly integrates deep and shallow natural language processing to increase robustness and accuracy. We discuss the implementation and application of the SOBA system within the SmartWeb multimodal dialog system. In addition, we present a thorough evaluation of the different components of the system. However, an end-to-end evaluation of the whole SmartWeb system is out of the scope of this paper and has been presented elsewhere by the SmartWeb consortium.

Middleware mechanisms for supporting multimodal interactions in smart environments

Smart and intelligent environments can proficiently benefit a new generation of multimodal dialog systems, which enable users to interact in natural ways like voice, gesture, facial motions, and so on. However, in order to make this possible, it is necessary to integrate such systems and services, possibly in a transparent way. In the present paper, we propose an architectural model for multimodal smart environments; then, an agent-based middleware infrastructure that provides basic components for the integration of multimodal interaction systems; and, a set of ontologies and rules for semantically handling interacting properties of devices and resources available in smart environments.

Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot

In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.

Information Seeking Spoken Dialogue Systems— Part II: Multimodal Dialogue

For pt.1see ibid., vol. 9, p. 3 (2007). In this paper, the task and user interface modules of a multimodal dialogue system development platform are presented. The main goal of this work is to provide a simple, application-independent solution to the problem of multimodal dialogue design for information seeking applications. The proposed system architecture clearly separates the task and interface components of the system. A task manager is designed and implemented that consists of two main submodules: the electronic form module that handles the list of attributes that have to be instantiated by the user, and the agenda module that contains the sequence of user and system tasks. Both the electronic forms and the agenda can be dynamically updated by the user. Next a spoken dialogue module is designed that implements the speech interface for the task manager. The dialogue manager can handle complex error correction and clarification user input, building on the semantics and pragmatic modules presented in Part I of this paper. The spoken dialogue system is evaluated for a travel reservation task of the DARPA Communicator research program and shown to yield over 90% task completion and good performance for both objective and subjective evaluation metrics. Finally, a multimodal dialogue system which combines graphical and speech interfaces, is designed, implemented and evaluated. Minor modifications to the unimodal semantic and pragmatic modules were required to build the multimodal system. It is shown that the multimodal system significantly outperforms the unimodal speech-only system both in terms of efficiency (task success and time to completion) and user satisfaction for a travel reservation task

Perspectives on Dialogue in the New Millennium (review)

Reviewed by: Perspectives on dialogue in the new millennium ed. by Peter Kühnlein, Hannes Rieser, and Henk Zeevat Eric McCready Perspectives on dialogue in the new millennium. Ed. by Peter Kühnlein, Hannes Rieser, and Henk Zeevat. (Pragmatics & beyond new series 114.) Amsterdam: John Benjamins, 2003. Pp. xii, 395. ISBN 0922842X. $150 (Hb). This volume collects nineteen papers selected from the BI-DIALOG workshop on dialogue held at the University of Bielefeld in 2001, along with an introduction by the editors. Most of the papers are concerned with issues relating to the semantics and pragmatics of dialogue, though many of the problems and solutions raised are also relevant to other contexts, such as single-author discourse and individual sentences. A wide range of approaches to dialogue is represented, but the primary focus is on semantics, pragmatics, and computational linguistics. Alex Lascarides and Nicholas Asher are concerned with the interpretation of imperatives and their interaction with discourse context. Jonathan Ginzburg, Ivan Sag, and Matthew Purver provide a means of integrating conversational moves into a grammar for dialogue within the type hierarchy of an HPSG framework, within which they propose a semantics for interjections. Along somewhat similar lines, Claudia Sassen proposes an HPSG treatment of illocutionary acts based on the propositional logic for speech acts proposed in John Searle and Daniel Vanderveken’s Foundations of illocutionary logic (Cambridge: Cambridge University Press, 1985). Continuing the discussion of speech acts, Rob van der Sandt analyzes denial acts, taken to be presuppositional, within a logical system based on his DRT-based system for presupposition projection. Jennifer Spenader also discusses presupposition; she examines the literature on bridging definite descriptions and argues that linguistic context should be given a larger role in their analysis. The last paper on presupposition, by Alessandro Capone, presents a critical discussion of existing analyses of presupposition as a prologue to an analysis of Italian presuppositional clitics. Next, Etsuko Oishi uses speech-act related distinctions made by J. L. Austin to analyze generics, referential and attributive uses of definite descriptions, and ‘aboutness’. Marina Terkourafi’s paper is also about pragmatic issues, analyzing politeness as stemming from particularized implicature and properties of the discourse context. William Mann’s contribution is a survey of different models of intention, focusing on attributes of intentions that have been proposed in the literature. Jörn Kreutel and Colin Matheson’s contribution begins the computationally oriented section of the book. They consider dialogue acts in terms of update of information states. Kersten Fischer uses an experiment in human-robot communication to determine what elements of the common ground are drawn on most by speakers in communication with unfamiliar partners. Alois Knoll discusses a multimodal dialogue system for communicating with robot systems. Oliver Lemon, Anne Bracy, Alexander Gruenstein, and Stanley Peters discuss a multimodal system for interaction between humans and mobile robots that also makes use of dynamic information state models. Berne Ludwig presents a dialogue modeling system that incorporates information about participant beliefs and intentions and supports [End Page 687] belief update of participants and inferences about discourse relations. Negotiative dialogue is the topic of Robin Cooper, Stina Ericsson, Staffan Larsson, and Ian Lewin’s contribution, which points out difficulties with earlier analyses of such dialogue and uses the Questions Under Discussion model of information states to remedy them. The implementation-oriented part of the book closes with the contribution of David Schlangen, Alex Lascarides, and Ann Copestake, which describes a component of dialogue systems that resolves underspecification in speech-act types, goals, and the denotation of temporal expressions. The last part of the book returns to more purely theoretical concerns. Anton Benz’s paper uses a formulation of bidirectional optimality theory to model epistemic contexts and expectations, which are shown to play a role in anaphora resolution. Elena Karagjosova analyzes German modal particles as marking a speaker...