Multimodal Dialogue Research Articles

Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

The way we see the objects around us determines speech and gestures we use to refer to them. The gestures we produce structure our visual perception. The words we use have an influence on the way we see. In this manner, visual perception, language and gesture present multiple interactions between each other. The problem is global and has to be tackled as a whole in order to understand the complexity of reference phenomena and to deduce a formal model. This model may be useful for any kind of human–machine dialogue system that focuses on deep comprehension. We show how a referring act takes place in a contextual subset of objects. This subset is called ‘reference domain’ and is implicit. It can be deduced from a lot of clues. Among these clues are those which come from the visual context and those which come from the multimodal utterance. We present the ‘multimodal reference domain’ model that takes these clues into account and that can be exploited in a multimodal dialogue system when interpreting.

Human-GIS interaction issues in crisis response

Geospatial information systems (GIS) provide a central infrastructure for computer supported crisis management in terms of database, analytical models and visualisation tools, but the user interfaces of such systems are still hard to use, and do not address the special needs of crisis managers who often work in teams and make judgements and decisions under stress. This paper articulates the overall challenges for effective GIS interfaces to support crisis management in three dimensions: immediacy, relevance and sharing. These three requirements are addressed by an integrated approach, taking a human-GIS interaction perspective. To demonstrate the feasibility of this approach, we cite our prototype system, DAVE_G (Dialogue-Assisted Visual Environment for Geoinformaton), as an example. DAVE_G uses a large screen display to create a shared workspace among team members, and allows risk managers to interact with a GIS through natural multimodal (speech/gesture) dialogues.

Constraining User Response via Multimodal Dialog Interface

This paper presents the results of an experiment comparing two different designs of an automated dialog interface. We compare a multimodal design utilizing text displays coordinated with spoken prompts to a voice-only version of the same application. Our results show that the text-coordinated version is more efficient in terms of word recognition and number of out-of-grammar responses, and is equal to the voice-only version in terms of user satisfaction. We argue that this type of multimodal dialog interface effectively constrains user response to allow for better speech recognition without increasing cognitive load or compromising the naturalness of the interaction.

Generic Multimedia Multimodal Agents Paradigms and Their Dynamic Reconfiguration at the Architectural Level

The multimodal fusion for natural human-computer interaction involves complex intelligent architectures which are subject to the unexpected errors and mistakes of users. These architectures should react to events occurring simultaneously, and possibly redundantly, from different input media. In this paper, intelligent agent-based generic architectures for multimedia multimodal dialog protocols are proposed. Global agents are decomposed into their relevant components. Each element is modeled separately. The elementary models are then linked together to obtain the full architecture. The generic components of the application are then monitored by an agent-based expert system which can then perform dynamic changes in reconfiguration, adaptation, and evolution at the architectural level. For validation purposes, the proposed multiagent architectures and their dynamic reconfiguration are applied to practical examples, including a W3C application.

Generation and evaluation of user tailored responses in multimodal dialogue

When people engage in conversation, they tailor their utterances to their conversational partners, whether these partners are other humans or computational systems. This tailoring, or adaptation to the partner takes place in all facets of human language use, and is based on a mental model or a user model of the conversational partner. Such adaptation has been shown to improve listeners’ comprehension, their satisfaction with an interactive system, the efficiency with which they execute conversational tasks, and the likelihood of achieving higher level goals such as changing the listener’s beliefs and attitudes. We focus on one aspect of adaptation, namely the tailoring of the content of dialogue system utterances for the higher level processes of persuasion, argumentation and advice-giving. Our hypothesis is that algorithms that adapt content for these processes, according to a user model, will improve the usability, efficiency, and effectiveness of dialogue systems. We describe a multimodal dialogue system and algorithms for adaptive content selection based on multi-attribute decision theory. We demonstrate experimentally the improved efficacy of system responses through the use of user models to both tailor the content of system utterances and to manipulate their conciseness.

Generation and evaluation of user tailored responses in multimodal dialogue

AbstractWhen people engage in conversation, they tailor their utterances to their conversational partners, whether these partners are other humans or computational systems. This tailoring, or adaptation to the partner takes place in all facets of human language use, and is based on a mental model or a user model of the conversational partner. Such adaptation has been shown to improve listeners' comprehension, their satisfaction with an interactive system, the efficiency with which they execute conversational tasks, and the likelihood of achieving higher level goals such as changing the listener's beliefs and attitudes. We focus on one aspect of adaptation, namely the tailoring of the content of dialogue system utterances for the higher level processes of persuasion, argumentation and advice‐giving. Our hypothesis is that algorithms that adapt content for these processes, according to a user model, will improve the usability, efficiency, and effectiveness of dialogue systems. We describe a multimodal dialogue system and algorithms for adaptive content selection based on multi‐attribute decision theory. We demonstrate experimentally the improved efficacy of system responses through the use of user models to both tailor the content of system utterances and to manipulate their conciseness.

Large-scale software integration for spoken language and multimodal dialog systems

The development of large-scale dialog systems requires a flexible architecture model and adequate software support to cope with the challenge of presents a general framework for building integrated natural-language and multimodal dialog systems. Our approach relies on a distributed component model that enables flexible re-use and extension of existing software modules and is able to deal with a heterogeneous software environment. A practical result of our research is the development of a sophisticated integration platform, called MULTIPLATFORM, which is based on the proposed framework. This MULTIPLATFORM testbed has been used in various large and mid-size research projects to develop integrated system prototypes.

Using speech and dialogue for interactive TV navigation

Interaction techniques for interactive television (iTV) are currently complex and difficult to use for a wide-range of viewers. Few previous studies have dealt with the potential benefits of multimodal dialogue interaction in the context of iTV for the purpose of flexibility, usability, efficiency, and accessibility. This paper investigates the benefits of introducing speech and connected dialogue for iTV interaction, and presents a case study in which a prototype system was built allowing users to navigate the information space and control the operation of the TV by a speech-based natural language interface. The system was evaluated by analysing the user experience in five categories capturing essential aspects of iTV interaction: interaction style, information load, data access, effectiveness and initiative. Design considerations relevant for speech and dialogue information systems for TV interfaces also emerged from the analysis.

Toward cognitive robot companions

The talk will explore the challenges of developing a new generation of robots that would be acting along with humans in their daily life and sharing their space. Such a robot companion will have to exhibit the cognitive capacities which are necessary to interact efficiently with humans in changing situations and tasks. It will have to understand its spatial surroundings, to make decisions according to its own evaluation of its situation, and to interact closely with humans both physically and through multimodal dialogue. A robot companion must also adapt to its human partner and learn new skills to be able to respond to the humans' needs.

Evaluation and usability of multimodal spoken language dialogue systems

With the technical advances and market growth in the field, the issues of evaluation and usability of spoken language dialogue systems, unimodal as well as multimodal, are as crucial as ever. This paper discusses those issues by reviewing a series of European and US projects which have produced major results on evaluation and usability. Whereas significant progress has been made on unimodal spoken language dialogue systems evaluation and usability, the emergence of, among others, multimodal, mobile, and domain-oriented systems continues to pose entirely new challenges to research in evaluation and usability.

Resolution of referring expressions in a Korean multimodal dialogue system

Referring expressions in multimodal dialogues have different aspects compared to those in language-only dialogues. They often refer to the items signified by either a gesture or visual means. In this article we classify referring expressions into two types (i.e., a deictic reference and an anaphoric reference), and propose two general methods to resolve these referring expressions. One method is a simple mapping algorithm that can find items referred with/without pointing gestures on a screen. The other is the centering algorithm with a dual cache model, to which Walker's centering algorithm is extended for a multimodal dialogue system. The extended algorithm is appropriate for resolving various anaphoric references in a multimodal dialogue. In the experiments, the proposed system correctly resolved 376 out of 405 referring expressions in 40 dialogues (0.54 referring expressions per utterance) showing 92.84 percent correctness.

Situated multimodal interaction in S martK om

In this paper, we give a short overview of the S martK om system, a flexible and adaptive multimodal dialog system. The system uses a common dialog backbone to realize three interaction scenarios. Through the use of generic methods and knowledge sources for multimodal fusion and fission, we were able to develop a generic dialog system for flexible multimodal interaction.

The evaluation of an innovative intelligent thermostat interface

Usability trials were carried out with a simulation of an early version of an intelligent thermostat interface. The multimodal interaction dialog took place in speech, sound, graphics and touch. Us...

The evaluation of an innovative intelligent thermostat interface: universal usability and age differences

Usability trials were carried out with a simulation of an early version of an intelligent thermostat interface. The multimodal interaction dialog took place in speech, sound, graphics and touch. Usability of new concepts of interaction was assessed. Most user behavior and problems of use were equal for old and young subjects (between 25 and 73 years of age). Relative high usability was observed with the new interaction styles, in particular with the task-based dialog and with the suggestions given by the thermostat. Even the oldest subjects could accomplish complicated programming. Age-dependent differences in usability were found for certain interface details, such as finding a hidden menu item. The summarized typical "modern" interaction details, which caused problems, were significantly less usable for the older cohort (over 59 years of age), compared to the young (under 46 years of age).

An Evolutionary Learning Community Network: How Evolutionary Artscience Emerges Through Evolutionary Systems Design

What is the experience of creating a synergistic approach to arts and sciences practice in a learning community focused on the notion of sustainable development? In this article, I answer this question through an evolutionary approach to societal transformation. My social research inquiry integrates the arts and sciences, a learning and design community, sustainable development, and Internet networking. Codesigners created the conditions to explore in multimodal dialogue and engage, guide, and design the emergence of what I call evolutionary artscience through portfolios and a new systems design space-the ArtSciLAB (an online artscience laboratory). Eighteen individuals working in nine countries participated in the Evolutionary Learning Community Network. Our learning and design experience, guided by Evolutionary Systems Design inquiry supports General Evolution Theory. We guided the Evolutionary Learning Community Network and designed the ArtSciLAB to incubate localized knowledge for development in learning regions worldwide.

Universal remote control allowing natural language modality for television and multimedia searches and requests

The remote control unit supports multi-modal dialog with the user, through which the user can easily select programs for viewing or recording. The remote control houses a microphone into which the user can input natural language speech. The input speech is recognized and interpreted by a natural language parser that extracts the semantic content of the user's speech. The parser works in conjunction with an electronic program guide, through which the remote control system is able to ascertain what programs are available for viewing or recording and supply appropriate prompts to the user. In one embodiment, the remote control includes a touch screen display upon which the user may view prompts or make selections by pen input or tapping. Selections made on the touch screen automatically limit the context of the ongoing dialog between user and remote control, allowing the user to interact naturally with the unit. The remote control unit can control virtually any audio-video component, including those designed before the current technology. The remote control system can be packaged entirely within the remote control handheld unit, or components may be distributed in other systems attached to the user's multimedia equipment.

An interaction mechanism of multimodal dialogue systems

AbstractWe propose a mechanism to handle ellipsis/anaphora resolution effectively for a dialogue system with multimodal interfaces. This mechanism uses a new method of storing/searching through the history of a dialogue. We express the object referred to by the current utterance as the packed data of all the related terms appearing in the preceding utterances. We formalize the mechanism and present an evaluation of a multimodal guidance system that has this mechanism. Experimental results show that this mechanism achieves natural dialogue with high interaction. We also discuss a method of evaluating interactive spoken dialogue systems. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(11): 70–79, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.1165

A Spoken Dialogue robot, Named Wonder, to Aid Senior Citizens Who living Alone with Communication

In an aging society, many kinds of welfare apparatus have been developed with highly sophisticated technology in order to support seniors and their helpers physically. But mental care is also very important because Japanese seniors living alone are rather reluctant to participate in their local community. We have designed an information system using robotic pets terminal controlled by PC which can communicate with the seniors. It includes speech recognition, multi-modal dialogue control and telecommunication software. In this system, one can deliver daily information through the pets from a distant place such as local government office or welfare institution. At the same time, one can also watch safety of their lives without intruding their privacy. Now, we carried out an field experiment in Ikeda City, Osaka. And we evaluated our system at a welfare institution and confirmed its effectiveness.

A hierarchical tag-graph search scheme with layered grammar rules for spontaneous speech understanding

It has always been difficult for language understanding systems to handle spontaneous speech with satisfactory robustness, primarily due to such problems as the fragments, disfluencies, out-of-vocabulary words, and ill-formed sentence structures. Also, the search schemes used are usually not flexible enough in accepting different input linguistic units, and great efforts are therefore required when they are used with different acoustic front ends in different tasks, specially in multi-modal and multi-lingual systems. In this paper, a new hierarchical tag-graph-based search scheme for spontaneous speech understanding is proposed. This scheme is based on a layered hierarchy of grammar rules, and therefore can integrate all the statistical and rule-based knowledge including acoustic scores, language model scores and grammar rules into the search process. More robust speech understanding is thus achievable. In addition, this scheme can accept graphs of different linguistic units such as phonemes, syllables, characters, words, spotted keywords, or phrases as the input, thus compatible to different acoustic front ends and multi-modal and multi-lingual applications can be easily developed. This search scheme has been successfully applied to a multi-domain, multi-modal dialogue system.

A SEMANTIC INTERPRETER AND A COOPERATIVE RESPONSE GENERATOR FOR A ROBUST SPOKEN DIALOGUE SYSTEM

In this paper, we describe a semantic interpreter and a cooperative response generator for a multimodal dialogue system which consists of speech input, touch screen input, speech output and graphical output. The dialogue system understands spontaneous speech which has many ambiguous phenomena such as interjections, ellipses, inversions, repairs, unknown words and so on, and responses to user for the utterance. But some utterances fail to be analyzed. This is caused due to "misrecognition" with the speech recognizer, "incompleteness" with the semantic interpreter and "lack of database" with the response generator. Therefore, we improved the semantic interpreter to build a more robust one. If a user's query does not have enough conditions/information to answer the question with the system, the dialogue manager should query the user to get the necessary conditions or to select the candidate. Further, if the system cannot retrieve any information related to the user's question, the generator should propose an alternative plan. Based on these considerations, we developed a cooperative response generator in the dialogue system. We report the evaluation results of the semantic interpreter and cooperative response generator in our dialogue system.