Natural Language Dialogue Research Articles

Dynamic Bayesian networks and variable length genetic algorithm for designing cue-based model for dialogue act recognition

The automatic recognition of dialogue act is a task of crucial importance for the processing of natural language dialogue at discourse level. It is also one of the most challenging problems as most often the dialogue act is not expressed directly in speaker’s utterance. In this paper, a new cue-based model for dialogue act recognition is presented. The model is, essentially, a dynamic Bayesian network induced from manually annotated dialogue corpus via dynamic Bayesian machine learning algorithms. Furthermore, the dynamic Bayesian network’s random variables are constituted from sets of lexical cues selected automatically by means of a variable length genetic algorithm, developed specifically for this purpose. To evaluate the proposed approaches of design, three stages of experiments have been conducted. In the initial stage, the dynamic Bayesian network model is constructed using sets of lexical cues selected manually from the dialogue corpus. The model is evaluated against two previously proposed models and the results confirm the potentiality of dynamic Bayesian networks for dialogue act recognition. In the second stage, the developed variable length genetic algorithm is used to select different sets of lexical cues to constitute the dynamic Bayesian networks’ random variables. The developed approach is evaluated against some of the previously used ranking approaches and the results provide experimental evidences on its ability to avoid the drawbacks of the ranking approaches. In the third stage, the dynamic Bayesian networks model is constructed using random variables constituted from the sets of lexical cues generated in the second stage and the results confirm the effectiveness of the proposed approaches for designing dialogue act recognition model.

Developing Natural Language Enabled Games in SCXML

The World Wide Web Consortium (W3C) has a standard in the pipeline – called SCXML – that may turn out to be very suitable for the design and implementation of games, in particular games featuring (possibly multimodal) natural language dialogue. We see three main reasons why SCXML may be a good fit for the game industry: 1) SCXML is all about statecharts – a powerful extension of finite-state machines – and we argue that statecharts has the right kind of expressivity for game design and de-velopment, 2) SCXML is an XML dialect (soon to be) endorsed by the W3C, and will thus become a part of a web infrastructure comprising speech technology and telephony, as well as other useful technologies for building games of certain genres, and 3) SCXML is designed for extensibility and it appears that it would be fairly straightforward – and very worthwhile – to build a game oriented extension (“profile”) around the SCXML core. The paper also presents an experimental implementation of SCXML, accessible from a user-friendly web-interface.

Advances in Education and Consumer Health Informatics

To summarize current excellent research in the field of education and consumer health informatics. Synopsis of the articles on education and consumer health informatics selected for the IMIA Yearbook of Medical Informatics 2007. The consumer health informatics papers selected reflect the growing importance of communication and information retrieval systems in health care both for patients and professionals. Sound exemplary studies point out both the benefit for patients as well as the economic advantages of such systems. On the education sector, an intelligent tutoring system for medical students based on natural language dialogue serves as an example for the advancement and refinement of methods. The selected articles demonstrate the potential of advanced communication and information systems in health care. The physician-patient relationship though must not be affected by the introduction of these systems in order to ensure acceptance by both patients and physicians. Therefore these tools should be used in addition to current processes, and not as a replacement.

Effect of modality on collaboration with a dialogue system

The aim of this study was to investigate the influence of modality on collaboration processes between human and computer. Spoken and written interactions with a natural language dialogue system were compared using two real information-retrieval systems. In order to look for a restaurant (Experiment 1) or plan a trip (Experiment 2), participants performed several task-oriented dialogue scenarios. Although the spoken interaction mode was less efficient, it promoted collaboration, the use of personal pronouns and the literal form of the system's command utterances. Overall, in the written mode, the emphasis was on the task and its performance, rather than on dialogue. These findings are discussed with respect to the effect of communication mode on collaboration in human–computer dialogue.

Erratum: "Assisting E-Government Users with Animated Talking Faces" by P. Foglia, F. Giuntoli, C. A. Prete, and M. Zanda

The references were inadvertently removed from an article entitled "Assisting E-Government Users with Animated Talking Faces" in the January-February 2007 issue of <interactions> . We apologize for this oversight. The information is available below. 1. Cullen, R., O'Connor D., Veritt A. An Evaluation of Local Government Web sites in New Zealand. In The World of EGovernment (ed. Curtin G.C., Sommer M.H., Sommer V.), Haworth, 2003. 2. Foglia P., Prete C.A., Zanda M. Modelling Public Administration Portals. In Encyclopaedia of Portal Technology and Application (ed. Tatnall A.), Idea Group (in press). 3. Haxby J.V., Gobbini M.I., Furey M.L., Ishai A., Schouten, J.L., Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, Sept. 28; 293(5539):2405-7, 2001. 4. Nielsen J. Talking-Head video is boring online, AlertBox, Dec. 5, 2005. 5. Qiu L., Benbasat I. An Investigation into the effects of text-to-speech voice and 3D avatars on the perception of presence and flow of live help in electronic commerce. ACM Transactions on Computer Human Interaction (TOCHI), 12(4); 2005. 6. Reis F. E-Government: Internet based interaction with the European businesses and citizens; Statistics in Focus; Eurostat, 2005. 7. Stewart O., Huerta J.M. Transition relevance place: a proposal for adaptive user interface in natural language dialog management systems. ACM SIGCHI Conference Work in progress, 2006.

Abstract models for dialogue protocols

We examine a variety of dialogue protocols, taking inspiration from two fields: natural language dialogue modelling and multiagent systems. In communicative interaction, one can identify different features that may increase the complexity of the dialogue structure. This motivates a hierarchy of abstract models for protocols that takes as a starting point protocols based on deterministic finite automata. From there, we proceed by looking at particular examples that justify either an enrichment or a restriction of the initial model.

Sequence Package Analysis: A New Method for Intelligent Mining of Patient Dialog, Blogs and Help-line Calls

The ambiguities, repetitions and ellipses commonly found in natural language dialog continue to hinder speech (and text) analytic mining programs that glean business intelligence data from consumer help-line calls, or extract important medical diagnostic information from doctor-patient interviews or consumer-generated health-related blogs. This poses an even greater problem when such mining programs attempt to extract critical emotional data from natural language dialog. At present, speech (and text) analytic programs that mine natural language dialog for signs of distress, frustration, anger or other human emotions are still largely ineffective, because conventional speech systems that are limited to a set of key words and phrases cannot process speech as it actually occurs; if a speaker or blogger fails to use the word(s) found in the speech application’s vocabulary, the program yields a poor statistical word match (or no match). This paper shows how Sequence Package Analysis is informed by a set of algorithms – representing some of the more complex semantic aspects of communication in addition to syntax – that can interpret less than perfect natural speech, enhancing intelligent mining of recordings of doctor-patient interviews, customer care help-line calls, and consumer generated health-related blogs.

Toward virtual humans

This article describes the virtual humans developed as part of the Mission Rehearsal Exercise project, a virtual reality-based training system. This project is an ambitious exercise in integration, both in the sense of integrating technology with entertainment industry content, but also in that we have joined a number of component technologies that have not been integrated before. This integration has not only raised new research issues, but it has also suggested some new approaches to difficult problems. We describe the key capabilities of the virtual humans, including task representation and reasoning, natural language dialogue, and emotion reasoning, and show how these capabilities are integrated to provide more human-level intelligence than would otherwise be possible.

Efficient Realization of Coordinate Structures in Combinatory Categorial Grammar

We describe a chart realization algorithm for Combinatory Categorial Grammar (CCG), and show how it can be used to efficiently realize a wide range of coordination phenomena, including argument cluster coordination and gapping. The algorithm incorporates three novel methods for improving the efficiency of chart realization: (i) using rules to chunk the input logical form into sub-problems to be solved independently prior to further combination; (ii) pruning edges from the chart based on the n-gram score of the edge’s string, in comparison to other edges with equivalent categories; and (iii) formulating the search as a best-first anytime algorithm, using n-gram scores to sort the edges on the agenda. The algorithm has been implemented as an extension to the OpenCCG open source CCG parser, and initial performance tests indicate that the realizer is fast enough for practical use in natural language dialogue systems.

Uptake and joint action

In natural language dialogue, the way a responder ‘takes up’ the initiative of a participant, largely influences the further course of the dialogue. This uptake mechanism can be understood as a negotiation at a meta level: an initiative counts as a bid of a dialogue game; an appropriate response counts as an acceptance of the bid. We propose to extend this account of uptake to other conventional joint activities besides dialogue. We show that for an uptake mechanism to be effective, a joint activity must be characterisable in terms of initiatives and responses, with projection rules that indicate what initiatives count as a bid for a joint action, and appropriateness rules that indicate what responses count as appropriate.

An intelligent tutoring system that generates a natural language dialogue using dynamic multi-level planning

The objective of this research was to build an intelligent tutoring system capable of carrying on a natural language dialogue with a student who is solving a problem in physiology. Previous experiments have shown that students need practice in qualitative causal reasoning to internalize new knowledge and to apply it effectively and that they learn by putting their ideas into words. Analysis of a corpus of 75 hour-long tutoring sessions carried on in keyboard-to-keyboard style by two professors of physiology at Rush Medical College tutoring first-year medical students provided the rules used in tutoring strategies and tactics, parsing, and text generation. The system presents the student with a perturbation to the blood pressure, asks for qualitative predictions of the changes produced in seven important cardiovascular variables, and then launches a dialogue to correct any errors and to probe for possible misconceptions. The natural language understanding component uses a cascade of finite-state machines. The generation is based on lexical functional grammar. Results of experiments with pretests and posttests have shown that using the system for an hour produces significant learning gains and also that even this brief use improves the student's ability to solve problems more then reading textual material on the topic. Student surveys tell us that students like the system and feel that they learn from it. The system is now in regular use in the first-year physiology course at Rush Medical College. We conclude that the CIRCSIM-Tutor system demonstrates that intelligent tutoring systems can implement effective natural language dialogue with current language technology.

Scaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART

It is well-documented that most students do not have adequate proficiencies in inquiry and metacognition, particularly at deeper levels of comprehension that require explanatory reasoning. The proficiencies are not routinely provided by teachers and normal tutors so it is worthwhile to turn to computer-based learning environments. This article describes some of our recent computer systems that were designed to facilitate explanation-centered learning through strategies of inquiry and metacognition while students learn science and technology content. Point&Query augments hypertext, hypermedia, and other learning environments with question-answer facilities that are under the learner control. AutoTutor and iSTART use animated conversational agents to scaffold strategies of inquiry, metacognition, and explanation construction. AutoTutor coaches students in generating answers to questions that require explanations (e.g., why, what-if, how) by holding a mixed-initiative dialogue in natural language. iSTART models and coaches students in constructing self-explanations and in applying other metacomprehension strategies while reading text. These systems have shown promising results in tests of learning gains and learning strategies.

Austinian Truth, Attitudes and Type Theory

This paper is part of a broader project whose aim is to present a coherent unified approach to natural language dialogue semantics using tools from type theory. Here we explore aspects of our approach which relate to situation theory and situation semantics. We first point out a relationship between type theory and the Austinian notion of truth. We then consider how records in type theory might be used to represent situations and how dependent record types can be used to model constraints on situations. We then sketch treatments of attitude phenomena for which Barwise and Perry proposed situation semantic analyses (perception complements, belief, the Pierre puzzle) as well as two other intensional phenomena (intensional verbs and intentional identity). Finally we give a characterisation of the type theory used and a small illustrative fragment of English.

Towards efficient human machine speech communication

This research investigates the design and performance of the Speech Graffiti interface for spoken interaction with simple machines. Speech Graffiti is a standardized interface designed to address issues inherent in the current state-of-the-art in spoken dialog systems such as high word-error rates and the difficulty of developing natural language systems. This article describes the general characteristics of Speech Graffiti, provides examples of its use, and describes other aspects of the system such as the development toolkit. We also present results from a user study comparing Speech Graffiti with a natural language dialog system. These results show that users rated Speech Graffiti significantly better in several assessment categories. Participants completed approximately the same number of tasks with both systems, and although Speech Graffiti users often took more turns to complete tasks than natural language interface users, they completed tasks in slightly less time.

Mode and modal transfer effects on performance and discourse organization with an information retrieval dialogue system in natural language

This paper describes an experiment on the effects of learning, mode of interaction (written vs. spoken) and transfer mode on user performance and discourse organization during interaction with a natural language dialogue system. Forty-eight participants took part in a series of 12 dialogues with an information retrieval system presented either in the written or the spoken mode during the first six dialogues. The next six dialogues were then presented either in the same interaction mode or in another mode. The analysis of the results showed that performance (time, number of effective turns) improved throughout the dialogues whatever the mode of interaction. Nevertheless, performance was higher in the written mode. Moreover, mode-specific characteristics were observed. These consisted in greater use of subject pronouns and articles in the spoken mode. Similarly, in the spoken mode, the users found it easier to re-use the formulations presented in the system speech than in the written mode. Furthermore, the analysis also revealed a positive transfer effect on performance and discourse organization when the individuals first interacted in the spoken mode and then in the written mode. Both positive and negative transfer effects were observed when the individuals interacted first in the written mode followed by the spoken mode. The implications of the results are discussed in terms of direct and indirect consequences of modality effects on natural language dialogue interaction.

Affect and learning: An exploratory look into the role of affect in learning with AutoTutor

The role that affective states play in learning was investigated from the perspective of a constructivist learning framework. We observed six different affect states (frustration, boredom, flow, confusion, eureka and neutral) that potentially occur during the process of learning introductory computer literacy with AutoTutor, an intelligent tutoring system with tutorial dialogue in natural language. Observational analyses revealed significant relationships between learning and the affective states of boredom, flow and confusion. The positive correlation between confusion and learning is consistent with a model that assumes that cognitive disequilibrium is one precursor to deep learning. The findings that learning correlates negatively with boredom and positively with flow are consistent with predictions from Csikszentmihalyi's analysis of flow experiences.

UIMA: an architectural approach to unstructured information processing in the corporate research environment

IBM Research has over 200 people working on Unstructured Information Management (UIM) technologies with a strong focus on Natural Language Processing (NLP). These researchers are engaged in activities ranging from natural language dialog, information retrieval, topic-tracking, named-entity detection, document classification and machine translation to bioinformatics and open-domain question answering. An analysis of these activities strongly suggested that improving the organization's ability to quickly discover each other's results and rapidly combine different technologies and approaches would accelerate scientific advance. Furthermore, the ability to reuse and combine results through a common architecture and a robust software framework would accelerate the transfer of research results in NLP into IBM's product platforms. Market analyses indicating a growing need to process unstructured information, specifically multilingual, natural language text, coupled with IBM Research's investment in NLP, led to the development of middleware architecture for processing unstructured information dubbed UIMA. At the heart of UIMA are powerful search capabilities and a data-driven framework for the development, composition and distributed deployment of analysis engines. In this paper we give a general introduction to UIMA focusing on the design points of its analysis engine architecture and we discuss how UIMA is helping to accelerate research and technology transfer.

Searching information with a natural language dialogue system: a comparison of spoken vs. written modalities

This paper examines the effects of spoken vs. written dialogue modalities on the effectiveness of information search with a computerized retrieval system. Forty-eight adults familiar with the use of computers were asked to carry out six information retrieval tasks, engaging with the system using either spoken or written communication. The written modality was more efficient with regard to the number of dialogue turns, length of interaction with the system and mental workload. Even though the turns lasted longer in the written mode, they appeared to yield less mental workload. Moreover, spoken and written dialogues did not differ as regards the use of pronouns and articles. The implications for the development of natural-language dialogue systems are discussed.

AutoTutor: a tutor with dialogue in natural language.

AutoTutor is a learning environment that tutors students by holding a conversation in natural language. AutoTutor has been developed for Newtonian qualitative physics and computer literacy. Its design was inspired by explanation-based constructivist theories of learning, intelligent tutoring systems that adaptively respond to student knowledge, and empirical research on dialogue patterns in tutorial discourse. AutoTutor presents challenging problems (formulated as questions) from a curriculum script and then engages in mixed initiative dialogue that guides the student in building an answer. It provides the student with positive, neutral, or negative feedback on the student's typed responses, pumps the student for more information, prompts the student to fill in missing words, gives hints, fills in missing information with assertions, identifies and corrects erroneous ideas, answers the student's questions, and summarizes answers. AutoTutor has produced learning gains of approximately .70 sigma for deep levels of comprehension.

Learning from a Computer Tutor with Natural Language Capabilities

CIRCSIM-Tutor is a computer tutor designed to carry out a natural language dialogue with a medical student. Its domain is the baroreceptor reflex, the part of the cardiovascular system that is responsible for maintaining a constant blood pressure. CIRCSIM-Tutor’s interaction with students is modeled after the tutoring behavior of two experienced human tutors. The effectiveness of CIRCSIM-Tutor has been tested by 50 first-year medical students. Using a pre-test/post-test paradigm we have demonstrated that significant learning occurs during a 1 hr interaction with the program. Students were also surveyed and indicated considerable satisfaction with the program.