Mixed-initiative Dialogue Research Articles

Obituary: Martin Kay

Obituary: Martin Kay

Toward Data-Driven Collaborative Dialogue Systems: The JILDA Dataset

Today’s goal-oriented dialogue systems are designed to operate in restricted domains and with the implicit assumption that the user goals fit the domain ontology of the system. Under these assumptions dialogues exhibit only limited collaborative phenomena. However, this is not necessarily true in more complex scenarios, where user and system need to collaborate to align their knowledge of the domain in order to improve the conversation and achieve their goals.To foster research on data-driven collaborative dialogues, in this paper we present JILDA, a fully annotated dataset of chat-based, mixed-initiative Italian dialogues related to the job-offer domain. As far as we know, JILDA is the first dialogic corpus completely annotated in this domain. The analysis realised on top of the semantic annotations clearly shows the naturalness and greater complexity of JILDA’s dialogues. In fact, the new dataset offers a large number of examples of pragmatic phenomena, such as proactivity (i.e., providing information not explicitly requested) and grounding, which are rarely investigated in AI conversational agents based on neural architectures. In conclusion, the annotated JILDA corpus, given its innovative characteristics, represents a new challenge for conversational agents and an important resource for tackling more complex scenarios, thus advancing the state of the art in this field.

Exploring Mixed-Initiative Interaction for Learning with Situated Instruction in Cognitive Agents

Human-agent interaction for learning with instruction canbe viewed on a continuum of instructor/agent control. At one extreme are systems that learn by instructor-driveninteractions, such as learning by demonstration, examples,or imitation. The other extreme of the continuum isoccupied by systems where instructor interaction is limited to responding to the questions posed by the agent or toproviding feedback on agent’s performance. There are advantages to an approach that explores a mixed-initiative instructional dialog. We describe an approach to mixed-initiative interaction based on collaborative discourse theory that allows Soar agents to learn from situated instruction.

Computational Psycholinguistics

Computational Psycholinguistics

A computational model for the emergence of turn-taking behaviors in user-agent interactions

We propose a computational model that endows conversational agents with the capability to coordinate their speaking turns (turn-taking management) in the context of mixed-initiative two-party dialogs. In human conversations, participants are continuously adjusting their verbal and non-verbal productions for ensuring the effective coordination of speaking turns. In our model, the decision making is a continuous process based on the intrinsic current goal of the agent with respect to turn-taking, namely its motivation to keep-or to leave-its current role (speaker or listener), and on its perception of the intentions of its partner. Concurrently, the agent is also producing signals indicating its willingness to maintain or leave its current role. Our model is based on two models from cognitive psychology: the drift-diffusion model and the theory of behavioral dynamics. After presenting simulations showing how our model makes the coordination emerge from the interactions, we propose a SAIBA-Compliant architecture, named BeAware, created to support the implementation of our model. Finally, using our model, we investigate how an agent’s turn-taking strategy may impact the user’s experience and the effectiveness of the coordination.

Virtual training and coaching of health behavior: Example from mindfulness meditation training

ObjectiveComputer-based virtual coaches are increasingly being explored for patient education, counseling, and health behavior training and coaching. The objective of this research was to develop and evaluate a Virtual Mindfulness Coach for training and coaching in mindfulness meditation. MethodsThe coach was implemented as an embodied conversational character, providing mindfulness training and coaching via mixed initiative, text-based, natural language dialog with the student, and emphasizing affect–adaptive interaction. (The term ‘mixed initiative dialog’ refers to a human–machine dialog where either can initiate a conversation or a change in the conversation topic.) ResultsFindings from a pilot evaluation study indicate that the coach-based training is more effective in helping students establish a regular practice than self-administered training using written and audio materials. The coached group also appeared to be in more advanced stages of change in terms of the transtheoretical model, and have a higher sense of self-efficacy regarding establishment of a regular mindfulness practice. ConclusionThese results suggest that virtual coach-based training of mindfulness is both feasible, and potentially more effective, than a self-administered program. Of particular interest is the identification of the specific coach features that contribute to its effectiveness. Practice implicationsVirtual coaches could provide easily accessible and cost-effective customized training for a range of health behaviors. The affect–adaptive aspect of these coaches is particularly relevant for helping patients establish long-term behavior changes.

End-user programming of a social robot by dialog

One of the main challenges faced by social robots is how to provide intuitive, natural and enjoyable usability for the end-user. In our ordinary environment, social robots could be important tools for education and entertainment (edutainment) in a variety of ways. This paper presents a Natural Programming System (NPS) that is geared to non-expert users. The main goal of such a system is to provide an enjoyable interactive platform for the users to build different programs within their social robot platform. The end-user can build a complex net of actions and conditions (a sequence) in a social robot via mixed-initiative dialogs and multimodal interaction. The system has been implemented and tested in Maggie, a real social robot with multiple skills, conceived as a general HRI researching platform. The robot’s internal features (skills) have been implemented to be verbally accessible to the end-user, who can combine them into others that are more complex following a bottom-up model. The built sequence is internally implemented as a Sequence Function Chart (SFC), which allows parallel execution, modularity and re-use. A multimodal Dialog Manager System (DMS) takes charge of keeping the coherence of the interaction. This work is thought for bringing social robots closer to non-expert users, who can play the game of “teaching how to do things” with the robot.

Initiative conflicts in task-oriented dialogue

An initiative conflict is an overlap of speech in which both conversants try to steer the conversation in different directions. In this paper, we investigate how conversants in human-human dialogue deal with such conflicts. First, in investigating why initiative conflicts occur, we find that the offsets of the utterances involved in initiative conflicts tend to be very short, and that initiative conflicts seem more likely to occur when one of the conversants has an urgent conversational goal. These findings strongly suggest that initiative conflicts are unintentional collisions and that conversants try to prevent them from even occurring, unless there is an urgent reason. Second, in investigating how initiative conflicts are resolved, we find that the overlaps tend to last less than two syllables, that volume correlates with who wins initiative conflicts, and that for longer overlaps, the volume of the winner increases in the second half of the overlaps. These findings strongly suggest that initiative conflicts are quickly resolved through an interactive process, using volume as one of the devices. Third, we find that after an initiative conflict is resolved, the winner sometimes repeats the words involved in the overlap; and this happens more when the overlap is more likely to interfere with the other conversant’s understanding. These findings will help us build next-generation mixed-initiative spoken dialogue systems that are natural and efficient to use.

Compositional specification and realisation of mixed-initiative web dialogs

We present DialogXML – a markup language approach to specifying and realising mixed-initiative web dialogs on mobile devices. By capturing the functional structure of the dialog independent of the modalities used to realise it, DialogXML facilitates the implementation of web interfaces that integrate hyperlink and speech modes of interaction. It enables the creation of websites that adapt to the needs of users yet permits the designer fine-grained control over what interactions to support. The framework uses an algorithm based on staging transformations – an approach that represents dialogs by programs and uses program transformations to simplify them based on user input. Design methodology, implementation details, and two case studies are presented.

Mixed-Initiative Systems for Collaborative Problem Solving

Mixed-initiative systems are a popular approach to building intelligent systems that can collaborate naturally and effectively with people. But true collaborative behavior requires an agent to possess a number of capabilities, including reasoning, communication, planning, execution, and learning. We describe an integrated approach to the design and implementation of a collaborative problem-solving assistant based on a formal theory of joint activity and a declarative representation of tasks. This approach builds on prior work by us and by others on mixed-initiative dialogue and planning systems.

Scaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART

It is well-documented that most students do not have adequate proficiencies in inquiry and metacognition, particularly at deeper levels of comprehension that require explanatory reasoning. The proficiencies are not routinely provided by teachers and normal tutors so it is worthwhile to turn to computer-based learning environments. This article describes some of our recent computer systems that were designed to facilitate explanation-centered learning through strategies of inquiry and metacognition while students learn science and technology content. Point&Query augments hypertext, hypermedia, and other learning environments with question-answer facilities that are under the learner control. AutoTutor and iSTART use animated conversational agents to scaffold strategies of inquiry, metacognition, and explanation construction. AutoTutor coaches students in generating answers to questions that require explanations (e.g., why, what-if, how) by holding a mixed-initiative dialogue in natural language. iSTART models and coaches students in constructing self-explanations and in applying other metacomprehension strategies while reading text. These systems have shown promising results in tests of learning gains and learning strategies.

Error-correction detection and response generation in a spoken dialogue system

Speech understanding errors in spoken dialogue systems can be frustrating for users and difficult to recover from in a mixed-initiative spoken dialogue system. Handling such errors requires both detecting error conditions and adjusting the response generation strategy accordingly. In this paper, we show that different response wording choices tend to be associated with different user behaviors that can impact word recognition performance in a telephone-based dialogue system. We leverage these findings in a system that integrates an error correction detection module with a modified dialogue strategy in order to drive the response generation module. In a user study, we find slight preferences for a dialogue system using this error handling strategy over a simple reprompting strategy.

A natural language help system shell through functional programming

This paper investigates the development of a natural language (NL) interface for mixed initiative dialogues within a constrained domain and demonstrates the applicability of the functional approach to NL system development. The system consists of two major components, a natural language subsystem comprises a general-purpose parser that interprets a 'plug and play' tagged BNF grammar (which may be ambiguous), to parse natural language input and extract semantic information. The knowledge-based subsystem uses the semantic tags extracted by the natural language subsystem to generate a focused query to select the most appropriate script for a guided dialogue with the user. The system was written entirely in a purely functional language, which resulted in a surprisingly small and simple program.

Experiences with Commercial Telephone-based Dialogue Systems (Erfahrungen mit kommerziellen Telefon-Sprachdialogsystemen)

Summary In this publication experiences with commercial spoken dialogue systems are discussed and guidelines for achieving high usability are pointed out. Different from most commercially deployed IVR (Interactive Voice Response) systems, the systems discussed in this paper belong to a new generation of real mixed-initiative spoken dialogue systems, i. e., the user may take the initiative, using full sentences, at virtually any point in time during the dialogue. We use three commercially deployed systems as example applications: the automated switchboard of a large German company, the movie information system operated by Germany´s largest multiplex cinema, and a football Bundesliga information system operated by a German media company.

AutoTutor: a tutor with dialogue in natural language.

AutoTutor is a learning environment that tutors students by holding a conversation in natural language. AutoTutor has been developed for Newtonian qualitative physics and computer literacy. Its design was inspired by explanation-based constructivist theories of learning, intelligent tutoring systems that adaptively respond to student knowledge, and empirical research on dialogue patterns in tutorial discourse. AutoTutor presents challenging problems (formulated as questions) from a curriculum script and then engages in mixed initiative dialogue that guides the student in building an answer. It provides the student with positive, neutral, or negative feedback on the student's typed responses, pumps the student for more information, prompts the student to fill in missing words, gives hints, fills in missing information with assertions, identifies and corrects erroneous ideas, answers the student's questions, and summarizes answers. AutoTutor has produced learning gains of approximately .70 sigma for deep levels of comprehension.

A framework of synthesizing tutoring conversation capability with web-based distance education courseware

Whereas existing learning environments on the Web lack high level interactivity, we have developed a human tutor-like tutorial conversation system for the Web that enhances educational courseware through mixed-initiative dialog with natural language processing. The conversational tutoring agent is composed of an animated tutor, a Latent Semantic Analysis (LSA) module, a database with curriculum scripts, and a dialog manager. As in the case of human tutors, the meaning of learner's contributions in natural language are compared with the content of expected answers to questions or problems specified in curriculum scripts. LSA is used to evaluate the conceptual matches between learner input and tutor expectations, whereas the dialog manager determines how the tutor adaptively responds to the learner by selecting content from the curriculum script. The integration of available courseware with the tutorial dialog system guarantees the reusability of existing Web tutorials with minimal effort in the modification of the curriculum script and LSA module. This development thereby simplifies the change into more valuable Web based training courseware.

The use of belief networks for mixed-initiative dialog modeling

This paper proposes the use of Belief Networks (BN) for mixed-initiative dialog modeling. The BN-based framework was previously used for natural language understanding, where BNs infer the informational goal of the user's query based on its parsed semantic concepts. We extended this framework with the technique of backward inference that can automatically detect missing or spurious concepts based on the inferred goal. This is, in turn, used to drive the mixed-initiative dialog model that prompts for missing concepts and clarifies for spurious concepts. Applicability is demonstrated for a simple foreign exchange domain, and our framework's mixed-initiative interactions were shown to be superior to the system-initiative and user-initiative interactions. We also investigate the scalability and portability of the BN-based framework to the more complex air travel (ATIS) domain. Backward inference detected an increased number of missing and spurious concepts, which led to redundancies in the dialog model. We experimented with several remedial measures that showed promise in reducing the redundancies. We also present a set of principles for hand-assigning "degrees of belief" to the BN to reduce the demand for massive training data when porting to a new domain. Experimentation with the ATIS data also gave promising results.

Detecting and extracting named entities from spontaneous speech in a mixed-initiative spoken dialogue context: How May I Help You? sm, tm

The understanding module of a spoken dialogue system must extract, from the speech recognizer output, the kind of request expressed by the caller (the call type) and its parameters (numerical expressions, time expressions or proper-names). Such expressions are called Named Entities and their definitions can be either generic or linked to the dialogue application domain. Detecting and extracting such Named Entities within a mixed-initiative dialogue context like How May I Help You? sm, tm (HMIHY) is the subject of this study. After reviewing standard methods based on hand-written grammars and statistical tagging, we propose a new approach, combining the advantages of both in a 2-step process. We also propose a novel architecture which exploits understanding to improve recognition accuracy: the output of the Automatic Speech Recognition module is now a word lattice and the understanding module is responsible for transcribing the word strings which are useful to the Dialogue Manager. All the methods proposed are trained and evaluated on a corpus comprising utterances from live customer traffic.

Response planning and generation in the mercury flight reservation system

This paper describes the response planning and generation components of the mercury flight reservation system, a mixed-initiative spoken dialogue system that supports both voice-only interaction and multi-modal interaction augmenting spoken inputs with typing or clicking at a displayed Web page. mercury is configured using the Galaxy Communicator architecture ( Seneff, Hurley, Lau, Schmid, & Zue, 1998), where a suite of servers interact via program control mediated by a central hub. Language generation is performed in two steps: response planning, or deep-structure generation, is carried out by the dialogue manager, and is well-integrated with other aspects of dialogue control; control flow is specified by a dialogue control table ( Seneff & Polifroni, 2000a). Response generation, or surface-form generation, is executed by a separate language generation server, under the guidance of a set of recursive generation rules and an associated lexicon ( Baptist & Seneff, 2000). The generation of the textual string for the graphical interface and the marked-up synthesis string for spoken outputs are controlled by a shared set of generation rules ( Seneff & Polifroni, 2000b). Thus there is a direct meaning-to-speech mapping that eliminates the need to analyze linguistic structure for synthesis. To date, we have collected over 25 000 utterances from users interacting with the mercury system. We report here on both the results of user satisfaction studies conducted by the National Institute of Standards and Technology (NIST), and on our own tabulation of a number of different measures of dialogue success.

Classifying Student Initiatives and Tutor Responses in Human Keyboard-to-Keyboard Tutoring Sessions

This study analyzed twenty-eight 1-hr-long tutoring sessions that were carried out keyboard-to-keyboard with tutor and student in different rooms. The tutors were professors of physiology at Rush Medical College. The students were 1st-year medical students. We classified student initiatives and tutor responses in human tutoring sessions with the goal of making our intelligent tutoring system capable of handling mixed-initiative dialogue. Student initiatives were classified along 4 dimensions: communicative goal, surface form, focus of attention, and degree of certainty (i.e., does the student hedge or not?). Student goals included request for confirmation, request for information, challenge, refusal to answer, and conversational repair. Tutor responses were classified along 3 dimensions: communicative goal, surface form, and delivery mode. The tutor goals included causal explanation, acknowledgment, conversational repair, instruction in the rules of the game, teaching the problem-solving algorithm, and teaching the language of physiology. Our interrater reliability studies supported these categories in the domain of tutoring.