Dialogue Management Research Articles

A Comprehensive Reinforcement Learning Framework for Dialogue Management Optimization

Reinforcement learning is now an acknowledged approach for optimizing the interaction strategy of spoken dialogue systems. If the first considered algorithms were quite basic (like SARSA), recent works concentrated on more sophisticated methods. More attention has been paid to off-policy learning, dealing with the exploration-exploitation dilemma, sample efficiency or handling non-stationarity. New algorithms have been proposed to address these issues and have been applied to dialogue management. However, each algorithm often solves a single issue at a time, while dialogue systems exhibit all the problems at once. In this paper, we propose to apply the Kalman Temporal Differences (KTD) framework to the problem of dialogue strategy optimization so as to address all these issues in a comprehensive manner with a single framework. Our claims are illustrated by experiments led on two real-world goal-oriented dialogue management frameworks, DIPPER and HIS.

An Example-Based Approach to Ranking Multiple Dialog States for Flexible Dialog Management

This paper presents a new hybrid dialog management framework that integrates a statistical ranking algorithm into an example-based dialog management approach for chat-like dialogs. The proposed model uses ranking features that consider various aspects of dialogs, including the relative importance of speech acts, dialog history sequences, and the causal relationships among speech acts and slot-filling states. The ranking algorithm enables one to aggregate these feature scores systematically and to generate diverse system responses. Additionally, the model provides detailed feedback by analyzing the causal relationships among speech acts and predicting the user's possible intentions associated with a given dialog states. Simulated experimental results demonstrate that our approach is effective for task-oriented dialogs and chat-like dialogs. Additionally, a case study using elementary school students implies that the proposed system can be used for language learning purposes in addition to task-oriented services.

Naturalistic Dialogue Management for Noisy Speech Recognition

With naturalistic dialogue management, a spoken dialogue system behaves as a human would under similar conditions. This paper reports on an experiment to develop naturalistic clarification strategies for noisy speech recognition in the context of spoken dialogue systems. We collected a wizard-of-Oz corpus in which human wizards with access to a rich set of clarification actions made clarification decisions online, based on human-readable versions of system data. The experiment compares an evaluation of calls to a baseline system in a library domain with calls to an enhanced version of the system. The new system has a clarification module based on the wizard data that is a decision tree constructed from three machine-learned models. It replicates the wizards' ability to ground partial understandings of noisy input and to build upon them. The enhanced system has a significantly higher rate of task completion, greater task success and improved efficiency.

Building Adaptive Dialogue Systems Via Bayes-Adaptive POMDPs

Recent research has shown that effective dialogue management can be achieved through the Partially Observable Markov Decision Process (POMDP) framework. However past research on POMDP-based dialogue systems usually assumed the parameters of the decision process were known a priori. The main contribution of this paper is to present a Bayesian reinforcement learning framework for learning the POMDP parameters online from data, in a decision-theoretic manner. We discuss various approximations and assumptions which can be leveraged to ensure computational tractability, and apply these techniques to learning observation models for several simulated spoken dialogue domains.

A survey on metrics for the evaluation of user simulations

AbstractUser simulation is an important research area in the field of spoken dialogue systems (SDSs) because collecting and annotating real human–machine interactions is often expensive and time-consuming. However, such data are generally required for designing, training and assessing dialogue systems. User simulations are especially needed when using machine learning methods for optimizing dialogue management strategies such as Reinforcement Learning, where the amount of data necessary for training is larger than existing corpora. The quality of the user simulation is therefore of crucial importance because it dramatically influences the results in terms of SDS performance analysis and the learnt strategy. Assessment of the quality of simulated dialogues and user simulation methods is an open issue and, although assessment metrics are required, there is no commonly adopted metric. In this paper, we give a survey of User Simulations Metrics in the literature, propose some extensions and discuss these metrics in terms of a list of desired features.

Towards building intelligent speech interfaces through the use of more flexible, robust and natural dialogue management solutions

In this paper a Bayesian Networks-based solution for dialogue modelling is presented. This solution is combined with carefully designed contextual information handling strategies. With the purpose of validating these solutions, and introducing a spoken dialogue system for controlling a Hi-Fi audio system as the selected prototype, a real-user evaluation has been conducted. Two different versions of the prototype are compared. Each version corresponds to a different implementation of the algorithm for the management of the actuation order, the algorithm for deciding the proper order to carry out the actions required by the user. The evaluation is carried out in terms of a battery of both subjective and objective metrics collected from speakers interacting with the Hi-Fi audio box through predefined scenarios. Defined metrics have been specifically adapted to measure: first, the usefulness and the actual relevance of the proposed solutions, and, secondly, their joint performance through their intelligent combination mainly measured as the level achieved with regard to the user satisfaction. A thorough and comprehensive study of the main differences between both approaches is presented. Two-way analysis of variance (ANOVA) tests are also included to measure the effects of both: the system used and the type of scenario factors, simultaneously. Finally, the effect of bringing this flexibility, robustness and naturalness into our home dialogue system is also analyzed through the results obtained. These results show that the intelligence of our speech interface has been well perceived, highlighting its excellent ease of use and its good acceptance by users, therefore validating the approached dialogue management solutions and demonstrating that a more natural, flexible and robust dialogue is possible thanks to them.

An Analysis of the Ideological Content of Internationally-Developed British and American ELT Textbooks

This study examines the ideological content of six internationally-used ELT textbooks, three British and three American. Applying the theory and procedures of critical discourse analysis (CDA), three aspects of the conversations in the textbooks, i.e. content, relations, and positions were examined. Results suggest that in terms of content, British textbooks tend to entertain the students while American textbooks include more occupational and business-related issues. Regarding relations, characters in the conversations were placed in socially equal roles. Moreover, inequalities were rarely addressed in the interactions and there was little attempt to make learners aware of issues such as dialogue management strategies. Finally, characters in the conversations took mostly societal positions.

On the dynamic adaptation of language models based on dialogue information

We present an approach to adapt dynamically the language models (LMs) used by a speech recognizer that is part of a spoken dialogue system. We have developed a grammar generation strategy that automatically adapts the LMs using the semantic information that the user provides (represented as dialogue concepts), together with the information regarding the intentions of the speaker (inferred by the dialogue manager, and represented as dialogue goals). We carry out the adaptation as a linear interpolation between a background LM, and one or more of the LMs associated to the dialogue elements (concepts or goals) addressed by the user. The interpolation weights between those models are automatically estimated on each dialogue turn, using measures such as the posterior probabilities of concepts and goals, estimated as part of the inference procedure to determine the actions to be carried out. We propose two approaches to handle the LMs related to concepts and goals. Whereas in the first one we estimate a LM for each one of them, in the second one we apply several clustering strategies to group together those elements that share some common properties, and estimate a LM for each cluster. Our evaluation shows how the system can estimate a dynamic model adapted to each dialogue turn, which helps to significantly improve the performance of the speech recognition, which leads to an improvement in both the language understanding and the dialogue management tasks.

Building Autonomous Sensitive Artificial Listeners

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

Multimodal Language Specification for Human Adaptive Mechatronics

Designing and building automated systems with which people can interact naturally is one of the emerging objective of Mechatronics. In this perspective multimodality and adaptivity represent focal issues, enabling users to communicate more freely and naturally with automated systems. One of the basic problem of multimodal interaction is the fusion process. Current approaches to fusion are mainly two: the former implements the multimodal fusion at dialogue management level, whereas the latter at grammar level. In this paper, we propose a multimodal attribute grammar, that provides constructions both for representing input symbols from different modalities and for modeling semantic and temporal features of multimodal input symbols, enabling the specification of multimodal languages. Moreover, an application of the proposed approach in the context of a multimodal language specification to control a driver assistance system, as robots using different integrated interaction modalities, is given.

Research and Implementation of Embedded Voice Interaction System Based on ARM in Intelligent Space

This paper presents the design and implementation of voice interaction system based on ARM in service robots intelligent space. ARM Cortex-M3 core based STM32F103 is used for main controller. And real-time embedded operating system μC/OS-II is used to schedule different tasks and manage peripheral devices. LD3320 and XFS4041CN are used for speech recognition and speech synthesis, respectively. Establish a dialogue set by defining the set of two-dimensional array, and then intelligent space updates the dialogue set dynamically via ZigBee wireless network for multiple scenes. If the recognition result is got by speech recognition module, dialogue management module will send corresponding texts to speech synthesis module, enabling text-to-speech output. At the same time, intelligent space decision support system will send corresponding commands to equipments by ZigBee. From lots of experiments and practical applications, we can conclude that voice interaction systems designed in the paper can gracefully satisfy the current request of voice interaction in service robots intelligent space, having great application value.

The context aware workflow execution framework

The development of self-adapting web applications based on composite architectures, such as service-oriented architectures (SOA), is challenged by the lack of support to the specification of explicit adaptation policies for the context aware management of the business, interaction and presentation logics. To address this limitation, we propose a vertical architecture extending SOA with advanced adaptation features. This paper presents the Context Aware Workflow Execution framework (CAWE), which enriches SOA with (1) context-aware workflow management; (2) dialogue management capabilities supporting the adaptation of the interaction with the user and (3) context-dependent user interface generation. This paper also briefly presents a prototype application developed by exploiting the CAWE framework.

BBN TransTalk: Robust multilingual two-way speech-to-speech translation for mobile platforms

In this paper we present a speech-to-speech (S2S) translation system called the BBN TransTalk that enables two-way communication between speakers of English and speakers who do not understand or speak English. The BBN TransTalk has been configured for several languages including Iraqi Arabic, Pashto, Dari, Farsi, Malay, Indonesian, and Levantine Arabic. We describe the key components of our system: automatic speech recognition (ASR), machine translation (MT), text-to-speech (TTS), dialog manager, and the user interface (UI). In addition, we present novel techniques for overcoming specific challenges in developing high-performing S2S systems. For ASR, we present techniques for dealing with lack of pronunciation and linguistic resources and effective modeling of ambiguity in pronunciations of words in these languages. For MT, we describe techniques for dealing with data sparsity as well as modeling context. We also present and compare different user confirmation techniques for detecting errors that can cause the dialog to drift or stall.

End-user programming of a social robot by dialog

One of the main challenges faced by social robots is how to provide intuitive, natural and enjoyable usability for the end-user. In our ordinary environment, social robots could be important tools for education and entertainment (edutainment) in a variety of ways. This paper presents a Natural Programming System (NPS) that is geared to non-expert users. The main goal of such a system is to provide an enjoyable interactive platform for the users to build different programs within their social robot platform. The end-user can build a complex net of actions and conditions (a sequence) in a social robot via mixed-initiative dialogs and multimodal interaction. The system has been implemented and tested in Maggie, a real social robot with multiple skills, conceived as a general HRI researching platform. The robot’s internal features (skills) have been implemented to be verbally accessible to the end-user, who can combine them into others that are more complex following a bottom-up model. The built sequence is internally implemented as a Sequence Function Chart (SFC), which allows parallel execution, modularity and re-use. A multimodal Dialog Manager System (DMS) takes charge of keeping the coherence of the interaction. This work is thought for bringing social robots closer to non-expert users, who can play the game of “teaching how to do things” with the robot.

Creating a general collaborative dialogue agent with lounge strategy feature

In this paper, we focus on an agent-based architecture for task-oriented dialogue management. We present our deliberation process based on optimizing the length of a dialogue in terms of its turns needed to accomplish a task. As innovative features, the manager accommodates a “lounge strategy” for passing initiative back to the user, and a two-layered structure for dialogue context representation. We show that applying the lounge strategy during a dialogue leads to improving the information exchange rate by approximately 5% compared to “common” dialogue strategies alternative. Furthermore, we present two of our dialogues and analyze them with respect to the agent description. At the end of the paper, we suggest future extensions and modifications to the architecture, and conclude.

Developing Speech Input for Virtual Applications

This paper contextualises the position of speech input from a user-centred human factors perspective. It is presented as a position paper so that researchers and designers can consider the underlying and future factors of a people-orientated approach to speech input for virtual applications. A number of key areas are explored including: human factors for speech input; speech input for virtual applications; speech as a spare mode of interaction; user acceptance and uptake; incorporating speech in the development of virtual applications; and speech input as an interaction tool. Given the user-centred perspective of this paper, this paper does not set out to address issues associated with spoken dialogue technologies, dialogue, and dialogue management; recent work on conversational agents in virtual environments; or multimodal interaction. This paper places the focus more fundamentally within human factors by looking at the user first as a basis for developing usable virtual applications incorporating speech input rather than to review the current state of the art in interaction design. A particular point this paper makes, however, is that speech input should be designed and used as another interaction tool that users need to learn to use, rather than assuming it will offer a natural or intuitive interface.

Predicting user mental states in spoken dialogue systems

In this paper we propose a method for predicting the user mental state for the development of more efficient and usable spoken dialogue systems. This prediction, carried out for each user turn in the dialogue, makes it possible to adapt the system dynamically to the user needs. The mental state is built on the basis of the emotional state of the user and their intention, and is recognized by means of a module conceived as an intermediate phase between natural language understanding and the dialogue management in the architecture of the systems. We have implemented the method in the UAH system, for which the evaluation results with both simulated and real users show that taking into account the user's mental state improves system performance as well as its perceived quality.

Sample-efficient batch reinforcement learning for dialogue management optimization

Spoken Dialogue Systems (SDS) are systems which have the ability to interact with human beings using natural language as the medium of interaction. A dialogue policy plays a crucial role in determining the functioning of the dialogue management module. Handcrafting the dialogue policy is not always an option, considering the complexity of the dialogue task and the stochastic behavior of users. In recent years approaches based on Reinforcement Learning (RL) for policy optimization in dialogue management have been proved to be an efficient approach for dialogue policy optimization. Yet most of the conventional RL algorithms are data intensive and demand techniques such as user simulation. Doing so, additional modeling errors are likely to occur. This paper explores the possibility of using a set of approximate dynamic programming algorithms for policy optimization in SDS. Moreover, these algorithms are combined to a method for learning a sparse representation of the value function. Experimental results show that these algorithms when applied to dialogue management optimization are particularly sample efficient , since they learn from few hundreds of dialogue examples. These algorithms learn in an off-policy manner, meaning that they can learn optimal policies with dialogue examples generated with a quite simple strategy. Thus they can learn good dialogue policies directly from data , avoiding user modeling errors.

Effective handling of dialogue state in the hidden information state POMDP-based dialogue manager

Effective dialogue management is critically dependent on the information that is encoded in the dialogue state. In order to deploy reinforcement learning for policy optimization, dialogue must be modeled as a Markov Decision Process. This requires that the dialogue state must encode all relevent information obtained during the dialogue prior to that state. This can be achieved by combining the user goal, the dialogue history, and the last user action to form the dialogue state. In addition, to gain robustness to input errors, dialogue must be modeled as a Partially Observable Markov Decision Process (POMDP) and hence, a distribution over all possible states must be maintained at every dialogue turn. This poses a potential computational limitation since there can be a very large number of dialogue states. The Hidden Information State model provides a principled way of ensuring tractability in a POMDP-based dialogue model. The key feature of this model is the grouping of user goals into partitions that are dynamically built during the dialogue. In this article, we extend this model further to incorporate the notion of complements. This allows for a more complex user goal to be represented, and it enables an effective pruning technique to be implemented that preserves the overall system performance within a limited computational resource more effectively than existing approaches.

Natural actor and belief critic

This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet. The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better.