Dialogue Policy Research Articles

IFDDS: An Anti-fraud Outbound Robot

With the rapid growth of internet finance and e-payment, payment fraud has attracted increasing attention. To prevent customers from being cheated, systems often block risky payments depending on a risk factor. However, this may also inadvertently block cases which are not actually risky. To solve this problem, we present IFDDS, a system that proactively chats with customers through intelligent speech interaction to precisely determine the actual payment risk. Our system adopts imitation learning to learn dialogue policies. In addition, it encompasses a dialogue risk detection module which identifies fraud probability every turn based on the dialogue state. We create a web-based user interface which simulates a practical voice-based dialogue system.

Converse, Focus and Guess - Towards Multi-Document Driven Dialogue

We propose a novel task, Multi-Document Driven Dialogue (MD3), in which an agent can guess the target document that the user is interested in by leading a dialogue. To benchmark progress, we introduce a new dataset of GuessMovie, which contains 16,881 documents, each describing a movie, and associated 13,434 dialogues. Further, we propose the MD3 model. Keeping guessing the target document in mind, it converses with the user conditioned on both document engagement and user feedback. In order to incorporate large-scale external documents into the dialogue, it pretrains a document representation which is sensitive to attributes it talks about an object. Then it tracks dialogue state by detecting evolvement of document belief and attribute belief, and finally optimizes dialogue policy in principle of entropy decreasing and reward increasing, which is expected to successfully guess the user's target in a minimum number of turns. Experiments show that our method significantly outperforms several strong baseline methods and is very close to human's performance.

Efficient Dialog Policy Learning With Hindsight, User Modeling, and Adaptation

Efficient Dialog Policy Learning With Hindsight, User Modeling, and Adaptation

SUMBT+LaRL: Effective Multi-Domain End-to-End Neural Task-Oriented Dialog System

The recent advent of neural approaches for developing each dialog component in task-oriented dialog systems has remarkably improved, yet optimizing the overall system performance remains a challenge. Besides, previous research on modeling complicated multi-domain goal-oriented dialogs in end-to-end fashion has been limited. In this paper, we present an effective multi-domain end-to-end trainable neural dialog system <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SUMBT+LaRL</i> that incorporates two previous strong models and facilitates them to be fully differentiable. Specifically, the SUMBT+ estimates user-acts as well as dialog belief states, and the LaRL models latent system action spaces and generates responses given the estimated contexts. We emphasize that the training framework of three steps significantly and stably increase dialog success rates: separately pretraining the SUMBT+ and LaRL, fine-tuning the entire system, and then reinforcement learning of dialog policy. We also introduce new reward criteria of reinforcement learning for dialog policy training. Then, we discuss experimental results depending on the reward criteria and different dialog evaluation methods. Consequently, our model achieved the new state-of-the-art success rate of 85.4% on corpus-based evaluation, and a comparable success rate of 81.40% on simulator-based evaluation provided by the DSTC8 challenge. To our best knowledge, our work is the first comprehensive study of a modularized E2E multi-domain dialog system that learning from each component to the entire dialog policy for task success.

Weighted User Goal Sampling for Dialog Policy Learning

A deep reinforcement learning algorithm has been widely used for learning dialog policy in a task-oriented dialog system. Dialog agent collects training data and improves its policy by interacting with users. Interact with real users is time-consuming and not realistic. Consequently, we usually build a user simulator instead of a real user. At the beginning of each dialogue, the simulator will sample a user goal extract from training data. Then the simulator will communicate with the dialog agent to accomplish this user goal. The existing user simulator usually samples this user goal randomly. This will cause the dialog agent to waste a lot of time to learn what it already learned. To solve this problem, we propose two user goal weighting methods which give relatively large weight to the user goal the current dialog agent can’t accomplish. This lets the dialog agent pay more attention to those user goals. Experiment result in a movie-ticket booking task shows that the proposed weighted user goal sampling method can effectively accelerate the policy learning progress compares to the random sampling method.

World monies or money-worlds: A new perspective on cryptocurrencies and their moneyness

AbstractThis essay makes the case that current debates about the ‘moneyness’ of Bitcoin and other cryptocurrencies are occurring at the incorrect scale. Rather than being some form of transnational digital money to be used alongside or compete with national fiat currencies, I argue that, instead, each cryptocurrency represents its own self-contained ‘money-world’. Put differently, a cryptocurrency is the uniquely specified unit of account and medium of exchange within the socio-technical bounds of its own blockchain. This new perspective can open new lines of intellectual dialogue and inform better policy choices for regulating cryptocurrencies.

DisBot: A Portuguese Disaster Support Dynamic Knowledge Chatbot

This paper presents DisBot, the first Portuguese speaking chatbot that uses social media retrieved knowledge to support citizens and first-responders in disaster scenarios, in order to improve community resilience and decision-making. It was developed and tested using Design Science Research Methodology (DSRM), being progressively matured with field specialists through several design and development iterations. DisBot uses a state-of-the-art Dual Intent Entity Transformer (DIET) architecture to classify user intents, and makes use of several dialogue policies for managing user conversations, as well as storing relevant information to be used in further dialogue turns. To generate responses, it uses real-world safety knowledge, and infers a dynamic knowledge graph that is dynamically updated in real-time by a disaster-related knowledge extraction tool, presented in previous works. Through its development iterations, DisBot has been validated by field specialists, who have considered it to be a valuable asset in disaster management.

Photographing the Aspiration of the Hindu-Islam as Interfaith Dialogue Policy

Managing plural and multicultural diversity of citizens is an obligation mandated by the constitution and the third precept of Pancasila. One of the efforts to care for diversity within the framework of Indonesian unity is through interfaith dialogue which is packaged in fostering religious harmony. With the issuance of Village law 16 of 2014 has provided space for aspirations to manage the diversity of citizens through the deleberative democracy process in the village budget policy mechanism in Village Development Conference. This research has found grassroots aspirations that are included in the Village Fund policy in Klinting Village, Banyumas district. Although it is not the main aspiration in the village budget policy priorities, it is at least a quality aspiration that is included in the village fund policy. The issue of religion in the form of interreligious dialogue between Hindu-Islamic religions is still a common thread in the realities of policy and the aspirations of people's lives, that is the aspiration demonstration that can be achieved from this research.&#x0D;

Recent advances and challenges in task-oriented dialog systems

Due to the significance and value in human-computer interaction and natural language processing, task-oriented dialog systems are attracting more and more attention in both academic and industrial communities. In this paper, we survey recent advances and challenges in task-oriented dialog systems. We also discuss three critical topics for task-oriented dialog systems: (1) improving data efficiency to facilitate dialog modeling in low-resource settings, (2) modeling multi-turn dynamics for dialog policy learning to achieve better task-completion performance, and (3) integrating domain ontology knowledge into the dialog model. Besides, we review the recent progresses in dialog evaluation and some widely-used corpora. We believe that this survey, though incomplete, can shed a light on future research in task-oriented dialog systems.

Civil society aid as balancing act – navigating between managerial and social transformative principles

ABSTRACT The civil society policy of the Ministry of Foreign Affairs of the Netherlands shifted from service delivery to lobby and advocacy in 2016 with the new Dialogue & Dissent policy framework (D&D), representing a large change in thinking from a managerial to a social transformative perspective. This article discusses how D&D has made this shift and navigates tensions arising from it: (1) by stressing the political role of CSOs; (2) by working in strategic partnerships; and (3) by employing flexible Theories of Change focused on learning. It concludes with lessons learnt for D&D’s successor.

Government and civil society organizations: Close but comfortable? Lessons from creating the Dutch “Strategic Partnerships for Lobby and Advocacy”

AbstractMotivationGovernments commonly support the advocacy role of civil society organizations (CSOs) in development, but studies argue that close linkages between government and CSOs are problematic. The Netherlands’ Dialogue and Dissent policy programme brings together the Netherlands Ministry of Foreign Affairs (NMFA) and 25 (alliances of) CSOs as advocacy partners. The programme is set up from an awareness of the challenging nature of such collaboration.PurposeWe contribute to understanding the ways in which issues with donor–CSO collaboration can be engaged in donor policies.This article addresses two questions: To what extent and how does this programme confront and overcome the challenges of close collaboration between government and CSOs? What practical lessons can be learned?ApproachWe conducted 33 (group) interviews with CSOs and policy officers involved with the Dialogue and Dissent programme, exploring their understandings, expectations and strategies as partners in the programme. We also analysed CSO programmes, policy documents and publicly available information.FindingsIn Dialogue and Dissent, space for dissent, flexibility and relative equality between government and CSOs positively address mutuality and institutional pressures. Challenges remain as estimated strategic significance, diverging capacities and risks to autonomy work against mutuality. Certain challenges are engaged with, but we identified no strategies countering pressures that stem from managerialism within the NMFA, external political pressures and conflicting government objectives. While the programme counters tendencies towards institutionalization of CSOs as insiders, some important challenges to public engagement identified in the literature remain insufficiently addressed.ConclusionsConditions built into policy can address challenges identified in the literature. However, challenges remain that are rooted in wider organizational and political realities. Lessons for practice are: (1) the advocacy role of CSOs can be advanced by building in certain formal conditions and advancing these informally; (2) flexibility allows for collaboration to develop as government and CSOs negotiate their roles, cognizant of pressures that get in the way; (3) challenges rooted beyond the support policy and government agency involved in the collaboration will constrict the power of policy design; (4) public engagement as a foundation for CSOs’ advocacy roles deserves much more attention.

A hierarchical approach for efficient multi-intent dialogue policy learning

This paper proposes a hierarchical method for learning an efficient Dialogue Management (DM) strategy for task-oriented conversations serving multiple intents of a domain. Deep Reinforcement Learning (DRL) networks specializing in individual intents communicate with each other, having the capability of sharing overlapping information across intents. The sharing of information across state space and the presence of global slot tracker prohibits the agent to reask known information. Thus, the system is able to handle sub-dialogues based on subset of intents covered by different Reinforcement Learning (RL) models, thereby, completing the dialogue without again asking already provided information common across intents. The developed system has been demonstrated for “Air Travel” domain. The experimental results indicate that the developed system is efficient, scalable and can serve multiple intents based dialogues adequately. The proposed system when applied to 5-intent dialogue systems attains an improvement of 41% in terms of dialogue length as compared to a single-intent based system serving the same 5-intents.

Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning.

PurposeDeveloping a Dialogue/Virtual Agent (VA) that can handle complex tasks (need) of the user pertaining to multiple intents of a domain is challenging as it requires the agent to simultaneously deal with multiple subtasks. However, majority of these end-to-end dialogue systems incorporate only user semantics as inputs in the learning process and ignore other useful user behavior and information. Sentiment of the user at the time of conversation plays an important role in securing maximum user gratification. So, incorporating sentiment of the user during the policy learning becomes even more crucial, more so when serving composite tasks of the user.MethodologyAs a first step towards enabling the development of sentiment aided VA for multi-intent conversations, this paper proposes a new dataset, annotated with its corresponding intents, slot and sentiment (considering the entire dialogue history) labels, named SentiVA, collected from open-sourced dialogue datasets. In order to integrate these multiple aspects, a Hierarchical Reinforcement Learning (HRL) specifically options based VA is proposed to learn strategies for managing multi-intent conversations. Along with task success based immediate rewards, sentiment based immediate rewards are also incorporated in the hierarchical value functions to make the VA user adaptive.FindingsEmpirically, the paper shows that task based and sentiment based immediate rewards cumulatively are required to ensure successful task completion and attain maximum user satisfaction in a multi-intent scenario instead of any of these rewards alone.Practical implicationsThe eventual evaluators and consumers of dialogue systems are users. Thus, to ensure a fulfilling conversational experience involving maximum user satisfaction requires VA to consider user sentiment at every time-step in its decision making policy.OriginalityThis work is the first attempt in incorporating sentiment based rewards in the HRL framework.

Towards integrated dialogue policy learning for multiple domains and intents using Hierarchical Deep Reinforcement Learning

Creation of Expert and Intelligent Dialogue/Virtual Agent (VA) that can serve complicated and intricate tasks (need) of the user related to multiple domains and its various intents is indeed quite challenging as it necessitates the agent to concurrently handle multiple subtasks in different domains. This paper presents an expert, unified and a generic Deep Reinforcement Learning (DRL) framework that creates dialogue managers competent for managing task-oriented conversations embodying multiple domains along with their various intents and provide the user with an expert system which is a one stop for all queries. In order to address these multiple aspects, the dialogue exchange between the user and the VA is split into hierarchies, so as to isolate and identify subtasks belonging to different domains. The notion of Hierarchical Reinforcement Learning (HRL) specifically options is employed to learn optimal policies in these hierarchies that operate at varying time steps to accomplish the user goal. The dialogue manager encompasses a top-level domain meta-policy, intermediate-level intent meta-policies in order to select amongst varied and multiple subtasks or options and low-level controller policies to select primitive actions to complete the subtask given by the higher-level meta-policies in varying intents and domains. Sharing of controller policies among overlapping subtasks enables the meta-policies to be generic. The proposed expert framework has been demonstrated in the domains of “Air Travel” and “Restaurant”. Experiments as compared to several strong baselines and a state of the art model establish the efficiency of the learned policies and the need for such expert models capable of handling complex and composite tasks.

Investment Case for a Comprehensive Package of Interventions Against Hepatitis B in China: Applied Modeling to Help National Strategy Planning.

BackgroundIn 2016, the first global viral hepatitis elimination targets were endorsed. An estimated one-third of the world’s population of individuals with chronic hepatitis B virus (HBV) infection live in China and liver cancer is the sixth leading cause of mortality, but coverage of first-line antiviral treatment was low. In 2015, China was one of the first countries to initiate a consultative process for a renewed approach to viral hepatitis. We present the investment case for the scale-up of a comprehensive package of HBV interventions.MethodsA dynamic simulation model of HBV was developed and used to simulate the Chinese HBV epidemic. We evaluated the impact, costs, and return on investment of a comprehensive package of prevention and treatment interventions from a societal perspective, incorporating costs of management of end-stage liver disease and lost productivity costs.ResultsDespite the successes of historical vaccination scale-up since 1992, there will be a projected 60 million people still living with HBV in 2030 and 10 million HBV-related deaths, including 5.7 million HBV-related cancer deaths between 2015 and 2030. This could be reduced by 2.1 million by highly active case-finding and optimal antiviral treatment regimens. The package of interventions is likely to have a positive return on investment to society of US$1.57 per US dollar invested.ConclusionsIncreases in HBV-related deaths for the next few decades pose a major public health threat in China. Active case-finding and access to optimal antiviral treatment are required to mitigate this risk. This investment case approach provides a real-world example of how applied modeling can support national dialog and inform policy planning.

Task-Oriented Dialog Systems That Consider Multiple Appropriate Responses under the Same Context

Conversations have an intrinsic one-to-many property, which means that multiple responses can be appropriate for the same dialog context. In task-oriented dialogs, this property leads to different valid dialog policies towards task completion. However, none of the existing task-oriented dialog generation approaches takes this property into account. We propose a Multi-Action Data Augmentation (MADA) framework to utilize the one-to-many property to generate diverse appropriate dialog responses. Specifically, we first use dialog states to summarize the dialog history, and then discover all possible mappings from every dialog state to its different valid system actions. During dialog system training, we enable the current dialog state to map to all valid system actions discovered in the previous process to create additional state-action pairs. By incorporating these additional pairs, the dialog policy learns a balanced action distribution, which further guides the dialog model to generate diverse responses. Experimental results show that the proposed framework consistently improves dialog policy diversity, and results in improved response diversity and appropriateness. Our model obtains state-of-the-art results on MultiWOZ.

MTSS: Learn from Multiple Domain Teachers and Become a Multi-Domain Dialogue Expert

How to build a high-quality multi-domain dialogue system is a challenging work due to its complicated and entangled dialogue state space among each domain, which seriously limits the quality of dialogue policy, and further affects the generated response. In this paper, we propose a novel method to acquire a satisfying policy and subtly circumvent the knotty dialogue state representation problem in the multi-domain setting. Inspired by real school teaching scenarios, our method is composed of multiple domain-specific teachers and a universal student. Each individual teacher only focuses on one specific domain and learns its corresponding domain knowledge and dialogue policy based on a precisely extracted single domain dialogue state representation. Then, these domain-specific teachers impart their domain knowledge and policies to a universal student model and collectively make this student model a multi-domain dialogue expert. Experiment results show that our method reaches competitive results with SOTAs in both multi-domain and single domain setting.

MOSS: End-to-End Dialog System Framework with Modular Supervision

A major bottleneck in training end-to-end task-oriented dialog system is the lack of data. To utilize limited training data more efficiently, we propose Modular Supervision Network (MOSS), an encoder-decoder training framework that could incorporate supervision from various intermediate dialog system modules including natural language understanding, dialog state tracking, dialog policy learning and natural language generation. With only 60% of the training data, MOSS-all (i.e., MOSS with supervision from all four dialog modules) outperforms state-of-the-art models on CamRest676. Moreover, introducing modular supervision has even bigger benefits when the dialog task has a more complex dialog state and action space. With only 40% of the training data, MOSS-all outperforms the state-of-the-art model on a complex laptop network trouble shooting dataset, LaptopNetwork, that we introduced. LaptopNetwork consists of conversations between real customers and customer service agents in Chinese. Moreover, MOSS framework can accommodate dialogs that have supervision from different dialog modules at both framework level and model level. Therefore, MOSS is extremely flexible to update in real-world deployment.

Scalable diagnostic screening of mild cognitive impairment using AI dialogue agent

The search for early biomarkers of mild cognitive impairment (MCI) has been central to the Alzheimer’s Disease (AD) and dementia research community in recent years. To identify MCI status at the earliest possible point, recent studies have shown that linguistic markers such as word choice, utterance and sentence structures can potentially serve as preclinical behavioral markers. Here we present an adaptive dialogue algorithm (an AI-enabled dialogue agent) to identify sequences of questions (a dialogue policy) that distinguish MCI from normal (NL) cognitive status. Our AI agent adapts its questioning strategy based on the user’s previous responses to reach an individualized conversational strategy per user. Because the AI agent is adaptive and scales favorably with additional data, our method provides a potential avenue for large-scale preclinical screening of neurocognitive decline as a new digital biomarker, as well as longitudinal tracking of aging patterns in the outpatient setting.

Policy Reuse for Dialog Management Using Action-Relation Probability

We study the problem of policy adaptation for reinforcement-learning-based dialog management. Policy adaptation is a commonly used technique to alleviate the problem of data sparsity when training a goal-oriented dialog system for a new task (the target task) by using knowledge when learning policies in an existing task. The methods used by current works in dialog policy adaptation need much time and effort for adapting because they use reinforcement learning algorithms to train a new policy for the target task from scratch. In this paper, we show that a dialog policy can be learned without training by reinforcement learning in the target task. In contrast to existing works, our proposed method learns the relation in the form of probability distribution between the action sets of the source and the target tasks. Thus, we can immediately derive a policy for the target task, which significantly reduces the adaptation time. Our experiments show that the proposed method learns a new policy for the target task much more quickly. In addition, the learned policy achieves higher performance than policies created by fine-tuning when the amount of available data on the target task is limited.