Dialogue Agent Research Articles

Discourse Behavior of Older Adults Interacting with a Dialogue Agent Competent in Multiple Topics

We present a conversational agent designed to provide realistic conversational practice to older adults at risk of isolation or social anxiety, and show the results of a content analysis on a corpus of data collected from experiments with elderly patients interacting with our system. The conversational agent, represented by a virtual avatar, is designed to hold multiple sessions of casual conversation with older adults. Throughout each interaction, the system analyzes the prosodic and nonverbal behavior of users and provides feedback to the user in the form of periodic comments and suggestions on how to improve. Our avatar is unique in its ability to hold natural dialogues on a wide range of everyday topics—27 topics in three groups, developed in collaboration with a team of gerontologists. The three groups vary in “degrees of intimacy,” and as such in degrees of cognitive difficulty for the user. After collecting data from nine participants who interacted with the avatar for seven to nine sessions over a period of 3 to 4 weeks, we present results concerning dialogue behavior and inferred sentiment of the users. Analysis of the dialogues reveals correlations such as greater elaborateness for more difficult topics, increasing elaborateness with successive sessions, stronger sentiments in topics concerned with life goals rather than routine activities, and stronger self-disclosure for more intimate topics. In addition to their intrinsic interest, these results also reflect positively on the sophistication and practical applicability of our dialogue system.

Building Goal-Oriented Dialogue Systems with Situated Visual Context

Goal-oriented dialogue agents can comfortably utilize the conversational context and understand its users' goals. However, in visually driven user experiences, these conversational agents are also required to make sense of the screen context in order to provide a proper interactive experience. In this paper, we propose a novel multimodal conversational framework where the dialogue agent's next action and their arguments are derived jointly conditioned both on the conversational and the visual context. We demonstrate the proposed approach via a prototypical furniture shopping experience for a multimodal virtual assistant.

Dual Task Framework for Improving Persona-Grounded Dialogue Dataset

This paper introduces a simple yet effective data-centric approach for the task of improving persona-conditioned dialogue agents. Prior model-centric approaches unquestioningly depend on the raw crowdsourced benchmark datasets such as Persona-Chat. In contrast, we aim to fix annotation artifacts in benchmarking, which is orthogonally applicable to any dialogue model. Specifically, we augment relevant personas to improve dialogue dataset/agent, by leveraging the primal-dual structure of the two tasks, predicting dialogue responses and personas based on each other. Experiments on Persona-Chat show that our approach outperforms pre-trained LMs by an 11.7 point gain in terms of accuracy.

Knowing What to Say: Towards knowledge grounded code-mixed response generation for open-domain conversations

Inculcating knowledge in the dialogue agents is an important step towards creating any agent more human-like. Hence, the use of knowledge while conversing is crucial for building interactive and engaging systems. Most existing works for developing social conversation systems focus on monolingual discussions, with little research on multilingual or code-mixed conversations. Therefore, in this work, we propose generating knowledge-aware code-mixed responses for building end-to-end code-mixed dialogue systems. We design a reinforced transformer framework that uses task-specific rewards for training the entire system. In addition, we utilize a knowledge selection module that captures the appropriate knowledge and generates responses using a deliberation decoder. We introduce a Knowledge aware Code-Mixed (KCM) dataset that consists of conversations grounded in knowledge for four Indian languages (Hindi, Bengali, Gujarati, and Telugu) and two European languages (Spanish and French). Quantitative and qualitative analysis show that the proposed framework on the newly created KCM dataset performs superior to the existing baselines for all the metrics.

Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems

Abstract As more users across the world are interacting with dialog agents in their daily life, there is a need for better speech understanding that calls for renewed attention to the dynamics between research in automatic speech recognition (ASR) and natural language understanding (NLU). We briefly review these research areas and lay out the current relationship between them. In light of the observations we make in this article, we argue that (1) NLU should be cognizant of the presence of ASR models being used upstream in a dialog system’s pipeline, (2) ASR should be able to learn from errors found in NLU, (3) there is a need for end-to-end data sets that provide semantic annotations on spoken input, (4) there should be stronger collaboration between ASR and NLU research communities.

Exploring Self-Care Needs Of African American And Hispanic/Latino Heart Failure Patients Outside Clinical Setting

The Health Belief Model is a theoretical model in which perceived barriers to action influence health behaviors. Unavailability of personalized, clear and timely healthcare information is a potential perceived barrier to effective patient self-care. Technological solutions like dialogue agents (DA) emulating health educators (HE) can address this barrier, reducing poor health outcomes in African American (AA) and Hispanic/Latino (H/L) heart failure (HF) patients. Objective: Use information collected inside and outside the clinical setting to develop a technological solution addressing the lack of personalized, clear, and timely information for AA and H/L HF patients, thereby reducing perceived barriers to self-care. Methods: We developed a prototype DA by combining existing literature with themes extracted from 20 transcripts of HE-HF patient interactions (18 AA & 2 H/L). Previously, we found that HE-HF patient interaction in clinical environments is controlled by HE and may not reveal information gaps of the patient. To identify these gaps and assess our prototype DA's performance, we conducted four 1:1 interviews with HF patients (3 AA & 1 H/L). In the interviews, we asked 12 questions on self-care routines (food, medication, understanding their condition & support systems) and then observed HF patient interaction with the DA. Interview content was compared to prior transcripts to uncover unique interview topics. We assessed the effectiveness of the DA by measuring the number of questions asked by patients and the accuracy of responses. Results: Participants reported a high level of satisfaction with their HE and displayed curiosity about topics not covered during HE-patient interactions, e.g., triggers for HF, impact on family, etc. A participant reported difficulty coming up with questions for the HE, attributing it to the clinical environment and language barriers. Questions asked by participants to the DA thematically matched our previous findings but had more personalized content, e.g., in the transcripts, patient questions addressed food items from the handout used by HE (e.g., ham, canned food, etc.) but when interacting with the DA, they were more likely to ask about foods they regularly ate (e.g., chocolate, fruit, etc.) When interacting with the DA, participants asked 4.25 questions on average. The DA responded correctly 44% of the time. Incorrect answers were either content errors (highly personalized question content or structure) and system errors (DA could not understand the user's speech.) Conclusion: There is a qualitative difference in the questions asked outside and inside a clinical setting by AA & H/L HF patients. Tech-based solutions built using information only from controlled clinical settings will replicate shortcomings in the existing HE process. By addressing needs of HF patients outside clinical settings, our DA is uniquely poised to reduce barriers to good self-care with personalized and timely information.

Say What? Learner Reactions to Unexpected Agent Dialogue Moves

For maximally efficient and effective conversation-based intelligent tutoring systems, designers must understand the expectations carried by their intended learners. Strategies programmed into the agents may be interpreted differently by the learners. For example, conversational heuristics in these systems may be biased against false alarms in identifying wrong answers (potentially accepting more incorrect answers), or they may avoid directly answering learner-generated questions in an attempt to encourage more open-ended input. Regardless of pedagogical merit, the learner may view these agents’ dialogue moves as bugs rather than features and respond by disengaging or distrusting future interactions. We test this effect by orchestrating situations in agent-based instruction of electrical engineering topics (through an intelligent tutoring system called AutoTutor) where the pedagogical agent behaves in ways likely counter to learner expectations. To better understand the learning experience of the user, we then measure learner response via think-aloud protocol, eye-tracking, and direct interview. We find that, with few exceptions, learners do not reason that the actions are meant as instructional or technical strategies, but instead broadly understood as errors. This indicates a need for either alteration of agent dialogue strategies, or else additional (implicit or explicit) introduction of the strategies to productively shape learners’ interactions with the system.

A knowledge infused context driven dialogue agent for disease diagnosis using hierarchical reinforcement learning

Disease diagnosis is an essential and critical step in any disease treatment process. Automatic disease diagnosis has gained immense popularity in recent years, owing to its effiacy, easy accessability and reliablity. The major challenges for the diagnosis agent are inevitably large action space (symptoms) and varieties of diseases, which demand either rich domain knowledge or an intelligent learning framework. We propose a novel knowledge-infused context-driven (KI-CD) hierarchical reinforcement learning (HRL) based diagnosis dialogue system, which leverages a bayesian learning-inspired symptom investigation module called potential candidate module (PCM) for aiding context-aware, knowledge grounded symptom investigation. The PCM module serves as a context and knowledge guiding companion for lower-level policies, leveraging current context and disease-symptom knowledge to identify candidate diseases and potential symptoms, and reinforcing the agent for conducting an intelligent and context guided symptom investigation with the information enriched state and an additional critic known as learner critic. The knowledge-guided symptom investigation extracts an adequate set of symptoms for disease identification, whereas the context-aware symptom investigation aspect substantially improves topic (symptom) transition and enhances user experiences. Furthermore, we also propose and incorporate a hierarchical disease classifier (HDC) with the model for alleviating symptom state sparsity issues, which has led to a significant improvement in disease classification accuracy. The proposed framework outperforms the current state-of-the-art method on the multiple benchmarked datasets and, in all evaluation metrics other than dialogue length (diagnosis success rate, average match rate, symptom identification rate, and disease classification accuracy by 7.1 %, 0.23 %, 19.67 % and 8.04 %, respectively), which firmly establishes the effiacy of the proposed bayesian learning-inspired context-driven symptom investigation and disease diagnosis methodology 11The code is available at https://github.com/NLP-RL/KI-CD..

Investigating a neural all pass warp in modern TTS applications

We present a neural implementation of the all pass warp (APW) previously used for vocal tract length normalisation. This includes an efficient back-propagation, which can easily be integrated in modern neural network frameworks. The APW offers a low-dimensional control to alter the spectrum, which by design generalises over different speakers. We investigate the APW in two tasks required for future dialogue or translation agents, and provide a fairly thorough literature review for both: (1) Zero-shot speaker adaptation to allow keeping the source speaker identity with very small amounts of data. Experiments show increased speaker similarity and prove that the APW increases the generalisability of a multi-speaker model. (2) Emotional speech synthesis to translate or produce affective cues. To the best of our knowledge this is the first attempt on emotional speech synthesis with an APW. While the APW is not able to increase expressiveness or audio quality, our analysis shows that the warping correlates with the level of valence in the emotion. This work should enable future research on emotion translation during machine translation.

A persona aware persuasive dialogue policy for dynamic and co-operative goal setting

Contextualization:In recent years, the popularity of virtual agents particularly task-oriented dialogue agents has increased immensely due to their effectiveness and simplicity in various domains such as industry, e-commerce, and health. Problem:In real-world, users do not have always a predefined and immutable goal, i.e., they may upgrade/downgrade/update their task goal dynamically depending upon their utility and the serving capability of the assisting agent. However, the existing Virtual Agents (VAs) in the dialogue literature relinquish and yield dialogue failure if they find any dynamic goal setting or goal unavailability scenarios. Contributions and methodology:Motivated by these inabilities of existing VAs, we propose some intelligent and expert Dialogue Agents (A Unified Dialogue Agent and Multi-agent Dialogue system) that can deal with dynamic and goal unavailability situations to elevate both user satisfaction and the agent’s utility particularly task success rate. The proposed architecture incorporates a goal guiding module namely Dynamic and Co-Operative Goal Driven Module (DyCoGDM), which traces goal status and resolves goal discrepancy through dynamic goal setting (Goal Formulator) and personalized persuasion (Goal Persuader) mechanisms. We also created and annotated a dialogue corpus because of unavailability of such corpus featured with dynamic and goal unavailability scenarios. Findings and implications:Our proposed method outperforms several baselines and state of the art methods in all evaluation metrics. The proposed VA is capable of dealing with dynamic goals and goal unavailability scenarios effectively. The study found that the persona aware persuasive dialogue agent outperforms generalized persuasive dialogue agent by a large margin. Furthermore, we also observed that the task oriented reward is the most essential reward for training a reinforcement learning based agent and agents trained without task based reward do not even converge.

Development of a listening system by guiding the robot's gaze to the nodding images of spoken words

For the elderly, the act of “speaking” is important to maintain their physical function, and it is known that listening with nods and giving responses are important to encourage utterances. Also, in elderly nursing homes, conversations are often started when they look at images. So far, we proposed a listening system that promotes utterances by recognizing the user’s speech contents by voice input, and presenting images related to the words in speech on the display, and giving nodding reaction from the listener’s InterActor and images, and demonstrated the effectiveness. In this study, we have developed a listening system that guides gaze to the image presented by utterance using a robot that is familiar as a dialogue agent. In this development, we have developed two methods, one is to guide the user to gaze at the image while communicating face-to-face with the robot to the user, and the other is to guide the robot to gaze at the screen by the robot taking a position diagonally to the right of the user.

Say What? Learner Reactions to Unexpected Agent Dialogue Moves

For maximally efficient and effective conversation-based intelligent tutoring systems, designers must understand the expectations carried by their intended learners. Strategies programmed into the agents may be interpreted differently by the learners. For example, conversational heuristics in these systems may be biased against false alarms in identifying wrong answers (potentially accepting more incorrect answers), or they may avoid directly answering learner-generated questions in an attempt to encourage more open-ended input. Regardless of pedagogical merit, the learner may view these agents’ dialogue moves as bugs rather than features and respond by disengaging or distrusting future interactions. We test this effect by orchestrating situations in agent-based instruction of electrical engineering topics (through an intelligent tutoring system called AutoTutor) where the pedagogical agent behaves in ways likely counter to learner expectations. To better understand the learning experience of the user, we then measure learner response via think-aloud protocol, eye-tracking, and direct interview. We find that, with few exceptions, learners do not reason that the actions are meant as instructional or technical strategies, but instead broadly understood as errors. This indicates a need for either alteration of agent dialogue strategies, or else additional (implicit or explicit) introduction of the strategies to productively shape learners’ interactions with the system.

事前学習済みモデルと音声対話エージェントを用いた顔識別システムの開発

Our research group is currently studying and developing listening services using spoken dialogue agent and IoT technologies to assist the "mind" of the elderly at home. However, the user identification function, an essential part of the service, has not yet been realized. It is difficult to determine the identity of the person who interacts with the spoken dialogue agent. Although with the rapid development of the artificial intelligence field, various smart devices and services using deep learning have appeared in the face recognition technology, problems exist, including costs and computational resources to build and apply a recognition model. This study aims to develop a facial identification system using pre-learned model and spoken dialogue agent. Our key ideas include the automatic generation of learning data by spoken dialogue between the user and the agent and the acquisition and comparison of facial features using a pre-trained model. In this way, our face identification system can be quickly introduced and expected with only a general-purpose computer and a Web camera, without needing a conventional Internet connection and manual labeling of learning data.

Abstract 10698: Towards a Dialogue Agent to Assist African American and Hispanic/Latino Heart Failure Patients with Dietary Restrictions

Background: Restricting dietary sodium is a common recommendation to address symptoms in heart failure (HF) patients. However, using the information provided on nutrition labels can be complex, leading to lapses in adherence. This problem is compounded by insufficient information on ethnic preferences in dietary habits. Based on our previous findings we developed a prototype dialogue agent (HF-DA) to assist African American (AA) and Hispanic Latino (H/L) patients with their HF self-care needs, including understanding nutrition labels and providing practical information per FDA recommendations. Objective: Compare HF-DA against existing technology in the public domain in assisting AA and H/L HF patients to make healthy food choices. Methods: We conducted a review of existing technology in two ways. 1: Google search engine: We measured the number of actions (typing, scrolling, clicking) required to find a source for a food’s sodium content matching the U.S. Department of Agriculture’s recommendations and used a calculator to understand the total sodium in a desired serving size. We identified relevant search terms by analyzing the commonly used words in the transcripts from our HF patient education sessions with their health educators (18 AA and 2 H/L) and 1:1 interviews with HF patients (3 AA and 1 H/L). 2: Google Play Store: We downloaded the top 15 apps using the search term “heart failure” and explored their functionality. Results: Google’s search engine required users to perform 1 action to identify the sodium content and percent daily value but then needed 6-8 actions (dependent on query length) to calculate sodium content of total serving size. Our HF-DA provides this information in 4 actions. Of the 15 apps available in the Play Store, none had any features for understanding nutrition labels. Conclusion: Our preliminary analysis revealed there are very few culturally sensitive technological resources available to assist HF patients in understanding nutrition labels. These resources require a patient to understand nutrition labels, perform arithmetic, and recall dietary restrictions. HF-DA is capable of promoting healthy eating habits by aiding HF patients' decision-making process by providing reliable information in fewer steps.

Emotion-sensitive deep dyna-Q learning for task-completion dialogue policy learning

In recent years, task-oriented dialogue systems have received extensive attention from academia and industry. Training a dialogue agent through reinforcement learning is often costly because it requires many interactions with real users. Although the Deep Dyna-Q (DDQ) framework uses simulation experience to alleviate the cost of direct reinforcement learning, it still suffers from challenges such as delayed rewards and policy degradation. This paper proposes an Emotion-Sensitive Deep Dyna-Q (ES-DDQ) model which: (1) presents an emotional world model that considers emotion-related cues to improve the ability of the traditional DDQ framework to model and simulate users, and (2) designs two kinds of emotion-related immediate rewards to mitigate the delayed reward problem. Experimental results show that our proposed approach effectively simulates users’ behaviors and is superior to the state-of-the-art benchmarks.

Impact of Chatbot in Transforming the Face of Retailing- An Empirical Model of Antecedents and Outcomes

Background: Chatbot has emerged as a significant resolution to the swiftly growing customer care demands in recent times. Chatbot has emerged as one of the biggest technological disruptions. In simple words, a software agent that facilitates interaction between computers and humans in natural language. It is a simulated, intellectual dialogue agent functional in a range of consumer engagement circumstances. It is the easiest and simplest means to enable interaction between the retailers and the customers. Aim: Most of the research work on chatbot is concerned with the technical aspects. The recent research on chatbot pays little attention on the impact it has created on users’ experience. Through this work, the author made an effort to know the customer-oriented impact that the chatbot has on the shoppers. The aim of this study was to develop and empirically test a framework that identifies the customer oriented attributes of chatbot and the impact these attributes create on customers. Objective: Objectives: The study intended to bridge the gap between conceptual and actual attributes and their applications on the subject of Chatbot. The following research objectives addressed the various aspects of Chatbot affecting the different characteristics of consumers’ shopping behaviors: Objective: a) Identification of various attributes of chatbot that bears an impression on consumer’s shopping behavior. Objective: b) Evaluation of the impact of chatbot on consumer’s shopping behavior that leads to the development of chatbot usage and adoption by the customer. Methodology: For the purpose of analysis, the author carried out Factor analysis and Multiple regression using SPSS version 23 for the identification of various attributes of Chatbot and their impact on shoppers. A self-administered questionnaire was developed. Industry experts in the field of retailing and academician evaluated the questionnaire. Primary information from the respondents was gathered using this questionnaire. The questionnaire comprised of Likert scale on a scale of 1 to 5 where 1 stands for strongly disagree and 5 stands for strongly agree. Data was collected from 126 respondents, out of which 111 respondents were finally considered for study and analysis purpose. Results/Findings: The empirical results showed that the study identified various attributes of chatbot like trust, usefulness, satisfaction, readiness to use and accessibility. It was also found that chatbot greatly influenced the customers in providing them with shopping experience, which can be very helpful to the businesses for increasing the sales and creating repurchase intention in the customers. Conclusion: The recent research on chatbot pays little attention on the impact it is creating on customers who are actually interacting with it on regular basis. The research paper extends information for understanding and appreciating the customer oriented attributes of artificially intelligent Chatbot. In this regard, the author developed a model framework and proposed the attributes identified. Through the work, author also made an effort to empirically test the impact of the identified attributes on the shoppers.

Visual Dialog Agent Based On Deep Q Learning and Memory Module Networks

In the past many years, it has been observed that there has been an increase in methods to solve problems and the solution involves a combination of Computer Vision and Natural Language Processing. New algorithms and systems are emerging and are being developed every day to solve the above-mentioned kind of problems. Visual Dialog Agent is one of them. This kind of system utilizes both Computer Vision and Natural Language Processing algorithms. With this technology many variants of Visual Dialog Agents have been designed till date and many exclusive algorithms are created for Visual Dialog Agent. In this paper we propose an idea to create a Visual Dialog Agent which utilizes the present state of art End to End Memory Module Networks along with Reinforcement Learning Policies to answer the questions prompted by the user and as well understand the inclination of the user in the conversation which it holds. The goal of the proposed Visual Dialog Agent is to have a more engaging conversation with the highest user inclination.

Automatic Curriculum Learning With Over-repetition Penalty for Dialogue Policy Learning

Dialogue policy learning based on reinforcement learning is difficult to be applied to real users to train dialogue agents from scratch because of the high cost. User simulators, which choose random user goals for the dialogue agent to train on, have been considered as an affordable substitute for real users. However, this random sampling method ignores the law of human learning, making the learned dialogue policy inefficient and unstable. We propose a novel framework, Automatic Curriculum Learning-based Deep Q-Network (ACL-DQN), which replaces the traditional random sampling method with a teacher policy model to realize the dialogue policy for automatic curriculum learning. The teacher model arranges a meaningful ordered curriculum and automatically adjusts it by monitoring the learning progress of the dialogue agent and the over-repetition penalty without any requirement of prior knowledge. The learning progress of the dialogue agent reflects the relationship between the dialogue agent's ability and the sampled goals' difficulty for sample efficiency. The over-repetition penalty guarantees the sampled diversity. Experiments show that the ACL-DQN significantly improves the effectiveness and stability of dialogue tasks with a statistically significant margin. Furthermore, the framework can be further improved by equipping with different curriculum schedules, which demonstrates that the framework has strong generalizability.

Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection

Real-life applications, heavily relying on machine learning, such as dialog systems, demand for out-of-domain detection methods. Intent classification models should be equipped with a mechanism to distinguish seen intents from unseen ones so that the dialog agent is capable of rejecting the latter and avoiding undesired behavior. However, despite increasing attention paid to the task, the best practices for out-of-domain intent detection have not yet been fully established. This paper conducts a thorough comparison of out-of-domain intent detection methods. We prioritize the methods, not requiring access to out-of-domain data during training, gathering of which is extremely time- and labor-consuming due to lexical and stylistic variation of user utterances. We evaluate multiple contextual encoders and methods, proven to be efficient, on three common datasets for intent classification, expanded with out-of-domain utterances. Our main findings show that fine-tuning Transformer-based encoders on in-domain data leads to superior results. Mahalanobis distance, together with utterance representations, derived from Transformer-based encoders, outperform other methods by a wide margin(1-5% in terms of AUROC) and establish new state-of-the-art results for all datasets. The broader analysis shows that the reason for success lies in the fact that the fine-tuned Transformer is capable of constructing homogeneous representations of in-domain utterances, revealing geometrical disparity to out of domain utterances. In turn, the Mahalanobis distance captures this disparity easily.

Decontextualization: Making Sentences Stand-Alone

Abstract Models for question answering, dialogue agents, and summarization often interpret the meaning of a sentence in a rich context and use that meaning in a new context. Taking excerpts of text can be problematic, as key pieces may not be explicit in a local window. We isolate and define the problem of sentence decontextualization: taking a sentence together with its context and rewriting it to be interpretable out of context, while preserving its meaning. We describe an annotation procedure, collect data on the Wikipedia corpus, and use the data to train models to automatically decontextualize sentences. We present preliminary studies that show the value of sentence decontextualization in a user-facing task, and as preprocessing for systems that perform document understanding. We argue that decontextualization is an important subtask in many downstream applications, and that the definitions and resources provided can benefit tasks that operate on sentences that occur in a richer context.