Response Generation Model Research Articles

Optimal economic configuration by sharing hydrogen storage while considering distributed demand response in hydrogen-based renewable microgrid

With the issues of the greenhouse effect and environmental pollution globally, a high proportion of renewable energy has been integrated into microgrids. Its interconnection brings many challenges to the electric power industry. This paper proposes a new distributed response strategy through sharing hydrogen storage resources, aiming to solve the supply-demand imbalance in microgrids. First, the uneven power distribution from shared energy storage stations necessitates ensuring a fair power dispatch among users. Next, a microgrid demand response model with multi-constraint conditions and high penetration of wind and solar power is established, incorporating constraints like user power consumption, energy storage capacity, and charging/discharging power while considering differences in user electricity behaviors. Finally, the optimal economic configuration uses Lagrange duality theory to process the integrated models, including the demand response, hydrogen sharing, and power generation models, to make the optimization goals fairer. Results demonstrate that our strategy can adjust user electricity usage according to real-time supply-demand conditions, reducing users' total costs by 39.4 %. Compared with the existing configurations of sharing hydrogen storage configurations, our model uses more accurate utility functions, demonstrating better economic advantages, with users' total costs reduced by 4.89 % and revenue of the shared storage station increased by 4.02 %.

Position-based focal loss for diverse and relevant response generation

Response generation models trained with cross entropy loss suffer from over-general responses due to their preference for high-frequent tokens. Focal loss and anti-focal loss are candidates to solve this problem, but they have their own limitation that they exaggerate only one of relevancy or diversity of responses. Therefore, this paper proposes two novel losses of positional focal loss and adaptive positional focal loss which emphasize relevancy or diversity flexibly according to the position of a target token. The positional focal loss introduces a position function as a weight to the token position, but it tends to underestimate the relevancy for low confident predictions. To tackle this problem, the adaptive positional focal loss balances relevancy and diversity by limiting the effect of over-confident predictions.

EmpCI: Empathetic response generation with common sense and empathetic intent

Empathy plays an important role in human conversations as an ability that enables individuals to understand the emotions and situations of others. Integrating empathy into dialogue systems is a crucial step in making them humanized. Relevant psychological studies have shown that a complete, high-quality empathetic dialogue should consist of the following two stages: (1) Empathetic Perception: the listener needs to perceive the emotional state of the speaker from both cognitive and affective aspects; (2) Empathetic Expression: the appropriate expression is chosen to respond to the perceived information. However, many existing studies on empathetic response generation only focus on one of these stages, resulting in incomplete and insufficiently empathetic responses. To this end, we propose the EmpCI, a two-stage empathetic response generation model that utilizes commonsense knowledge and mixed empathetic intent, respectively. Specifically, we use commonsense knowledge in the first stage to enhance the model’s perception of the user’s emotion and introduce mixed empathetic intent in the second stage to generate responses with appropriate expressions for the perceived information. Finally, we evaluated the EmpCI on the EmpatheticDialogues dataset, and extensive experiment results show that the proposed model outperforms the baselines in both perceiving users’ emotions and generating empathetic responses.

Virtual Assistant by using Large Language Model

Abstract— This paper presents the development of a virtual assistant named “Pragyan,” utilizing a Large Language Model (LLM) for natural language understanding and response generation. The virtual assistant is designed to interact with users through text and speech inputs, providing informative and conversational responses. Pragyan is built with a frontend user interface, a backend server, and a database for efficient communication and data management. The performance evaluation and result analysis demonstrate the effectiveness and user-friendliness of the virtual assistant Keywords: Virtual Assistant, Large Language Model, Natural Language Processing, Frontend, Backend, Database, Performance Evaluation, Result Analysis.

An Adaptive Contextual Relation Model for Improving Response Generation

An Adaptive Contextual Relation Model for Improving Response Generation

Towards Building an Open-Domain Dialogue System Incorporated With Internet Memes

In recent years, Internet memes have been widely used in online chatting. Compared with text-based communication, conversations become more expressive and attractive when Internet memes are incorporated. This paper presents our solutions for the Meme incorporated Open-domain Dialogue (MOD) Challenge of DSTC10, where three tasks are involved: text response modeling, meme retrieval, and meme emotion classification. Firstly, we leverage a large-scale pre-trained dialogue model for coherent and informative response generation. Secondly, based on interaction-based text-matching, our approach can retrieve appropriate memes with good generalization ability. Thirdly, we propose to model the emotion flow (EF) in conversations and introduce an auxiliary task of emotion description prediction (EDP) to boost the performance of meme emotion classification. Experimental results on the MOD dataset demonstrate that our methods can incorporate Internet memes into dialogue systems effectively.

Dual variational generative model and auxiliary retrieval for empathetic response generation by conversational robot

Empathy in human-robot conversations aims to endow the robot with the ability to comprehend user emotion and experience, and then respond to it appropriately. Generally, empathy is embodied in the aspects of both contextual understanding and affective expression, which occur when there exist content and emotion consistencies between context and response. However, previous studies only focus on either aspect. In this paper, we propose a dual variational generative model (DVG) for empathetic response generation to achieve both. Specifically, we integrate an emotion classifier and a variational autoencoder (VAE) into a dual response and context generative model to learn the emotion and content consistencies efficiently. DVG utilizes VAE to mimic the process of context/response understanding. In addition to the generative model, our model can effectively switch to another retrieval system as a fallback solution. Automatic and human evaluations on Japanese and English EmpatheticDialogue datasets demonstrate the effectiveness of our method for empathetic response generation. Furthermore, we evaluate our model's ability in general response generation, which is not specific to empathetic but also chitchatting dialogue system.

Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities

Empathy is a vital factor that contributes to mutual understanding, and joint problem-solving. In recent years, a growing number of studies have recognized the benefits of empathy and started to incorporate empathy in conversational systems. We refer to this topic as empathetic conversational systems. To identify the critical gaps and future opportunities in this topic, this paper examines this rapidly growing field using five review dimensions: (i) conceptual empathy models and frameworks, (ii) adopted empathy-related concepts, (iii) datasets and algorithmic techniques developed, (iv) evaluation strategies, and (v) state-of-the-art approaches. The findings show that most studies have centered on the use of the EMPATHETICDIALOGUES dataset, and the text-based modality dominates research in this field. Studies mainly focused on extracting features from the messages of the users and the conversational systems, with minimal emphasis on user modeling and profiling. Notably, studies that have incorporated emotion causes, external knowledge, and affect matching in the response generation models, have obtained significantly better results. For implementation in diverse real-world settings, we recommend that future studies should address key gaps in areas of detecting and authenticating emotions at the entity level, handling multimodal inputs, displaying more nuanced empathetic behaviors, and encompassing additional dialogue system features.

Aoba_v3 bot: a multimodal chatbot system combining rules and various response generation models

In this paper, we present a multimodal dialogue system combining a neural response generation mechanism, a reranking mechanism, and a rule-based avatar control mechanism. Our system was submitted to the open track at the Fifth Dialogue System Live Competition and won second place. Remarkably, our system received the best human evaluation performance for visual information control (i.e. speaking style of avatar) in the preliminary round. The assessment of the competition evaluators revealed that our system generates natural utterances appropriate to the conversational context and topic with an appealing speaking style. Through the analysis, we found that our devices, such as post-processing on speech recognition results and the final response selection method, are effective, but we also found room for improvement, such as speech recognition errors and challenges in the reranking module.

Keep and Select: Improving Hierarchical Context Modeling for Multi-Turn Response Generation.

Hierarchical context modeling plays an important role in the response generation for multi-turn conversational systems. Previous methods mainly model context as multiple independent utterances and rely on attention mechanisms to obtain the context representation. They tend to ignore the explicit responds-to relationships between adjacent utterances and the special role that the user's latest utterance (the query) plays in determining the success of a conversation. To deal with this, we propose a multi-turn response generation model named KS-CQ, which contains two crucial components, the Keep and the Select modules, to produce a neighbor-aware context representation and a context-enriched query representation. The Keep module recodes each utterance of context by attentively introducing semantics from its prior and posterior neighboring utterances. The Select module treats the context as background information and selectively uses it to enrich the query representing process. Extensive experiments on two benchmark multi-turn conversation datasets demonstrate the effectiveness of our proposal compared with the state-of-the-art baselines in terms of both automatic and human evaluations.

KPT: Keyword-Guided Pre-training for Grounded Dialog Generation

Incorporating external knowledge into the response generation process is essential to building more helpful and reliable dialog agents. However, collecting knowledge-grounded conversations is often costly, calling for a better pre-trained model for grounded dialog generation that generalizes well w.r.t. different types of knowledge. In this work, we propose KPT (Keyword-guided Pre-Training), a novel self-supervised pre-training method for grounded dialog generation without relying on extra knowledge annotation. Specifically, we use a pre-trained language model to extract the most uncertain tokens in the dialog as keywords. With these keywords, we construct two kinds of knowledge and pre-train a knowledge-grounded response generation model, aiming at handling two different scenarios: (1) the knowledge should be faithfully grounded; (2) it can be selectively used. For the former, the grounding knowledge consists of keywords extracted from the response. For the latter, the grounding knowledge is additionally augmented with keywords extracted from other utterances in the same dialog. Since the knowledge is extracted from the dialog itself, KPT can be easily performed on a large volume and variety of dialogue data. We considered three data sources (open-domain, task-oriented, conversational QA) with a total of 2.5M dialogues. We conduct extensive experiments on various few-shot knowledge-grounded generation tasks, including grounding on dialog acts, knowledge graphs, persona descriptions, and Wikipedia passages. Our comprehensive experiments and analyses demonstrate that KPT consistently outperforms state-of-the-art methods on these tasks with diverse grounding knowledge.

Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery

Abstract In this paper, we conduct the first study on spurious correlations for open-domain response generation models based on a corpus CGDialog curated by ourselves. The current models indeed suffer from spurious correlations and have a tendency to generate irrelevant and generic responses. Inspired by causal discovery algorithms, we propose a novel model-agnostic method for training and inference using a conditional independence classifier. The classifier is trained by a constrained self-training method, coined ConSTrain, to overcome data sparsity. The experimental results based on both human and automatic evaluation show that our method significantly outperforms the competitive baselines in terms of relevance, informativeness, and fluency.

GERP: A Personality-Based Emotional Response Generation Model

It is important for chatbots to emotionally communicate with users. However, most emotional response generation models generate responses simply based on a specified emotion, neglecting the impacts of speaker’s personality on emotional expression. In this work, we propose a novel model named GERP to generate emotional responses based on the pre-defined personality. GERP simulates the emotion conversion process of humans during the conversation to make the chatbot more anthropomorphic. GERP adopts the OCEAN model to precisely define the chatbot’s personality. It can generate the response containing the emotion predicted based on the personality. Specifically, to select the most-appropriate response, a proposed beam evaluator was integrated into GERP. A Chinese sentiment vocabulary and a Chinese emotional response dataset were constructed to facilitate the emotional response generation task. The effectiveness and superiority of the proposed model over five baseline models was verified by the experiments.

Open-Domain Response Generation in Low-Resource Settings using Self-Supervised Pre-Training of Warm-Started Transformers

Learning response generation models constitute the main component of building open-domain dialogue systems. However, training open-domain response generation models requires large amounts of labeled data and pre-trained language generation models that are often nonexistent for low-resource languages. In this article, we propose a framework for training open-domain response generation models in low-resource settings. We consider Dialectal Arabic (DA) as a working example. The framework starts by warm-starting a transformer-based encoder-decoder with pre-trained language model parameters. Next, the resultant encoder-decoder model is adapted to DA by employing self-supervised pre-training on large-scale unlabeled data in the desired dialect. Finally, the model is fine-tuned on a very small labeled dataset for open-domain response generation. The results show significant performance improvements on three spoken Arabic dialects after adopting the framework’s three stages, highlighted by higher BLEU and lower Perplexity scores compared with multiple baseline models. Specifically, our models are capable of generating fluent responses in multiple dialects with an average human-evaluated fluency score above 4. Our data is made publicly available.

Toward Explainable Dialogue System Using Two-stage Response Generation

In recent years, neural networks have achieved impressive performance on dialogue response generation. However, most of these models still suffer from some shortcomings, such as yielding uninformative responses and lacking explainable ability. This article proposes a Two-stage Dialogue Response Generation model (TSRG), which specifies a method to generate diverse and informative responses based on an interpretable procedure between stages. TSRG involves a two-stage framework that generates a candidate response first and then instantiates it as the final response. The positional information and a resident token are injected into the candidate response to stabilize the multi-stage framework, alleviating the shortcomings in the multi-stage framework. Additionally, TSRG allows adjusting and interpreting the interaction pattern between the two generation stages, making the generation response somewhat explainable and controllable. We evaluate the proposed model on three dialogue datasets that contain millions of single-turn message-response pairs between web users. The results show that, compared with the previous multi-stage dialogue generation models, TSRG can produce more diverse and informative responses and maintain fluency and relevance.

Neural response generation for task completion using conversational knowledge graph.

Effective dialogue generation for task completion is challenging to build. The task requires the response generation system to generate the responses consistent with intent and slot values, have diversity in response and be able to handle multiple domains. The response also needs to be context relevant with respect to the previous utterances in the conversation. In this paper, we build six different models containing Bi-directional Long Short Term Memory (Bi-LSTM) and Bidirectional Encoder Representations from Transformers (BERT) based encoders. To effectively generate the correct slot values, we implement a copy mechanism at the decoder side. To capture the conversation context and the current state of the conversation we introduce a simple heuristic to build a conversational knowledge graph. Using this novel algorithm we are able to capture important aspects in a conversation. This conversational knowledge-graph is then used by our response generation model to generate more relevant and consistent responses. Using this knowledge-graph we do not need the entire utterance history, rather only the last utterance to capture the conversational context. We conduct experiments showing the effectiveness of the knowledge-graph in capturing the context and generating good response. We compare these results against hierarchical-encoder-decoder models and show that the use of triples from the conversational knowledge-graph is an effective method to capture context and the user requirement. Using this knowledge-graph we show an average performance gain of 0.75 BLEU score across different models. Similar results also hold true across different manual evaluation metrics.

Open Domain Response Generation Guided by Retrieved Conversations

Open domain response generation is the task of creating a response given a user query in any topics/domain. Limited by context and reference information, responses generated by current systems are often “bland” or generic. In this paper, we combine a response generation model with a retrieval system that searches for relevant utterances and responses. The generation model has two main components: a keyword extraction module and a two-stage transformer. The keyword extraction module aims to extract two types of keywords in an unsupervised fashion from the retrieved results: (1) keywords in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">query</i> not found in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">retrieved utterances</i> (DiffKey), and (2) overlapping keywords among the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">retrieved responses</i> (SimKey). Given these keywords, the two-stage transformer first decides where to insert the keywords in the response, and the second generates the full response given the location of the keywords. The keyword extraction module and the two-stage transformer are connected in a single network, and so our system is trained end-to-end. Experimental results on Cornell Movie-Dialog corpus, Douban and Weibo demonstrate that our model outperforms state-of-the-art systems in terms of ROUGE, relevance scores and human evaluation.

Analyzing the Capabilities of a Hybrid Response Generation Model for an Empathetic Conversational Agent

Research on the use of conversational agents or chatbots to provide alternative and accessible mental health interventions has gained much interest in recent years. Designed to engage human users through natural and empathetic conversations, these chatbots have shown their potential applications in pre-emptive healthcare that emphasizes the importance of helping individuals maintain their optimal mental health and well-being. The ability of chatbots to process user input and produce pertinent and sympathetic responses to dynamically adjust to the context of a discussion is, however, constrained by the usage of retrieval-based models. Neural-based generative models, currently applied in open-domain dialogue and text generation systems, may be able to address the limitations of retrieval-based models. In this paper, we present Virtual Hope (VHope), a conversational agent that combines retrieval-based and generative models to perform its role as a therapist capable of generating empathetic responses to enrich the conversation. The best performing generative model, derived from training DialoGPT with the EmpatheticDialogues dataset and a local mental well-being dataset, yielded a perplexity score of 9.977. Two variations for embedding retrieval and generative models in the chatbot’s conversation flow yielded insignificant differences based on users’ evaluation of VHope’s performance. Results from experts’ analysis of the conversation logs showed that the responses generated by VHope were 67% relevant, 78% human-like, and 79% empathic. Future improvements may include the use of a larger, human-based empathetic dataset for enhanced retrieval model’s conversation design and generative model’s fine-tuning.

Knowledge-grounded dialogue modelling with dialogue-state tracking, domain tracking, and entity extraction

As knowledge-grounded dialogue systems are attracting intense research interest, technology that facilitates reference to various types of external knowledge as dialogue-system data is also developing. A knowledge-grounded dialogue system must be capable of (1) accurately interpret the conversation content, (2) determining whether external knowledge should be referenced for the current turn, (3) identifying the external knowledge to be referenced, and (4) generating a response. If the referenced external knowledge is in the form of a question–answer pair, this pair is closely related to the domain to which the external knowledge belongs and the entity to which it pertains. This study proposes a knowledge-grounded dialogue system that predicts the domain and entity associated with a question–answer pair referenced by the dialogue system; the system then leverages this information to effectively implement the above three external-knowledge-based capabilities. As the dialogue data associated with external knowledge is diverse, adaptation to the slots and entities of different dialogue datasets is challenging. In this study, the Triple Copy (TripPy) model, which is one of the leading benchmark models for dialogue-state tracking with the Multi-domain Wizard-of-Oz (MultiWOZ) dataset, is further developed to adapt to DSTC10 data for the external knowledge-based dialogue system; hence, dialogue content is effectively interpreted. The developed knowledge-grounded dialogue system incorporates knowledge-seeking turn detection, knowledge selection, and knowledge-grounded response generation models. The model, in DSTC10 dataset, achieves 15.49%p, 12.54%p, and 36.45%p improvements over the baseline, the state-of-the-art model for the previous version of the data (DSTC9 dataset), in terms of the F1 score, Recall@1 score, and BLEU-1 score, respectively. Moreover, in a dialogue-state tracking task for DSTC10 dataset, a 15.41%p improvement in joint goal accuracy score is achieved compared to the TripPy model.

Multi-Dimension Attention for Multi-Turn Dialog Generation (Student Abstract)

We present a generative neural model for open and multi-turn dialog response generation that relies on a multi-dimension attention process to account for the semantic interdependence between the generated words and the conversational history, so as to identify all the words and utterances that influence each generated response. The performance of the model is evaluated on the wide scope DailyDialog corpus and a comparison is made with two other generative neural architectures, using several machine metrics. The results show that the proposed model improves the state of the art for generation accuracy, and its multi-dimension attention allows for a more detailed tracking of the influential words and utterances in the dialog history for response explainability by the dialog history.