Task-oriented Dialogue Research Articles

Investigating the capabilities of large language model-based task-oriented dialogue chatbots and the impact of “suggest replies” function from a learner’s perspective

Investigating the capabilities of large language model-based task-oriented dialogue chatbots and the impact of “suggest replies” function from a learner’s perspective

WaterGPT: Training a Large Language Model to Become a Hydrology Expert

This paper introduces WaterGPT, a language model designed for complex multimodal tasks in hydrology. WaterGPT is applied in three main areas: (1) processing and analyzing data such as images and text in water resources, (2) supporting intelligent decision-making for hydrological tasks, and (3) enabling interdisciplinary information integration and knowledge-based Q&A. The model has achieved promising results. One core aspect of WaterGPT involves the meticulous segmentation of training data for the supervised fine-tuning phase, sourced from real-world data and annotated with high quality using both manual methods and GPT-series model annotations. These data are carefully categorized into four types: knowledge-based, task-oriented, negative samples, and multi-turn dialogues. Additionally, another key component is the development of a multi-agent framework called Water_Agent, which enables WaterGPT to intelligently invoke various tools to solve complex tasks in the field of water resources. This framework handles multimodal data, including text and images, allowing for deep understanding and analysis of complex hydrological environments. Based on this framework, WaterGPT has achieved over a 90% success rate in tasks such as object detection and waterbody extraction. For the waterbody extraction task, using Dice and mIoU metrics, WaterGPT’s performance on high-resolution images from 2013 to 2022 has remained stable, with accuracy exceeding 90%. Moreover, we have constructed a high-quality water resources evaluation dataset, EvalWater, which covers 21 categories and approximately 10,000 questions. Using this dataset, WaterGPT achieved the highest accuracy to date in the field of water resources, reaching 83.09%, which is about 17.83 points higher than GPT-4.

OPERA: Harmonizing Task-Oriented Dialogs and Information Seeking Experience

Existing studies in conversational AI mostly treat task-oriented dialog (TOD) and question answering (QA) as separate tasks. Towards the goal of constructing a conversational agent that can complete user tasks and support information seeking, it is important to develop a system that can handle both TOD and QA with access to various external knowledge sources. In this work, we propose a new task, Open-Book TOD (OB-TOD), which combines TOD with QA and expands the external knowledge sources to include both explicit sources (e.g., the web) and implicit sources (e.g., pre-trained language models). We create a new dataset OB-MultiWOZ, where we enrich TOD sessions with QA-like information-seeking experience grounded on external knowledge. We propose a unified model OPERA ( Op en-book E nd-to-end Task-o r iented Di a log) which can appropriately access explicit and implicit external knowledge to tackle the OB-TOD task. Experimental results show that OPERA outperforms closed-book baselines, highlighting the value of both types of knowledge. 1

Optimizing pipeline task-oriented dialogue systems using post-processing networks

Many studies have proposed methods for optimizing the dialogue performance of an entire pipeline task-oriented dialogue system by jointly training modules in the system using reinforcement learning. However, these methods are limited in that they can only be applied to modules implemented using trainable neural-based methods. To solve this problem, we propose a method for optimizing the dialogue performance of a pipeline system that consists of modules implemented with arbitrary methods for dialogue. With our method, neural-based components called post-processing networks (PPNs) are installed inside such a system to post-process the output of each module. All PPNs are updated to improve the overall dialogue performance of the system by using reinforcement learning, not necessitating that each module be differentiable. Through dialogue simulations and human evaluations on two well-studied task-oriented dialogue datasets, CamRest676 and MultiWOZ, we show that our method can improve the dialogue performance of pipeline systems consisting of various modules. In addition, a comprehensive analysis of the results of the MultiWOZ experiments reveals the patterns of post-processing by PPNs that contribute to the overall dialogue performance of the system.

Combining replay and LoRA for continual learning in natural language understanding

Large language models have significantly improved dialogue systems through enhanced capabilities in understanding queries and generating responses. Despite these enhancements, task-oriented dialogue systems—which power many intelligent assistants—face challenges when adapting to new domains and applications. This challenge arises from a phenomenon known as catastrophic forgetting, where models forget previously acquired knowledge when learning new tasks. This paper addresses this issue through continual learning techniques to preserve previously learned knowledge while seamlessly integrating new tasks and domains. We propose Experience Replay Informative-Low Rank Adaptation or ERI-LoRA, a hybrid continual learning method for natural language understanding in dialogue systems that effectively combines the replay-based methods with parameter-efficient techniques. Our experiments on intent detection and slot-filling tasks demonstrate that ERI-LoRA significantly outperforms competitive baselines in continual learning. The results of our catastrophic forgetting experiments demonstrate that ERI-LoRA maintains robust memory stability in the model, demonstrating its effectiveness in mitigating these effects.

A multi-agent collaborative algorithm for task-oriented dialogue systems

A multi-agent collaborative algorithm for task-oriented dialogue systems

Enhancing Task-Oriented Dialogue Systems through Synchronous Multi-Party Interaction and Multi-Group Virtual Simulation

This paper presents two innovative approaches: a synchronous multi-party dialogue system that engages in simultaneous interactions with multiple users, and multi-group simulations involving virtual user groups to evaluate the resilience of this system. Unlike most other chatbots that communicate with each user independently, our system facilitates information gathering from multiple users and executes 17 administrative tasks for group requests adeptly by leveraging a state machine-based framework for complete control over dialogue flow and a large language model (LLM) for robust context understanding. Assessing such a unique dialogue system poses challenges, as it requires many groups of users to interact with the system concurrently for an extended duration. To address this, we simulate various virtual groups using an LLM, each comprising 10–30 users who may belong to multiple groups, in order to evaluate the efficacy of our system; each user is assigned a persona and allowed to interact freely without scripts. As a result, our system shows average success rates of 87% for task completion and 89% for natural language understanding. Comparatively, our virtual simulation, which has an average success rate of 80%, is juxtaposed with a group of 15 human users, depicting similar task diversity and error trends. To our knowledge, it is the first work to show the LLM’s potential in both task execution and the simulation of a synchronous dialogue system to fully automate administrative tasks.

Dialogue agents 101: a beginner’s guide to critical ingredients for designing effective conversational systems

Abstract Sharing ideas through communication with peers is the primary mode of human interaction. Consequently, extensive research has been conducted in the area of conversational AI, leading to an increase in the availability and diversity of conversational tasks, datasets, and methods. However, with numerous tasks being explored simultaneously, the current landscape of conversational AI has become fragmented. Consequently, initiating a well-thought-out model for a dialogue agent can pose significant challenges for a practitioner. Toward highlighting the critical ingredients needed for a practitioner to design a dialogue agent from scratch, the current study provides a comprehensive overview of the primary characteristics of a dialogue agent, the supporting tasks, their corresponding open-domain datasets, and the methods used to benchmark these datasets. We observe that different methods have been used to tackle distinct dialogue tasks. However, building separate models for each task is costly and does not leverage the correlation among the several tasks of a dialogue agent. As a result, recent trends suggest a shift toward building unified foundation models. To this end, we propose Unit, a Unified dialogue dataset constructed from conversations of varying datasets for different dialogue tasks capturing the nuances for each of them. We then train a Unified dialogue foundation model, GPT-2 $^{\textrm{U}}$ and present a concise comparative performance of GPT-2 $^{\textrm{U}}$ against existing large language models. We also examine the evaluation strategies used to measure the performance of dialogue agents and highlight the scope for future research in the area of conversational AI with a thorough discussion of popular models such as ChatGPT.

Enhancing Task-Oriented Dialogue Modeling through Coreference-Enhanced Contrastive Pre-Training

Pre-trained language models (PLMs) are proficient at understanding context in plain text but often struggle with the nuanced linguistics of task-oriented dialogues. The information exchanges in dialogues and the dynamic role-shifting of speakers contribute to complex coreference and interlinking phenomena across multi-turn interactions. To address these challenges, we propose Coreference-Enhanced Contrastive Pre-training (CECPT), an innovative pre-training framework specifically designed to enhance dialogue modeling. CECPT utilizes unsupervised dialogue datasets to capture both semantic richness and structural coherence. Our experimental results demonstrate that the CECPT model significantly outperforms established baselines in three critical applications: intent recognition, dialogue act prediction, and dialogue state tracking. These findings suggest that CECPT is more adept at following the information flow within dialogues and accurately linking statuses to their respective references.

An emotion-sensitive dialogue policy for task-oriented dialogue system

Reinforcement learning (RL) is an effective method in training dialogue policies to steer the conversation towards successful task completion. However, most RL-based methods only rely on semantic inputs that lack empathy as they ignore the user emotional information. Moreover, these methods suffer from delayed rewards caused by the user simulator returning valuable results only at dialogue end. Recently, some methods have been proposed to learn the reward function together with user emotions, but they omit considering user emotion in each dialogue turn. In this paper, we proposed an emotion-sensitive dialogue policy model (ESDP), it incorporates user emotions information into dialogue policy and selects the optimal action by the combination of top-k actions with the user emotions. The user emotion information in each turn is used as an immediate reward for the current dialogue state to solve sparse rewards and the dependency on termination. Extensive experiments validate that our method outperforms the baseline approaches when combined with different Q-Learning algorithms, and also surpasses other popular existing dialog policies’ performance.

Shift-Reduce Task-Oriented Semantic Parsing with Stack-Transformers

Intelligent voice assistants, such as Apple Siri and Amazon Alexa, are widely used nowadays. These task-oriented dialogue systems require a semantic parsing module in order to process user utterances and understand the action to be performed. This semantic parsing component was initially implemented by rule-based or statistical slot-filling approaches for processing simple queries; however, the appearance of more complex utterances demanded the application of shift-reduce parsers or sequence-to-sequence models. Although shift-reduce approaches were initially considered the most promising option, the emergence of sequence-to-sequence neural systems has propelled them to the forefront as the highest-performing method for this particular task. In this article, we advance the research on shift-reduce semantic parsing for task-oriented dialogue. We implement novel shift-reduce parsers that rely on Stack-Transformers. This framework allows to adequately model transition systems on the transformer neural architecture, notably boosting shift-reduce parsing performance. Furthermore, our approach goes beyond the conventional top-down algorithm: we incorporate alternative bottom-up and in-order transition systems derived from constituency parsing into the realm of task-oriented parsing. We extensively test our approach on multiple domains from the Facebook TOP benchmark, improving over existing shift-reduce parsers and state-of-the-art sequence-to-sequence models in both high-resource and low-resource settings. We also empirically prove that the in-order algorithm substantially outperforms the commonly used top-down strategy. Through the creation of innovative transition systems and harnessing the capabilities of a robust neural architecture, our study showcases the superiority of shift-reduce parsers over leading sequence-to-sequence methods on the main benchmark.

A novel model based on a transformer for intent detection and slot filling

Building task-oriented dialogue systems has become a topic of interest in the research community and industry. The task-oriented dialogue system is a closed-domain dialogue system that can perform specific tasks for users. The natural language understanding module of a task-oriented dialogue system is crucial because it is related to a task-oriented dialogue system that provides correctional services for users. The natural language understanding module of a task-oriented dialogue system performs two tasks: intent detection and slot filling. The intent detection task can be regarded as a text classification task; a classification model is trained to predict the intention of the user from the user’s input information. The slot filling task can be regarded as a sequence analysis task; a sequence analysis model is trained to predict the details of the user’s intention. In this paper, we proposed a novel model based on a transformer encoder for intent detection and slot filling. It follows the encoder-decoder structure, including a vanilla Transformer encoder, a bidirectional LSTM encoder, a linear classification decoder for intent detection, and a conditional random field decoder for slot filling. The experimental results on two public datasets show that our proposed model outperforms the existing methods based on the Transformer and can be combined with BERT to achieve better intent detection and slot filling results.

MCKP: Multi-aspect contextual knowledge-enhanced prompting for conversational recommender systems

Empowering conversational recommender systems (CRSs) with knowledge facilitates the generation of high-quality human-like recommendation proposals to users. Despite substantial endeavors in developing knowledge-based CRSs, they mainly focus on modeling entity knowledge by introducing external data, which cannot sufficiently contribute to understanding complex contextual semantics. Additionally, the source of knowledge relies on external data, ignoring the effective use of original training data, significantly increasing the cost of building efficient CRSs.To tackle the issues above, we explore how to leverage raw dialogue data for modeling and exploiting contextual knowledge to compensate for existing knowledge's shortcomings. Specifically, we first train knowledge generators by constructing multiple standard hybrid prompt templates and answer spaces, which is conducive to perceiving the reasons behind contextual connections from multiple aspects and storing corresponding contextual knowledge. Then, we design answer-guided prompt templates based on the answer space, closing the gap between inference and training of knowledge generators and facilitating the generation of aspect-specific contextual knowledge. Next, we extend our approach by harnessing external knowledge graphs to construct entity knowledge. We propose a local-to-global semantic fusion strategy to fuse knowledge representations from different sources better. Finally, based on the fused knowledge representations, we tailor contextual knowledge-enhanced prompt templates for the CRS to stimulate the reasoning ability of pre-trained language models on dialogue and recommendation tasks. We evaluate the performance of our method on separate evaluation datasets, including the ReDial and the INSPIRED datasets. Based on the Recall, MRR, NDCG, and Distinct evaluation indicators, we demonstrate the effectiveness of the proposed method in dialogue tasks and recommendation tasks. At the same time, we demonstrate that by constructing an interpretable multi-aspect hybrid prompt template, we can effectively mine contextual knowledge from multiple aspects of dialogue.

A multi-agent curiosity reward model for task-oriented dialogue systems

In practical decision-making dialogues, reinforcement learning methods face hurdles due to delays and sparse reward feedback for agents, and in some cases, lack of rewards altogether. These issues can impede efficient learning of dialogue strategies and compromise the performance of the model. To address this challenge, this paper introduces the Multi-Agent Curiosity Reward Model (MACRM) for task-oriented dialog systems. Firstly, in terms of dialog reward mechanisms, a forward dynamics model generates curiosity rewards, which are integrated with extrinsic rewards from the dialog environment feedback to mitigate the problem of sparse rewards resulting from inadequate agent exploration. Secondly, regarding the dialogue strategy training mechanism, an exploration-exploitation approach inspired by organismic exploration is adopted. This approach involves fully exploring the dialogue environment in the early stages and optimally exploiting learned knowledge later, thereby balancing exploration and exploitation and enhancing dialogue strategy learning efficiency. To assess the proposed model's effectiveness, experiments are conducted using the MultiWOZ corpus across three reward environments: (1) extrinsic rewards only, (2) curiosity rewards only, and (3) a combination of both. The experimental results demonstrate that agents employing MACRM exhibit faster learning of dialogue strategies compared to those relying on a single exploratory reward method, effectively addressing reward sparsity and delay issues in practical decision-making scenarios.

Knowledge-aware audio-grounded generative slot filling for limited annotated data

Manually annotating fine-grained slot-value labels for task-oriented dialogue (ToD) systems is an expensive and time-consuming endeavour. This motivates research into slot-filling methods that operate with limited amounts of labelled data. Moreover, the majority of current work on ToD is based solely on text as the input modality, neglecting the additional challenges of imperfect automatic speech recognition (ASR) when working with spoken language. In this work, we propose a Knowledge-Aware Audio-Grounded generative slot filling framework, termed KA2G, that focuses on few-shot and zero-shot slot filling for ToD with speech input. KA2G achieves robust and data-efficient slot filling for speech-based ToD by (1) framing it as a text generation task, (2) grounding text generation additionally in the audio modality, and (3) conditioning on available external knowledge (e.g. a predefined list of possible slot values). We show that combining both modalities within the KA2G framework improves the robustness against ASR errors. Further, the knowledge-aware slot-value generator in KA2G, implemented via a pointer generator mechanism, particularly benefits few-shot and zero-shot learning. Experiments, conducted on the standard speech-based single-turn SLURP dataset and a multi-turn dataset extracted from a commercial ToD system, display strong and consistent gains over prior work, especially in few-shot and zero-shot setups.

Mixture-of-languages Routing for Multilingual Dialogues

We consider multilingual dialogue systems and ask how the performance of a dialogue system can be improved by using information that is available in other languages than the language in which a conversation is being conducted. We adopt a collaborative chair-experts framework, where each expert agent can be either monolingual or cross-lingual, and a chair agent follows a mixture-of-experts procedure for globally optimizing multilingual task-oriented dialogue systems. We propose a mixture-of-languages routing framework that includes four functional components, i.e., input embeddings of multilingual dialogues, language model, pairwise alignment between the representation of every two languages, and mixture-of-languages. We quantify language characteristics of unity and diversity using a number of similarity metrics, i.e., genetic similarity, and word and sentence similarity based on embeddings. Our main finding is that the performance of multilingual task-oriented dialogue systems can be greatly impacted by three key aspects, i.e., data sufficiency, language characteristics, and model design in a mixture-of-languages routing framework.

TCLNet: Turn-level contrastive learning network with reranking for dialogue state tracking

Dialogue state tracking (DST) plays a crucial role in task-oriented dialogue systems, as it interprets and tracks user intentions throughout the dialogue. The accuracy of DST directly impacts system efficiency and user experience. While existing generation-based DST models have shown promising results, effectively modelling long dialogue sequences remains a challenge. Furthermore, the greedy search decoding strategy employed by most previous works for speed is suboptimal because the probability of ground truth tokens is not always the highest. In this study, we propose a novel learning framework that addresses these issues by incorporating turn-level contrastive learning and a reranking module. Specifically, we adopt turn-level contrastive learning to progressively rectify the intermediate states of lengthy dialogues by contrasting data points at a finer granularity. Additionally, the integration of the reranking module empowers the model to make more reliable decisions at each time step by further considering alternative tokens with high probabilities. Experimental results demonstrate that our approach consistently enhances the model’s capability to handle long dialogue sequences and achieves superior performance over strong baselines on both the MultiWOZ 2.1 and MultiWOZ 2.2 datasets.

Prompting large language models for user simulation in task-oriented dialogue systems

Large Language Models (LLMs) have gained widespread popularity due to their instruction-following abilities. In this study, we evaluate their ability in simulating user interactions for task-oriented dialogue (TOD) systems. Our findings demonstrate that prompting LLMs reveals their promising capabilities for training and testing dialogue policies, eliminating the need for domain expertise in crafting complex rules or relying on large annotated data, as required by traditional simulators. The results show that the dialogue system trained with the ChatGPT simulator achieves a success rate of 59%, comparable to a 62% success rate of the dialogue system trained with the manual-rules, agenda-based user simulator (ABUS). Furthermore, the dialogue system trained with the ChatGPT simulator demonstrates better generalization ability compared to the dialogue system trained with the ABUS. Its success rate outperforms that of the dialogue system trained with the ABUS by 4% on GenTUS, 5% on the ChatGPT Simulator, and 3% on the Llama simulator. Nevertheless, LLM-based user simulators provide challenging environment, lexically rich, diverse, and random responses. Llama simulator outperforms the human reference in all lexical diversity metrics with a margin of 0.66 in SE, 0.39 in CE, 0.01 in MSTTR, 0.04 in HDD, and 0.55 in MTLD, while the ChatGPT simulator achieves comparable results. This ultimately contributes to enhancing the system’s ability to generalize more effectively.

Position-aware Interactive Attention Network for multi-intent spoken language understanding

Spoken Language Understanding (SLU) is a crucial component of task-oriented dialog systems. In recent years, there has been increasing attention on multi-intent SLU due to its relevance to complex real-world applications. Most existing joint models only utilize multi-intent information to guide the slot filling, with only a small number of models achieving bidirectional interaction. Additionally, unlike traditional single-intent SLU, multi-intent SLU is scope-dependent, where each intent in a sentence has its specific dependency range. However, current bidirectional joint models often employ a pipeline approach to implement interaction between the two sub-tasks, which fails to fully leverage the semantic information between them and can lead to error propagation. Moreover, attention-based multi-intent joint models do not adequately model the positional relationships between words, resulting in suboptimal overall performance. In this paper, we propose a novel multi-intent SLU model called Position-aware Interactive Attention Network (PIAN), which consists of interactive attention and rotary position embedding. The aim of PIAN is to fully exploit the correlation between the two sub-tasks and capturing the positional dependencies between words. This facilitates mutual guidance and enhancement between the two sub-tasks. Additionally, to address the issue of uncoordinated slot problem generated by traditional slot filling decoders, we employ an MCRF slot filling decoder to constrain the slot labels. We evaluate our model on two public multi-intent SLU datasets, and the experimental results demonstrate that our model achieves state-of-the-art performance on key metrics. Code for this paper is publicly available at https://github.com/puhahahahahaha/SLU_with_Co_PRoE.

Mimetic schemas in multimodal contact-establishing communication

Contact-establishing is deservedly characterized as the major function of communication. Until recently, it has been mostly subjected to linguistic analysis aimed at identifying its discursive markers. Meanwhile, contact-establishing frequently appears in gesturing. The current work develops a cognitive view to gesture and speech alignment, and addresses multimodal contact-establishing communication as mediated by mimetic schemas or bodily schemas shared by communicants who engage in face-to-face and body-to-body interaction. Based on a multimodal experiment where participants engage into task-oriented expository dialogue, we identified two most common contact-establishing recurrent gestures, palm-up-open-hand (PUOH) and palm-down-open-hand (PDOH) gestures contingent on two mimetic schemas, SHOW and RESTRAIN. In the study, we explore the distribution of these two schemas in their sub-schemas in gesture and speech. Following the participants’ contact-establishing PUOH and PDOH gestures (manifesting SHOW and RESTRAIN sub-schemas) and verbal cues (cognitive, pragmatic, and functional semantic dimensions), we determined the multimodal alignment patterns mediated by SHOWand RESTRAIN mimetic schemas in contactestablishing communication. Additionally, the clusters of PUOH and PDOH contactestablishing gestures were determined via their linguistic correspondences since they account for common thinking-for-speaking growth points and the language profiles of PUOH and PDOH contact-establishing gestures. The results allowed to scale the mimetic sub-schemas as manifesting event and referent features such as situatedness, embodiment, performativity, referent definiteness, referent foregrounding, reification, dynamicity, addressing, agentivity, referentiality. Overall, we established that in cognitive dimension non-situated events prevail in PUOH gestures while situated events prevail in PDOH gestures. In pragmatic dimension there is the difference in constativity in PUOH gestures and in performativity in PDOH gestures. In functional semantic dimension PDOH gestures commonly occur with acts, while PUOH gestures are more frequented with attributes. Additionally, we identified that within-cluster distance in PDOH gestures is more obvious than in PUOH gestures, which signifies that the typological differences (mediated indirectly by linguistic characteristics) in PDOH gestures are more distinct. The data obtained provide new evidence in multimodal contact-establishing communication.