Task-oriented Dialogue Research Articles

KddRES: A Multi-level Knowledge-driven Dialogue Dataset for Restaurant Towards Customized Dialogue System

To alleviate the shortage of dialogue datasets for Cantonese, one of the low-resource languages, and facilitate the development of customized task-oriented dialogue systems, we propose KddRES, the first Cantonese Knowledge-driven dialogue dataset for REStaurants. It contains 834 multi-turn dialogues, 8000 utterances, and 26 distinct slots. The slots are hierarchical, and beneath the 26 coarse-grained slots are the additional 16 fine-grained slots. Annotations of dialogue states and dialogue actions at both the user and system sides are provided to suit multiple downstream tasks such as natural language understanding and dialogue state tracking. To effectively detect hierarchical slots, we propose a framework HierBERT by modelling label semantics and relationships between different slots. Experimental results demonstrate that KddRES is more challenging compared with existing datasets due to the introduction of hierarchical slots and our framework is particularly effective in detecting secondary slots and achieving a new state-of-the-art. Given the rich annotation and hierarchical slot structure of KddRES, we hope it will promote research on the development of customized dialogue systems in Cantonese and other conversational AI tasks, such as dialogue state tracking and policy learning.

Model discrepancy policy optimization for task-oriented dialogue

Task-oriented dialogue systems use deep reinforcement learning (DRL) to learn policies, and agent interaction with user models can help the agent enhance its generalization capacity. But user models frequently lack the language complexity of human interlocutors and contain generative errors, and their design biases can impair the agent’s ability to function well in certain situations. In this paper, we incorporate an evaluator based on inverse reinforcement learning into the model to determine the quality of the dialogue of user models in order to recruit high-quality user models for training. We can successfully regulate the quality of training trajectories while maintaining their diversity by constructing a sampling environment distribution to pick high-quality user models to participate in policy learning. The evaluation on the Multiwoz dataset demonstrates that it is capable of successfully improving the dialogue agents’ performance.

Research on discourse role recognition in task-oriented collaborative dialogue

Task-oriented collaborative dialogues have become an indispensable form of communication in our daily work and learning, in which participants exchange ideas and share information to advance goals. It is crucial to automatically analyze participants’ contributions and understand these dialogues relative to individuals with limited attention spans. In this paper, seven Discourse Role (DR) labels are designed to describe discourse’s different roles in collaborative dialogues for goal achievement. We collected about 11K discourses from a publicly available dialogue corpus and annotated them with DR tags to construct a dataset named MRDR (Meeting Recorder Discourse Role). In addition, this paper proposes a novel hierarchical model, STTAHM (Speaker Turn and Topic-Aware Hierarchical Model), for Discourse Role classification. The model is equipped to perceive speaker turn and dialogue topic and can effectively capture the discourse’s local and global semantic information. Experimental results show that our proposed method is effective on the constructed dataset, and the accuracy of Discourse Role classification reaches 86.99%.

Improving Dialogue Management Through Data Optimization

In task-oriented dialogue systems, the ability for users to effortlessly communicate with machines and computers through natural language stands as a critical advancement. Central to these systems is the dialogue manager, a pivotal component tasked with navigating the conversation to effectively meet user goals by selecting the most appropriate response. Traditionally, the development of sophisticated dialogue management has embraced a variety of methodologies, including rule-based systems, reinforcement learning, and supervised learning, all aimed at optimizing response selection in light of user inputs. This research casts a spotlight on the pivotal role of data quality in enhancing the performance of dialogue managers. Through a detailed examination of prevalent errors within acclaimed datasets, such as Multiwoz 2.1 and SGD, we introduce an innovative synthetic dialogue generator designed to control the introduction of errors precisely. Our comprehensive analysis underscores the critical impact of dataset imperfections, especially mislabeling, on the challenges inherent in refining dialogue management processes

Machine Learning-Assisted Speech Analysis for Early Detection of Parkinson's Disease: A Study on Speaker Diarization and Classification Techniques.

Parkinson's disease (PD) is a neurodegenerative disorder characterized by a range of motor and non-motor symptoms. One of the notable non-motor symptoms of PD is the presence of vocal disorders, attributed to the underlying pathophysiological changes in the neural control of the laryngeal and vocal tract musculature. From this perspective, the integration of machine learning (ML) techniques in the analysis of speech signals has significantly contributed to the detection and diagnosis of PD. Particularly, MEL Frequency Cepstral Coefficients (MFCCs) and Gammatone Frequency Cepstral Coefficients (GTCCs) are both feature extraction techniques commonly used in the field of speech and audio signal processing that could exhibit great potential for vocal disorder identification. This study presents a novel approach to the early detection of PD through ML applied to speech analysis, leveraging both MFCCs and GTCCs. The recordings contained in the Mobile Device Voice Recordings at King's College London (MDVR-KCL) dataset were used. These recordings were collected from healthy individuals and PD patients while they read a passage and during a spontaneous conversation on the phone. Particularly, the speech data regarding the spontaneous dialogue task were processed through speaker diarization, a technique that partitions an audio stream into homogeneous segments according to speaker identity. The ML applied to MFCCS and GTCCs allowed us to classify PD patients with a test accuracy of 92.3%. This research further demonstrates the potential to employ mobile phones as a non-invasive, cost-effective tool for the early detection of PD, significantly improving patient prognosis and quality of life.

Personality prediction from task-oriented and open-domain human–machine dialogues

If a dialogue system can predict the personality of a user from dialogue, it will enable the system to adapt to the user’s personality, leading to better task success and user satisfaction. In a recent study, personality prediction was performed using the Myers–Briggs Type Indicator (MBTI) personality traits with a task-oriented human–machine dialogue using an end-to-end (neural-based) system. However, it is still not clear whether such prediction is generally possible for other types of systems and user personality traits. To clarify this, we recruited 378 participants, asked them to fill out four personality questionnaires covering 25 personality traits, and had them perform three rounds of human–machine dialogue with a pipeline task-oriented dialogue system or an end-to-end task-oriented dialogue system. We also had another 186 participants do the same with an open-domain dialogue system. We then constructed BERT-based models to predict the personality traits of the participants from the dialogues. The results showed that prediction accuracy was generally better with open-domain dialogue than with task-oriented dialogue, although Extraversion (one of the Big Five personality traits) could be predicted equally well for both open-domain dialogue and pipeline task-oriented dialogue. We also examined the effect of utilizing different types of dialogue on personality prediction by conducting a cross-comparison of the models trained from the task-oriented and open-domain dialogues. As a result, we clarified that the open-domain dialogue cannot be used to predict personality traits from task-oriented dialogue, and vice versa. We further analyzed the effects of system utterances, task performance, and the round of dialogue with regard to the prediction accuracy.

Intelligent Customer Service System Optimization Based on Artificial Intelligence

To elevate the intelligence of customer service dialogue systems, this article proposes an intelligent customer service system comprising chat dialogue subsystems, task-oriented multi-turn dialogue subsystems, single-turn dialogue subsystems, and an integration model. Firstly, to enhance diversity of responses and improve user experience, particularly in casual chat scenarios, this article presents a Seq2Seq-based approach for multi-answer responses, allowing for more expressive emotional expression in responses. Secondly, to address situations where customers cannot articulate their needs in a single sentence during multi-turn dialogues, this article designs a task-oriented multi-turn dialogue module. It employs intent recognition and slot filling to maintain contextual information throughout the conversation, aiding customers in problem resolution. Lastly, to overcome the current limitation of intelligent customer service models providing relatively one-dimensional answers in specific domains.

Feedback quality and divided attention: exploring commentaries on alignment in task-oriented dialogue

Abstract While studies have shown the importance of listener feedback in dialogue, we still know little about the factors that impact its quality. Feedback can indicate either that the addressee is aligning with the speaker (i.e. ‘positive’ feedback) or that there is some communicative trouble (i.e. ‘negative’ feedback). This study provides an in-depth account of listener feedback in task-oriented dialogue (a director–matcher game), where positive and negative feedback is produced, thus expressing both alignment and misalignment. By manipulating the listener’s cognitive load through a secondary mental task, we measure the effect of divided attention on the quantity and quality of feedback. Our quantitative analysis shows that performance and feedback quantity remain stable across cognitive load conditions, but that the timing and novelty of feedback vary: turns are produced after longer pauses when attention is divided between two speech-focused tasks, and they are more economical (i.e. include more other-repetitions) when unrelated words need to be retained in memory. These findings confirm that cognitive load impacts the quality of listener feedback. Finally, we found that positive feedback is more often generic and shorter than negative feedback and that its proportion increases over time.

Actionable conversational quality indicators for improving task-oriented dialog systems

Abstract Automatic dialog systems have become a mainstream part of online customer service. Many such systems are built, maintained, and improved by customer service specialists, rather than dialog systems engineers and computer programmers. As conversations between people and machines become commonplace, it is critical to understand what is working, what is not, and what actions can be taken to reduce the frequency of inappropriate system responses. These analyses and recommendations need to be presented in terms that directly reflect the user experience rather than the internal dialog processing. This paper introduces and explains the use of Actionable Conversational Quality Indicators (ACQIs), which are used both to recognize parts of dialogs that can be improved and to recommend how to improve them. This combines benefits of previous approaches, some of which have focused on producing dialog quality scoring while others have sought to categorize the types of errors the dialog system is making. We demonstrate the effectiveness of using ACQIs on LivePerson internal dialog systems used in commercial customer service applications and on the publicly available LEGOv2 conversational dataset. We report on the annotation and analysis of conversational datasets showing which ACQIs are important to fix in various situations. The annotated datasets are then used to build a predictive model which uses a turn-based vector embedding of the message texts and achieves a 79% weighted average f1-measure at the task of finding the correct ACQI for a given conversation. We predict that if such a model worked perfectly, the range of potential improvement actions a bot-builder must consider at each turn could be reduced by an average of 81%.

Multi-domain gate and interactive dual attention for multi-domain dialogue state tracking

Multi-domain dialogue state tracking (MDST) is a crucial component of task-oriented dialogue systems. In the context of multi-turn dialogues between the user and the system, MDST necessitates the continuous keeping track of dialogue states based on the information present in the current dialogue utterance and the dialogue states from the preceding turn. Recent work achieves the successful execution of multi-domain dialogue tasks by adopting an approach that treats each state as an individual label, while regrettably neglecting the potential benefits of incorporating domain-specific information associated with these states. Simultaneous, existing models exhibit a deficiency in effectively modelling the explicit correlations between dialogue contextual semantics and dialogue states. In this paper, we introduce the modules of multi-domain gate and interactive dual attention as novel solutions to address the aforementioned concerns. For the efficient exploitation of domain-specific information within states, we leverage the multi-domain gate as indices to amplify the states pertinent to the current utterance domain while filtering out irrelevant states. Interactive dual attention comprises utterance attention and slot attention, effectively modelling the correlation between dialogue utterances and slots. Additionally, interactive dual attention ensures that each dialogue utterance interacts with the slots once to derive all state updates, thereby ensuring computational efficiency. Specifically, slot attention models the associations between slots by incorporating semantic features to forecast updates in slot values. Meanwhile, utterance attention captures the semantics of dialogue context and integrates it with slot name features to generate dialogue states. All the aforementioned modules are designed based on a slot-independent framework, enabling efficient scalability of slots and circumventing issues related to model input limitations. The experimental results on the multi-domain dialogues dataset MultiWOZ 2.4 demonstrate the superior performance of our model compared to the baselines. Additionally, we conduct a comprehensive analysis of the effectiveness of the multi-domain gate and interactive dual attention modules, elucidating their contribution to the performance of the model through visualization and case studies.

A tag-based methodology for the detection of user repair strategies in task-oriented conversational agents

Mutual comprehension is a crucial component that makes a conversation succeed. While it can be easily reached through the cooperation of the parties in human–human dialogues, such cooperation is often lacking in human–computer interaction due to technical problems, leading to broken conversations. Our goal is to work towards an effective detection of breakdowns in a conversation between humans and Conversational Agents (CA), as well as the different repair strategies users adopt when such communication problems occur. In this work, we propose a novel tag system designed to map and classify users’ repair attempts while interacting with a CA. We subsequently present a set of Machine Learning models11https://github.com/rogerferrod/boht. trained to automatize the detection of such repair strategies. The tags are employed in a manual annotation exercise, performed on a publicly available dataset 22The dataset is also available at https://github.com/rogerferrod/boht. of text-based task-oriented conversations. The batch of annotated data was then used to train the neural network-based classifiers. The analysis of the annotations provides interesting insights about users’ behaviour when dealing with breakdowns in a task-oriented dialogue system. The encouraging results obtained from neural models confirm the possibility of automatically recognizing occurrences of misunderstanding between users and CAs on the fly.

A plug-and-play adapter for consistency identification in task-oriented dialogue systems

Task-oriented Dialogue system (ToD) has gained significant attention due to its aim to assist users in accomplishing various tasks. However, the neural network-based dialogue system is like a black box, which may lead to erroneous responses and result in an unfriendly user experience. To address this issue, consistency identification is proposed to prevent generating inconsistent responses. However, the existing consistency identification methods require frequent interaction with the knowledge base, making them susceptible to the introduction of noise during the knowledge base fusion process, ultimately leading to a decline in performance. In this paper, we propose a plug-and-play method for consistency identification, which can introduce external knowledge into the internal reasoning process of the pre-trained language model (PLM) without modifying PLM’s structure. Additionally, we design a new fusion mechanism that effectively fuses the knowledge base information related to the current utterance, which helps the model avoid introducing noise from the irrelevant knowledge base. The experimental results demonstrate that our method achieves state-of-the-art performance on the consistency identification task, improving F1 scores by 2.9% absolute points over the previous methods. Finally, we investigate different knowledge base fusion methods and provide extensive experiments to show the advantages of our proposed method.

End-to-End Task-Oriented Dialogue with Entity-Based Knowledge Inference

End-to-End Task-Oriented Dialogue with Entity-Based Knowledge Inference

Task-Oriented Document-Grounded Dialog Systems by HLTPR@RWTH for DSTC9 and DSTC10

This paper summarizes our contributions to the document-grounded dialog tasks at the 9th and 10th Dialog System Technology Challenges (DSTC9 and DSTC10). In both iterations the task consists of three subtasks: first detect whether the current turn is knowledge seeking, second select a relevant knowledge document, and third generate a response grounded on the selected document. For DSTC9 we proposed different approaches to make the selection task more efficient. The best method, Hierarchical Selection, actually improves the results compared to the original baseline and gives a speedup of 24x. In the DSTC10 iteration of the task, the challenge was to adapt systems trained on written dialogs to perform well on noisy automatic speech recognition transcripts. Therefore, we proposed data augmentation techniques to increase the robustness of the models as well as methods to adapt the style of generated responses to fit well into the proceeding dialog. Additionally, we proposed a noisy channel model that allows for increasing the factuality of the generated responses. In addition to summarizing our previous contributions, in this work, we also report on a few small improvements and reconsider the automatic evaluation metrics for the generation task which have shown a low correlation to human judgments.

A Replaceable Curiosity-Driven Candidate Agent Exploration Approach for Task-Oriented Dialog Policy Learning

A Replaceable Curiosity-Driven Candidate Agent Exploration Approach for Task-Oriented Dialog Policy Learning

Decomposed Deep Q-Network for Coherent Task-Oriented Dialogue Policy Learning

Decomposed Deep Q-Network for Coherent Task-Oriented Dialogue Policy Learning

Two-Step Masked Language Model for Domain-Adapting Multi-Modal Task-Oriented Dialogue Systems

Two-Step Masked Language Model for Domain-Adapting Multi-Modal Task-Oriented Dialogue Systems

Unified Approach for Scalable Task-Oriented Dialogue System

Unified Approach for Scalable Task-Oriented Dialogue System

A Stack-Propagation Framework With Slot Filling for Multi-Domain Dialogue State Tracking.

Dialogue state tracking (DST) is a core component of task-oriented dialogue systems. Recent works focus mainly on end-to-end DST models that omit the spoken language understanding (SLU) module to directly obtain the dialogue state based on a user's dialogue. However, the slot information detected by slot filling in SLU is closely tied to the slot-value pair that needs to be updated in DST. Efficient use of the key slot semantic knowledge obtained by slot filling contributes to improving the performance of DST. Based on this idea, we introduce slot filling as a subtask and build an end-to-end joint model to explicitly integrate the slot information detected by slot filling, which further guides DST. In this article, a novel stack-propagation framework with slot filling for multidomain DST is proposed. The stack-propagation framework is introduced to jointly model slot filling and DST. The framework directly feeds the key slot semantic knowledge detected by slot filling into the DST module. In addition, a slot-masked attention mechanism is designed to enable DST to focus on the key slot information obtained by slot filling. When the slot value is updated, a slot-value softcopy mechanism is designed to enhance the influence of the words marked by key slots. Experiments show that our approach outperforms previous methods and performs outstandingly on two benchmark datasets.

Character expression of a conversational robot for adapting to user personality

User adaptation is important in conversational robots to enhance the user experience and engagement. One way of user adaptation is to change the dialogue behavior. A character (e.g. extrovert or agreeable) can be defined to achieve human-like behaviors and user adaptation, and the appropriate character differs depending on each user. In this study, we investigate user adaptation by character expression of conversational robots. Our character expression model adopts the Big Five traits for controlling four dialogue behaviors: amount of utterance, backchannel frequency, filler frequency, and switching pause length. We cluster the system character and user personality into four classes based on an analysis of a speed dating dialogue corpus, and we found the best combinations between the system character and user personality. We implemented the character expression model into a laboratory guide and a chit-chat robot and conducted subjective experiments, where the subjects talked with robots with four different characters and evaluated their impressions of each character. The results shows that the Role model character was preferred for the task-oriented dialogue of the laboratory guide, but a robot character complementary to the subject personality was preferred for the non-task-oriented dialogue of the speed dating and chit-chat.