Dialogue Systems Research Articles

The Evaluation of a Code-Switched Sepedi-English Automatic Speech Recognition System

Speech technology is a field that encompasses various techniques and tools used to enable machines to interact with speech, such as automatic speech recognition (ASR), spoken dialog systems, and others, allowing a device to capture spoken words through a microphone from a human speaker. End-to-end approaches such as Connectionist Temporal Classification (CTC) and attention-based methods are the most used for the development of ASR systems. However, these techniques were commonly used for research and development for many high-resourced languages with large amounts of speech data for training and evaluation, leaving low-resource languages relatively underdeveloped. While the CTC method has been successfully used for other languages, its effectiveness for the Sepedi language remains uncertain. In this study, we present the evaluation of the SepediEnglish code-switched automatic speech recognition system. This end-to-end system was developed using the Sepedi Prompted Code Switching corpus and the CTC approach. The performance of the system was evaluated using both the NCHLT Sepedi test corpus and the Sepedi Prompted Code Switching corpus. The model produced the lowest WER of 41.9%, however, the model faced challenges in recognizing the Sepedi only text

Multiparty conversation via multirobot system: incorporation of nonverbal user responses for continued conversation

Recent years have seen the advent of conversational humanoid robots. Implementing multiple robots is a promising approach for continuing human–robot conversations when a robot encounters speech-recognition errors. This strategy is effective for the user's verbal responses; however, people generally use nonverbal responses, such as nods or smiles, to respond to the interlocutor. In this study, we proposed a conversational strategy for twin robots in which the second robot recognized a human's nonverbal responses and interrupted the conversation by mentioning these responses. Moreover, we developed an interrogative dialogue system using simple nonverbal recognition modules. To verify the effectiveness of this conversational strategy, a subject-based experiment was conducted in which humans conversed with the two robots, and their impressions were evaluated through a questionnaire. We compared the outcomes under two conditions: whether the second robot's interruption was inserted or otherwise. Our results indicate that language-only ineffectiveness in responding to ambiguous responses regarding satisfaction and comprehension was mitigated. Thus, such a dialogue system is advantageous because it can facilitate robust conversation without relying solely on speech recognition.

Seq2Set2Seq: A Two-stage Disentangled Method for Reply Keyword Generation in Social Media

Social media produces large amounts of content every day. How to predict the potential influences of the contents from a social reply feedback perspective is a key issue that has not been explored. Thus, we propose a novel task named reply keyword prediction in social media, which aims to predict the keywords in the potential replies in as many aspects as possible. One prerequisite challenge is that the accessible social media datasets labeling such keywords remain absent. To solve this issue, we propose a new dataset, 1 to study the reply keyword prediction in social media. This task could be seen as a single-turn dialogue keyword prediction for open-domain dialogue system. However, existing methods for dialogue keyword prediction cannot be adopted directly, which has two main drawbacks. First, they do not provide an explicit mechanism to model topic complementarity between keywords which is crucial in social media to controllably model all aspects of replies. Second, the collocations of keywords are not explicitly modeled, which also makes it less controllable to optimize for fine-grained prediction since the context information is much less than that in dialogue. To address these issues, we propose a two-stage disentangled framework, which can optimize the complementarity and collocation explicitly in a disentangled fashion. In the first stage, we use a sequence-to-set paradigm via multi-label prediction and determinantal point processes, to generate a set of keyword seeds satisfying the complementarity. In the second stage, we adopt a set-to-sequence paradigm via seq2seq model with the keyword seeds guidance from the set, to generate the more-fine-grained keywords with collocation. Experiments show that this method can generate not only a more diverse set of keywords but also more relevant and consistent keywords. Furthermore, the keywords obtained based on this method can achieve better reply generation results in the retrieval-based system than others.

Margin and Shared Proxies: Advanced Proxy Anchor Loss for Out-of-Domain Intent Classification

Out-of-Domain (OOD) intent classification is an important task for a dialog system, as it allows for appropriate responses to be generated. Previous studies aiming to solve the OOD intent classification task have generally adopted metric learning methods to generate decision boundaries in the embedding space. However, these existing methods struggle to capture the high-dimensional semantic features of data, as they learn decision boundary using scalar distances. They also use generated OOD samples for learning. However, such OOD samples are biased, and they cannot include all real-world OOD intents, thus representing a limitation. In the current paper, we attempt to overcome these challenges by using Advanced Proxy-Anchor loss, which introduces a margin proxy and shared proxy. First, to generate a decision boundary that has the high-dimensional semantic features of training data, we use a margin proxy for learnable embedding vectors. Next, the shared proxy, which is shared by all In-Domain (IND) samples, is introduced to make it possible to learn the discriminative feature between IND intents and OOD intent, ultimately leading to the improved classification of OOD samples. We conduct evaluations of the proposed method using three benchmark datasets. The experimental results demonstrate that our method achieved an improved performance compared to the methods described in previous studies.

KddRES: A Multi-level Knowledge-driven Dialogue Dataset for Restaurant Towards Customized Dialogue System

To alleviate the shortage of dialogue datasets for Cantonese, one of the low-resource languages, and facilitate the development of customized task-oriented dialogue systems, we propose KddRES, the first Cantonese Knowledge-driven dialogue dataset for REStaurants. It contains 834 multi-turn dialogues, 8000 utterances, and 26 distinct slots. The slots are hierarchical, and beneath the 26 coarse-grained slots are the additional 16 fine-grained slots. Annotations of dialogue states and dialogue actions at both the user and system sides are provided to suit multiple downstream tasks such as natural language understanding and dialogue state tracking. To effectively detect hierarchical slots, we propose a framework HierBERT by modelling label semantics and relationships between different slots. Experimental results demonstrate that KddRES is more challenging compared with existing datasets due to the introduction of hierarchical slots and our framework is particularly effective in detecting secondary slots and achieving a new state-of-the-art. Given the rich annotation and hierarchical slot structure of KddRES, we hope it will promote research on the development of customized dialogue systems in Cantonese and other conversational AI tasks, such as dialogue state tracking and policy learning.

Model discrepancy policy optimization for task-oriented dialogue

Task-oriented dialogue systems use deep reinforcement learning (DRL) to learn policies, and agent interaction with user models can help the agent enhance its generalization capacity. But user models frequently lack the language complexity of human interlocutors and contain generative errors, and their design biases can impair the agent’s ability to function well in certain situations. In this paper, we incorporate an evaluator based on inverse reinforcement learning into the model to determine the quality of the dialogue of user models in order to recruit high-quality user models for training. We can successfully regulate the quality of training trajectories while maintaining their diversity by constructing a sampling environment distribution to pick high-quality user models to participate in policy learning. The evaluation on the Multiwoz dataset demonstrates that it is capable of successfully improving the dialogue agents’ performance.

Dialogue Generation Model with Hierarchical Encoding and Semantic Segmentation of Dialogue Context

Dialogue generation, as a crucial subtask of dialogue systems, is garnering increasing attention in the field of Natural Language Processing (NLP). The success of dialogue generation relies on effectively utilizing context information to ensure coherent and diverse responses. However, current approaches heavily rely on external sources rather than leveraging the inherent dialogue content. We propose a new approach to address this challenge by introducing semantic segmentation from the field of image processing into NLP. Our contribution lies in the development of a Dialogue Generation model with Hierarchical Encoding and Semantic segmentation of dialogue Context, which is called DGHESC. This model is topic and speaker-aware, capturing the flow of topic and speaker information within the dialogue context using a hierarchical transformer-based framework. Specifically, we extract semantic information at the word-level for each utterance, segment the dialogue context based on topic and speaker semantics, and employ attention mechanisms to model the context at the utterance-level. Experimental results on two open-domain datasets demonstrate the effectiveness of DGHESC. It enhances response quality and achieves state-of-the-art performances on the datasets.

Previous Experience Matters: An in-Person Investigation of Expectations in Human–Robot Interaction

The human–robot interaction (HRI) field goes beyond the mere technical aspects of developing robots, often investigating how humans perceive robots. Human perceptions and behavior are determined, in part, by expectations. Given the impact of expectations on behavior, it is important to understand what expectations individuals bring into HRI settings and how those expectations may affect their interactions with the robot over time. For many people, social robots are not a common part of their experiences, thus any expectations they have of social robots are likely shaped by other sources. As a result, individual expectations coming into HRI settings may be highly variable. Although there has been some recent interest in expectations within the field, there is an overall lack of empirical investigation into its impacts on HRI, especially in-person robot interactions. To this end, a within-subject in-person study (N=31\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N=31$$\\end{document}) was performed where participants were instructed to engage in open conversation with the social robot Pepper during two 2.5 min sessions. The robot was equipped with a custom dialogue system based on the GPT-3 large language model, allowing autonomous responses to verbal input. Participants’ affective changes towards the robot were assessed using three questionnaires, NARS, RAS, commonly used in HRI studies, and Closeness, based on the IOS scale. In addition to the three standard questionnaires, a custom question was administered to capture participants’ views on robot capabilities. All measures were collected three times, before the interaction with the robot, after the first interaction with the robot, and after the second interaction with the robot. Results revealed that participants to large degrees stayed with the expectations they had coming into the study, and in contrast to our hypothesis, none of the measured scales moved towards a common mean. Moreover, previous experience with robots was revealed to be a major factor of how participants experienced the robot in the study. These results could be interpreted as implying that expectations of robots are to large degrees decided before interactions with the robot, and that these expectations do not necessarily change as a result of the interaction. Results reveal a strong connection to how expectations are studied in social psychology and human-human interaction, underpinning its relevance for HRI research.

Improving Dialogue Management Through Data Optimization

In task-oriented dialogue systems, the ability for users to effortlessly communicate with machines and computers through natural language stands as a critical advancement. Central to these systems is the dialogue manager, a pivotal component tasked with navigating the conversation to effectively meet user goals by selecting the most appropriate response. Traditionally, the development of sophisticated dialogue management has embraced a variety of methodologies, including rule-based systems, reinforcement learning, and supervised learning, all aimed at optimizing response selection in light of user inputs. This research casts a spotlight on the pivotal role of data quality in enhancing the performance of dialogue managers. Through a detailed examination of prevalent errors within acclaimed datasets, such as Multiwoz 2.1 and SGD, we introduce an innovative synthetic dialogue generator designed to control the introduction of errors precisely. Our comprehensive analysis underscores the critical impact of dataset imperfections, especially mislabeling, on the challenges inherent in refining dialogue management processes

DSTEA: Improving Dialogue State Tracking via Entity Adaptive pre-training

Dialogue State Tracking (DST) is critical for comprehensively interpreting user and system utterances, thereby forming the cornerstone of efficient dialogue systems. Despite past research efforts focused on enhancing DST performance through alterations to the model structure or integrating additional features like graph relations, they often require additional pre-training with external dialogue corpora. In this study, we propose DSTEA, improving Dialogue State Tracking via Entity Adaptive pre-training, which can enhance the encoder through by intensively training key entities in dialogue utterances. DSTEA identifies these pivotal entities from input dialogues utilizing four different methods: ontology information, named-entity recognition, the spaCy toolkit, and the flair library. Subsequently, it employs selective knowledge masking to train the model effectively. Remarkably, DSTEA only requires pre-training without the direct infusion of extra knowledge into the DST model. This approach results in substantial performance improvements of four robust DST models on MultiWOZ 2.0, 2.1, and 2.2, with joint goal accuracy witnessing an increase of up to 2.69% (from 52.41% to 55.10%). Comparative experiments considering various entity types and different entity adaptive pre-training configurations, such as masking strategy and masking rate, further validated the efficacy of DSTEA.

Distinct but correct: generating diversified and entity-revised medical response

Medical Dialogue Generation (MDG) is intended to build a medical dialogue system for intelligent consultation, which can communicate with patients in real-time, thereby improving the efficiency of clinical diagnosis with broad application prospects. This paper presents our proposed framework for the Chinese MDG organized by the 2021 China conference on knowledge graph and semantic computing (CCKS) competition, which requires generating context-consistent and medically meaningful responses conditioned on the dialogue history. In our framework, we propose a pipeline system composed of entity prediction and entity-aware dialogue generation, by adding predicted entities to the dialogue model with a fusion mechanism, thereby utilizing information from different sources. At the decoding stage, we propose a new decoding mechanism named Entity-revised Diverse Beam Search (EDBS) to improve entity correctness and promote the length and quality of the final response. The proposed method wins both the CCKS and the International Conference on Learning Representations (ICLR) 2021 Workshop Machine Learning for Preventing and Combating Pandemics (MLPCP) Track 1 Entity-aware MED competitions, which demonstrate the practicality and effectiveness of our method.

An Overview of the Application of Convolutional Neural Networks inSentiment Analysis

The field of natural language processing, or NLP, uses its understanding of human language to find practical solutions to issues. It mainly includes two parts: the core task and the application. The core task represents the common problem that needs to be solved in various natural language application directions. It includes language models, morphology, grammar analysis, semantic analysis, etc. At the same time, the application section focuses on specific natural language processing tasks such as machine translation, information retrieval, question-answering systems, dialogue systems, etc. Natural language processing has made a significant contribution to the development of human society and the economy and provides strong support for all aspects of research work. Opinion mining, or sentiment analysis, is a subfield of natural language processing that develops systems for identifying and extracting ideas from text. Sentiment analysis is a hot topic since it has many practical applications. Many opinion-expressing texts are available on review sites, forums, blogs, and social media as the amount of publicly available information on the Internet grows. This unstructured information can then be automatically transformed into structured data about products, services, brands, politics, or other topics on which people can express their opinions using sentiment analysis systems. This information can be used for marketing analytics, public relations, product reviews, network sponsor ratings, product feedback, and customer service. With the rapid growth of labeled sample data sets and the notable enhancement in graphics processor (GPU) performance, convolutional neural network research has advanced rapidly and achieved remarkable leads to various computer vision tasks. By reviewing the application of CNN, we see that convolutional operations are naturally suitable for some text processing and, thus, naturally suitable for the background of sentiment analysis.

A multimodal dialogue system for customer service based on user personality adaptation and dialogue strategies

ABSTRACT This paper presents a multimodal dialogue system with user personality adaptation and multiple nonverbal behavior generation for a conversational android robot that plays a customer service role in recommending travel plans. It is useful to estimate the user's personality to realize a dialogue system that adapts the dialogue strategy to individual users. To improve the user's impression of the conversation robot as a customer service agent, we must control the nonverbal behavior of the robot, such as its facial expressions and motions. Moreover, we need to design the dialogue strategy to provide information and make users enjoy the conversation by adding an ice-breaking step. Against this background, we develop a dialogue system prototype that adjusts the dialogue strategy based on the user's personality, as estimated by a pre-trained multimodal sensing model. Furthermore, we implement appropriate nonverbal behavior patterns for each situation, including the voice, motion, and facial expressions of the android robot, to provide it appropriate and polite service agent behaviors. A personality assessment game is introduced into the dialogue to prevent the user from becoming bored. The game plays an ice-breaking role in the dialogue. To let the user engage with the robot, we implement a complement to the user as a dialogue act. We evaluate the proposed multimodal dialogue system for customer service by means of participation in a dialogue robot competition. The fair evaluation results from 26 dialogue users who conversed with multiple systems show that the proposed dialogue system achieved the best score among 13 teams participating in the preliminary round of the competition, indicating that the user personality adaptation and various elements implemented in our dialogue system improve the dialogue experience of general users. The results of a third-party evaluation and multiple regression analysis show that natural response is the most important factor contributing to user impression.

Personality prediction from task-oriented and open-domain human–machine dialogues

If a dialogue system can predict the personality of a user from dialogue, it will enable the system to adapt to the user’s personality, leading to better task success and user satisfaction. In a recent study, personality prediction was performed using the Myers–Briggs Type Indicator (MBTI) personality traits with a task-oriented human–machine dialogue using an end-to-end (neural-based) system. However, it is still not clear whether such prediction is generally possible for other types of systems and user personality traits. To clarify this, we recruited 378 participants, asked them to fill out four personality questionnaires covering 25 personality traits, and had them perform three rounds of human–machine dialogue with a pipeline task-oriented dialogue system or an end-to-end task-oriented dialogue system. We also had another 186 participants do the same with an open-domain dialogue system. We then constructed BERT-based models to predict the personality traits of the participants from the dialogues. The results showed that prediction accuracy was generally better with open-domain dialogue than with task-oriented dialogue, although Extraversion (one of the Big Five personality traits) could be predicted equally well for both open-domain dialogue and pipeline task-oriented dialogue. We also examined the effect of utilizing different types of dialogue on personality prediction by conducting a cross-comparison of the models trained from the task-oriented and open-domain dialogues. As a result, we clarified that the open-domain dialogue cannot be used to predict personality traits from task-oriented dialogue, and vice versa. We further analyzed the effects of system utterances, task performance, and the round of dialogue with regard to the prediction accuracy.

A Modular Framework for Domain-Specific Conversational Systems Powered by Never-Ending Learning

Complex and long interactions (e.g., a change of topic during a conversation) justify the use of dialog systems to develop task-oriented chatbots and intelligent virtual assistants. The development of dialog systems requires considerable effort and takes more time to deliver when compared to regular BotBuilder tools because of time-consuming tasks such as training machine learning models and low module reusability. We propose a framework for building scalable dialog systems for specific domains using the semi-automatic methods of corpus, ontology, and code development. By separating the dialog application logic from domain knowledge in the form of an ontology, we were able to create a dialog system for the banking domain in the Portuguese language and quickly change the domain of the conversation by changing the ontology. Moreover, by using the principles of never-ending learning, unsupported operations or unanswered questions create triggers for system knowledge demand that can be gathered from external sources and added to the ontology, augmenting the system’s ability to respond to more questions over time.

Special issue on multimodal processing and robotics for dialogue systems (Part II)

Special issue on multimodal processing and robotics for dialogue systems (Part II)

Phantom in the opera: adversarial music attack for robot dialogue system

This study explores the vulnerability of robot dialogue systems' automatic speech recognition (ASR) module to adversarial music attacks. Specifically, we explore music as a natural camouflage for such attacks. We propose a novel method to hide ghost speech commands in a music clip by slightly perturbing its raw waveform. We apply our attack on an industry-popular ASR model, namely the time-delay neural network (TDNN), widely used for speech and speaker recognition. Our experiment demonstrates that adversarial music crafted by our attack can easily mislead industry-level TDNN models into picking up ghost commands with high success rates. However, it sounds no different from the original music to the human ear. This reveals a serious threat by adversarial music to robot dialogue systems, calling for effective defenses against such stealthy attacks.

Dialogue System for Casual and Continuous Word Learning

Facilitating causal and continuous learning in individuals with low motivation presents a formidable challenge owing to the diverse methods and systems available for vocabulary acquisition. Therefore, this study developed a non‐task‐oriented dialogue system to facilitate casual and continuous word learning. Specifically, dialogue systems capable of processing text‐based inputs, encompassing words registered by learners, were formulated. Learners could engage in the acquisition of these words through informal conversation. The evaluation results demonstrated the effectiveness of the proposed method for fostering casual and continuous learning. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

Intelligent Customer Service System Optimization Based on Artificial Intelligence

To elevate the intelligence of customer service dialogue systems, this article proposes an intelligent customer service system comprising chat dialogue subsystems, task-oriented multi-turn dialogue subsystems, single-turn dialogue subsystems, and an integration model. Firstly, to enhance diversity of responses and improve user experience, particularly in casual chat scenarios, this article presents a Seq2Seq-based approach for multi-answer responses, allowing for more expressive emotional expression in responses. Secondly, to address situations where customers cannot articulate their needs in a single sentence during multi-turn dialogues, this article designs a task-oriented multi-turn dialogue module. It employs intent recognition and slot filling to maintain contextual information throughout the conversation, aiding customers in problem resolution. Lastly, to overcome the current limitation of intelligent customer service models providing relatively one-dimensional answers in specific domains.

Correction: A review of dialogue systems: current trends and future directions

Correction: A review of dialogue systems: current trends and future directions