Computer Dialogue Research Articles

UMLS-Based Approach for Developing VoiS: Voice-Activated Conversational Agent for Self-Management of Multiple Chronic Conditions

This abstract proposes a system design for an ontology-based conversational agent (CA) for the self-management of chronic conditions. The proposed system plans to integrate the largest medical ontology, the Unified Medical Language System (UMLS) (Bodenreider, 2004), aiming to narrow the vocabulary gaps between health professionals and patients and to make the agent more responsive to the users.
 Recently, conversational agents (CAs) like ChatGPT, or computer dialog systems that simulate human-to-human communication in natural language, have risen and gained popularity in various health contexts (Bin Sawad et al., 2022; Montenegro et al., 2019) including self-management of chronic conditions (Griffin et al., 2020). Despite their potential, the currently available CAs are often criticized for lacking the capability to understand natural language inputs (Montenegro et al., 2019). This limitation can be highlighted in medical areas due to the known vocabulary gaps between health professionals and health consumers. The knowledge-grounded dialog flow for CAs presents the potential to lift this limitation, making CAs naturally converse with their users.
 In the proposed voice-activated self-monitoring support (VoiS) application, the research team plans to integrate the UMLS to make the agent better understand lay terms from patients and properly map those terms to medical concepts. This automated process is expected to improve the user experience in two folds: a) promote the quality of communication between the patients and health providers and b) make the VoiS app more responsive to user inputs, overcoming accepting only constrained user inputs (e.g., multiple choice of utterance).

Affective Computing: Recent Advances, Challenges, and Future Trends

Affective computing is a rapidly growing multidisciplinary field that encompasses computer science, engineering, psychology, neuroscience, and other related disciplines. Although the literature in this field has progressively grown and matured, the lack of a comprehensive bibliometric analysis limits the overall understanding of the theory, technical methods, and applications of affective computing. This review presents a quantitative analysis of 33,448 articles published in the period from 1997 to 2023, identifying challenges, calling attention to 10 technology trends, and outlining a blueprint for future applications. The findings reveal that the emerging forces represented by China and India are transforming the global research landscape in affective computing, injecting transformative power and fostering extensive collaborations, while emphasizing the need for more consensus regarding standard setting and ethical norms. The 5 core research themes identified via cluster analysis not only represent key areas of international interest but also indicate new research frontiers. Important trends in affective computing include the establishment of large-scale datasets, the use of both data and knowledge to drive innovation, fine-grained sentiment classification, and multimodal fusion, among others. Amid rapid iteration and technology upgrades, affective computing has great application prospects in fields such as brain–computer interfaces, empathic human–computer dialogue, assisted decision-making, and virtual reality.

Intent recognition model based on sequential information and sentence features

In human–computer dialogue systems, intent recognition is crucial for determining the intentions or purposes of users during interactions with the system, enabling the system to provide appropriate responses or actions. This paper proposes an intent recognition model that integrates sequential information and sentence structural features. Specifically, the approach utilizes a CNN to capture local salient features in the text, followed by a BILSTM to extract sequential information within the local context. The sequential information is then fed into a multi-head attention mechanism to focus on more relevant sequential details. Additionally, the original data is processed by BERT to extract sentence structural features. Finally, the sequential information features and sentence structural features are concatenated and fused to achieve enhanced intent recognition performance. This approach effectively leverages contextual and semantic information within the text, leading to improved accuracy in intent recognition. Experimental results demonstrate the effectiveness of the proposed method in intent recognition and its high relevance for practical applications.

“Standard Text” Relational Classification Model Based on Concatenated Word Vector Attention and Feature Concatenation

The task of relation classification is an important pre-task in natural language processing tasks. Relation classification can provide a high-quality corpus for tasks such as machine translation, human–computer dialogue, and structured text generation. In the process of the digitalization of standards, identifying the entity relationship in the standard text is an important prerequisite for the formation of subsequent standard knowledge. Only by accurately labeling the relationship between entities can there be higher efficiency and accuracy in the subsequent formation of knowledge bases and knowledge maps. This study proposes a standard text relational classification model based on cascaded word vector attention and feature splicing. The model was compared and ablated on our labeled standard text Chinese dataset. At the same time, in order to prove the performance of the model, the above experiments were carried out on two general English datasets, SemEval-2010 Task 8 and KBP37. On standard text datasets and general datasets, the model proposed in this study achieved excellent results.

Understanding computational dialogue understanding.

In this paper, we first explain why human-like dialogue understanding is so difficult for artificial intelligence (AI). We discuss various methods for testing the understanding capabilities of dialogue systems. Our review of the development of dialogue systems over five decades focuses on the transition from closed-domain to open-domain systems and their extension to multi-modal, multi-party and multi-lingual dialogues. From being somewhat of a niche topic of AI research for the first 40 years, it has made newspaper headlines in recent years and is now being discussed by political leaders at events such as the World Economic Forum in Davos. We ask whether large language models are super-parrots or a milestone towards human-like dialogue understanding and how they relate to what we know about language processing in the human brain. Using ChatGPT as an example, we present some limitations of this approach to dialogue systems. Finally, we present some lessons learned from our 40 years of research in this field about system architecture principles: symmetric multi-modality, no presentation without representation and anticipation feedback loops. We conclude with a discussion of grand challenges such as satisfying conversational maxims and the European Language Equality Act through massive digital multi-linguality-perhaps enabled by interactive machine learning with human trainers. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'.

Introduction to the Special Issue on Computational Dialogue Modelling

Introduction to the Special Issue on Computational Dialogue Modelling

Comparison of Common Ground Models for Human--Computer Dialogue

Common ground processes [26] can improve performance in communication tasks [72, 42, 43, 24], and understanding these processes will likely benefit human--computer dialogue interfaces. However, there are multiple proposed theories with different implications for interface design. Fusaroli and Tylén [40] achieved a direct comparison by designing two models: one based on alignment theory and the other based on complementarity theory that encapsulated interpersonal synergy and audience design. The current research used these models, extending them to differentiate between interpersonal synergy and audience design. Few studies have tested multiple common ground models against tasks representative of envisioned human--computer interaction (HCI) applications. We report on four such tests, which allowed examination of generalizability of findings. Results supported the complementarity models over the alignment model, and were suggestive of the audience design variant of complementarity, providing guidance for HCI design that differs from contemporary approaches.

Using Eye Tracking and Heart-Rate Activity to Examine Crossmodal Correspondences QoE in Mulsemedia

Different senses provide us with information of various levels of precision and enable us to construct a more precise representation of the world. Rich multisensory simulations are thus beneficial for comprehension, memory reinforcement, or retention of information. Crossmodal mappings refer to the systematic associations often made between different sensory modalities (e.g., high pitch is matched with angular shapes) and govern multisensory processing. A great deal of research effort has been put into exploring cross-modal correspondences in the field of cognitive science. However, the possibilities they open in the digital world have been relatively unexplored. Multiple sensorial media (mulsemedia) provides a highly immersive experience to the users and enhances their Quality of Experience (QoE) in the digital world. Thus, we consider that studying the plasticity and the effects of cross-modal correspondences in a mulsemedia setup can bring interesting insights about improving the human computer dialogue and experience. In our experiments, we exposed users to videos with certain visual dimensions (brightness, color, and shape), and we investigated whether the pairing with a cross-modal matching sound (high and low pitch) and the corresponding auto-generated vibrotactile effects (produced by a haptic vest) lead to an enhanced QoE. For this, we captured the eye gaze and the heart rate of users while experiencing mulsemedia, and we asked them to fill in a set of questions targeting their enjoyment and perception at the end of the experiment. Results showed differences in eye-gaze patterns and heart rate between the experimental and the control group, indicating changes in participants’ engagement when videos were accompanied by matching cross-modal sounds (this effect was the strongest for the video displaying angular shapes and high-pitch audio) and transitively generated cross-modal vibrotactile effects.<?vsp -1pt?>

Data fusion methods in multimodal human computer dialog

In multimodal human computer dialog, non-verbal channels, such as facial expression, posture, gesture, etc, combined with spoken information, are also important in the procedure of dialogue. Nowadays, in spite of high performance of users’ single channel behavior computing, it is still great challenge to understand users’ intention accurately from their multimodal behaviors. One reason for this challenge is that we still need to improve multimodal information fusion in theories, methodologies and practical systems. This paper presents a review of data fusion methods in multimodal human computer dialog. We first introduce the cognitive assumption of single channel processing, and then discuss its implementation methods in human computer dialog; for the task of multi-modal information fusion, serval computing models are presented after we introduce the principle description of multiple data fusion. Finally, some practical examples of multimodal information fusion methods are introduced and the possible and important breakthroughs of the data fusion methods in future multimodal human-computer interaction applications are discussed.

Conceptual Effects of Audience Design in Human–Computer and Human–Human Dialogue

ABSTRACTSpeakers in conversation often tailor referring expressions in ways that reflect beliefs about the knowledge and perspectives of specific addressees. In this study we examined whether such audience design can influence speakers’ conceptualizations of referents beyond the conversation. In a Wizard-of-Oz paradigm, participants identified novel images either for an ostensible computer dialogue application or for a human partner as part of a referential communication task. Subsequently, they independently sorted the task images into discrete groups. Consistent with audience design, when interacting with the computer a greater proportion of participants’ descriptions in the dialogue task focused on literal, geometric features compared with descriptions for the human partner. Importantly, participants’ postdialogue sorts were also more shape-based after human–computer interaction. Different ways of attending to objects for communicative purposes can affect how speakers construe objects for themselves.

Deep Music: Towards Musical Dialogue

Computer dialogue systems are designed with the intention of supporting meaningful interactions with humans. Common modes of communication include speech, text, and physical gestures. In this work we explore a communication paradigm in which the input and output channels consist of music. Specifically, we examine the musical interaction scenario of call and response. We present a system that utilizes a deep autoencoder to learn semantic embeddings of musical input. The system learns to transform these embeddings in a manner such that reconstructing from these transformation vectors produces appropriate musical responses. In order to generate a response the system employs a combination of generation and unit selection. Selection is based on a nearest neighbor search within the embedding space and for real-time application the search space is pruned using vector quantization. The live demo consists of a person playing a midi keyboard and the computer generating a response that is played through a loudspeaker.

A Socratic epistemology for verbal emotional intelligence

We describe and experimentally validate a question-asking framework for machine-learned linguistic knowledge about human emotions. Using the Socratic method as a theoretical inspiration, we develop an experimental method and computational model for computers to learn subjective information about emotions by playing emotion twenty questions (EMO20Q), a game of twenty questions limited to words denoting emotions. Using human–human EMO20Q data we bootstrap a sequential Bayesian model that drives a generalized pushdown automaton-based dialog agent that further learns from 300 human–computer dialogs collected on Amazon Mechanical Turk. The human–human EMO20Q dialogs show the capability of humans to use a large, rich, subjective vocabulary of emotion words. Training on successive batches of human–computer EMO20Q dialogs shows that the automated agent is able to learn from subsequent human–computer interactions. Our results show that the training procedure enables the agent to learn a large set of emotion words. The fully trained agent successfully completes EMO20Q at 67% of human performance and 30% better than the bootstrapped agent. Even when the agent fails to guess the human opponent’s emotion word in the EMO20Q game, the agent’s behavior of searching for knowledge makes it appear human-like, which enables the agent to maintain user engagement and learn new, out-of-vocabulary words. These results lead us to conclude that the question-asking methodology and its implementation as a sequential Bayes pushdown automaton are a successful model for the cognitive abilities involved in learning, retrieving, and using emotion words by an automated agent in a dialog setting.

Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems

Human---computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural dialogue system evaluation. We present some of the different evaluation techniques used in natural dialogue systems, including black box and glass box, comparative, quantitative, and qualitative evaluation. Four aspects of NLP dialogue system evaluation are often overlooked: usefulness in terms of a user's qualitative needs, localizability to new genres and languages, humanness or naturalness compared to human---human dialogues, and language benefit compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers.

Voice anthropomorphism, interlocutor modelling and alignment effects on syntactic choices in human−computer dialogue

The growth of speech interfaces and speech interaction with computer partners has made it increasingly important to understand the factors that determine users' language choices in human-computer dialogue. We report two controlled experiments that used a picture-naming-matching task to investigate whether users in human-computer speech-based interactions tend to use the same grammatical structures as their conversational partners, and whether such syntactic alignment can impact strong default grammatical preferences. We additionally investigate whether beliefs about system capabilities that are based on partner identity (i.e. human or computer) and speech interface design cues (here, voice anthropomorphism) affect the magnitude of syntactic alignment in such interactions. We demonstrate syntactic alignment for both dative structures (e.g., give the waitress the apple vs. give the apple to the waitress), where there is no strong default preference for one or other structure (Experiment 1), and noun phrase structures (e.g., a purple circle vs. a circle that is purple), where there is a strong default preference for one structure (Experiment 2). The tendency to align syntactically was unaffected by partner identity (human vs. computer) or voice anthropomorphism. These findings have both practical and theoretical implications for HCI by demonstrating the potential for spoken dialogue system behaviour to influence users' syntactic choices in interaction. As well as verifying natural corpora findings, this work also highlights that priming and cognitive mechanisms that are unmediated by beliefs about partner identity could be important in understanding why people align syntactically in human-computer dialogue. Paper investigates syntactic alignment in spoken human-computer dialogue.The role of partner modelling through partner type and voice is also explored.Humans align similarly with human and computer partners, irrespective of voice.Priming is an important mechanism to consider in explaining our HCD choices.Syntactic alignment can affect strong default preferences and could be used to improve spoken dialogue technology.

Alignment to the Actions of a Robot

Alignment is a phenomenon observed in human conversation: Dialog partners’ behavior converges in many respects. Such alignment has been proposed to be automatic and the basis for communicating successfully. Recent research on human–computer dialog promotes a mediated communicative design account of alignment according to which the extent of alignment is influenced by interlocutors’ beliefs about each other. Our work aims at adding to these findings in two ways. (a) Our work investigates alignment of manual actions, instead of lexical choice. (b) Participants interact with the iCub humanoid robot, instead of an artificial computer dialog system. Our results confirm that alignment also takes place in the domain of actions. We were not able to replicate the results of the original study in general in this setting, but in accordance with its findings, participants with a high questionnaire score for emotional stability and participants who are familiar with robots align their actions more to a robot they believe to be basic than to one they believe to be advanced. Regarding alignment over the course of an interaction, the extent of alignment seems to remain constant, when participants believe the robot to be advanced, but it increases over time, when participants believe the robot to be a basic version.

Do (and Say) as I Say: Linguistic Adaptation in Human–Computer Dialogs

There is strong research evidence showing that people naturally align to each other’s vocabulary, sentence structure, and acoustic features in dialog, yet little is known about how the alignment mechanism operates in the interaction between users and computer systems let alone how it may be exploited to improve the efficiency of the interaction. This article provides an account of lexical alignment in human–computer dialogs, based on empirical data collected in a simulated human–computer interaction scenario. The results indicate that alignment is present, resulting in the gradual reduction and stabilization of the vocabulary-in-use, and that it is also reciprocal. Further, the results suggest that when system and user errors occur, the development of alignment is temporarily disrupted and users tend to introduce novel words to the dialog. The results also indicate that alignment in human–computer interaction may have a strong strategic component and is used as a resource to compensate for less optimal (visually impoverished) interaction conditions. Moreover, lower alignment is associated with less successful interaction, as measured by user perceptions. The article distills the results of the study into design recommendations for human–computer dialog systems and uses them to outline a model of dialog management that supports and exploits alignment through mechanisms for in-use adaptation of the system’s grammar and lexicon.Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/hhci.

Improving the Human–Computer Dialogue With Increased Temporal Predictability

An experiment was conducted to investigate the impacts of length and variability of system response time (SRT) on user behavior and user experience (UX) in sequential computing tasks. Length is widely considered to be the most important aspect of SRTs in human-computer interaction. Research on temporal attention shows that humans adjust to temporal structures and that performance substantially improves with temporal predictability. Participants performed a sequential task with simulated office software. Duration and variability, that is, the number of different SRTs, was manipulated. Lower variability came at the expense of on average higher durations. User response times, task execution times, and failure rates were measured to assess user performance. UX was measured with a questionnaire. A reduction in variability improved user performance significantly. Whereas task load and failure rates remained constant, responses were significantly faster. Although a reduction in variability came along with, on average, increased SRTs, no difference in UX was found. Considering SRT variability when designing software can yield considerable performance benefits for the users. Although reduced variability comes at the expense of overall longer SRTs, the interface is not subjectively evaluated to be less satisfactory or demanding. Time design should aim not only at reducing average SRT length but also at finding the optimum balance of length and variability. Our findings can easily be applied in any user interface for sequential tasks. User performance can be improved without loss of satisfaction by selectively prolonging particular SRTs to reduce variability.

A Rule-Based Framework for Modular Development of In-Game Interactive Dialogue Simulation

In this paper, we discuss approaches to dialogue in interactive video games and interactive narrative research. We propose that situating interactive dialogue in the simplified expectations of video games is a profitable way to investigate computational dialogue simulation. Taking cues from existing physical simulations such as combat, we propose a hypothetical game environment and design goals for an embedded interactive dialogue system. We present a modular framework targeted at that environment, which is designed to enable incremental development and exploration of dialogue concepts. We describe this framework together with a work-in-progress system for simulating simple in-game negotiation dialogues.

Performance Test Bench Development of Auto Transmission Control Mechanism

According to the capability require of the automotive transmission control mechanism（ATCM）, and entered into test method by the automotive accessory corporation and user，developed the performance test bench of ATCM，and introduce the makeup and function of test bench and control system. The test results showed that this test bench has many merits such as high test precision，stable anti-interfere capability，rapid fix test element，perfect function of man - computer dialogue，and see about history data facilely. It can provide reliable test method for the ATCM’s test and scientific reference for improve on ATCM.

Informal logic dialogue games in human–computer dialogue

AbstractInformal logic (IL) is an area of philosophy rich in models of communication and discourse with a heavy focus on argument and ‘dialogue games’. Computational dialectics is a maturing strand of research that is focused on implementing these dialogue games. The aim of this paper is to review research on applying IL dialogue games into human–computer dialogue design. We argue that IL dialogue games tend to have a number of attractive properties for human computer dialogue and that their computational utilization in this area has been increasing recently. Despite the strength of the case for IL, a number of important barriers need to be overcome if the potential of IL is to be fulfilled. These barriers are examined and means of overcoming them discussed.