Multiparty Conversations Research Articles

Gaze modeling in multi-party dialogues and extraversion expression through gaze aversion control

Gaze control plays a crucial role in the context of conversational robots involving multiple participants. Particularly, gaze aversion is not only important for improving human likeness but is also strongly associated with the expression of a robot's personality and should be accounted for dialogue interactions. While previous studies on gaze models considering multi-party conversations have been conducted, those considering gaze aversion are limited. In this study, we developed a gaze motion generation model, focusing on three features: (1) the gaze target (towards dialogue partners or gaze aversion), (2) the gaze duration, and (3) the eyeball direction during gaze aversion. We extracted the gaze model parameters of two individuals with distinct extraversion from a multimodal three-party dialogue database and created a model that can modulate extraversion expression by setting a gaze aversion ratio parameter. We implemented this model on an android robot, which can reproduce human-like gaze behaviors. Video-based experiments have been conducted where participants watched videos of the robot talking with two humans, replicating the three-party dialogue scenarios used in this study, and evaluated the impressions on the robot's extraversion. The results demonstrated that by controlling gaze aversion parameters, the robot could exhibit various levels of extraversion.

Behavioral Intervention for Adults With Autism on Distribution of Attention in Triadic Conversations: A/B-Tested Pre-Post Study.

Cross-neurotype differences in social communication patterns contribute to high unemployment rates among adults with autism. Adults with autism can be unsuccessful in job searches or terminated from employment due to mismatches between their social attention behaviors and society's expectations on workplace communication. We propose a behavioral intervention concerning distribution of attention in triadic (three-way) conversations. Specifically, the objective is to determine whether providing personalized feedback to each individual with autism based on an analysis of their attention distribution behavior during an initial conversation session would cause them to modify their orientation behavior in a subsequent conversation session. Our system uses an unobtrusive head orientation estimation model to track the focus of attention of each individual. Head orientation sequences from a conversation session are analyzed based on five statistical domains (eg, maximum exclusion duration and average contact duration) representing different types of attention distribution behavior. An intervention is provided to a participant if they exceeded the nonautistic average for that behavior by at least 2 SDs. The intervention uses data analysis and video modeling along with a constructive discussion about the targeted behaviors. Twenty-four individuals with autism with no intellectual disabilities participated in the study. The participants were divided into test and control groups of 12 participants each. Based on their attention distribution behavior in the initial conversation session, 11 of the 12 participants in the test group received an intervention in at least one domain. Of the 11 participants who received the intervention, 10 showed improvement in at least one domain on which they received feedback. Independent t tests for larger test groups (df>15) confirmed that the group improvements are statistically significant compared with the corresponding controls (P<.05). Crawford-Howell t tests confirmed that 78% of the interventions resulted in significant improvements when compared individually against corresponding controls (P<.05). Additional t tests comparing the first conversation sessions of the test and control groups and comparing the first and second conversation sessions of the control group resulted in nonsignificant differences, pointing to the intervention being the main effect behind the behavioral changes displayed by the test group, as opposed to confounding effects or group differences. Our proposed behavioral intervention offers a useful framework for practicing social attention behavior in multiparty conversations that are common in social and professional settings.

Enhancing Communication in CPS Using Graph-Based Reply Relationship Identification in Multi-Party Conversations

To enhance communication and collaborative work efficiency in cyber–physical systems (CPSs) within the Industry 4.0 environment, this study investigates a graph-based machine learning approach aimed at optimizing information interaction during multi-party conversations. Devices within CPSs must efficiently exchange information in real time to synchronize operations and responses. This research treats these interactions as intricate graph structures and uses graph learning techniques to accurately identify communication links and dependencies among devices. This improvement leads to more accurate decision-making and smoother operations. Our methodology involves a real-time analysis of structural patterns and node attributes within conversations, improving information flow and comprehension. The empirical findings demonstrate that this approach significantly enhances production efficiency, system adaptability, and minimizes delays attributed to communication misunderstandings. Our method can effectively identify the communication relationships between devices, significantly improving the efficiency and accuracy of information transmission. This improved communication capability leads to an enhanced production efficiency of the entire system.

Extended repair in SGD-mediated conversation involving speakers with dysarthria

Background: The ability to engage successfully in repair is central to accomplishing shared conversational activities. Based on two case studies, this paper describes one strategy individuals with dysarthria who use speech generating devices (SGD) may engage in to manage extended repair in multiparty conversation. Method: Conversation Analysis was applied to video data of two extended repair interactions involving an aided speaker and two interlocutors, one familiar, the other less familiar. Results: Aided speakers took an agentive role in the organization of repair when dyadic repair with the less familiar conversant was unsuccessful by selecting the more familiar partners to participate in repair. Aided speakers capitalized on the partner shift by immediately switching to use of idiosyncratic communication modalities and by making reference to past shared experiences relevant to the repair. Conclusion/Discussion: The use of this strategy warrants further exploration as a means for managing extended repair in SGD-mediated conversation.

Competitive points in Mandarin-speaking multiparty interaction: Speakership and epistemics

This study explores the use of hand points in competitive epistemic and speakership environments in Mandarin-speaking conversation. It examines incoming speakers’ points at a current speaker. This study employs multimodal conversation-analytic methods to analyse 334 min of triadic interaction. 40 points directed at a current speaker by an incoming speaker are the focus of analysis. This study finds that these points indicate an upcoming bid for the floor, thereby carrying out self-selection for speakership. In addition, the turn introduced by the incoming speaker's point suggests that they know at least as much as the current speaker about the issue at hand, and can implicate epistemic and speakership competition. As such, these points foreshadow delivery of a weakly aligning turn or a disaligning turn, and may accomplish either affiliation or disaffiliation. This study generates new knowledge about the interactional functions of points in multiparty conversation, and suggests potential phenomena for cross-linguistic comparison.

Having Different Dialog Roles in Telecommunication by Using Two Teleoperated Robots Reduces an Operator’s Guilt

In recent years, applications of social robots as the operator’s avatar have been widely studied for remote conversation with rich nonverbal information. Having another side-participant robot beside the avatar robot of the operator was found to be effective for providing long-lasting backchannels to the interlocutor. The side-participant robot is also expected to play a role in assisting human participation in multiparty conversations. However, such a focus has not been applied to remote conversations with multiple robots. Here, we propose a multiple-robot telecommunication system with which the operator can use a side-participant robot to assist conversation that is developed by the operator through the main speaker robot to verify its effectiveness. In the laboratory experiment where the subjects were made to feel stressed by being forced to provide rude questions to the interlocutor, the proposed system was shown to reduce guilt and to improve the overall mood of operators. The result encourages the application of a multi robot remote conversation system to allow the user to participate in remote conversations with less anxiety of potential failure in maintaining the conversation.

Multiparty conversation via multirobot system: incorporation of nonverbal user responses for continued conversation

Recent years have seen the advent of conversational humanoid robots. Implementing multiple robots is a promising approach for continuing human–robot conversations when a robot encounters speech-recognition errors. This strategy is effective for the user's verbal responses; however, people generally use nonverbal responses, such as nods or smiles, to respond to the interlocutor. In this study, we proposed a conversational strategy for twin robots in which the second robot recognized a human's nonverbal responses and interrupted the conversation by mentioning these responses. Moreover, we developed an interrogative dialogue system using simple nonverbal recognition modules. To verify the effectiveness of this conversational strategy, a subject-based experiment was conducted in which humans conversed with the two robots, and their impressions were evaluated through a questionnaire. We compared the outcomes under two conditions: whether the second robot's interruption was inserted or otherwise. Our results indicate that language-only ineffectiveness in responding to ambiguous responses regarding satisfaction and comprehension was mitigated. Thus, such a dialogue system is advantageous because it can facilitate robust conversation without relying solely on speech recognition.

"Your Turn!": Using Finger Pointing and PALM-UP Actions to Ask Questions in Norwegian Sign Language

Abstract: Researchers examining the structure of questions in signed languages, often using elicited data from informants, have proposed that there are specific manual and nonmanual actions produced by signers to indicate different question types (e.g., Zeshan 2004), for example, raised eyebrows for polar questions. In the current study, we add to this body of work with an investigation into the use of two common interactional behaviors—finger pointing and palm-up actions—as they occur in questions asked during naturalistic conversations in Norwegian Sign Language. The interactional work finger pointing and palm-up actions can achieve in various settings is well-documented (e.g., Bavelas 1994), and their presence in question contexts has been observed in Norwegian Sign Language (Ferrara 2020). In total, 455 questions produced by thirty-four deaf signers were examined as they appeared across dyadic, triadic, and multiparty conversations recorded in the Norwegian Sign Language Corpus (Ferrara and Ringsø 2021; Ferrara and Bø 2022; Ferrara in prep). These questions were then examined for the presence of interactional finger pointing and palm-up actions. Analyses firstly revealed that interactional finger pointing and palm-up actions appeared in many of the questions examined. Within this subset of the data, further analysis showed that these forms may be the only manual action that signals an utterance as a question, or they may occur in combination with other manual (e.g., wh-question forms: how, where) and nonmanual actions (e.g., furrowed or raised brows). In many of these questions, the two forms appear in utterance-final position, thereby indexing an interlocutor in order to solicit a response. Such initial observations suggest that these two manual actions must be considered as part of the constellation of behaviors that characterize the highly indexical and interactional nature of question-asking in Norwegian Sign Language.

The Effect of Eye Contact in Multi-Party Conversations with Virtual Humans and Mitigating the Mona Lisa Effect

The demand for kiosk systems with embodied conversational agents has increased with the development of artificial intelligence. There have been attempts to utilize non-verbal cues, particularly virtual human (VH) eye contact, to enable human-like interaction. Eye contact with VHs can affect satisfaction with the system and the perception of VHs. However, when rendered in 2D kiosks, the gaze direction of a VH can be incorrectly perceived, due to a lack of stereo cues. A user study was conducted to examine the effects of the gaze behavior of VHs in multi-party conversations in a 2D display setting. The results showed that looking at actual speakers affects the perceived interpersonal skills, social presence, attention, co-presence, and competence in conversations with VHs. In a second study, the gaze perception was further examined with consideration of the Mona Lisa effect, which can lead users to believe that a VH rendered on a 2D display is gazing at them, regardless of the actual direction, within a narrow range. We also proposed the camera rotation angle fine tuning (CRAFT) method to enhance the users’ perceptual accuracy regarding the direction of the VH’s gaze.The results showed that the perceptual accuracy for the VH gaze decreased in a narrow range and that CRAFT could increase the perceptual accuracy.

Multi-Party Conversation Modeling for Emotion Recognition

Multi-party conversation modeling plays a vital role in emotion recognition in conversation (ERC). Aside from the intra- and inter-speaker dependencies between different speakers, the difficulty also lies in the fact that each conversation may contain several to many utterances that compose a long text sequence. In this article, we present two approaches to effective multi-party conversation modeling. First, to encode long sequences and capture long-range dependency between utterances, we introduce a dialog-oriented language model, DialogXL, with enhanced memory to store longer conversation sequences and dialog-aware self-attention to deal with multi-party dependencies. Second, we present a directed acyclic neural network, namely DAG-ERC, to encode the utterances with a directed acyclic graph (DAG) to better capture the intrinsic structure within a conversation. DAG-ERC combines the advantages of recurrent models and graph models and provides a more intuitive way to model information flow between sequential utterances. Extensive experiments are conducted on four ERC benchmarks with state-of-the-art models employed for comparison, and empirical results demonstrate the superiority of the two models in multi-party conversation modeling.

A Multi-Party Conversation-Based Effective Robotic Navigation System for Futuristic Vehicle

In response to the growing need for advanced in-car navigation systems that prioritize user experience and aim to reduce driver cognitive workload, this study addresses the research question of how to enhance the interaction between drivers and navigation systems. The focus is on minimizing distraction while providing personalized and geographically relevant information. The research introduces an innovative in-car robotic navigation system comprising three subsystem models: geofencing,personalization, and conversation. The dynamic geofencing model acquires geographic details related to the user's current location and provides information about required destinations. The personalization model tailors suggestions based on user preferences, while the conversation model, employing two virtual robots, fosters interactive multiparty conversations aligned with the driver's interests. The study's scope is specifically confined to interactive conversations centered on nearby restaurants and the driver's dietary preferences. Evaluation of the system indicates a notable prevalence of neutral expressions amongparticipants during interaction, suggesting that the implemented system successfully mitigates cognitive workload. Participants in the experiments express higher usability and interactivity levels, as evidenced by feedback collected at the study's conclusion, affirming the system's effectiveness in enhancing the user experience while maintaining a driver-friendly environment.&#x0D; Keywords: Human-Robot Interaction, Multiparty Conversation, In-Car Navigation

Keeping track of who knows what in multiparty conversation despite severe memory impairment

Language use has long been understood to be tailored to the intended addressee, a process termed audience design. Audience design is reflected in multiple aspects of language use, including adjustments based on the addressee's knowledge about the topic at hand. In group settings, audience design depends on representations of multiple individuals, each of whom may have different knowledge about the conversational topic. A central question, then, concerns how these representations are encoded and retrieved in multiparty conversation where successful conversation requires keeping track of who knows what. In the present research, we probe the biological memory systems that are involved in this process of multiparty audience design. We present the results of two experiments that compare language use in persons with bilateral hippocampal damage and severe declarative memory impairment (amnesia), and demographically matched neurotypical comparison participants. Participants played a game in which they discussed abstract images with one partner in conversation, and then discussed the images again with the same partner or with a new partner in a three-party conversation. Neurotypical participants' language use reflected newly formed representations of which partner was familiar with which images. Participants with amnesia showed evidence of partner-specific audience design in multiparty conversation but it was attenuated, especially when success required rapid alternations between representations of common ground. The findings suggest partial independence of the formation and use of partner-specific representations from the hippocampal-dependent declarative memory system and highlight the unique contributions of the declarative memory system to flexible and dynamic language use.

"I See What You Mean"-A Case Study of the Interactional Foundation of Building a Working Alliance in Care Decisions Involving an Older Couple Living with Cognitive Decline.

Social workers have a key role in needs assessment meetings with families dealing with dementia, providing information, support, and advocacy, while also assessing needs and making decisions about care services for several parties. These contacts are especially important during the introduction of home care services, where often the person has previously relied on informal support from relatives. The needs assessment process entails the involvement of all present parties, with the aim to reach a mutual agreement, a working alliance, regarding which services to apply for. The aim of this case study is to explore how the participants, by means of different conversational practices, jointly create a working alliance between the different parties in one family. The study provides insights into the process of co-constructing a working alliance in the needs assessment process for elder care services. This article addresses the process by which social workers build a working alliance in a multi-party conversation with a family living with cognitive decline; a meeting that lasted 50 min. In this case study, we benefit from an inductive and detailed conversation analytic methodology. The theoretical framework of working alliances in institutional interaction has informed the analysis. The findings illustrate how the social worker in this case study involves all parties in the decision regarding care services and explores the use of the conversational practices of mitigations, positive framing, adding information, and positioning, as a "we" achieve mutual agreement toward the end of several sequences. Drawing on the results of this case study, we argue that multi-party interaction involving relatives enables diversity in role-taking, where the professional, for instance, can pursue a more empathic role. Also, our results indicate that minimal agreement to a proposal is sufficient in a multi-party interaction involving clients with cognitive decline.

Abstractive Meeting Summarization: A Survey

AbstractA system that could reliably identify and sum up the most important points of a conversation would be valuable in a wide variety of real-world contexts, from business meetings to medical consultations to customer service calls. Recent advances in deep learning, and especially the invention of encoder-decoder architectures, has significantly improved language generation systems, opening the door to improved forms of abstractive summarization—a form of summarization particularly well-suited for multi-party conversation. In this paper, we provide an overview of the challenges raised by the task of abstractive meeting summarization and of the data sets, models, and evaluation metrics that have been used to tackle the problems.

Tactile engagement of prospective next speakers in Indonesian multiparty conversations

AbstractThis article investigates the use of touch as a tool for engaging prospective next speakers within Indonesian multiparty conversation. We examine the lamination of touch onto questions directed towards specifically targeted recipients. First, we find that questions with touch are deployed when the physical environment complicates the attainment of mutual orientation. Second, when previously targeted recipients have failed to respond to a question, touch is added to follow-up questions that are deployed for pursuing a response. Third, touch is added to questions that are personal or that inquire about potentially delicate matters. This multimodal investigation of conversational turn-taking provides data from Colloquial Indonesian as basis for cross-linguistic comparison. In considering the volume of touches in these data we ask whether cultural and environmental factors might contribute to a haptic modification of ordinary turn-taking procedures. (Turn-taking, touch, multimodality, sociotopography)

SKIER: A Symbolic Knowledge Integrated Model for Conversational Emotion Recognition

Emotion recognition in conversation (ERC) has received increasing attention from the research community. However, the ERC task is challenging, largely due to the complex and unstructured properties of multi-party conversations. Besides, the majority of daily dialogues take place in a specific context or circumstance, which requires rich external knowledge to understand the background of a certain dialogue. In this paper, we address these challenges by explicitly modeling the discourse relations between utterances and incorporating symbolic knowledge into multi-party conversations. We first introduce a dialogue parsing algorithm into ERC and further improve the algorithm through a transfer learning method. Moreover, we leverage different symbolic knowledge graph relations to learn knowledge-enhanced features for the ERC task. Extensive experiments on three benchmarks demonstrate that both dialogue structure graphs and symbolic knowledge are beneficial to the model performance on the task. Additionally, experimental results indicate that the proposed model surpasses baseline models on several indices.

End-to-End Deep Reinforcement Learning for Conversation Disentanglement

Collaborative Communication platforms (e.g., Slack) support multi-party conversations which contain a large number of messages on shared channels. Multiple conversations intermingle within these messages. The task of conversation disentanglement is to cluster these intermingled messages into conversations. Existing approaches are trained using loss functions that optimize only local decisions, i.e. predicting reply-to links for each message and thereby creating clusters of conversations. In this work, we propose an end-to-end reinforcement learning (RL) approach that directly optimizes a global metric. We observe that using existing global metrics such as variation of information and adjusted rand index as a reward for the RL agent deteriorates its performance. This behaviour is because these metrics completely ignore the reply-to links between messages (local decisions) during reward computation. Therefore, we propose a novel thread-level reward function that captures the global metric without ignoring the local decisions. Through experiments on the Ubuntu IRC dataset, we demonstrate that the proposed RL model improves the performance on both link-level and conversation-level metrics.

The Priority Structure behind Overlap of Multi-Person Conversation

This article explores the priority structure underlying overlapping speech in multi-person conversations. While most research on overlapping talk has focused on resolution and management, there remains a gap in understanding multi-party conversations. Drawing on a self-collected corpus of dialogue materials, the author analyzes existing data to examine the influence of underlying priority structures on overlapping phenomena. By delving into the observable implicit communicative structure and regular characteristics, this study sheds light on the prioritization dynamics in multi-person conversations. The findings contribute to conversation analysis research and enhance our comprehension of the complexities involved in managing overlaps in communication.

Literature Review of Turn-Taking

Turn-taking theory has attracted a lot of attention from linguistics and other disciplines since its emergence. Numerous studies have been conducted with regard to turn system, presenting its theory and empirical evolution in foreign and domestic research. In terms of turn-taking theory, majority of scholars state interpretation of its definition and rules without breaking through the framework of notion of Sacks. As far as application of turn-taking theory, they mainly focus on news interviews, TV programs, class interaction and daily dialogue and other aspects at home and abroad. Few studies concern about turn features of WeChat language and WeChat multiparty conversation. Through the view of turn and turn-taking, characteristics of WeChat multiparty conversation will be figured out and mechanism and characteristics of WeChat language will be summarized through the analysis of turn system of the WeChat multiparty conversation.

Multi-Hypergraph Neural Networks for Emotion Recognition in Multi-Party Conversations

Emotion recognition in multi-party conversations (ERMC) is becoming increasingly popular as an emerging research topic in natural language processing. Recently, many approaches have been devoted to exploiting inter-dependency and self-dependency among participants. However, these approaches remain inadequate in terms of inter-dependency due to the fact that the effects among speakers are not individually captured. In this paper, we design two hypergraphs to deal with inter-dependency and self-dependency, respectively. To this end, we design a multi-hypergraph neural network for ERMC. In particular, we combine average aggregation and attention aggregation to generate hyperedge features, which can allow utterance information to be better utilized. The experimental results show that our method outperforms multiple baselines, indicating that further exploitation of inter-dependency is of great value for ERMC. In addition, we also achieved good results on the emotional shift issue.