Multimodal Dialogue Research Articles

Make new friends, leave my friends: A dialogical investigation into transition experiences and agency in children from UK Armed Forces families

AbstractThis article provides new perspectives on the transitions of children from military families. It examines the experiences and agency of a group of UK primary‐school Service children who were undergoing far‐reaching transitions while participating in an arts‐based research project. Transitions are conceived here not as events, such as school moves, but as processes of changing, the dialogical interplay between ever‐changing socio‐cultural and physical environments and the psychological work individuals undertake in response to change. This reconceptualisation of transitions shifts attention away from children's resilience, or lack thereof, and towards unique, nuanced understandings of their subjective experiences and priorities. Presenting multimodal pieces created by three children as they explored the question, ‘What's it like to be a Service child in this school?’, I describe their diverse and agentic responses to their changing circumstances, as they sought to mitigate anticipated and past losses and perceived disadvantage and to use their transitions as positive opportunities for self‐development. Although punctuated by observable moments of change, this transition work happened over an indefinite timescale, highlighting a need for long‐term support informed by understandings of children's agency and priorities. Such support and insight may be achieved through developing spaces for multimodal dialogue with Service children.

Multi-modal Anaphora and Broadcasting of Information by Gestural Post-holds

This paper deals with three interrelated topics, linguistic anaphora, multi-modal anaphora and the top-down broadcasting of information using gestural post-holds in multimodal dialogue. Initially, a new solution for definite, pronominal and pro-adverbial anaphora is given based on the idea that an existentially quantified general term may output a definite reference. This approach is extended to multimodal anaphora, where part or all of an anaphor’s meaning is contributed by some sequence of iconic or deictic gestures. Anaphora exploit the semantic potential of their antecedents, they work, as tradition has it, “bottom-up”. An inverse relation, more general than cataphora, and investigated here for the first time, is “broadcasting”, where information is freely distributed top down and input to receiving sites (ports). Anaphora are modelled with the same top-down mechanism and the same applies for coherence relations in dialogue which generally show an anaphora-like behaviour. “Broadcasting” can be used in the context of anaphors, for example, to provide their gestural meaning parts but also for a verb’s multi-modal arguments for referring to a location, a direction or an area. As to multi-modal data, broadcasting is shown to be frequently tied up with gestural post-holds, the holding of a gesture’s stroke information independently of semantically alignable speech. This leads to considering post-holds from a new perspective, stressing their speech-independent function and their relevance for indicating topic-continuity. We show that multi-modal anaphora and especially broadcasting cross single contributions and turns. The data which let us develop these perspectives come from the SaGA (Speech and Gesture Alignment) corpus, a set of route-description dialogues generated in a VR-setting incorporating marker-based eye-tracking facilities. The calculus used to model the anaphora and broadcasting dynamics is the concurrent λΨ-calculus, a recently developed two-tiered machinery using a Ψ-calculus for input-output, data transport and broadcasting. The data transported are in a typed λ-calculus format incorporating Neo-Davidsonian representations; these data can be linguistic, gestural only or multi-modal. Multi-modal informational chunks are modelled as communicating agents sending and receiving information via input-output-channels. They are introduced incrementally on an empirically motivated construction or gesture-plus-construction or gesture only basis. The λΨ-calculus is also used for the multi-modal fusion component unifying gestural and linguistic information; hence, the paper is also a contribution to multi-modal fusion of linguistic and gestural input. Finally, it is shown how the presented algorithm can capture multi-modal coherence relations or a multi-modal anaphora resolution based on PTT ideas.

Gaze modeling in multi-party dialogues and extraversion expression through gaze aversion control

Gaze control plays a crucial role in the context of conversational robots involving multiple participants. Particularly, gaze aversion is not only important for improving human likeness but is also strongly associated with the expression of a robot's personality and should be accounted for dialogue interactions. While previous studies on gaze models considering multi-party conversations have been conducted, those considering gaze aversion are limited. In this study, we developed a gaze motion generation model, focusing on three features: (1) the gaze target (towards dialogue partners or gaze aversion), (2) the gaze duration, and (3) the eyeball direction during gaze aversion. We extracted the gaze model parameters of two individuals with distinct extraversion from a multimodal three-party dialogue database and created a model that can modulate extraversion expression by setting a gaze aversion ratio parameter. We implemented this model on an android robot, which can reproduce human-like gaze behaviors. Video-based experiments have been conducted where participants watched videos of the robot talking with two humans, replicating the three-party dialogue scenarios used in this study, and evaluated the impressions on the robot's extraversion. The results demonstrated that by controlling gaze aversion parameters, the robot could exhibit various levels of extraversion.

Analysing educational dialogue around shared artefacts in technology-mediated contexts: a new coding framework

ABSTRACT Multimodal interactions that incorporate technology use can support rich new forms of educational dialogue that highlight differences between participants’ perspectives and make reasoning processes more explicit. Learning with technology is mediated by digitally represented knowledge artefacts that participants construct together and manipulate through dialogue. However, there is a need for research into the methods by which researchers and practitioners can understand the learning taking place in such complex environments where verbal and nonverbal interactions with technology are inter-connected and entangled with activities and material learning resources. This paper addresses that need. It offers a critical discussion of the underlying theory and presents a new 13-category coding scheme, ‘Tech-SEDA’. This is the first public version of a systematic, research-informed and practical tool for analysing verbal and nonverbal dialogic communication at the micro- or meso- level across diverse educational settings. Application is illustrated with examples of dialogic interaction across diverse technology contexts by learners located in classrooms or remotely – including online discussions, 3-dimensional modelling in augmented reality, software apps on tablets, Knowledge Forum, Scratch programming, interactive whiteboards, and more. The paper considers the methodological issues arising in capturing dynamic multimodal dialogue and in identifying the learning taking place.

Travel Agency Task Dialogue Corpus: A Multimodal Dataset with Age-Diverse Speakers

When individuals communicate, they use different vocabularies, speaking speeds, facial expressions, and gestural languages, depending on those with whom they are speaking. This study focuses on the age of the speaker as a factor that affects the style of communication. We collected a multimodal dialogue corpus with various speaker ages. We used travel as the topic, as it interests people of all ages, and we set up a task based on a tourism consultation between an operator and a customer at a travel agency. This paper presents the details of the dialogue task, collection procedures and annotations, and analysis of the characteristics of the dialogues and facial expressions, focusing on the age of the speakers. The results of the analysis suggest that the adult speakers have more independent opinions, the older speakers express their opinions more frequently than other age groups, and those in the operator role smiled more frequently at minors.

An Audio - Visual Virtual Personal Assistant

Current traits in smart assistants and smart home automation are currently attracting the interests of customers and researchers. Speech enabled smart virtual assistants (named smart speakers) offer a wide sort of network orientated services and, in some instances, can connect to smart environments, accordingly improving them with new and effective user interfaces. But, such gadgets also reveal new desires and a few weaknesses. Specially, they constitute faceless and blind assistants, not able to reveal a face, and therefore an emotion, and not able to ‘see’ the person. For this reason, the interaction is impaired and, in a few cases, ineffective. One of the goals of artificial intelligence is the realization of natural talk among humans and machines. In latest years, the dialogue systems, known as interactive conversational systems are the fastest developing vicinity in AI. Many companies have used the dialogue systems technology to set up various sorts of VPAs based on their application areas, including Apple’s Siri, Amazon Alexa, etc. To triumph over such troubles, in this project we combine a number of advance techniques. The proposed Assistant is powerful and resource-efficient, interactive and customizable. We use the multi-modal dialogue structures and screen projection which process two or more combined user input modes, which includes speech, image, video, touch, manual gestures, body movements with the intention to design the NextGeneration of VPAs. The new version of VPAs can be used to increase the interaction between humans and machines by the usage of distinctive technologies, consisting of gesture recognition, image/video recognition, speech recognition, dialogue system, conversational information base, and the overall knowledge base. Furthermore, this VPA device can be utilized in different areas such as education, medical assistance, disabilities systems, home automation and security access control.

Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements

The topic of multimodal conversation systems has recently garnered significant attention across various industries, including travel and retail, among others. While pioneering works in this field have shown promising performance, they often focus solely on context information at the utterance level, overlooking the context-aware dependencies of multimodal semantic elements like words and images. Furthermore, the ordinal information of images, which indicates the relevance between visual context and users’ demands, remains underutilized during the integration of visual content. Additionally, the exploration of how to effectively utilize corresponding attributes provided by users when searching for desired products is still largely unexplored. To address these challenges, we propose PMATE, a P osition-aware M ultimodal di A logue system with seman T ic E lements. Specifically, to obtain semantic representations at the element level, we first unfold the multimodal historical utterances and devise a position-aware multimodal element-level encoder. This component considers all images that may be relevant to the current turn and introduces a novel position-aware image selector to choose related images before fusing the information from the two modalities. Finally, we present a knowledge-aware two-stage decoder and an attribute-enhanced image searcher for the tasks of generating textual responses and selecting image responses, respectively. We extensively evaluate our model on two large-scale multimodal dialogue datasets, and the results of our experiments demonstrate that our approach outperforms several baseline methods.

BVA-Transformer: Image-text multimodal classification and dialogue model architecture based on Blip and visual attention mechanism

Multimodal tasks have become a hot research direction in recent years. The emergence of large-scale models has gradually elevated the extensive multimodal tasks, achieving remarkable achievements. However, when it comes to fusing multiple modalities for multimodal tasks, how to better integrate multimodal features is still a problem worth exploring. In tasks such as sentiment analysis targeting a wide range of social media content, the use of features derived solely from the [CLS] token may lead to insufficient information. This paper proposes BVA-Transformer model architecture for image-text multimodal classification and dialogue, which incorporates the EF-CaTrBERT method for feature fusion and introduces BLIP for the transformation of images to the textual space. This enables the fusion of images and text in the same information space, avoiding issues of information redundancy and conflict compared to traditional feature fusion methods. In addition, we proposed a Global Features Encoder (GFE) module based on visual attention in the BVA-Transformer, which can provide more global and targeted auxiliary features for the [CLS] token. This enables the model to utilize more feature information in classification tasks under this feature fusion method and dynamically select information to focus on. We also introduced the Trv structure from EVA-02 in the Decoder part of BVA-Transformer, investigating its impact on the model performance. Furthermore, we designed a three-stage training to further enhance the model’s performance. Experimental results demonstrate that BVA-Transformer achieves high-quality classification while generating dialogue sentences. Compared to existing multimodal classification models on our validation dataset, it exhibits excellent performance.

MODE: a multimodal open-domain dialogue dataset with explanation

MODE: a multimodal open-domain dialogue dataset with explanation

A multimodal dialogue system for customer service based on user personality adaptation and dialogue strategies

ABSTRACT This paper presents a multimodal dialogue system with user personality adaptation and multiple nonverbal behavior generation for a conversational android robot that plays a customer service role in recommending travel plans. It is useful to estimate the user's personality to realize a dialogue system that adapts the dialogue strategy to individual users. To improve the user's impression of the conversation robot as a customer service agent, we must control the nonverbal behavior of the robot, such as its facial expressions and motions. Moreover, we need to design the dialogue strategy to provide information and make users enjoy the conversation by adding an ice-breaking step. Against this background, we develop a dialogue system prototype that adjusts the dialogue strategy based on the user's personality, as estimated by a pre-trained multimodal sensing model. Furthermore, we implement appropriate nonverbal behavior patterns for each situation, including the voice, motion, and facial expressions of the android robot, to provide it appropriate and polite service agent behaviors. A personality assessment game is introduced into the dialogue to prevent the user from becoming bored. The game plays an ice-breaking role in the dialogue. To let the user engage with the robot, we implement a complement to the user as a dialogue act. We evaluate the proposed multimodal dialogue system for customer service by means of participation in a dialogue robot competition. The fair evaluation results from 26 dialogue users who conversed with multiple systems show that the proposed dialogue system achieved the best score among 13 teams participating in the preliminary round of the competition, indicating that the user personality adaptation and various elements implemented in our dialogue system improve the dialogue experience of general users. The results of a third-party evaluation and multiple regression analysis show that natural response is the most important factor contributing to user impression.

Emotion generation method in online physical education teaching based on data mining of teacher-student interactions.

Different from conventional educational paradigms, online education lacks the direct interplay between instructors and learners, particularly in the sphere of virtual physical education. Regrettably, extant research seldom directs its focus toward the intricacies of emotional arousal within the teacher-student course dynamic. The formulation of an emotion generation model exhibits constraints necessitating refinement tailored to distinct educational cohorts, disciplines, and instructional contexts. This study proffers an emotion generation model rooted in data mining of teacher-student course interactions to refine emotional discourse and enhance learning outcomes in the realm of online physical education. This model includes techniques for data preprocessing and augmentation, a multimodal dialogue text emotion recognition model, and a topic-expanding emotional dialogue generation model based on joint decoding. The encoder assimilates the input sentence into a fixed-length vector, culminating in the final state, wherein the vector produced by the context recurrent neural network is conjoined with the preceding word's vector and employed as the decoder's input. Leveraging the long-short-term memory neural network facilitates the modeling of emotional fluctuations across multiple rounds of dialogue, thus fulfilling the mandate of emotion prediction. The evaluation of the model against the DailyDialog dataset demonstrates its superiority over the conventional end-to-end model in terms of loss and confusion values. Achieving an accuracy rate of 84.4%, the model substantiates that embedding emotional cues within dialogues augments response generation. The proposed emotion generation model augments emotional discourse and learning efficacy within online physical education, offering fresh avenues for refining and advancing emotion generation models.

DialogMCF: Multimodal Context Flow for Audio Visual Scene-Aware Dialog

In recent years, Audio Visual Scene-Aware Dialog (AVSD) has been an active research task in the multimodal dialogue community and has also been a core part of the Dialog System Technology Challenge (DSTC). This task is an extension of conventional visual question answering, where video-relevant answers must be generated taking into account multimodal contextual information from previous dialogue rounds. Despite recent advances in the AVSD task, there are still two major challenges in developing such a system: how to model the multimodal contextual information of multiple rounds of dialogues and how to integrate audio-visual information into the generation of textual responses. To tackle these two challenges, in this paper we propose a novel model, named DialogMCF, which constructs a multimodal context flow model to generate responses that are relevant to video scenes. This proposed context flow modeling can track the dynamics of the topic information across multiple rounds of dialogue history. To achieve an effective fusion of multimodal information, we propose an audio-visual memory network with cross-modality aligned features to model long multimodal dialogue context, and thus enhance the flow modeling. Furthermore, we make attempts to improve the performance of the proposed DialogMCF model with manual descriptions and explore the incorporation of temporal reasoning information. Extensive experiments on the DSTC AVSD datasets show that, compared to a range of baseline methods, the proposed method can yield state-of-art dialogue generation performance on most metrics when integrating the video descriptions.

Two-Step Masked Language Model for Domain-Adapting Multi-Modal Task-Oriented Dialogue Systems

Two-Step Masked Language Model for Domain-Adapting Multi-Modal Task-Oriented Dialogue Systems

DER-GCN: Dialog and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialog Emotion Recognition.

With the continuous development of deep learning (DL), the task of multimodal dialog emotion recognition (MDER) has recently received extensive research attention, which is also an essential branch of DL. The MDER aims to identify the emotional information contained in different modalities, e.g., text, video, and audio, and in different dialog scenes. However, the existing research has focused on modeling contextual semantic information and dialog relations between speakers while ignoring the impact of event relations on emotion. To tackle the above issues, we propose a novel dialog and event relation-aware graph convolutional neural network (DER-GCN) for multimodal emotion recognition method. It models dialog relations between speakers and captures latent event relations information. Specifically, we construct a weighted multirelationship graph to simultaneously capture the dependencies between speakers and event relations in a dialog. Moreover, we also introduce a self-supervised masked graph autoencoder (SMGAE) to improve the fusion representation ability of features and structures. Next, we design a new multiple information Transformer (MIT) to capture the correlation between different relations, which can provide a better fuse of the multivariate information between relations. Finally, we propose a loss optimization strategy based on contrastive learning to enhance the representation learning ability of minority class features. We conduct extensive experiments on the benchmark datasets, Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD), which verify the effectiveness of the DER-GCN model. The results demonstrate that our model significantly improves both the average accuracy and the F1 value of emotion recognition. Our code is publicly available at https://github.com/yuntaoshou/DER-GCN.

Improving user's sense of participation in robot-driven dialogue

ABSTRACT In this study, we aimed to enhance the sense of dialogue participation (SDP), which reflects an individual's conscious engagement in dialogues, and investigated the multimodal dialogue elements that contribute to its improvement. To achieve this, we developed a dialogue system utilizing an android robot specifically for tour spot recommendations at a travel agency. The system was designed with various components aimed at enhancing SDP and was evaluated by 29 participants during the Dialogue Robot Competition 2022. After conceptually defining SDP, we employed the author's subjective evaluation to visualize the time-series changes in SDP at approximately 20-second intervals, using video recordings of the dialogues. Initially, we expected an overall decrease in SDP, as the robot had a speaking duration of around two minutes during the middle of the dialogue session when presenting the tour plan. However, the analysis of five participants' data revealed an increase in SDP during segments that involved ‘Dealing with content that increases the user's interest’ and ‘Multimodal response requests.’ Consequently, we can conclude that these elements have a discernible impact on enhancing SDP.

Response generation in multi-modal dialogues with split pre-generation and cross-modal contrasting

Due to the natural multi-modal occurrence format (text, audio, vision) of the dialogues, textual response generation in dialogues should rely on the multi-modal contexts beyond text only. However, most existing studies normally ignore the rich information of other modalities, such as audio. To investigate the importance of the acoustic contexts, we explore the multi-modal dialogue scenario with aligned text and audio temporal sequences for textual response generation of an assumed system, namely RGMD task. To this end, we construct a new multi-modal dataset for this task based on TV shows, which contains 84.9K utterances. Considering the response diversity limited by the context and modality interactions for RGMD, we attempt the split pre-generation (SPG) strategy and the cross-modal contrastive learning (CCL) strategy in multi-modal pre-training for better response generation. On the one hand, with SPG, we can obtain many diverse responses without the restrictions of too many historical mixed multi-modal contexts. On the other hand, with CCL, we can capture the interactions between text and audio. Extensive experiments demonstrate that our approach based on BART can consistently perform better than the state-of-the-art textual approach DP by 4.17%, 8.96%, 2.43%, 1.04% and 7.54% regarding metrics of BLEU, DIST, ROUGE, METEOR and NIST, respectively. Moreover, our approach based on GPT can outperform the state-of-the-art multi-modal approach RLM by 6.79%, 9.25%, 7.49%, 9.31% and 13.75% regarding metrics of BLEU, DIST, ROUGE, METEOR and NIST, respectively. Besides, we conduct much in-depth analysis, showing the necessity of audio for response generation and further verifying the effectiveness of our approach.

Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model

Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: (1) overlook the benefit of generative pretraining and (2) ignore the textual context-related knowledge . To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language mode for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection , dual knowledge-enhanced context learning , and knowledge-enhanced response generation . To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly, integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.

Aspect-level sentiment-controlled knowledge grounded multimodal dialog generation using generative models for reviews

Aspect-level sentiment-controlled knowledge grounded multimodal dialog generation using generative models for reviews

A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations.

The rise of depression, anxiety, and suicide rates has led to increased demand for telemedicine-based mental health screening and remote patient monitoring (RPM) solutions to alleviate the burden on, and enhance the efficiency of, mental health practitioners. Multimodal dialog systems (MDS) that conduct on-demand, structured interviews offer a scalable and cost-effective solution to address this need. This study evaluates the feasibility of a cloud based MDS agent, Tina, for mental state characterization in participants with depression, anxiety, and suicide risk. Sixty-eight participants were recruited through an online health registry and completed 73 sessions, with 15 (20.6%), 21 (28.8%), and 26 (35.6%) sessions screening positive for depression, anxiety, and suicide risk, respectively using conventional screening instruments. Participants then interacted with Tina as they completed a structured interview designed to elicit calibrated, open-ended responses regarding the participants' feelings and emotional state. Simultaneously, the platform streamed their speech and video recordings in real-time to a HIPAA-compliant cloud server, to compute speech, language, and facial movement-based biomarkers. After their sessions, participants completed user experience surveys. Machine learning models were developed using extracted features and evaluated with the area under the receiver operating characteristic curve (AUC). For both depression and suicide risk, affected individuals tended to have a higher percent pause time, while those positive for anxiety showed reduced lip movement relative to healthy controls. In terms of single-modality classification models, speech features performed best for depression (AUC = 0.64; 95% CI = 0.51-0.78), facial features for anxiety (AUC = 0.57; 95% CI = 0.43-0.71), and text features for suicide risk (AUC = 0.65; 95% CI = 0.52-0.78). Best overall performance was achieved by decision fusion of all models in identifying suicide risk (AUC = 0.76; 95% CI = 0.65-0.87). Participants reported the experience comfortable and shared their feelings. MDS is a feasible, useful, effective, and interpretable solution for RPM in real-world clinical depression, anxiety, and suicidal populations. Facial information is more informative for anxiety classification, while speech and language are more discriminative of depression and suicidality markers. In general, combining speech, language, and facial information improved model performance on all classification tasks.

Remote Assessment for ALS using Multimodal Dialog Agents: Data Quality, Feasibility and Task Compliance.

We investigate the feasibility, task compliance and audiovisual data quality of a multimodal dialog-based solution for remote assessment of Amyotrophic Lateral Sclerosis (ALS). 53 people with ALS and 52 healthy controls interacted with Tina, a cloud-based conversational agent, in performing speech tasks designed to probe various aspects of motor speech function while their audio and video was recorded. We rated a total of 250 recordings for audio/video quality and participant task compliance, along with the relative frequency of different issues observed. We observed excellent compliance (98%) and audio (95.2%) and visual quality rates (84.8%), resulting in an overall yield of 80.8% recordings that were both compliant and of high quality. Furthermore, recording quality and compliance were not affected by level of speech severity and did not differ significantly across end devices. These findings support the utility of dialog systems for remote monitoring of speech in ALS.