Multimodal Interaction Research Articles

Fish behavior recognition based on an audio-visual multimodal interactive fusion network

In light of the challenges imposed by fish behavior recognition, which arise from environmental noise and dim lighting in aquaculture environments and adversely affect the effectiveness of unimodal recognition methods based on either sound or visual cues, this paper proposes a fish behavior recognition model, Mul-SEResNet50, based on the fusion of audio and visual information. To address issues such as image blurring and indistinct sounds in aquaculture environments, which hinder the effectiveness of multimodal fusion and complementary modalities, a multimodal interaction fusion (MIF) module is introduced. This module integrates audio-visual modalities at multiple stages to achieve a more comprehensive joint feature representation. To enhance complementarity during the fusion process, we designed a U-shaped bilinear fusion structure to fully utilize multimodal information, capture cross-modal associations, and extract high-level features. Furthermore, to address the potential loss of key features, a temporal aggregation and pooling (TAP) layer is introduced to preserve more fine-grained features by extracting both the maximum and average values within pooling regions. To validate the effectiveness of the proposed model, both ablation experiments and comparative experiments are conducted. The results demonstrate that Mul-SEResNet50 achieves a 5.04 % accuracy improvement over SEResNet50 without sacrificing detection speed. Compared to the state-of-the-art U-FusionNet-ResNet50 +SENet model, Mul-SEResNet50 achieves accuracy and F1 score improvements of 0.47 % and 1.32 %, respectively. These findings confirm the efficacy of the proposed model in terms of accurately recognizing fish behavior, facilitating the precise monitoring of fish behavior.

Global-aware Fragment Representation Aggregation Network for image-text retrieval

Image-text retrieval is an important kind of cross-modal retrieval method and has recently attracted much attention. Existing image-text retrieval methods often ignore the relative importance of each fragment (region in an image or word in a sentence) on the global semantic of image or text when aggregating features of image or text fragments, resulting in the ineffectiveness of the learned image and text representations. To address this problem, we propose an image-text retrieval method named Global-aware Fragment Representation Aggregation Network (GFRAN). Specifically, it first designs a fine-grained multimodal information interaction module based on the self-attention mechanism to model both the intra-modality and inter-modality relationships between image regions and words. Then, with the guidance of the global image or text feature, it aggregates image or text fragment features conditioned on their attention weights over global feature, to highlight fragments that contribute more to the overall semantics of images and texts. Extensive experiments on two benchmark datasets Flickr30K and MS-COCO demonstrate the superiority of the proposed GFRAN model over several state-of-the-art baselines.

Virtual Obstacle Avoidance Strategy: Navigating through a Complex Environment While Interacting with Virtual and Physical Elements.

Robotic walking devices can be used for intensive exercises to enhance gait rehabilitation therapies. Mixed Reality (MR) techniques may improve engagement through immersive and interactive environments. This article introduces an MR-based multimodal human-robot interaction strategy designed to enable shared control with a Smart Walker. The MR system integrates virtual and physical sensors to (i) enhance safe navigation and (ii) facilitate intuitive mobility training in personalized virtual scenarios by using an interface with three elements: an arrow to indicate where to go, laser lines to indicate nearby obstacles, and an ellipse to show the activation zone. The multimodal interaction is context-based; the presence of nearby individuals and obstacles modulates the robot's behavior during navigation to simplify collision avoidance while allowing for proper social navigation. An experiment was conducted to evaluate the proposed strategy and the self-explanatory nature of the interface. The volunteers were divided into four groups, with each navigating under different conditions. Three evaluation methods were employed: task performance, self-assessment, and observational measurement. Analysis revealed that participants enjoyed the MR system and understood most of the interface elements without prior explanation. Regarding the interface, volunteers who did not receive any introductory explanation about the interface elements were mostly able to guess their purpose. Volunteers that interacted with the interface in the first session provided more correct answers. In future research, virtual elements will be integrated with the physical environment to enhance user safety during navigation, and the control strategy will be improved to consider both physical and virtual obstacles.

The complexity of interpersonal physiology during rupture and repair episodes in the treatment of borderline personality disorder: a proof-of-concept multimethod single case study of verbal and non-verbal interactional dynamics.

The aim of this proof-of-concept multimethod exploratory single case study is to increase knowledge of the underlying mechanisms of alliance ruptures and repairs in Borderline Personality Disorder treatment across and within the psychotherapeutic treatment process. The multimethod includes outcome assessment of patient self-reporting questionnaires (the Affect Integration Inventory, the Hopkins Symptom Checklist), observation-based ratings of sessions with the Rupture Resolution Rating System, quantitative analysis of heart rate variability using recurrence quantification analysis, and a qualitative multimodal interaction analysis of within-session dynamics. Results reveal how patterns of heart rate synchrony between patient and therapist reflect periodical patterns of emotional interaction corresponding to key therapeutic alliance processes throughout the treatment process. Particularly, heart rate synchronization and desynchronization correspond with increasing rupture resolution ratings and positive outcome measures in the last part of the therapy process, indicating increased productivity, affectivity, and positive change. The qualitative microanalysis highlights context sensitivity to alliance management within sessions. Physiological arousal is found to underlie important alliance processes, including emotion regulation, relatedness, security, empathic responding, sense-making, and validation in correspondence with different therapist verbal and non-verbal markerbs. Clinical implications and study limitations are discussed. Recommendations are made for future directions in relation to applying multimethod approaches when studying rupture and repair processes in psychotherapy.

Referring Image Segmentation with Multi-Modal Feature Interaction and Alignment Based on Convolutional Nonlinear Spiking Neural Membrane Systems.

Referring image segmentation aims to accurately align image pixels and text features for object segmentation based on natural language descriptions. This paper proposes NSNPRIS (convolutional nonlinear spiking neural P systems for referring image segmentation), a novel model based on convolutional nonlinear spiking neural P systems. NSNPRIS features NSNPFusion and Language Gate modules to enhance feature interaction during encoding, along with an NSNPDecoder for feature alignment and decoding. Experimental results on RefCOCO, RefCOCO[Formula: see text], and G-Ref datasets demonstrate that NSNPRIS performs better than mainstream methods. Our contributions include advances in the alignment of pixel and textual features and the improvement of segmentation accuracy.

Multilevel attention imitation knowledge distillation for RGB-thermal transmission line detection

Transmission line detection (TLD) plays a crucial role in ensuring the safety and stability of electricity supply. Applying RGB-thermal convolutional neural networks (CNNs) to unmanned aerial vehicles (UAVs) photography is a valuable alternative for diagnosing transmission line faults. However, existing CNNs struggle to generalize to TLD due to the clustered backgrounds and variable weather conditions. In addition, the limited computational resources and storage space of UAVs pose challenges for the lightweight design of models. In the present study, we developed a novel multilevel attention imitation knowledge distillation structure comprising a high-performing teacher model called MAINet-T and a compact student model called MAINet-S. We aimed to 1) improve the accuracy and robustness of TLD and 2) optimize the performance and capacity of the model for deployment on UAVs. The MAINet-T has a three-stage feature aggregation module and a detailed enhancement module to facilitate the processes of multi-modal and multilevel feature complement and interaction. To balance model performance and capacity for deployment, we proposed a novel KD strategy, including response distillation and feature distillation, to obtain an optimized model called MAINet-S*. Within feature distillation, we proposed a multilevel attention imitation module to integrate the advantages of the attention maps in different stages of the encoder. In experiments based on the VITLD dataset, MAINet-S* outperformed 15 state-of-the-art methods, with a 66.2% reduction in the number of weight parameters (Params) and a 69.9% increase in floating-point operations (FLOPs) compared with MAINet-T.

Specific Emitter Identification Algorithm Based on Time–Frequency Sequence Multimodal Feature Fusion Network

Specific emitter identification is a challenge in the field of radar signal processing. Its aims to extract individual fingerprint features of the signal. However, early works are all designed using either signal or time–frequency image and heavily rely on the calculation of hand-crafted features or complex interactions in high-dimensional feature space. This paper introduces the time–frequency multimodal feature fusion network, a novel architecture based on multimodal feature interaction. Specifically, we designed a time–frequency signal feature encoding module, a wvd image feature encoding module, and a multimodal feature fusion module. Additionally, we propose a feature point filtering mechanism named FMM for signal embedding. Our algorithm demonstrates high performance in comparison with the state-of-the-art mainstream identification methods. The results indicate that our algorithm outperforms others, achieving the highest accuracy, precision, recall, and F1-score, surpassing the second-best by 9.3%, 8.2%, 9.2%, and 9%. Notably, the visual results show that the proposed method aligns with the signal generation mechanism, effectively capturing the distinctive fingerprint features of radar data. This paper establishes a foundational architecture for the subsequent multimodal research in SEI tasks.

Zero-shot visual grounding via coarse-to-fine representation learning

Visual grounding (VG) locates target objects in visual scenes by understanding given natural language queries. Current methods for VG mainly focus on grounding referring expressions or noun phrases covered in the labeled training samples. Despite their grounding prowess, these approaches struggle in grounding novel query–image pairs excluded from the training data. This shortage is usually caused by the deficiency of discriminative representation learning both in images and queries. To address these issues, we propose a one-stage coarse-to-fine framework for zero-shot VG to ground novel query-image samples. Specifically, in the coarse stage, we mine the global context information in the visual features and query embeddings by employing a multi-head self-attention block, strengthening the intra-modality relations in the visual and textual features. In the fine stage, we first learn the query-aware visual representations based on the acquired global context information via a multi-modal relation-enhanced transformer block, which explores the informative information by modeling the cross-modal interaction among visual and textual domains. We further excavate target-oriented discriminative representations from the acquired query-aware visual representations by a noun phrase-guided multi-modal interaction network, which augments the interaction between target-related phrases and the obtained query-aware visual representations to enhance the distinction of target regions, enhancing the subsequent referred target grounding and generalizing. In order to validate the proposed approach, we implement extensive experiments and ablation studies on public benchmark datasets, including RefCOCO, RefCOCO+, RefCOCOg, Flickr30K Entity, Flickr-Split-0 and Flickr-Split-1. Experimental results demonstrate that our approach substantially improves the grounding accuracy and achieves new state-of-the-art performance under single-stage training and testing.

NARRATIVE MULTIMODAL TEXT AS A TYPE OF MULTIMODAL TEXT

The modern communication environment is saturated with multimodal information with which a person actively interacts in everyday life. Researchers of multimodal texts have proven that the perception of these texts differs from the perception of linear monomodal texts, since the semantic load in the multimodal text is distributed between verbal and non-verbal modes. However, in addition to the multimodality itself, such texts may have other features, for example, narrative. Currently, quite a few works are devoted to the study of narrative multimodal texts in general, as a type of multimodal texts. In our work, we pay attention to narrative multimodal texts. The article analyses the concept of narrative multimodal text, provides a classification of these texts. Features of interaction of narrative and multimodality within one text are considered, influence of narrative on correlation of modes in the text is revealed, as well as influence of two given features on perception of such texts. The issue of further study of narrative and multimodality interaction within one text is actualised.

Robust Stochastic Model Predictive Control for Autonomous Vehicle Motion Planning

This work presents a Robust Stochastic Model Predictive Control (RSMPC) framework for real-time motion planning autonomous vehicles, addressing the complex multi-modal vehicle interactions. The proposed framework involves adding expert policy from observations to the dataset and applying the Data Aggregation (DAgger) method to filter unsafe demonstrations and resolve expert conflicts. A Dual-Stage Attention-based Recurrent Neural Network (DA-RNN) model is integrated to predict dual class variables from the dataset, producing a set containing constraints collision-avoidance predicted to be active. The RSMPC framework enhances formulation optimization by eliminating irrelevant collision avoidance constraints, resulting in faster control signals. The framework is applied iteratively, continuously updating observations and solving the RSMPC optimization formulation in real-time. Evaluation of the DA-RNN model achieved a recall value of 0.97 and a high accuracy rate of 98.1% in predicting dual interactions, with a minimal false negative rate of 0.026, highlighting its effectiveness in capturing interaction intricacies. Validated through simulations of interactive traffic intersections, the proposed framework demonstrably excels, showing high feasibility of 99.84% and a 15-fold increase in response speed compared to the baseline. This approach ensures autonomous vehicles navigate safely and efficiently in complex traffic scenarios, paving the way for more reliable and scalable autonomous driving solutions.

Learnable and Expressive Visualization Authoring through Blended Interfaces.

A wide range of visualization authoring interfaces enable the creation of highly customized visualizations. However, prioritizing expressiveness often impedes the learnability of the authoring interface. The diversity of users, such as varying computational skills and prior experiences in user interfaces, makes it even more challenging for a single authoring interface to satisfy the needs of a broad audience. In this paper, we introduce a framework to balance learnability and expressivity in a visualization authoring system. Adopting insights from learnability studies, such as multimodal interaction and visualization literacy, we explore the design space of blending multiple visualization authoring interfaces for supporting authoring tasks in a complementary and fexible manner. To evaluate the effectiveness of blending interfaces, we implemented a proof-of-concept system, Blace, that combines four common visualization authoring interfaces-template-based, shelf confguration, natural language, and code editor-that are tightly linked to one another to help users easily relate unfamiliar interfaces to more familiar ones. Using the system, we conducted a user study with 12 domain experts who regularly visualize genomics data as part of their analysis workfow. Participants with varied visualization and programming backgrounds were able to successfully reproduce unfamiliar visualization examples without a guided tutorial in the study. Feedback from a post-study qualitative questionnaire further suggests that blending interfaces enabled participants to learn the system easily and assisted them in confdently editing unfamiliar visualization grammar in the code editor, enabling expressive customization. Refecting on our study results and the design of our system, we discuss the different interaction patterns that we identifed and design implications for blending visualization authoring interfaces.

Effects of Self-Learning and Exploration for XR-based Interactions

This research explores the overtime learning trends of multimodal gaze-based interactions in tasks involving the movement of augmented objects within extended reality (XR) environments. This study employs three interactions, including two multimodal gaze-based approaches, and compares them with an unimodal hand-based interaction. The underlying hypothesis posits that gaze-based interactions outperform other modalities, promising improved performance, lower learnability rates, and enhanced efficiency. These assertions serve as the foundation for investigating the dynamics of self-learning and exploration within XR-based environments. To this end, the study addresses questions related to the temporal evolution of learnability, post-learning efficiency, and users’ subjective preferences regarding these interaction modalities. This research shows that gaze-based interactions enhance performance, exhibit a lower learnability rate, and demonstrate higher efficiency compared to an unimodal hand-based interaction. Our results contribute to the design and refinement of more effective, user-friendly, and adaptive XR user interfaces.

Robot Control Platform for Multimodal Interactions with Humans Based on ChatGPT

This paper presents the architecture of a multimodal human–robot interaction control platform that leverages the advanced language capabilities of ChatGPT to facilitate more natural and engaging conversations between humans and robots. Implemented on the Pepper humanoid robot, the platform aims to enhance communication by providing a richer and more intuitive interface. The motivation behind this study is to enhance robot performance in human interaction through cutting-edge natural language processing technology, thereby improving public attitudes toward robots, fostering the development and application of robotic technology, and reducing the negative attitudes often associated with human–robot interactions. To validate the system, we conducted experiments measuring negative attitude robot scale and their robot anxiety scale scores before and after interacting with the robot. Statistical analysis of the data revealed a significant improvement in the participants’ attitudes and a notable reduction in anxiety following the interaction, indicating that the system holds promise for fostering more positive human–robot relationships.

Translanguaging space through pointing gestures: Multilingual family literacy at a science museum

Translanguaging theory highlights the dynamic use of multiple languages and communication modes by multilingual people in their daily experiences. Museums are informal family learning spaces where multilingual families use languages and other semiotic resources to create learning opportunities for their children. Using a microethnographic approach to discourse analysis and multimodal interaction analysis, I examined how a multilingual family uses translanguaging practices to organize their family learning in museums and the role of pointing gestures as part of their translanguaging repertoires in multilingual family learning. The analysis of two literacy events highlights that a child and his mother translanguaged with various semiotic resources to organize museum performances, joint attention, and telling, and that pointing gestures played a role in constructing a translanguaging space as they organized the two performances. Pointing involved the family in reading signage texts and allowed the mother to translate them for the child. Viewing translation as part of the translanguaging repertoire, this study recognizes the importance of the role of pointing gestures in constructing family learning at museums, enriching children’s schooling and literacy learning in classrooms. I argue that recognizing pointing as a critical component of translanguaging allows educators to develop strategies that leverage families’ unique repertoires to support multilingual students’ language and literacy learning.

Who are the best contributors? Designing a multimodal science communication interface based on the ECM, TAM and the Taguchi methods

Science communication conducted through mobile devices and mobile applications is an efficient and widespread phenomenon that requires communicators and design practitioners to further develop suitable design elements and strategies for such platforms. The effective application of multimodal or multisensory design in interfaces provides users with rich experiences. However, there is a lack of guiding recommendations for user interface design in the citizen science community. This study investigated factors affecting users’ perceptions and behavioral intentions toward multimodal scientific communication interface designs and identified the optimal combinations of such factors for such designs. Through a focus group, we defined three design dimensions of a science communication interface: visual, auditory, and haptic. An online experiment involving 916 participants was then conducted and integrated the technology acceptance model, expectation–confirmation model, and Taguchi method to examine the hierarchical combinations with the greatest influence in each dimension. The results indicated that interface design combinations primarily focusing on visual elements, with auditory and haptic as secondary elements, can serve as effective tools for science communication. Moreover, layout, color tones, vibration intensity, and sound volume significantly affected users’ perceptions and behavioral intentions. As one of the few studies using the Taguchi method to explore the design of science communication interfaces, the present findings enrich the multimodal theory from the perspectives of design and communication, highlighting its value in science communication. This paper simultaneously provides insights into how to select and combine multimodal design elements in science communication interfaces, demonstrating the potential of such designs to affect a user perception, satisfaction, confirmation, and continued usage intention.

A tactile oral pad based on carbon nanotubes for multimodal haptic interaction

A tactile oral pad based on carbon nanotubes for multimodal haptic interaction

MRehab: A Mixed Reality Rehabilitation System Supporting Integrated Speech and Hand Training

Integrated speech and hand-motor training is an effective post-stroke rehabilitation method. However, few interactive systems and assistive technologies were developed in this field. Driven by this challenge, we leverage Mixed Reality technology, which merges immersive virtual scenarios with physical hands-on tools in the real world, to provide patients with multi-modal interactions and engaging training experiences. Following a user-centered design approach, we first interviewed seven therapists to identify user requirements and design considerations. We further designed MRehab, an interactive rehabilitation system that allows patients to regain speech and hand skills through MR scenarios that depict daily living activities. We conducted a preliminary user test with 12 patients and 5 therapists to validate the feasibility and understand the user experience with MRehab. The results confirmed its feasibility for hand-motor training. Additionally, the patients expressed high motivation, engagement, and a positive attitude toward using MRehab. Our findings demonstrate the potential of MR technology in integrated speech and hand function rehabilitation training.

Building and coordinating multimodal interaction with gaze across two communicative situations

Building and coordinating multimodal interaction with gaze across two communicative situations

Analysing educational dialogue around shared artefacts in technology-mediated contexts: a new coding framework

ABSTRACT Multimodal interactions that incorporate technology use can support rich new forms of educational dialogue that highlight differences between participants’ perspectives and make reasoning processes more explicit. Learning with technology is mediated by digitally represented knowledge artefacts that participants construct together and manipulate through dialogue. However, there is a need for research into the methods by which researchers and practitioners can understand the learning taking place in such complex environments where verbal and nonverbal interactions with technology are inter-connected and entangled with activities and material learning resources. This paper addresses that need. It offers a critical discussion of the underlying theory and presents a new 13-category coding scheme, ‘Tech-SEDA’. This is the first public version of a systematic, research-informed and practical tool for analysing verbal and nonverbal dialogic communication at the micro- or meso- level across diverse educational settings. Application is illustrated with examples of dialogic interaction across diverse technology contexts by learners located in classrooms or remotely – including online discussions, 3-dimensional modelling in augmented reality, software apps on tablets, Knowledge Forum, Scratch programming, interactive whiteboards, and more. The paper considers the methodological issues arising in capturing dynamic multimodal dialogue and in identifying the learning taking place.

A Multilayer Architecture towards the Development and Distribution of Multimodal Interface Applications on the Edge.

Today, Smart Assistants (SAs) are supported by significantly improved Natural Language Processing (NLP) and Natural Language Understanding (NLU) engines as well as AI-enabled decision support, enabling efficient information communication, easy appliance/device control, and seamless access to entertainment services, among others. In fact, an increasing number of modern households are being equipped with SAs, which promise to enhance user experience in the context of smart environments through verbal interaction. Currently, the market in SAs is dominated by products manufactured by technology giants that provide well designed off-the-shelf solutions. However, their simple setup and ease of use come with trade-offs, as these SAs abide by proprietary and/or closed-source architectures and offer limited functionality. Their enforced vendor lock-in does not provide (power) users with the ability to build custom conversational applications through their SAs. On the other hand, employing an open-source approach for building and deploying an SA (which comes with a significant overhead) necessitates expertise in multiple domains and fluency in the multimodal technologies used to build the envisioned applications. In this context, this paper proposes a methodology for developing and deploying conversational applications on the edge on top of an open-source software and hardware infrastructure via a multilayer architecture that simplifies low-level complexity and reduces learning overhead. The proposed approach facilitates the rapid development of applications by third-party developers, thereby enabling the establishment of a marketplace of customized applications aimed at the smart assisted living domain, among others. The supporting framework supports application developers, device owners, and ecosystem administrators in building, testing, uploading, and deploying applications, remotely controlling devices, and monitoring device performance. A demonstration of this methodology is presented and discussed focusing on health and assisted living applications for the elderly.