Multimodal Interaction Research Articles

Collaborative Robot Teleoperation in Mixed Reality Environment for Inspection Tasks

Abstract Collaborative robots are increasing their presence in the industrial production context due to their flexibility in the tasks they can perform. However, programming and teleoperating them could be a difficult task from the perspective of an unqualified worker. In cases where the worker has to collaborate remotely with a robotic arm, the probability of collisions between the robotic arm and the environment increase. In this work, we describe the design and implementation of a mixed-reality multimodal interface for a worker to teleoperate a robotic arm by interacting with its holographic digital twin, while remaining aware of surrounding obstacles to avoid any collisions. To teleoperate the robot, we have developed a multimodal interface with which the user can communicate naturally with voice (through a semanticsbased task-oriented dialogue system), and by performing manual actions on holographic objects. The application also provides audiovisual information to the user about the status of the collaborative robot as well as the actions commanded. We compute and command the behavior of the robotic arm with ROS (Robot Operating System) and “MoveIt!” (motion planning framework), and we represent computed trajectories and the real-time pose of the robot in the mixed reality environment.

Languaging in Multimodal Digital Interaction: An Enactive Exploration of Emoji Metaphors and Meanings

Emojis, small pictographic characters commonly used in electronic messages, have become a global phenomenon. This study utilizes key enactive concepts to demonstrate how emojis reflect the ongoing interplay between technology, language, and culture. A central theme of the analysis is the role of metaphors in justifying the design, use, and interpretation of emojis, illustrating how these symbols originate, evolve, and adapt to the contextual nuances of our environment. The ultimate goal of this study is to enhance our understanding of the evolving meanings of emojis, emphasizing their dynamic nature, and enriching our understanding of their role and interpretation in contemporary human expression and interaction. In this context, emojis emerge as cultural artifacts that are both shaped by and continually shape how we communicate and interact in the digital age.

Considering Ornaments, Necklaces and Floors: Negotiations of Everyday Family Activities Through the ‘Collective Action’ of Multiple Actors

This paper explores negotiations of everyday family activities in Swedish family homes between multiple actors such as children, parents, and material things. The family activities involve decorating the Christmas tree, brushing teeth and engaging with necklaces, and siblings’ engagements with cleaning equipment. The data comprises 45 hours of video recordings in 12 Swedish family homes. This study draws on multimodal interaction analysis, Bruno Latour’s concept ‘collective action’ and on key Child Studies sensibilities. Such sensibilities involve approaching children’s everyday lives as emerging from the engagements between multiple entities. By considering how multiple actors accomplish negotiations of family activities (e.g., children, ornaments, necklaces, floors, parents), the results shows that negotiations arise as complex interactional accomplishments. Different types of activities achieve currency in those negotiations which range from more straightforward to more complex and uncertain. The study also found that intimate human–nonhuman relationships are central to how such negotiations are accomplished.

Enhancing Human-Computer Interaction in Augmented Reality (AR) and Virtual Reality (VR) Environments: The Role of Adaptive Interfaces and Haptic Feedback Systems

Human-Computer Interaction (HCI) in Augmented Reality (AR) and Virtual Reality (VR) environments has emerged as a critical area of research, shaping how users interact with immersive technologies. This study explores the design and development of advanced HCI frameworks in AR/VR environments, with a particular focus on adaptive interfaces and haptic feedback systems. Adaptive interfaces employ real-time data to personalize user experiences, while haptic feedback systems provide tactile sensations to bridge the gap between digital and physical interactions. The integration of these technologies enhances user engagement, immersion, and efficiency across applications such as gaming, training simulations, healthcare, and remote collaboration. This research investigates the challenges in creating intuitive, responsive systems and examines cutting-edge solutions in multimodal interactions, gesture recognition, and real-time feedback processing. By analyzing user performance and satisfaction, this study contributes to the refinement of AR/VR systems, paving the way for next-generation immersive environments.

High-Order Multimodal Interaction Network for Efficient Prediction of Drug-Drug Interaction

The pharmaceutical industry places a lot of focus on preventing drug-drug interactions (DDIs). Medication interaction detection has been the primary emphasis of machine learning-based DDI prediction methods. Since studies have shown that DDIs can cause different future occurrences, it is more beneficial to predict DDI connected events while examining the mechanism behind combination medicine consumption or adverse reactions. The areas of medication development and illness diagnostics are seeing increased usage of a developing approach that predicts DDIs-associated occurrences. We examine potential interactions between the two medications as well as the kinds of interactions that can occur in this research. Additionally, we provide a strategy that relies on learning and use High-Order Multimodal Interaction Network (HOMIN) to forecast DDIs by learning feature representations. The purpose of this work is to provide HOMIN-DDI, a new method for drug-drug interaction predictions that we have developed utilizing a HOMIN architecture. First, we develop feature vectors from medication categories, targets, paths, and digestive enzymes, and we use the Jaccard similarity to evaluate how similar drugs appear. After that, we build a new HOMIN using the feature representation to predict DDIs. The results of the experiments show that the drug categories feature type, when used to the HOMIN -DDI approach, is effective. Furthermore, compared to employing only one feature, combining numerous features yields more useful and informative results. Compared to other algorithms, HOMIN-DDI performs better when it comes to predicting DDIs.

A Multilingual Preschooler’s School Belonging: The Role of Translanguaging Pedagogy

ABSTRACT To engage in learning, students must feel a sense of belonging. And yet, children often go to schools where language practices are different from those of their homes and where they often feel alienated. In this article, we look closely at the effects of translanguaging pedagogical practices on one multilingual young child’s sense of belonging in a Luxembourgish preschool, and ultimately in society. The focused video analysis identifies seven significant moments of translanguaging pedagogical interactions between the teacher and the child as she (a) develops the child’s voice as creator/teller of stories, (b) raises the class’s consciousness of multilingualism, (c) emphasizes multilingualism as resource, (d) engages with children’s wondering, (e) elaborates children’s wonderings, (f) creates a sense of being a collaborative ensemble, and finally (g) develops a sense that Luxembourgish belongs to all. Within each moment, we zoom in on the manifestations of the child’s developing sense of belonging by identifying his multimodal interactions with the teacher and with his classmates in relationship to components of belonging: engagement, attachment, connectedness, and community. The analysis shows that the teachers’ use of translanguaging pedagogical practices is related to the child’s increased relationship to people (the teacher and peers), as well as to the place (the pre-school). Beyond the focused child, the analysis shows that translanguaging pedagogical practices also made possible all the children’s sense of belonging as multilingual Luxembourgers.

Affording Social Experience for Adolescents Using Immersive Virtual Reality: A Moderated Mediation Analysis.

Immersive virtual reality (IVR) serves as a promising tool to provide adolescents with enriched social experience due to its high-fidelity simulations and multimodal interaction. This study aims to design and develop a multi-user IVR collaborative game utilizing simultaneous localization and mapping (SLAM)-based inside-out tracking technique to foster social experience among students. Also, this study explored the mechanism by which technology acceptance affected social experience in the IVR collaboration game, focusing on the mediating effects of presence, collective efficacy, and group effectiveness, as well as the moderating effect of social-emotional competence (SEC). A total of 104 seventh graders from a middle school in Central China participated in this study and completed the questionnaire. Finally, 87 valid questionnaire responses were retrieved. The results revealed that technology acceptance both directly and indirectly influenced social experience. The mediation analysis revealed a key pathway influencing social experience: technology acceptance → presence → collective efficacy → group effectiveness → social experience. However, no moderating effect of SEC was found in the relationship between technology acceptance and social experience, group effectiveness, and social experience. Based on these results, more appropriate IVR interventions could be developed for social-emotional learning among children and adolescents.

Audiovisual Imagery in Multimodal Perception and Performance of Qigang Chen’s Er Huang Concerto

In the realm of musical performance, the role of a pianist extends beyond the mere execution of musical scores to become a pivotal contributor to artistic expression. This study delves into the auditory and visual imagination through the perception and performance of piano works by the Chinese-French composer Qigang Chen from the early 21st century, involving the performer’s meticulous response to the nuances of musical sound as well as the interplay of visual synaesthesia.The study also explores the inherent conveyance chain within the performance process, experimenting with multimodal forms of expression. These creative choices not only enrich the narrative of the arts but also contribute to crafting a comprehensive, immersive concert experience for the audience. The principal aim of this research is to explore the prismatic view within Qigang Chen’s piano work, offering multiple possibilities of meaning and emphasizing the crucial role of the pianist in forming and elucidating the fusion of auditory and visual senses in performative action.By transcending traditional performance boundaries, this exploration invites the audience to witness the multimodal interactions inherent in musical performances, providing a fresh perspective on dynamic exchanges between different sensory elements and cultural influences.

Multimodal Locomotion and Dynamic Interaction of Hydrogel Microdisks at the Air-Water Interface under Magnetic and Light Stimuli.

The potential applications of hydrogel microrobots in biomedicine and environmental exploration have sparked significant interest in understanding their behavior under multiphysical fields. This study explores the multimodal locomotion and dynamic interaction of hydrogel microrobots at the air-water interface under magnetic and light stimuli. A pair of hydrogel microrobots at the air-water interface exhibits a transition from cooperative, combined rotation to interactive behavior, involving both rotation and revolution under the influence of a rotating magnetic field (RMF), and a shift from attraction to separation under near-infrared (NIR) light, demonstrating the dynamic modulation of their behaviors in response to different stimuli. Notably, the behavioral patterns of multiple hydrogel microrobots under multiphysical fields indicate that NIR light can enhance interactive motion behaviors under RMFs and extend the range of motion trajectories. Dynamic models for each condition are established and analyzed based on dynamic equilibrium, and their behavior can be modulated by parameters such as magnetic particle concentration, magnetic field frequency, and NIR light intensity. This work introduces a novel strategy for regulating and controlling the dynamic behaviors of hydrogel microrobots, offering new insights into their multiphysical field locomotion.

Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question answering

Medical Visual Question Answering (VQA) is a task that aims to provide answers to questions about medical images, which utilizes both visual and textual information in the reasoning process. The absence of large-scale annotated medical VQA datasets presents a formidable obstacle to training a medical VQA model from scratch in an end-to-end manner. Existing works have been using image captioning dataset in the pre-training stage and fine-tuning to downstream VQA tasks. Following the same paradigm, we use a collection of public medical image captioning datasets to pre-train multimodality models in a self-supervised setup, and fine-tune to downstream medical VQA tasks. In the work, we propose a method that featured with Cross-Modal pre-training with Multiple Objectives (CMMO), which includes masked image modelling, masked language modelling, image-text matching, and image-text contrastive learning. The proposed method is designed to associate the visual features of medical images with corresponding medical concepts in captions, for learning aligned vision and language feature representations, and multi-modal interactions. The experimental results reveal that our proposed CMMO method outperforms state-of-the-art methods on three public medical VQA datasets, showing absolute improvements of 2.6%, 0.9%, and 4.0% on the VQA-RAD, PathVQA, and SLAKE dataset, respectively. We also conduct comprehensive ablation studies to validate our method, and visualize the attention maps which show a strong interpretability. The code and pre-trained weights will be released at https://github.com/pengfeiliHEU/CMMO.

Multimodal Co-Attention Fusion Network With Online Data Augmentation for Cancer Subtype Classification.

It is an essential task to accurately diagnose cancer subtypes in computational pathology for personalized cancer treatment. Recent studies have indicated that the combination of multimodal data, such as whole slide images (WSIs) and multi-omics data, could achieve more accurate diagnosis. However, robust cancer diagnosis remains challenging due to the heterogeneity among multimodal data, as well as the performance degradation caused by insufficient multimodal patient data. In this work, we propose a novel multimodal co-attention fusion network (MCFN) with online data augmentation (ODA) for cancer subtype classification. Specifically, a multimodal mutual-guided co-attention (MMC) module is proposed to effectively perform dense multimodal interactions. It enables multimodal data to mutually guide and calibrate each other during the integration process to alleviate inter- and intra-modal heterogeneities. Subsequently, a self-normalizing network (SNN)-Mixer is developed to allow information communication among different omics data and alleviate the high-dimensional small-sample size problem in multi-omics data. Most importantly, to compensate for insufficient multimodal samples for model training, we propose an ODA module in MCFN. The ODA module leverages the multimodal knowledge to guide the data augmentations of WSIs and maximize the data diversity during model training. Extensive experiments are conducted on the public TCGA dataset. The experimental results demonstrate that the proposed MCFN outperforms all the compared algorithms, suggesting its effectiveness.

Multimodal human-computer interaction in interventional radiology and surgery: a systematic literature review.

As technology advances, more research dedicated to medical interactive systems emphasizes the integration of touchless and multimodal interaction (MMI). Particularly in surgical and interventional settings, this approach is advantageous because it maintains sterility and promotes a natural interaction. Past reviews have focused on investigating MMI in terms of technology and interaction with robots. However, none has put particular emphasis on analyzing these kind of interactions for surgical and interventional scenarios. Two databases were included in the query to search for relevant publications within the past 10 years. After identification, two screening steps followed which included eligibility criteria. A forward/backward search was added to identify more relevant publications. The analysis incorporated the clustering of references in terms of addressed medical field, input and output modalities, and challenges regarding the development and evaluation. A sample of 31 references was obtained (16 journal articles, 15 conference papers). MMI was predominantly developed for laparoscopy and radiology and interaction with image viewers. The majority implemented two input modalities, with voice-hand interaction being the most common combination-voice for discrete and hand for continuous navigation tasks. The application of gaze, body, and facial control is minimal, primarily because of ergonomic concerns. Feedback was included in 81% publications, of which visual cues were most often applied. This work systematically reviews MMI for surgical and interventional scenarios over the past decade. In future research endeavors, we propose an enhanced focus on conducting in-depth analyses of the considered use cases and the application of standardized evaluation methods. Moreover, insights from various sectors, including but not limited to the gaming sector, should be exploited.

A Multi-Hierarchical Complementary Feature Interaction Network for Accelerated Multi-Modal MR Imaging

Magnetic resonance (MR) imaging is widely used in the clinical field due to its non-invasiveness, but the long scanning time is still a bottleneck for its popularization. Using the complementary information between multi-modal imaging to accelerate imaging provides a novel and effective MR fast imaging solution. However, previous technologies mostly use simple fusion methods and fail to fully utilize their potential sharable knowledge. In this study, we introduced a novel multi-hierarchical complementary feature interaction network (MHCFIN) to realize joint reconstruction of multi-modal MR images with undersampled data and thus accelerate multi-modal imaging. Firstly, multiple attention mechanisms are integrated with a dual-branch encoder–decoder network to represent shared features and complementary features of different modalities. In the decoding stage, the multi-modal feature interaction module (MMFIM) acts as a bridge between the two branches, realizing complementary knowledge transfer between different modalities through cross-level fusion. The single-modal feature fusion module (SMFFM) carries out multi-scale feature representation and optimization of the single modality, preserving better anatomical details. Extensive experiments are conducted under different sampling patterns and acceleration factors. The results show that this proposed method achieves obvious improvement compared with existing state-of-the-art reconstruction methods in both visual quality and quantity.

Diverse Role of Buffer Mediums and Protein Concentrations to Mediate the Multimodal Interaction of Phenylalanine-Functionalized Gold Nanoparticle and Lysozyme Protein at Same Nominal pH.

Recently, buffer molecules have been known to affect intermolecular protein-protein interactions at physiological pH. However, the roles of buffer molecules and different monolayer protein concentrations remain elusive in controlling the interaction of gold nanoparticles (Au NPs) with protein molecules. Herein, for the first time taking phenylalanine functionalized gold nanoparticles (Au-Phe NPs) and lysozyme (Lyz) protein as model systems, we report that buffer molecules of different charges (at a particular pH) play diverse roles in protein-Au NPs interaction, particularly in protein induced Au NPs aggregation. Among different buffers, negatively charged buffer (citrate and phosphate) induces aggregation of both Au-Phe NPs and Lyz protein, whereas zwitterionic and positive buffer (HEPES, MOPS, and Tris) only cause the Au NPs aggregation. Taking the diverse role of buffer into account, we propose multimodal models for stability and protein induced aggregation mechanism of NPs at different monolayer (sub-, near-, and excess) concentrations of Lyz in different medium.

Multimodal Academic Discourse Socialization: Examining Geoscience Students’ Disciplinary Knowledge Construction and Socialization at a Canadian University

Abstract This ethnographic multiple-case study examines how undergraduate students are socialized into the disciplinary norms, values, and practices of a geoscience course at a Canadian university. Transcending logocentric assumptions about academic discourse, this article advances a broader domain of inquiry––multimodal academic discourse socialization––which foregrounds the polysemiotic nature of academic socialization. This approach examines not only linguistic but also a wider range of semiotic resources, including gestural, visual, material, and spatial ones, among others. To understand geoscientists’ disciplinary norms, values, and communicative practices, ethnographic data (classroom observations, semi-structured interviews, course-related artefacts) were thematically analysed. Focal students’ geoscience poster presentation performances were also analysed using multimodal interaction analysis to scrutinize micro-level instantiations of disciplinary practices. Findings highlight how students were socialized into geoscience ‘observations and interpretations’ through a recurrent multimodal classroom activity, which was also reflected in micro-level multimodal practices enacted in students’ geoscience poster presentations. This study emphasizes that multimodal enactments constitute a crucial dimension of disciplinary practices and values connected with learning to think, view, and represent knowledge as geoscientists.

Multimodal human computer interaction of wheelchairs supporting lower limb active rehabilitation

Currently, an important challenge in stroke rehabilitation is how to effectively restore motor functions of lower limbs. This paper presents multimodal human computer interaction (HCI) of wheelchairs supporting lower limb active rehabilitation. First, multimodal HCI incorporating motor imagery electroencephalography (EEG), electromyography (EMG) and speech is designed. Second, prototype supporting wheelchair active rehabilitation method is illustrated in details. Third, the preliminary brain-computer interfaces (BCI) and speech recognition task experiments are carried out respectively, and the results are obtained. Finally, discussion is conducted and conclusion is drawn. This study has important practical significance in auxiliary movements and neurorehabilitation for stroke patients.

Social Media Sentiment Analysis

Social media sentiment analysis is the computational detection and extraction of human subjective evaluation of objects embedded on social media. Previous sentiment analysis was conducted on isolated written texts, and typically classified sentiment into positive, negative, and neutral states. Social media sentiment analysis has included multi-modal texts, temporal dynamics, interactions, network relationships, and sentiment propagation. Specific emotions and sentiment intensity are also detected.

Reconstructing representations using diffusion models for multimodal sentiment analysis through reading comprehension

The primary challenge in multimodal sentiment analysis (MSA), which utilizes textual, audio, and visual information to analyze speakers' emotions, lies in constructing representation vectors that incorporate both unimodal semantic and multimodal interaction information. While existing research has extensively focused on multimodal fusion strategies, there remains insufficient exploration of the intrinsic potential within concurrently enhancing unimodal and multimodal representations. To address this gap, we introduce two additional steps to traditional three-step MSA: text modality enhancement through the machine reading comprehension (MRC) framework and multimodal representation reconstruction via proposing the diverse diffusion denoising autoencoder (D3AE) module. The MRC queries are integrated to locate sentiment-related prior knowledge, thereby deepening textual semantic understanding from a pretrained language model. Meanwhile, D3AE employs a single-step denoising strategy along with diffusion models across multiple time intervals, enabling efficient reconstruction and enhancement of multimodal representations. Extensive experiments conducted on two benchmark datasets, CMU-MOSI and CMU-MOSEI, validate that our model, MRC-D3AE, achieves state-of-the-art performance. The superiority of our model over existing baselines is primarily attributed to integrating MRC for enhancing text modality and D3AE for reconstructing multimodal representations.

A Parent’s Multimodal Support to Bilingual Child’s Reading: A Case Study of Korean Immigrant Family

ABSTRACT Shared reading is a practice for promoting children’s literacy that parents may utilize at home. In particular, multimodal interactions between a parent and a child can play a role in fostering bilingual children’s reading experiences. This qualitative case study explores how a Korean bilingual mother multimodally supports her child’s shared reading practices. Data sources include an online survey, a monthly reading log, two home visit video observations, and a parent interview. Four multimodal cues are classified throughout the reading: pointing, hand gestures, nodding, and audio language. The three emerging themes of the parent’s nonverbal language in support of the child’s literacy practices include assisting in guessing vocabulary and meaning, drawing attention, and encouraging confidence. The implications of the study are discussed with regards to better promoting a home literacy environment, along with the open communication of relevant messages to parents and educators to offer more support for bilingual students.

Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images.

Large language models (LLMs) are pivotal in artificial intelligence, demonstrating advanced capabilities in natural language understanding and multimodal interactions, with significant potential in medical applications. This study explores the feasibility and efficacy of LLMs, specifically ChatGPT-4o and Claude 3-Opus, in classifying thyroid nodules using ultrasound images. This study included 112 patients with a total of 116 thyroid nodules, comprising 75 benign and 41 malignant cases. Ultrasound images of these nodules were analyzed using ChatGPT-4o and Claude 3-Opus to diagnose the benign or malignant nature of the nodules. An independent evaluation by a junior radiologist was also conducted. Diagnostic performance was assessed using Cohen's Kappa and receiver operating characteristic (ROC) curve analysis, referencing pathological diagnoses. ChatGPT-4o demonstrated poor agreement with pathological results (Kappa = 0.116), while Claude 3-Opus showed even lower agreement (Kappa = 0.034). The junior radiologist exhibited moderate agreement (Kappa = 0.450). ChatGPT-4o achieved an area under the ROC curve (AUC) of 57.0% (95% CI: 48.6-65.5%), slightly outperforming Claude 3-Opus (AUC of 52.0%, 95% CI: 43.2-60.9%). In contrast, the junior radiologist achieved a significantly higher AUC of 72.4% (95% CI: 63.7-81.1%). The unnecessary biopsy rates were 41.4% for ChatGPT-4o, 43.1% for Claude 3-Opus, and 12.1% for the junior radiologist. While LLMs such as ChatGPT-4o and Claude 3-Opus show promise for future applications in medical imaging, their current use in clinical diagnostics should be approached cautiously due to their limited accuracy.